Scaling Temporal beyond a single cluster

Temporal is an incredibly powerful platform for building resilient, highly scalable distributed applications. By abstracting away the complexities of distributed state management, it allows developers to focus entirely on business logic.

However, as you scale out to handle extremely high throughput or look to deploy your application in a true multi-region, active-active fashion, you might hit some architectural challenges. Out of the box, Temporal doesn't natively support multi-cluster active-active deployments.

The Problem: Scaling Beyond a Single Cluster

To bridge this gap, there are several ways:

Implement dynamic shard splitting into Temporal core, which is more complicated and I can not handle it 😅
Introduce another abstraction layer to "group" several physical entities into one unified access layer. This layer should serve Temporal SDK APIs and compatible with existing worker implementation.

I chose the 2nd path, the problem becomes more easy to reason about:

A workflow is a "sticky session", once created, it will stays in the physical location forever (until retention passed, or replicated to another physical location, in that case, it still essentially a new "session", just with the same "session name").
Following request that is scoped to a workflow, must be routed to the workflow's birthright location.
A workflow is processed when worker execute a poll request from Temporal server. The worker then executes its workflow logic and issue another RPC says that "I'm done, here is the result, tell me what next". This polling request is not workflow scoped, only contains namespace and task queue info.

Enter Tempura 🍤

Tempura is my implementation to solves these challenges by acting as an intelligent routing layer sitting in front of your Temporal backend clusters. It intercepts gRPC calls and inspects the protobuf payloads to make appropriate routing decisions.

Tempura introduces the concept of a VirtualNamespace. From your client's perspective, they interact with a single namespace. Under the hood, Tempura maps this to multiple physical namespaces distributed across different Temporal clusters.

Tempura inspects gRPC calls (like StartWorkflowExecution or RespondWorkflowTaskCompleted) to extract the semantic context (WorkflowId and Namespace). Several important design decisions behind Tempura:

Workflow Stickiness: Once a workflow is assigned and started on a specific cluster, Tempura's internal resolver caches this mapping. Any subsequent signals, updates, or task completions for that WorkflowId are strictly and reliably routed back to the exact physical cluster where the execution lives.
Connection Parking: When your workers issue poll requests (PollWorkflowTaskQueue and PollActivityTaskQueue), Tempura fans out these requests to multiple backend clusters. It parks the connection and routes the task back to the worker from whichever cluster has pending tasks available.

The primary advantage of using Tempura is that it allows you to achieve massive scalability and fault tolerance by horizontally scaling workloads across multiple independent backend clusters.

Most importantly, you get this active-active architecture without requiring any configuration changes or custom sharding logic in your client applications or workers**.** To your clients, it looks like they are talking to a single, infinitely scalable Temporal cluster.

Things to Keep in Mind

Because Tempura sits between your clients/workers and the Temporal server, it only sees traffic that goes through the frontend API.

There is one limitation to be aware of: APIs that issue direct RPCs into another history shard within the same cluster (bypassing the Temporal frontend server) will not pass through Tempura. Examples include:

Signaling another workflow from inside a workflow.
Canceling external workflows from inside a workflow.

Since these internal APIs are handled by Temporal's internal queues (timer/transfer), you simply need to wrap these calls inside an Activity. By wrapping the call in an Activity, the request is directed through Tempura's mapping layer, ensuring it gets routed to the correct destination cluster.

Give it a try, if you encounter any issues, please kindly help to submit also. Github repo is here