Demonstrate how Tempura solves a typical Temporal scaling bottleneck (namespace overload) by horizontally scaling across multiple physical namespaces/clusters under load, avoiding the need to re-provision a cluster with higher shard counts.

Background

You've maybe already familiar with Scaling Temporal: The basics, the canonical load test exercise for Temporal. The article mentioned a common bottleneck for Temporal under heavy load is high number of state transition & request latency caused by shard lock contention. The typical resolution is increasing the History Shard count. However, the shard count cannot be changed after a cluster is built. If your namespace is overloaded, you are normally forced to migrate to a new cluster with a larger shard count or suffer degraded performance.

In this article, we win re-exercise the stress-test, but with Tempura in the middle to see how Tempura solves this gracefully by abstracting a single logical namespace to multiple physical namespaces across different clusters, Tempura allows you to add additional capacity on the fly without altering the client configuration or rebuilding your primary cluster.

Test Scenario Setup

Step 1: Initial Setup with a Single Physical Namespace

First, we start the compose stack, the stack includes:

• 2 Temporal clusters

• 1 Redis instance

• 1 Prometheus and 1 Grafana instance

• 1 Tempura instance (our routing layer)

docker-compose -f docker-compose.yml up -d 

We configure Tempura to route our target Virtual Namespace exclusively to Cluster A.

# Create the Virtual Namespace
curl -X POST http://localhost:8089/namespace/virtual/create \
  -d '{"name": "my-app-namespace"}'

# Map Cluster A's physical namespace to the Virtual Namespace
curl -X POST http://localhost:8089/namespace/physical/add \
  -d '{
      "virtual_namespace": "my-app-namespace",
      "physical_namespace": "namespace-a",
      "cluster_address": "cluster-a.example.com:7233"
  }'

Virtual namespace 'my-app-namespace' created successfully
Physical namespace 'namespace-a' added to virtual namespace 'my-app-namespace'

Step 2: Apply Load (The Overload Condition)

Using the benchmark-workers image from the Temporal load testing guide, we start applying a heavy load to the Tempura endpoint. Ensure your workers are pointed to Tempura instead of the physical Temporal clusters.

# Start Workers pointing to Tempura
docker run --network=sandbox_temporal-sandbox-network --env TEMPORAL_TASK_QUEUE=benchmark --env TEMPORAL_NAMESPACE=my-app-namespace --env TEMPORAL_GRPC_ENDPOINT=tempura.example.com:8088 ghcr.io/temporalio/benchmark-workers:main


# Start the Runner to generate workflows
# -c is the number of concurrent workflows
docker run --network=sandbox_temporal-sandbox-network --env TEMPORAL_TASK_QUEUE=benchmark --env TEMPORAL_WORKFLOW_TASK_POLLERS=16 --env TEMPORAL_ACTIVITY_TASK_POLLERS=8 --env TEMPORAL_NAMESPACE=my-app-namespace --env TEMPORAL_GRPC_ENDPOINT=tempura.example.com:8088 ghcr.io/temporalio/benchmark-workers:main runner -c 50 -t ExecuteActivity '{ "Count": 10, "Activity": "Echo", "Input": { "Message": "test" } }'

Note: Adjust the runner parallelism to generate enough load to stress Cluster A.

Step 3: Measure & Observe the Bottleneck

Observe the Prometheus metrics for Cluster A:

• State Transitions: sum(rate(state_transition_count_count{namespace="namespace-a"}[1m]))

• Shard lock spike:

  histogram_quantile(0.95, sum by (instance, le)(rate(lock_latency_bucket[1m])))
  

All workflows are allocated to cluster A

Expected Result: As the load increases, the StartWorkflowExecution latency on Cluster A will spike above our acceptable SLO (e.g. > 150ms). This indicates that namespace-a is overloaded (due to shard lock contention or database limits) and cannot comfortably handle the load.

Step 4: Scale Out with Tempura

Instead of rebuilding Cluster A with a higher shard count (which requires downtime and complex migrations), we seamlessly add Cluster B's physical namespace to the Tempura Virtual Namespace.

# Map Cluster B's physical namespace to the existing Virtual Namespace
curl -X POST http://localhost:8089/namespace/physical/add \
  -d '{
      "virtual_namespace": "my-app-namespace",
      "physical_namespace": "namespace-b",
      "cluster_address": "cluster-b.example.com:7233"
  }'

Physical namespace 'namespace-b' added to virtual namespace 'my-app-namespace'

Step 5: Observe Load Sharing and Recovery

Continue the benchmark-worker load tests and monitor both clusters in Grafana. After adding a new namespace to share the load, the load in cluster A are shared to cluster B

State transition count steadily back to equilibrium

What Tempura does behind the scenes:

Tempura’s proxy implementation acts as a Temporal-aware proxy layer that intercepts Temporal requests to dynamically route them based on a semantic mapping cache. When a request (like StartWorkflowExecution or a signal) arrives, the proxy decodes the protobuf payload to extract the WorkflowId and VirtualNamespace. It then queries its mapping cache using the WorkflowId to check for an existing assignment. If a mapping exists (a cache hit), it means the workflow is already running, and the proxy applies "workflow stickiness" by rewriting the request to target that exact physical cluster. If it's a cache miss (a new workflow), the proxy's resolver uses a load balancer to assign a physical namespace, stores this new WorkflowId -> PhysicalNamespace mapping in the cache, and forwards the rewritten request. This guarantees that all subsequent updates, queries, or signals for that workflow are consistently routed to the correct backend cluster without the client needing to know the physical topology.

TTL (Time-to-Live) handling in Tempura is specifically designed to manage the lifecycle of the semantic mapping cache (Redis) so it doesn't grow indefinitely, while ensuring that mappings exist exactly as long as they are needed. Some key things:

1. Indefinite Storage for Active Workflows: When a workflow is first started and a new mapping is created (WorkflowId -> PhysicalNamespace), it is stored in Redis with no expiration. This ensures the mapping persists for the entire duration of the workflow execution, regardless of whether it takes seconds or months.

2. Intercepting Completion: Tempura inspects the payloads of RespondWorkflowTaskCompleted requests as workers send them back to the server. It scans the list of commands within the payload looking for COMMAND_TYPE_COMPLETE_WORKFLOW_EXECUTION (which indicates the workflow has successfully finished).

3. Dynamic TTL Application: When a workflow completion is detected, Tempura calls UpdateTTL(workflowID) in the mapping layer.

4. Namespace Retention Synchronization: Inside UpdateTTL, Tempura maintains namespace retention information. This layer retrieves the exact retention period configured on the underlying physical Temporal namespace (e.g., 7 days, 30 days).

Expected Result:

1. The State Transitions metric will show the workload being distributed across both Cluster A and Cluster B.

2. Cluster B's resources are actively utilized, effectively doubling the horizontal scalability of the single logical my-app-namespace.

Conclusion

This scenario exercises Tempura's core value proposition: dynamically scaling Temporal namespaces beyond the limits of a single cluster's architecture (like fixed History Shard counts) without downtime or client-side reconfiguration.

However, as you scale out to handle extremely high throughput or look to deploy your application in a true multi-region, active-active fashion, you might hit some architectural challenges. Out of the box, Temporal doesn't natively support multi-cluster active-active deployments.

The Problem: Scaling Beyond a Single Cluster

To bridge this gap, there are several ways:

• Implement dynamic shard splitting into Temporal core, which is more complicated and I can not handle it 😅

• Introduce another abstraction layer to "group" several physical entities into one unified access layer. This layer should serve Temporal SDK APIs and compatible with existing worker implementation.

I chose the 2nd path, the problem becomes more easy to reason about:

• A workflow is a "sticky session", once created, it will stays in the physical location forever (until retention passed, or replicated to another physical location, in that case, it still essentially a new "session", just with the same "session name").

• Following request that is scoped to a workflow, must be routed to the workflow's birthright location.

• A workflow is processed when worker execute a poll request from Temporal server. The worker then executes its workflow logic and issue another RPC says that "I'm done, here is the result, tell me what next". This polling request is not workflow scoped, only contains namespace and task queue info.

Enter Tempura 🍤

Tempura is my implementation to solves these challenges by acting as an intelligent routing layer sitting in front of your Temporal backend clusters. It intercepts gRPC calls and inspects the protobuf payloads to make appropriate routing decisions.

Tempura introduces the concept of a VirtualNamespace. From your client's perspective, they interact with a single namespace. Under the hood, Tempura maps this to multiple physical namespaces distributed across different Temporal clusters.

Tempura inspects gRPC calls (like StartWorkflowExecution or RespondWorkflowTaskCompleted) to extract the semantic context (WorkflowId and Namespace). Several important design decisions behind Tempura:

1. Workflow Stickiness: Once a workflow is assigned and started on a specific cluster, Tempura's internal resolver caches this mapping. Any subsequent signals, updates, or task completions for that WorkflowId are strictly and reliably routed back to the exact physical cluster where the execution lives.

2. Connection Parking: When your workers issue poll requests (PollWorkflowTaskQueue and PollActivityTaskQueue), Tempura fans out these requests to multiple backend clusters. It parks the connection and routes the task back to the worker from whichever cluster has pending tasks available.

The primary advantage of using Tempura is that it allows you to achieve massive scalability and fault tolerance by horizontally scaling workloads across multiple independent backend clusters.

Most importantly, you get this active-active architecture without requiring any configuration changes or custom sharding logic in your client applications or workers. To your clients, it looks like they are talking to a single, infinitely scalable Temporal cluster.

Things to Keep in Mind

Because Tempura sits between your clients/workers and the Temporal server, it only sees traffic that goes through the frontend API.

There is one limitation to be aware of: APIs that issue direct RPCs into another history shard within the same cluster (bypassing the Temporal frontend server) will not pass through Tempura. Examples include:

• Signaling another workflow from inside a workflow.

• Canceling external workflows from inside a workflow.

Since these internal APIs are handled by Temporal's internal queues (timer/transfer), you simply need to wrap these calls inside an Activity. By wrapping the call in an Activity, the request is directed through Tempura's mapping layer, ensuring it gets routed to the correct destination cluster.

Give it a try, if you encounter any issues, please kindly help to submit also. Github repo is here

• Things you want to do

• Things you want to learn

• Things you want to be involved with

Work efficiency:

• How fast can you deliver

• The relative quality of your work compared to your co-workers in the vicinities (think of team level, organization level)

What is often misunderstood (from my experience): these 2 collides. But it is not, unless you are lucky. Otherwise, let's consider these scenarios normal:

• You are interested in database internals and your work require you to build a React Web using Claude.

• You want to implement a cool white paper, but your work require you to spend the whole day to communicate with stakeholders to integrate a new system that only require 3 lines of code change.

• You can make a feature/API more sophisticated, the kind of architecture that are being taught in every system design books. Don't do it. Implement just enough to get by, same as it was implemented everywhere else in your company, save the scaling capability for next sprint/next half OKR.

That's normal and that's life. 9-18 you are paid to work for others's dreams. The stakeholders that you are complaining about, he or she is also probably hate this as much as you do. Instead of being hatred so much, let's coordinate and get the work done. Leave your fancy optimization idea behind, leave the rocket-science RFC that you want to do behind, do it for the cash.

What about the things that your are eager to get involved with? Wake up a bit earlier, find some spare time here and there. There is no magic formula for this. If you want more you just have to invest more.

If a bug only manifests once in tens of millions of requests, reproducing it locally using traditional testing methods is practically impossible. Engineers are often left analyzing ambiguous log lines without a reliable path to reproduction.

To solve this, infrastructure engineering has increasingly turned to Deterministic Simulation Testing (DST). While application-level frameworks like Tokio’s Turmoil (for Rust) have brought this capability to specific language ecosystems, achieving language-agnostic, infrastructure-wide determinism requires a lower-level approach by operating at the hypervisor layer.

This article breaks down the engineering mechanics of building a deterministic hypervisor, which is a system capable of freezing time, controlling randomness, and snapshotting virtual states to eliminate non-determinism from distributed systems testing.

The Sources of Non-Determinism

To make a system completely deterministic, a simulation environment must neutralize four core sources of environmental variation at the hardware and operating system levels:

1. Multi-core Thread Scheduling: When multiple CPU cores execute instructions simultaneously, the exact order in which they read or write to shared memory locations depends on hardware-level variables like cache line invalidations and internal power management.

2. Asynchronous Hardware Interrupts: Physical hardware devices (such as network interface cards or storage controllers) notify the operating system of completed tasks by firing hardware interrupts. These interrupts halt the CPU execution flow at unpredictable instruction boundaries.

3. System Clocks: Software that queries the physical system clock (time.Now() or std::time::Instant::now()) receives a value tied to real-world wall-clock time, which changes across test runs.

4. Hardware Entropy: Calls to system random number generators draw from hardware-level thermal noise inside the silicon, yielding unpredictable outputs.

Architecture of a Deterministic Hypervisor

Achieving determinism without requiring developers to rewrite their application code or swap out network primitives requires dropping down to the virtualization layer.

By modifying an open-source hypervisor, such as FreeBSD’s minimalist bhyve or AWS’s Rust-based Firecracker, you can construct a virtual machine execution loop that intercepts and controls these hardware behaviors.

1. Enforcing Single-Threaded Context Switching

To eliminate the unpredictability of multi-core scheduling, a deterministic hypervisor provisions the guest virtual machine (VM) with exactly one virtual CPU (vCPU).

The hypervisor binds this single vCPU execution loop to a single dedicated thread on the host machine. Even if the guest operating system (for example, Linux running multiple Docker containers) spawns hundreds of internal application threads, the hypervisor forces them to execute sequentially on that single core. Because the hypervisor completely controls when the vCPU switches contexts between these guest threads, physical concurrency noise is eliminated.

2. Instruction Counting via Hardware PMCs

To control execution with instruction-level precision, the hypervisor must be able to instruct the CPU to execute a precise number of instructions and then halt.

This is implemented using the CPU's physical Performance Monitoring Counters (PMCs), configured specifically to track "instructions retired." The hypervisor sets a countdown value within the PMC; when the counter hits zero, the hardware triggers a VM Exit, dropping control back out of the guest and into the hypervisor.

The Hardware Skid Challenge: Modern out-of-order CPU pipelines can occasionally experience a 1-to-2 instruction "skid" when tripping these counters. Production-grade simulators implement a correction layer within the hypervisor to detect these skids, roll back the execution state, and realign it to maintain perfect reproducibility across runs.

3. Intercepting Time and Randomness

Whenever the guest operating system or an application executes an instruction to read external state, the CPU triggers a VM Exit, allowing the hypervisor to intercept and mock the response:

• Logical Time: When code calls instructions like RDTSC (Read Time-Stamp Counter) on x86 architectures, the hypervisor intercepts the call. It returns a synthetic time value calculated strictly based on the number of CPU instructions executed so far. If the system is idle, virtual time stops moving.

• Controlled Entropy: System calls like getrandom() or hardware instructions like RDRAND are intercepted. The hypervisor feeds them data derived from a mathematical pseudo-random number generator (PRNG) tied to a fixed master seed.

4. Serializing Virtual I/O

Standard virtual hardware devices (like virtio-net or virtio-blk) operate asynchronously, firing interrupts back to the OS as soon as the host completes a disk or network task.

In a deterministic hypervisor, virtual hardware operations are strictly serialized. When a virtual disk write occurs, the hypervisor queues the response and delivers the completion interrupt to the guest kernel only at a pre-calculated instruction boundary.

State Snapshotting and Branching Realities

Once 100% deterministic execution is achieved, meaning an entire network topology runs inside a single-threaded loop bounded by a single master seed, the hypervisor can implement highly efficient, copy-on-write memory snapshotting.

As the distributed system executes inside the hypervisor, an external coordination engine monitors variables like code coverage, system logs, and data invariants. When the system enters a complex state, such as right before a database consensus vote, the hypervisor takes an in-memory snapshot of the entire virtual machine's RAM and disk state.

From that exact microsecond, the orchestration engine can branch execution into multiple test paths:

                       [ Snapshot at State A ]
                            /          \
                           /            \
       [ Test Path Alpha ]               [ Test Path Beta ]
    Injected Network Partition           Normal Network Path
              │                                   │
              ▼                                   ▼
      (Bug Discovered!)                   (System Healthy)

In Test Path Alpha, the hypervisor rolls back to the snapshot and simulates a sudden network drop between nodes. In Test Path Beta, it allows the packets to flow normally. Because the underlying state transition is deterministic, if a consistency bug is discovered in Path Alpha, that exact failure sequence can be reproduced identically on every subsequent execution by reusing the same seed and snapshot.

Practical Application: Time-Travel Debugging

The primary practical benefit of a deterministic hypervisor is the shift in how distributed systems are debugged.

In a traditional multi-node staging environment, finding a race condition requires adding verbose logging, running a stress test, and parsing the resulting traces after the failure occurs.

In a deterministic hypervisor loop, when a system invariant breaks (for instance, a distributed key-value store encounters a split-brain state), the testing platform isolates the exact seed that caused the failure. Because the entire execution path is bounded by that seed, engineers can load it into a time-travel debugger. This allows them to step both forward and backward through the entire operating system's memory space instruction-by-instruction, across multiple isolated containers, tracing pointer or memory corruptions directly back to the exact CPU cycle where the fault occurred.

Conclusion

Building a deterministic hypervisor requires bypassing many of the parallel execution optimizations built into modern out-of-order CPUs and modifying core scheduling primitives within operating system kernels.

While the upfront engineering complexity of this approach is high, it provides a language-agnostic mechanism to test distributed infrastructure. By moving the identification of distributed edge cases into an automated, single-threaded simulation loop, infrastructure teams can transform unpredictable distributed bugs into predictable, reproducible software defects.

You fix a small issue because it is urgent. You take ownership of a messy system because no one else wants it. You stay late once, then twice, then it becomes expected. At some point, you realize your workload keeps growing, but your role, influence, and recognition stay the same.

That is what being leveraged looks like.

The first step to avoiding it is clarity. If your responsibilities are vague, people will fill the gaps for you. Not out of malice, but out of convenience. Make it clear what you own, what you support, and what is not yours to carry. Writing things down helps more than people think.

Another common trap is becoming the hero. The engineer who always jumps in, fixes production, and saves the day feels valuable. In reality, heroes are easy to overuse. Instead of optimizing for being busy, optimize for making problems go away. Automate, simplify, and remove recurring pain. Engineers who reduce chaos are harder to exploit than engineers who absorb it.

You also want to be careful about becoming the only person who understands something. It can feel like job security, but it often turns into a cage. When you cannot step away, the organization leans on you harder. Share context, document systems, and bring others in early. Being valuable without being indispensable is a healthier place to be.

Saying no matters too. You do not need to be confrontational. A professional no sounds like explaining trade-offs. If you take on this work, something else will slip. If a task does not fit your role, say so clearly. Framing boundaries as decisions makes them easier for others to accept.

Many engineers get leveraged because their impact is invisible. Effort alone does not speak for itself. Make a habit of communicating what you solved, why it mattered, and what risk or cost you removed. This is not self-promotion. It is basic hygiene.

There is also a subtle form of leverage that comes from taking responsibility without authority. Mentoring, planning, and coordinating are real work. If you are doing leadership work, you should either have the authority that comes with it or consciously pull back. Otherwise, you are donating free labor.

Finally, protect your energy. Your time and focus are not infinite. Treat them like production capacity, not an emergency fund that anyone can withdraw from. Set expectations around availability and urgency. Burnout is not a badge of honor, and it rarely benefits the people who wear it.

Not getting leveraged does not mean doing less or caring less. It means being intentional about where your effort goes and making sure your value compounds instead of being quietly extracted.

Session will try to discover hosts for control connection pool first by resolving the provided hosts with DNS lookup. It will then use this host to discover other members in the cluster by querying system.peers and system.local for host info. (https://github.com/apache/cassandra-gocql-driver/blob/7279756d7311211939f1466d20d1a029ab69f4a4/host_source.go#L711)

Control connection pool also maintains a go routine to periodically heartbeat the connected host and reinitialize the connection when the heartbeat fails. (https://github.com/apache/cassandra-gocql-driver/blob/7279756d7311211939f1466d20d1a029ab69f4a4/control.go#L125)

When a node join/leave the cluster, Cassandra will send a corresponding event to the connected client via the mentioned above control connection. The driver when then issue a ring refresh request for a new host list. (https://github.com/apache/cassandra-gocql-driver/blob/7279756d7311211939f1466d20d1a029ab69f4a4/events.go#L166)

However, I feel like using the MacBook touchpad (or any other kind of touchpad) made me hunch over a lot since I'm already a guy with a huge rib cage. My back hurt, and my shoulders were also stiff because of the excessive internal rotation of my shoulders.

That’s when I searched the internet for an “ergonomic mouse” and decided to buy a Logitech (yes, Logitech again) MX Master 3, the most ergonomic mouse at that time.

Do you see those 2 navigation buttons? I find them very useful when binding IDE navigation keys to them so I don’t have to switch back and forth between the keyboard and the mouse (yes, I’m that kind of person who likes to use the mouse only). 2 buttons are great, but how about more, I asked myself.

That is when I decided to go all out with a full T9 keyboard on the rear of the mouse. Meet Razer Naga HyperSpeed.

If you are like me, here are some notes:

• Razer does not have an official driver for macOS, but the binding can be installed directly on the firmware. So a workaround is to install your bindings on another Windows machine.

• In order for the flashy light on the back of the mouse to work, you will need to use another driver: https://github.com/1kc/razer-macos

To be honest, I was impressed with Temporal’s capabilities, but it stopped there. Like, I know how to build a workflow on top of Temporal, but I don’t know how it works, where the pitfalls are, or how it operates at a larger scale.

Somehow that brings me to my next adventure, where I’m actually responsible for a workflow system. I chose to learn Temporal “the hard way,” and the gains are quite amazing. Now I feel more comfortable navigating a large codebase and can grasp the design and implementations of other systems quicker. I’m excited to see where my decision will take me. But until then, cerca trova. 👨‍🚀

Background story

In our daily, our cloud operation team operates more than a hundred accounts within our AWS Organization. The process of granting user access to an account might appear straightforward with just a few accounts, but as we continue to grow, managing a vast array of permissions presents its own distinct challenges.

For example, an operator may need to request temporary access to a project team's AWS account to perform maintenance work. With hundreds of requests like this daily, our team burnt out. We realized that action needed to be taken.

In this blog post, we will explore JIT, our internal access manager that helps make cloud access more secure.

Solution

Why not use an existing solution, you may ask. Although there are some solutions out there from CyberArk and even AWS itself, we have a few constraints that are convincing enough to create our own masterpiece:

• We need a more customizable approval step. An approval step might come from different people, with the ability for an approver to delegate their job to others while they're busy, for example, on vacation.

• We need to have a mechanism for grouping permission sets. In AWS, permission sets are created and ready to use for any account in its AWS organization.

• We want to allow ticket requesters to extend their ticket time frame.

• Our requester might be a team manager who wants to request tickets for other users.

  With that being said, let's make a quick rusty product requirement:

  • As a requester, I want to be able to:

    • Create a ticket for myself, another person(s), or both. The ticket should include: start time, end time, the desired permission set, the account to be accessed, and a brief description explaining the rationale for the ticket.

    • If my ticket is rejected, I want to edit and resend it to the approver(s).

    • If I need more time to complete my task, I can request that my approvers extend the end time of my ticket.

  • As a system administrator, I want to be able to:

    • Register and unregister accounts in my AWS organization to this system (JIT).

    • Register and unregister permission sets along with their access scopes to this system (JIT).

    • Configure approval rules for registered accounts: the number of approval stages, users in each stage, and the number of approvals required in each stage for it to be considered approved.

  • As an approver, I want to be able to:

    • Approve or reject tickets related only to the account(s) I am responsible for.

    • Have full control over the ticket I have approved: terminate them any time and audit user activities within the requested time frame for the targeted account.

Let's dive deeper.

We will split JIT into smaller services:

• Ticket service

• Approval service

• Approval rule service

• Permission set manager service

• Account manager service

• Notification service

• Audit service

Requester flow

Approver flow

Administrator flow

Demo

The requester requests a ticket with the necessary informations.

Approver will receive notification via email, they can log in to JIT portal via an Identity Provider that supports OpenID Connect. After the ticket is approved, user will be granted access.

Approver can track user activity in this time frame.

Discussions

• At the heart of JIT is Temporal, which we use to orchestrate background processes handling notifications, approval verification logic, and resource creation logic. Temporal greatly assists us in building durable workflows with multiple parallel logic branches. Additionally, Temporal's workflow status query and workflow signal help simplify a significant amount of implementation code. We can see what's happening in the workflow of each ticket and send signals to it.

• Since we are only creating account assignments behind the scenes, we've decided that Terraform (a popular IaC tool written in Go) is overkill. Instead, we create resources directly with the AWS Go SDK v2 and utilize Temporal workflows with a simple stack to record which resources were created, allowing us to clean them up afterward.

• Authorization in JIT is implemented with Casbin, a popular authorization library also written in Go. Each approval rule/permission set access scope is converted to a Casbin policy. With a delicate use of Casbin, we can enforce our own customizable logic into AWS permission set 🎉

• Building the UI was also a challenge for us, as the UI needed to be as familiar as possible to end users. Since we work in an enterprise that just adopted AWS recently, we want users to feel at home whether they are using the actual AWS console or the JIT console. Fortunately, AWS has its design system open-sourced under the name Cloudscape. Users will have an AWS-like experience when using JIT.

Conclusion

Managing access permissions in an AWS Organization can be a daunting task, especially with a growing number of accounts. Creating a custom solution like JIT, that is tailored to the specific needs of your organization, can significantly streamline the process. By breaking down the system into smaller services and leveraging existing tools and libraries like Temporal, Terraform, AWS Go SDK, and Casbin, we can build a robust, efficient, and secure access management system. Furthermore, using an AWS-like UI design system such as Cloudscape can provide a familiar user experience. We look forward to sharing JIT with the wider community soon.