Beyond Kubernetes at Modal: How to Scale 1 Million Concurrent Sandboxes in Seconds

8.8 relevance

Modal's custom sandbox architecture at scale is a deep infrastructure case study.

Cloud infoq.com

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For a solutions architect focused on cloud infrastructure and AI/ML agent orchestration, this demonstrates a concrete architectural pattern to escape Kubernetes' coordination overhead when building low-latency, high-density execution environments for sandboxes or agents.

Key Takeaways

Decouple coordination from execution by replacing centralized scheduling with horizontal load-balancing and direct worker RPC, and use worker self-reporting to a fast shared stream for state.

Beyond Kubernetes at Modal: How to Scale 1 Million Concurrent Sandboxes in Seconds

Summary

Modal engineers rebuilt their sandbox infrastructure to support 1 million concurrent sandboxes, abandoning Kubernetes because its centralized etcd store and serialized scheduler become bottlenecks at high pod churn. Their new architecture uses a fleet of parallel scheduling servers that load-balance across workers via direct RPC, each worker self-reporting state to Redis. This design achieved 50,000 sandbox creations per second with median startup-to-code time under 0.5 seconds, while the sole Redis bottleneck scales beyond 100,000 workers.

Author

Sergio De Simone