How Modal Launches a Million Sandboxes Instantly

Engineers at Modal rebuilt their infrastructure to launch millions of concurrent sandboxes, creating tens of thousands of them every single second.

Engineers at the cloud computing company Modal have detailed how they rebuilt their core infrastructure to achieve a massive new scale, according to a report in InfoQ. Staff engineers Colin Weld and Connor Adams explained the project was designed to support millions of concurrent sandboxes—isolated, secure environments for running code—and create tens of thousands of them every second. This level of performance was necessary to power Modal's platform, which offers developers instant access to GPUs and other cloud resources. The engineering team found that existing tools were insufficient for their needs, prompting a ground-up redesign that moves beyond the capabilities of standard container orchestrators like Kubernetes.

The decision to move "beyond Kubernetes" stems from the inherent trade-offs in general-purpose orchestration systems at extreme scale. While Kubernetes is the industry standard for managing containerized applications, its architecture can introduce bottlenecks when dealing with an exceptionally high rate of very short-lived tasks. Key components like the central etcd key-value store and the general-purpose scheduler are not optimized for creating and destroying millions of containers in minutes. Modal's custom solution likely involves a highly specialized, distributed control plane that avoids these single points of contention. This would include a purpose-built scheduler optimized purely for speed and resource placement, a streamlined container runtime interface that minimizes startup latency, and a simplified networking model designed for ephemeral environments. By stripping away features not essential to their specific use case, Modal’s engineers could focus entirely on reducing the overhead associated with each sandbox's lifecycle, from request to termination.

Modal's approach places it in a small but influential group of technology companies that have outgrown mainstream infrastructure solutions. While managed services like Amazon's EKS and Google's GKE provide a powerful and flexible foundation for most businesses, they are designed to serve a wide variety of workloads. Companies with unique, hyper-scale demands often find that a bespoke system offers a crucial competitive edge in performance and cost. For example, Cloudflare built its own orchestration system around v8 Isolates to power its Workers platform, and Netflix developed its Titus container manager to handle its massive media-processing pipelines. These custom platforms trade the broad compatibility of off-the-shelf tools for superior efficiency and control over a specific, business-critical function. Modal’s work is a contemporary example of this trend, tailored for the new wave of AI and serverless computing that demands instant, on-demand resource allocation.

For CTOs and infrastructure teams, Modal's story is not a directive to abandon Kubernetes but a valuable lesson in understanding its limits and knowing when to consider a custom build. The key takeaway is the importance of aligning infrastructure choices with specific workload characteristics. For the vast majority of companies, the feature richness and robust ecosystem of Kubernetes remain the optimal choice. However, for businesses whose core product depends on performance metrics that push beyond industry norms—like sub-second cold starts for tens of thousands of concurrent functions—a specialized solution can become a strategic necessity. The critical question for leaders is to identify the point at which the cost of working around a generic tool's limitations exceeds the investment in building a custom one. Going forward, the industry will be watching to see if the patterns developed by Modal are open-sourced or influence the next generation of cloud-native orchestration platforms.

For engineers and CTOs, Modal's architecture provides a rare, detailed look at the breaking points of standard tools like Kubernetes at massive scale. It demonstrates a practical approach for building custom container orchestration when off-the-shelf solutions are no longer sufficient for hyper-growth performance and cost-efficiency.

Modal's ability to instantly provision millions of sandboxes creates a significant competitive advantage, enabling new AI and data processing products that were previously unfeasible. This engineering feat directly translates to a superior user experience and potentially lower operational costs compared to scaling traditional cloud infrastructure.