Why Your System Fails Like a Collapsing Building

Learn how a 1960s building collapse holds the key to preventing catastrophic software outages and building more resilient distributed systems.

In a presentation for InfoQ, systems expert Sam Newman explained how a concept from civil engineering, known as "progressive collapse," is a critical lens for understanding modern software outages. He drew a direct line from the 1968 Ronan Point apartment building collapse, where a small gas explosion led to the failure of an entire side of the tower, to large-scale cloud service disruptions like those experienced by AWS. The core idea is that in a complex, interconnected system, a minor, localized fault can trigger a chain reaction that results in a catastrophic, system-wide failure. This isn't just a rare accident; it's an inherent risk in the highly distributed architectures that power today's digital services. Newman argues that by studying these structural failures, software leaders can learn valuable lessons about building more resilient digital infrastructure that can withstand unexpected stress and prevent a small problem from becoming a total disaster.

The mechanism behind a cascading failure in software often starts with a single, seemingly isolated event, such as a misconfiguration, a network partition, or a single service becoming overloaded. When one component fails, the services that depend on it begin to experience errors or timeouts. These dependent services might then retry their requests, increasing the load on the already struggling component or putting pressure on other parts of the system. This creates a domino effect that spreads across the architecture. To combat this, Newman highlights three key strategies borrowed from engineering. The first is strengthening individual components to make them more fault-tolerant. The second, and more crucial, is isolating failures using patterns like circuit breakers, which stop requests to a failing service, or bulkheads, which partition resources to prevent one failure from consuming all available capacity. Finally, he advises reducing unnecessary interconnections to simplify the system and limit the potential pathways for a failure to propagate.

Understanding and preventing cascading failures is no longer an academic exercise; it is a fundamental responsibility for anyone building or managing software. For CTOs and lead developers, this means shifting architectural focus from pure performance and feature velocity to resilience and graceful degradation. A system that can partially function during an outage is vastly superior to one that collapses entirely. For IT and security teams, a cascading failure can obscure the root cause of an incident, making it difficult to diagnose and resolve. It can also be triggered by a minor security event, amplifying its impact significantly. For business leaders, the consequences are direct and severe: major outages lead to lost revenue, damage to brand reputation, and a loss of customer trust that can take years to rebuild. The high-profile cloud outages of recent years demonstrate that even the most sophisticated technology companies are not immune to this threat.

The industry's widespread adoption of microservices and cloud-native architectures has made this problem more pressing than ever. While these designs offer benefits like scalability and independent deployment, they also create a vast network of dependencies where a failure in one small service can have unforeseen consequences across the entire ecosystem. The practical takeaway from Newman's analysis is that resilience must be an explicit design goal, not an afterthought. Teams must proactively plan for failure by identifying critical dependencies, implementing fault isolation patterns, and establishing clear service level objectives. This requires a cultural shift towards embracing failure as an inevitable part of complex systems. Instead of asking "if" a component will fail, resilient engineering asks "what happens when" it fails, and builds the necessary safeguards to ensure the system as a whole survives.

This engineering philosophy has given rise to the discipline of Chaos Engineering, a practice that directly applies these principles. Pioneered by companies like Netflix, Chaos Engineering involves deliberately injecting failures into a production system to test its ability to withstand unexpected disruptions. By proactively breaking things in a controlled environment, teams can uncover hidden weaknesses, validate their assumptions about system behavior, and fix vulnerabilities before they cause a real-world cascading failure. This approach treats resilience not as a static property but as an ongoing practice of experimentation and learning. As systems continue to grow in complexity, adopting such proactive, evidence-based methods for building and testing resilient software will become an essential strategy for ensuring services remain available and reliable for users.