Your Incident Response Plan Is a Fantasy

Marathon outages expose the gap between planned work and reality, revealing hidden weaknesses in your systems, teams, and overall organization.

According to a presentation at InfoQ by Vanessa Huerta Granda, the most challenging and prolonged technical incidents do more than just take down a service—they expose the fundamental gap between how an organization plans to work and how it actually functions under pressure. These "marathon incidents" act as a powerful diagnostic tool, revealing hidden organizational fragility, the real limits of human endurance, and the complex, often undocumented, interdependencies within modern software systems. The key insight is that resolving these crises is less about a single heroic engineer finding a silver bullet and more about the entire organization's ability to sustain a coordinated, long-term response. The failure to prepare for this reality is why many well-documented incident response plans crumble when faced with a truly complex, unending outage. This perspective shifts the focus from purely technical solutions to a more holistic view that incorporates human factors and organizational design as critical components of resilience.

The core concept Huerta Granda highlights is the disconnect between "work as imagined" and "work as done." Work as imagined is captured in our runbooks, flowcharts, and on-call policies; it assumes a linear, predictable process where a known problem has a known solution. Work as done, however, is the messy, adaptive reality of navigating a crisis with incomplete information. During a marathon incident, engineers aren't just following a script. They are actively discovering the nature of the problem, which often involves cascading failures across multiple systems that no single person fully understands. This process is mentally taxing and exposes the hard limits of human cognition and stamina. Fatigue leads to errors, communication breaks down, and institutional knowledge that exists only in certain individuals' heads becomes a critical bottleneck. The incident becomes a test not of the system's design, but of the team's ability to learn and adapt under extreme duress.

This challenge is not new, but it has been amplified by the prevailing trends in software architecture. The widespread adoption of microservices, distributed systems, and complex cloud infrastructure has created applications that are more powerful and scalable, but also significantly more opaque. A failure in one seemingly minor service can trigger a chain reaction with unpredictable consequences, making root cause analysis incredibly difficult. This landscape is precisely what the field of Resilience Engineering studies: how to build and manage systems that can function reliably despite inevitable component failures and human error. The ideas presented by Huerta Granda build on this foundation, arguing that true resilience isn't about preventing all failures. Instead, it's about building socio-technical systems—combinations of people, processes, and technology—that can gracefully absorb, adapt to, and recover from unexpected events, especially those that last for days rather than hours.

The practical takeaway for leaders and teams is to shift their incident response strategy from one of pure reaction to one of structured endurance. This begins with acknowledging that people are the most critical and most fragile part of the response. Implementing "humane rotations" is a crucial first step, which involves creating clear protocols for handing off responsibilities between on-call engineers to ensure they get adequate rest. This prevents burnout and reduces the likelihood of costly mistakes made by exhausted responders. Furthermore, teams should foster "holistic cross-functional coordination" long before a crisis strikes. This means running realistic drills that involve not just engineering and operations, but also customer support, legal, and communications teams. By practicing together, these disparate groups build the shared context and trust necessary to function as a cohesive unit during a high-stakes, long-running incident. The ultimate goal is to cultivate an organization that is prepared to endure, adapt, and learn, rather than one that simply hopes a crisis will never happen.

For engineers and SREs, this highlights that technical skill alone cannot resolve complex incidents. Understanding human factors, cognitive load, and organizational dynamics is crucial for building resilient systems and avoiding burnout during high-stakes, prolonged outages that affect core services.

Extended downtime directly impacts revenue, customer trust, and brand reputation. Failing to prepare for marathon incidents creates significant operational risk, as exhausted teams make more mistakes. This framework helps leaders build organizational resilience, reducing the financial cost of major outages.