Uber Finds What Distributed Tracing Misses

Uber developed a new system to automatically find rare failures that traditional distributed tracing often misses across its massive service mesh.

Uber has engineered a new system to automatically map and analyze the complex web of dependencies within its massive microservices architecture. According to a report from InfoQ, the company developed this tool to address a critical gap in modern observability: the inability of standard methods, like sampled distributed tracing, to reliably detect rare but significant failures. In an environment with thousands of interconnected services, a minor error in one component can trigger a cascade of problems elsewhere. Uber's system is designed to find these hidden connections by correlating failures between services, providing engineers with a clearer understanding of how their systems behave—and break—under real-world conditions. This moves beyond simply knowing that a service failed; it aims to answer why it failed and what other services were impacted. The goal is to create a dynamic, data-driven map of dependencies that reflects the actual operational state of the network, rather than relying on static documentation or incomplete trace data. By automating this analysis, Uber can continuously monitor the health of its service mesh and identify potential weak points before they escalate into widespread outages that affect customers. This represents a significant investment in proactive reliability, shifting the focus from post-incident forensics to pre-emptive risk mitigation.

The system's novelty lies in its approach, which combines data from two key sources: RPC (Remote Procedure Call) middleware and production metrics. Instead of relying solely on sampled traces, which might miss infrequent errors, Uber's tool instruments the very communication layer that connects its services. Every time one service calls another, the middleware logs critical metrics about the success or failure of that request. The system then aggregates this data at scale, using statistical analysis to find correlations between a caller service experiencing errors and a downstream service it depends on also showing errors. This allows it to infer a dependency and, more importantly, classify its nature. Dependencies are categorized as either "fail-close," meaning a failure in the downstream service directly causes the caller to fail, or "fail-open," where the caller has built-in resilience (like a fallback or cache) and can continue operating despite the downstream issue. A third category, "unknown," is used for dependencies that require further investigation. This classification is crucial, as it allows engineers to immediately prioritize fixing the high-risk "fail-close" dependencies that pose the greatest threat to system stability.

Uber's work fits into a broader industry trend toward more sophisticated reliability engineering, a discipline pioneered by companies like Google and Netflix. As more organizations adopt microservice architectures, they face the same challenges of scale and complexity that Uber does. The tools and practices for managing these systems, collectively known as observability, have evolved rapidly, but gaps remain. Distributed tracing, for example, is a powerful tool for debugging specific requests, but its reliance on sampling makes it less effective for understanding systemic, low-frequency failure modes. A one-in-a-million error is statistically invisible to most sampling strategies, but in a system handling billions of requests, it can still impact thousands of users. Uber's approach acknowledges this limitation and represents a move toward a more holistic, metric-driven form of analysis. It complements tracing by providing a macro-level view of system-wide failure patterns. This reflects a growing understanding that no single observability tool is a silver bullet; true system resilience requires combining data from multiple sources—metrics, logs, and traces—to build a comprehensive and accurate picture of system health.

The key lesson for other engineering organizations is the power of building observability directly into the foundational layers of their infrastructure, such as the RPC framework. By capturing detailed metrics at the point of communication, teams can unlock insights that are difficult to obtain from application-level instrumentation alone. Uber's system demonstrates that a deep understanding of dependencies is not just an architectural exercise but a critical, ongoing operational requirement. For teams managing their own service meshes or using platforms like Kubernetes, this approach serves as a blueprint for moving beyond reactive incident response. The focus shifts to proactively identifying and hardening the most critical failure paths in the system. Looking ahead, we can expect to see these concepts influence the development of open-source observability tools and commercial platforms. The demand for automated failure correlation and dependency classification will likely grow, pushing the industry toward solutions that provide a more complete and actionable view of how complex distributed systems truly operate and fail.

For engineers managing large-scale microservices, this offers a new blueprint for reliability. Uber's method moves beyond sampled tracing to provide a more complete picture of dependencies, helping teams proactively identify and classify critical failure points before they cause major outages.

This system directly reduces the risk and cost of downtime in complex digital platforms. By automatically identifying high-risk dependencies, companies can better allocate engineering resources, improve system resilience, and protect revenue by preventing cascading failures.