How to Find Hidden Delays in Your Systems

Gearset engineers used OpenTelemetry tracing to find and fix hidden performance bottlenecks in their complex distributed queueing systems.

Engineers at the software company Gearset were struggling with a common but frustrating problem: performance bottlenecks hidden deep within their distributed asynchronous queues. According to a presentation at InfoQ by Oli Lane and Julian Wreford, traditional metrics like queue length and processing rates were not providing the answers they needed. These high-level indicators showed that something was wrong, but they couldn't explain why certain tasks were slow or failing. This lack of visibility led to guesswork and lengthy, often fruitless, debugging sessions. To break this cycle, the team decided to implement OpenTelemetry tracing, a modern observability framework, to get a detailed, end-to-end view of how tasks moved through their systems. This shift allowed them to move beyond simply monitoring symptoms and start diagnosing the root causes of their performance issues with concrete data.

The fundamental limitation of traditional metrics is that they provide an aggregated, outside-in view of a system's health. A metric might show that the average task processing time is acceptable, but it completely hides the experience of the slowest five percent of users who are facing significant delays. OpenTelemetry tracing works differently by following a single unit of work, or "trace," as it travels across various services and components. When a task enters a queue, it is assigned a unique ID that is passed along at every step of its journey. This creates a detailed timeline showing exactly how long the task spent waiting in the queue, how long it took to be processed, and how much time was spent in any downstream services it called. By analyzing these individual traces, Gearset's engineers could finally see the hidden delays, unnecessary retries, and inefficient processing logic that were invisible to their old monitoring dashboards. This granular, inside-out perspective is what transforms debugging from an art of intuition into a science of evidence.

Gearset's experience reflects a broader industry-wide shift from monitoring to observability. As companies increasingly adopt complex, distributed architectures like microservices, the old approach of monitoring the health of individual servers or applications is no longer sufficient. A single user request might touch dozens of services, and a problem in one can cause a cascade of issues that are nearly impossible to track with isolated metrics. Observability addresses this by providing tools to understand a system's internal state by analyzing its outputs—namely traces, metrics, and logs. OpenTelemetry has emerged as the de facto open-source standard for collecting this data, offering a unified set of APIs and libraries that work across different programming languages and platforms. This standardization prevents vendor lock-in with specific observability platforms and creates a common language for developers and operations teams to discuss and solve performance problems, making it a cornerstone of modern reliability engineering.

The most practical lesson from Gearset's journey is the importance of aligning engineering efforts with user experience through Service Level Objectives (SLOs). Instead of tracking abstract system metrics, the team began defining SLOs based on the latency distribution of their queues. For example, they set a goal that 99% of all critical background jobs must complete within a specific time frame. OpenTelemetry tracing provided the precise data needed to measure and enforce these user-centric goals. For other engineering teams, the takeaway is to start instrumenting their most critical asynchronous workflows to gain similar insights. This doesn't require a complete overhaul; teams can begin with one key business process to prove the value of tracing. This approach not only helps in debugging current issues but also fosters a proactive culture of reliability, where performance is treated as a core feature of the product, not an afterthought.

Many engineering teams struggle to diagnose issues in complex asynchronous systems where problems are hard to trace. This case study provides a concrete blueprint for using OpenTelemetry to gain deep visibility, define meaningful performance goals, and improve overall system reliability.

Unseen system delays directly harm user experience, leading to slow features and customer frustration. Pinpointing these bottlenecks helps companies improve product reliability, reduce engineering time wasted on guesswork, and protect revenue by ensuring critical background tasks complete on time.