Is Your Observability Stack Counting Everything Twice?

A common cloud-native setup can accidentally double your traffic metrics. Here's the guide from CNCF on how to get accurate observability data.

A new guide from the Cloud Native Computing Foundation (CNCF) blog addresses a subtle but critical issue in modern cloud infrastructure. Many teams using service meshes like Istio for observability are finding their metrics are wrong. Specifically, they are seeing double the traffic they actually have, a problem the guide calls "zero+zero=two." This happens when the service mesh's own monitoring is combined with other application-level monitoring tools, such as OpenTelemetry, leading to duplicated data in systems like Prometheus. This isn't a bug in any single tool, but an integration problem that can mislead engineers during critical incidents.

A service mesh works by injecting a "sidecar" proxy next to each application container. This proxy intercepts all network traffic, automatically generating detailed metrics about requests, latency, and errors. This is powerful because it requires no changes to the application code. However, many teams also instrument their applications directly using libraries like OpenTelemetry to gather business-specific metrics or trace requests through the system. When both the service mesh sidecar and the application's OpenTelemetry agent report on the same HTTP request, monitoring systems like Prometheus scrape both sets of metrics, effectively counting each event twice. The result is inflated request counts and error rates, painting a false picture of system health.

This issue directly impacts Site Reliability Engineers (SREs), DevOps teams, and developers responsible for maintaining the health of distributed systems. Inaccurate metrics can trigger false alarms, wasting valuable engineering time on phantom problems. During a real outage, doubled error rates could send teams down the wrong diagnostic path, prolonging downtime. For CTOs and engineering leaders, this erodes trust in the observability platform, which is a significant investment. If the data can't be trusted, it becomes impossible to set accurate Service Level Objectives (SLOs) or make informed decisions about scaling and performance optimization. The problem is widespread because using a service mesh alongside application performance monitoring (APM) is a standard best practice.

The practical solution, as outlined in the CNCF guide, involves carefully configuring the monitoring system to de-duplicate this data. The primary technique is to use specific labels in Prometheus to distinguish between metrics originating from the service mesh proxy and those from the application itself. Engineers can then write queries that select only one source of truth for standard traffic metrics, while still using the other for its unique data. For example, they might rely on the mesh for network-level data and use OpenTelemetry for application-specific data. The key takeaway for businesses is that simply deploying powerful tools isn't enough; careful integration and configuration are essential to realize their value.

This double-counting problem is a symptom of a larger trend in the cloud-native world: the convergence of different observability pillars and layers. As stacks become more complex, the potential for overlapping and conflicting data sources increases. The industry is moving towards standards like OpenTelemetry to unify data collection, but as this guide shows, integration at the platform level remains a significant challenge. Teams should anticipate these "seams" between tools and proactively design their observability strategy to handle them, rather than waiting for an incident to reveal the flaws in their data.

Inaccurate metrics from double-counting can mislead engineers during outages, trigger false alarms, and erode trust in the entire observability system. This makes it difficult to set reliable performance goals (SLOs) and diagnose real problems quickly.

Businesses relying on flawed data risk making poor decisions about scaling, resource allocation, and system reliability. The fix requires careful configuration, highlighting that the value of expensive observability tools depends on expert integration to avoid costly errors and wasted engineering time.