How a 20-Year-Old Game Solved Modern Observability

Eve Online uses smart, dynamic sampling to monitor its massive game architecture without degrading performance or breaking the bank on telemetry.

In a presentation at InfoQ, Nicholas Herring of CCP Games detailed how the company behind the 20-year-old MMO Eve Online tackles the immense challenge of observability. The game runs on a complex, high-throughput architecture that simulates a real-time universe for thousands of concurrent players. The engineering team needed a way to implement distributed tracing to understand system behavior, but faced a critical problem: collecting enough data to be useful without degrading game performance or generating astronomical telemetry costs. Their solution combines several advanced techniques, including head-based sampling, a dynamic exponential moving average (EMA), and deterministic sampling to create an intelligent and cost-effective monitoring system.

At the core of Eve Online's strategy is a smarter way to decide which data to keep. Instead of sampling randomly at the beginning of a request, their system uses head-based sampling. This method allows the system to begin a trace and then make a sampling decision based on events that happen early in its lifecycle. For example, if a trace contains an error or shows unusually high latency, it is flagged for collection, ensuring that the most important diagnostic data is never missed. This is coupled with a dynamic sampling rate controlled by an EMA algorithm, which automatically adjusts how aggressively it samples based on current system load. During massive in-game fleet battles, the system can reduce its sampling rate to avoid being overwhelmed, then increase it again during quieter periods.

The challenges faced by Eve Online are an extreme example of a problem plaguing the entire tech industry: the observability data deluge. As applications become more distributed and complex, the volume of logs, metrics, and traces they generate can explode, leading to massive bills from monitoring platforms and overwhelming engineering teams with noise. Many organizations resort to simple probabilistic sampling, like keeping only 1% of all traces, but this is a blunt instrument. This approach often fails to capture the rare but critical events that cause production incidents. Eve Online's approach represents a significant evolution, moving from brute-force data collection to a surgical, context-aware strategy that prioritizes high-signal data over high-volume data.

For CTOs, developers, and SREs, this case study offers a powerful lesson in modern system management. An effective observability strategy must be as sophisticated as the architecture it monitors. Simply enabling a tool and shipping all possible data is no longer a viable or financially responsible approach. Teams should instead evaluate whether their current sampling methods are truly serving their needs or just inflating their costs. By implementing intelligent, dynamic policies like head-based sampling, organizations can gain deeper insights into the issues that actually impact users and system health. This allows them to debug faster and maintain high performance, all while keeping their observability budget under control.

This case study provides a rare, concrete blueprint for managing observability in high-throughput, legacy systems. For SREs and platform engineers, Eve Online's use of dynamic, head-based sampling demonstrates a practical way to gain deep system insights without incurring prohibitive performance penalties or telemetry costs.

Eve Online's approach directly translates to lower operational costs by reducing telemetry data volume and storage needs. For any company running large-scale services, this method offers a path to better system reliability and performance diagnostics while maintaining a predictable and controlled budget for observability tooling.