Datadog Now Tests AI Models on Live Traffic
TL;DR: Datadog's new Online Experiments feature lets developers test different AI agent versions directly on live user traffic. This helps teams see how models perform in the real world, not just in controlled lab environments.
Key facts
- Category
- AI
- Impact
- High
- Published
- Source
- InfoQ
Full summary
Datadog's new feature lets teams test AI agent versions on live production traffic, seeing how they perform in the real world.
Datadog has expanded its AI Agent Observability platform with a new feature called Online Experiments, according to a report from InfoQ. The tool is designed to help engineering teams move beyond traditional testing methods by allowing them to evaluate and compare different versions of their AI agents using live production traffic. For years, developers have relied on static, predefined datasets to gauge a model's performance before deployment. However, this approach often fails to capture the unpredictable nature of real-world user interactions and edge cases. Online Experiments aims to solve this problem by providing a framework for testing in the very environment where the AI will ultimately operate, offering a more accurate picture of how a model will behave after it ships. This shift from lab-based evaluation to live-environment testing represents a significant step in maturing the development lifecycle for AI-powered applications.
The new feature works by enabling teams to conduct controlled experiments, similar to A/B tests used in web development. Developers can configure Online Experiments to route a specific percentage of their production traffic to two or more different AI agent versions simultaneously. For example, a team could send 5% of user queries to a new, experimental model while the other 95% continues to be handled by the stable, current version. Datadog's platform then automatically collects and analyzes key performance indicators for each version in a side-by-side view. These metrics can include operational data like latency and cost per inference, as well as quality scores based on user feedback or automated evaluations. This allows teams to make direct, data-backed comparisons on how each model version performs under the exact same real-world conditions, eliminating the guesswork involved in offline testing.
Datadog's move is part of a larger industry trend focused on bringing the discipline and tooling of modern DevOps to the world of Machine Learning Operations (MLOps). While software engineering has long benefited from mature continuous integration, continuous delivery (CI/CD), and experimentation frameworks, the AI development lifecycle has been comparatively immature. The non-deterministic and complex nature of AI models, especially large language models (LLMs), makes them difficult to evaluate with simple pass/fail tests. This has created a critical gap between model development and reliable production deployment. Observability platforms are now racing to fill this void, building tools that help teams understand and manage the inherent uncertainty of AI. By providing a structured way to experiment in production, Datadog is addressing a core MLOps challenge and helping to standardize how AI products are safely and effectively rolled out.
For CTOs, developers, and IT teams, the practical implication of Online Experiments is a more confident and less risky path to deploying AI features. This capability directly tackles the common problem where a model performs perfectly in a controlled environment but fails or underperforms when exposed to the chaos of real user behavior. It provides the empirical data needed to decide which model version is truly better for the business, based on metrics that matter like user satisfaction, cost-efficiency, and response speed. Looking ahead, the next evolution in this space will likely involve greater automation. We can expect to see MLOps platforms not only facilitate these experiments but also begin to automate the decision-making process, such as automatically promoting a winning model version to 100% of traffic or triggering an alert and rollback if a new version's performance degrades below a certain threshold. This will further close the loop between development, testing, and operations for AI systems.
Why it matters
For developers and MLOps teams, this closes a critical feedback loop. Evaluating AI agents only on static datasets can miss real-world edge cases and user behaviors. Live traffic testing provides more accurate performance data, leading to faster iteration and more reliable AI products in production.
Business impact
Companies can now reduce the risk and cost of deploying ineffective AI models. By identifying performance issues with live data before a full rollout, businesses can avoid poor user experiences, protect revenue, and make more confident decisions about which AI versions to promote to all users.
Tags
Related on Notifire
Related stories
Primary source: InfoQ
