Enterprise

Datadog Adds Online Experiments for AI Agent Testing in Production

New capability lets teams compare agent versions against real user traffic, not just static test datasets.

Omega Editorial · November 17, 2026 · 3 min read

Key takeaways

  • Datadog's Online Experiments feature enables teams to test AI agent changes against real production traffic, not just static datasets.
  • The platform combines agent quality scores with operational metrics like latency, token consumption, cost, and errors in side-by-side comparisons.
  • One experiment showed a 44.3% cost reduction with a newer model but revealed increased invalid tool parameters through trace inspection.
  • Online experiments create a feedback loop where production failures become new regression test cases for offline evaluation.
  • The approach shifts agent testing from passing predefined evaluations to proving better performance under real user conditions.

Datadog has introduced Online Experiments to its Agent Observability platform, enabling engineering teams to test AI agent modifications against live production traffic rather than relying solely on static evaluation datasets. The feature combines agent quality scores with operational metrics including latency, token usage, execution cost, and error rates.

The capability addresses a fundamental limitation in agent development: changes that perform well in controlled tests may behave unpredictably when exposed to real user requests. By integrating controlled experimentation with production telemetry and execution traces, teams gain visibility into both outcomes and the underlying behaviors that produce them.

How online experiments work

Datadog's approach allows teams to run controlled comparisons by randomly distributing production traffic between a baseline agent and a modified variant. This reduces bias from differences in user request patterns and provides a realistic assessment of performance under actual workloads.

The platform integrates its Agent Observability evaluators with its Experiments capability, enabling side-by-side comparison of evaluation scores and conventional operational metrics. Teams can track response quality, accuracy, and completeness while simultaneously monitoring execution duration, token consumption, estimated cost, and workflow errors. Feature flags control traffic allocation, allowing engineers to expose changes to a small user percentage before broader rollout.

Trade-offs revealed through production data

According to InfoQ, one experiment compared GPT-5.6 Luna against GPT-6 Luna with evenly split traffic. An LLM-based evaluator scored investigation quality on a one-to-five scale, considering accuracy, completeness, and evidence support. Results showed a 44.3% reduction in estimated cost for the newer model, with quality scores remaining nearly identical at 3.84 and 3.86.

However, operational metrics revealed potential concerns. The newer model showed estimated increases in execution duration and workflow errors, though confidence intervals were too broad for definitive conclusions. Trace inspection revealed the newer model supplied invalid tool parameters more frequently, typically recovering after retries—behavior not sufficiently visible in overall quality scores.

Datadog used these findings to improve tool handling before proceeding with the model change. The failed calls also provided candidates for additional offline regression tests.

Why it matters

Offline evaluations remain valuable but are limited by the representativeness of test datasets and scoring criteria. Real user workflows, unexpected requests, and changing tool responses can expose weaknesses that existing test cases never anticipated. Online experiments create a feedback loop between production behavior and development testing, where unexpected failures become new regression cases and subsequent experiments establish whether changes actually resolve them. For organizations deploying AI agents at scale, this shifts the question from whether an agent passes predefined evaluations to whether it performs better under real conditions.

Datadog connects these capabilities within its existing observability ecosystem, linking evaluation results directly to production experiments and execution evidence. Other tools including LangSmith, Langfuse, and LaunchDarkly offer related capabilities for agent tracing, evaluation, and feature rollouts.

These details were first reported by InfoQ.

  • #ai agents
  • #observability
  • #production testing
  • #datadog
  • #llm evaluation
  • #feature flags

This is an original analysis by the Omega editorial team. Source reporting: AI Watch.

Want systems like this working for your business?

Book a Call

More in Enterprise