NVIDIA GB300 Delivers 20x Gain on First Agentic AI Benchmark

First Standard for Measuring Agentic Workloads

AI agents that write code, call tools, and make sequential decisions have created inference workloads fundamentally different from traditional chatbot queries. Until now, no standard existed to measure how well hardware handles these complex, non-deterministic patterns. Artificial Analysis has introduced AA-AgentPerf, the industry's first multi-vendor benchmark designed specifically for agentic coding tasks, and NVIDIA's GB300 NVL72 platform has set the opening performance bar with results up to 20 times better than the prior-generation H200.

AA-AgentPerf measures how many concurrent AI agents an inference system can support while meeting service level objectives for output token speed and time-to-first-token. The benchmark uses prerecorded trajectories—complete sequences of actions, decisions, and tool calls—drawn from real coding tasks across public repositories, multiple programming languages, and frontier model responses. Request lengths range from 5,000 to 131,000 tokens with a mean around 27,000, and tool calls are simulated with representative CPU-side delays averaging one second.

The test harness sends thousands of concurrent requests to GPUs and measures the highest concurrency level that satisfies strict latency thresholds across an entire agent session. Results are normalized per accelerator and per megawatt, enabling direct comparison across hardware configurations. The test set remains private to prevent vendors from optimizing specifically for the benchmark.

Why It Matters

As enterprises deploy AI agents for software development, customer support, and workflow automation, infrastructure teams need reliable metrics to plan capacity and compare platforms. AA-AgentPerf provides the first apples-to-apples standard for evaluating how systems handle long-context reasoning, tool use, and sustained multi-turn sessions—workloads that differ sharply from single-shot inference. The 20x performance gap between generations signals that purpose-built hardware for agentic workloads delivers step-function economic advantages at data center scale.

NVIDIA's Architecture for Agent Scale

NVIDIA GB300 NVL72 achieved 61,400 concurrent agents per megawatt in initial testing, compared to 2,600 for H200. Per-GPU concurrency reached 57.5 agents versus 1.4 on the previous platform. These gains stem from tight integration across the stack.

The system links 72 GPUs into a single NVLink fabric, allowing rapid sharing of parameters, key-value cache, and intermediate results critical for coordinated agent execution. Agent runtimes including SGLang, TensorRT LLM, and vLLM apply optimizations like WideEP and DeepEP to distribute mixture-of-experts computation across the full 72-GPU domain, maximizing batch sizes and utilization.

DeepGEMM and Mega MoE optimizations use MXFP4 and MXFP8 kernels with fused operations that overlap NVLink communication with tensor core computation, boosting throughput for reasoning and code generation phases.

NVIDIA's upcoming Vera Rubin platform is expected to extend these results further with 50 petaflops of NVFP4 compute and a Vera CPU designed to accelerate LLM tool calls, addressing the CPU-side bottlenecks that currently limit end-to-end agent performance.

Details of the benchmark methodology and results were first reported by NVIDIA in a technical blog post.