NVIDIA NeMo Switchyard Routes AI Agent Tasks Across Models

Dynamic model routing for AI agents

AI agents rarely benefit from sending every task to the same large language model. A single agentic workflow may require classification in one step, complex reasoning in another, and routine text generation for follow-up—each with different accuracy, latency, and cost requirements. NVIDIA has released NeMo Switchyard, an open-source orchestration layer that routes agent workloads across multiple models based on task complexity, model capability, and deployment constraints.

The system evaluates each request at runtime and directs it to the most appropriate model in a defined pool. According to details first reported by NVIDIA, this approach can reduce costs substantially while maintaining quality thresholds for production workloads.

Why it matters

As enterprises deploy AI agents for coding, customer support, and workflow automation, the economics of model selection become critical. Routing every request to a frontier model inflates costs and latency; routing everything to a smaller model sacrifices quality on complex tasks. NeMo Switchyard addresses this tradeoff by making model orchestration practical without requiring developers to rebuild applications around each provider or model choice. Early benchmarks show meaningful cost reductions—up to 74% in some evaluations—with controlled accuracy tradeoffs, making agent deployments more economically viable at scale.

How the routing system works

NeMo Switchyard uses a provider-agnostic SDK that separates routing logic from specific model endpoints. Each model target receives a semantic name, while the underlying client maps that name to the actual provider and model ID. This architecture allows teams to update models, change providers, or adjust endpoints without modifying routing integration.

The system draws routing signals from three areas: model capabilities for specific task types, cost and latency profiles for each model, and infrastructure-level signals such as load and error rates. Routing decisions can occur at the request level, at each turn in a multi-turn conversation, or at individual steps within a complex task.

NeMo Switchyard supports both tuning-free and tunable routers. Tuning-free options include an LLM classifier that uses a model as a judge to select candidates, a stage router that examines tool activity to match agent phases with appropriate model capability, and an escalation router that starts with a lower-cost model and moves to a more capable one when sustained difficulty is detected.

Tunable routers learn from workload data. The prefill router, for example, extracts signals from an LLM's residual stream during the prefill phase to estimate query complexity, then predicts the likelihood each candidate model will successfully complete the task. A policy layer blends predicted accuracy with cost and latency constraints to select the optimal model.

Benchmark results and partner integrations

LangChain tested NeMo Switchyard on 145 multi-turn agentic tasks spanning customer support, incident investigation, and workflow automation. Using the escalation router to route between NVIDIA Nemotron 3.5 Lightning and Claude Opus 4.8, the system achieved a 74% cost reduction compared to using only the frontier model, sending just 7% of calls to the more capable model with approximately a 6-point accuracy tradeoff.

Cognition deployed NeMo Switchyard's staged-routing methodology in Devin Desktop for production-grade coding tasks. Routing between Opus 5 and Kimi K2.7, the system achieved 50.6% accuracy on the FrontierCode Main benchmark at a mean cost of $3.11—within 2.8 percentage points of Opus 5 accuracy at roughly 28% lower cost.

NVIDIA is integrating NeMo Switchyard with partners including Cognition, Nous Research, Ramp, LangChain, LiteLLM, Kong, Classmethod, Boomi, Cadence, and Siemens. The system is fully open source and available on GitHub.

These details were first reported by NVIDIA in a technical post on the NVIDIA Developer Blog.