How Zepto Cut Support Costs 65% With Evaluation-First AI Agents

How Zepto Cut Support Costs 65% With Evaluation-First AI Agents

Zepto, one of India's fastest-growing quick-commerce platforms, processes more than 100,000 customer support tickets daily through a multi-agent AI system. The company reduced support costs by 65 percent and achieved payback in under one month by treating evaluation as core infrastructure rather than an afterthought.

The approach centers on what Zepto calls a dual-loop architecture: a development loop where agents are designed and tested against quality gates, and a production loop where live behavior is monitored and failures are fed back to improve future iterations. The system runs on Databricks and MLflow, with evaluation serving as the primary mechanism for building, testing, and operating agents at scale.

Why it matters

Most organizations ship AI agents quickly but struggle when volume scales. At 100,000 tickets per day, even a one percent error rate creates thousands of bad customer experiences and measurable revenue leakage. Zepto's framework offers a concrete pattern for engineering reliability into agentic systems before problems reach customers, turning evaluation from a nice-to-have into the foundation that makes high-volume AI operations viable.

The assurance gap in agentic systems

Agentic workflows operate as multi-step processes—classifying intent, retrieving knowledge, reasoning through decisions, calling tools, and generating responses. Failures can emerge at any step, not just in the final answer. According to the engineering team at Zepto, this creates an assurance gap where problems remain invisible until customers complain, fixes take too long, and teams lack a principled way to balance cost against quality.

The company partnered with Databricks to close that gap by making evaluation the primary way agents get built and operated, not by shipping more agents.

Building the dual-loop framework

The development loop is where engineers design and iterate on agent versions. Every change triggers automated regression testing against a golden dataset—a curated collection of normal, edge, and failure cases that represents real-world scenarios. The dataset grew from 500 examples with an eight-point accuracy gap between development and production to 5,247 examples with a 0.4-point gap over six months.

The production loop monitors live agent behavior using stratified sampling. Rather than evaluating all traffic uniformly, the system samples at higher rates for high-value customers, new features, negative sentiment signals, and fraud-prone interactions. This approach captures 45 to 60 percent of edge cases while evaluating only 18 to 20 percent of total traffic, reducing review costs by 86 percent compared to uniform sampling.

A quality gate sits between the loops. New agent versions must meet thresholds across multiple dimensions—customer experience, operational efficiency, risk, and cost—and perform at least as well as the production baseline before deployment. If a version fails, it gets rejected and the existing agent continues serving traffic.

Tracing and scoring at scale

Every agent invocation emits a detailed execution trace using MLflow, capturing prompts, completions, retrieved documents, tool calls, and decision paths. The system uses what Zepto calls an "AI jury"—a combination of built-in judges for standard metrics, custom LLM-based judges calibrated to 80 to 90 percent agreement with human labels, and code-based scorers for deterministic checks like latency.

Critical alerts monitor intent accuracy, groundedness violations, escalation risk, and latency every five minutes. When the system detected riders stuck in traffic while the agent repeated "arriving in 10 mins" from cached data, token-usage and escalation-risk scorers surfaced the issue within five minutes. That insight led to a new feature: proactive cancellation offers when riders remain stationary for more than ten minutes.

Results and architecture

The framework now manages more than 80 percent of support tickets with human oversight, improved customer satisfaction by 20 percent, and cut time to resolution by four times. Development cycles accelerated three times. The system uses vertical agents for specific intents—order tracking, missing items, returns, payment failures—and horizontal agents for cross-cutting concerns like image deduplication and fraud detection.

The details were first reported by Gireesh Sreedhar KP, Deepak Dhankani, and Eash Sharma in a post on the Databricks blog.