Zepto在Databricks和MLflow上以评估优先构建客服AI Agent

Zepto's Push for Reliable, Real-Time Customer Support

Zepto is one of India's fastest-growing quick-commerce platforms, with more than thousands of products, a presence in over 60 cities, and delivery windows measured in minutes. In a business where speed is the product, customer support has to move just as fast.

To meet that expectation, Zepto runs customer support on a multi-agent AI system that processes over a hundred thousand tickets a day. Early on, the team could build and ship agents quickly. The harder question was how to keep those agents reliable as volume grew, categories expanded, and customer behavior kept changing. Zepto partnered with Databricks to answer that question, not by shipping more agents, but by making evaluation the primary way agents get built, tested, and operated.

This blog walks through that journey: the system architecture, the evaluation framework on Databricks and MLflow, the production stories where it earned its keep, and the results and lessons that came out of it.

Why "Just Ship the Agent" Breaks at Scale

In a high-velocity business, "just ship the agent" works right up until it breaks at scale. At more than 100,000 AI-agent tickets a day, even a 1% error rate creates thousands of bad outcomes and real revenue leakage every single day.

The pressure arrived in uneven waves. Weather events, Diwali, and the start of summer drove sharp spikes in ticket volume. Expansion from groceries into apparel, electronics, and beauty introduced new refund, exchange, and return journeys. Meanwhile, a more diverse, multilingual customer base brought a wider range of support requests—and new failure modes surfaced every few weeks.

The deeper problem is the assurance gap. Agentic systems operate as multi-step workflows—classifying intent, retrieving knowledge, analyzing inputs, reasoning through decisions, calling transactional tools, and generating responses—so failures can emerge anywhere along the way, not only in the final answer.

This assurance gap translated into concrete problems:

• Failures were invisible until customers complained
• Fixes were slow
• The final answer hid internal errors
• Lacked a principled way to balance cost, performance, and quality for agent
• Agent design did not capture all critical stakeholder perspectives
• Reliability was hard to assure in the face of rapid agent evolution

The objective became clear: engineer an evaluation framework on Databricks and MLflow so it functions as core AI infrastructure on which Agents are built and operated.

Why Evaluation Framework and its Outcomes

A strong evaluation framework directly affects five axes of production readiness:

• Reliability: system-level guarantees that agents behave correctly across steps, not just “sound right”
• Velocity: faster, safer iteration on prompts, policies, and models because changes are regression-tested automatically
• Cost vs Quality vs Performance control: ability to choose optimal models, Prompt strategies or hybrid routing strategies to hit sweet spot on constraints, backed by hard evaluation data
• Governance: auditable traces, versioned evaluation baselines, and well-defined thresholds for deployment and rollback. Moving decisions from gut-feel (“this version feels better”) to evidence (“this version beats the baseline on agreed metrics”)
• Stakeholder collaboration: capturing success and reliability criteria from stakeholders perspective, and making trade-offs explicit and measurable for everyone

With Databricks + MLflow as the evaluation backbone and an evaluation-first agent architecture, Zepto achieved

Cost & efficiency

• 80% plus tickets fully managed by AI agents with human oversight
• 65% reduction in support cost or support tickets
• Payback period of less than one month

Quality & reputation

• 20% improvement in customer satisfaction (CSAT)
• 8% improvement in accuracy

Performance & operations

• 3x faster development cycles
• 4x faster time to resolution

Building the Framework: A Dual Loop for Confidence and Control

At the core of this approach is the dual-loop model: a development loop and a production loop, connected by a quality gate. This section outlines how those loops work together.

• Development loop: where you design, iterate, and evaluate agent versions to build with confidence before shipping
• Production loop: where you monitor live behavior, and detect failures to operate agents with confidence
• Feedback loop: where production failures fed back to development to enrich next iteration
• Quality gate: controlling movement between loops, it decides which versions are allowed into production and which are pushed back to development for better iterations

Together, these two loops ensure that agents are built and operated with control. Any failure is automatically captured, fed back, and corrected. As a result, agents are built with confidence, run with control, and continuously improve to handle production failures better over time. The dual loop lies at the core of our framework.

Phase 0: Enable Tracing: Transparent Agents by design

Every agent invocation emits a rich execution trace that captures prompts, completions, retrieved documents, tool calls, latencies, and decision paths, so the whole workflow is observable at granular level rather than an input and final output.

We enabled this with a hybrid approach using MLflow. A single line, mlflow.<library>.autolog(), turns on automatic tracing, and the @mlflow.trace decorator adds custom spans wherever we need more detail. Traces are emitted in real-time as OpenTelemetry spans with unique IDs so they stay composable, and MLflow's integration with Unity Catalog centralizes logging into Delta tables.

Phase 1: Set Evaluation Dimensions: Pillars and Gates

With tracing enabled, the next step is to capture, from each stakeholder’s perspective, “What does success for this agent mean to you?”. We formalize this as evaluation pillars, each with specific gates.

This turns a multi-stakeholder debate into a shared, measurable contract. Agents are evaluated along the dimensions that actually matter for each stakeholder. Typical pillars include customer experience, operational efficiency, risk and compliance, and financial impact; each pillar has clear numeric thresholds that must be met before deployment.

Phase 2: The Golden Dataset: Cornerstone of Reliability

The golden dataset is the single source of truth for evaluating agent behavior in the development loop. It should:

• Cover normal, edge, and failure cases the agent must handle
• Include expectations from different stakeholders captured as examples
• Be richly annotated with metadata (scenario type, business line, risk level, etc.)

Every stakeholder comes together to shape the dataset, for example the security team contributing examples on adversarial patterns such as prompt injection, identity attacks, data-exfiltration attempts, this ensures that reliability is measured against all real-world scenarios as well as ordinary use.

Datasets are a living asset, and their quality compounds over time. The gap between development and production accuracy is itself a dataset quality signal. Zepto invested steadily in MLflow evaluation datasets over six months, moving from 500 examples and an 8-point dev–prod accuracy gap, to 2,000 examples and a 2-point gap, to 5,247 examples and a 0.4-point gap. Every hour spent on dataset quality saves roughly ten hours of production debugging, so the golden dataset becomes a 10x multiplier: every production failure adds failure traces to the golden dataset and makes the system more robust for all future versions.

Phase 3: Automate Prompt Engineering: Auto generate and auto optimize

Rather than hand-writing prompts, we made prompt engineering a data-driven, automated process. Prompt design is the critical phase where engineers spend most of their time, and the quality of prompts has a disproportionate impact on the quality and performance of agent outputs.

Using MLflow prompt optimization, we register an initial prompt, generate and optimize variants against the same scorers that gate deployment, run A/B evaluations automatically, and deploy the best result. The optimizer reflects with a strong model and production scores candidates with a cheaper one so the search itself stays cost-aware in production. This reduced manual prompt experimentation, improved accuracy, and ensured that prompt improvements were always measured against the golden dataset before reaching production.

Phase 4: Define Scorers and setup AI Jury

With traces flowing in the production loop, we need to score them along the dimensions that matter for agent quality (Evaluation dimensions). Think of this as an AI jury, where each scorer plays to its strengths. MLflow provides three options for creating scorers.

• Built-in judges - out of the box scorers
• Custom judges - create customized judges to meet unique needs
• Code-based scorers - for deterministic metrics like tool latency or number of tool calls

We use LLM-based scorers only where human-like judgment is necessary and rely on simple rules where deterministic logic is enough. We calibrate the judges against human labels to reach 80–90% agreement and use multiple judges for high-stakes decisions.

Phase 5: Set up model optionality

Modal optionality is a critical component which allows the framework to switch between many proprietary and open-source models simply by changing model names in Databricks. This means the development loop can continuously search for a better combination to find the best trade-off between cost, performance, and quality.

Phase 6: Build auto-regression

We automate regression evaluation to create a repeatable, configurable, and scalable development loop. Any change triggers auto-regression and, when reliability is guaranteed, auto-deployment.

Putting the pieces together, a typical change follows this path:

1. Someone changes the agent’s logic, prompt, or model.
2. The evaluation workflow auto-triggers with three inputs: the new agent version, the golden dataset, and the current production baseline.
3. The system runs evaluations and produces metrics across all scorers and pillars.
4. Does the new version meet all gates (cost, quality, performance, etc.)?
5. Does it perform at least as well as, or better than, the production baseline?

If yes, the new version is promoted to production; if not, it is rejected and the existing agent continues serving traffic.

Phase 7: Build Production Loop - Setup real-time safety net

Evaluating 100% of traffic is expensive, but naive 10% uniform sampling misses most edge cases. We implemented stratified sampling where evaluation sampling rates depend on high-value customers, new features or recently changed flows, negative sentiment or high escalation risk, and image-based or fraud-prone interactions. This yields an effective 18–20% evaluation sample (~14,400 traces per day) at manageable cost, while capturing 45–60% of edge cases and detecting issues within 4–6 minutes.

The financial logic is compelling: Compared to a uniform sampling approach, the stratified methodology achieved an 86% reduction in review cost per issue identified while delivering a 9X improvement in edge-case detection, making the quality assurance process significantly more efficient and scalable.

Evaluation results are written into Delta tables and surfaced via dashboards and alert rules in Databricks. Critical alerts (checked every 5 minutes) monitor for intent accuracy drops, groundedness violations, high escalation risk, and P95 latency breaches. High/medium alerts track empathy degradation, cost spikes, tool failure rates, CSAT trends, fraud detection rate, and multimodal latency. This enables SRE-like operations for AI agents: fast detection, triage, and mitigation.

The Composable Architecture Agent Stack

A good evaluation framework works far better when the agent architecture is designed to be observable and decomposable in the first place. Zepto's support stack is built around that idea.

A customer query, in chat or image form, first passes through an agentic orchestrator and router. The router can hand off to a human at any point. Underneath it, the system separates into two kinds of agents.

Vertical agents are specialists, each owning a single, well-defined intent family:

• WIMO: for order tracking and ETA questions
• Missing: for undelivered or partial orders
• Expiry: for expired packaged goods
• Returns: for refund status and processing
• Quality: for stale or rotten produce
• Unable to Pay: for wallet, promo, and payment failures
• General, as the fallback

Horizontal agents act as oversight layers that cut across use cases:

• Image Deduplication: catches reused images across claims
• Item Matching and Image Manipulation Detection: verify uploaded images match catalog items and have not been edited

This separation pays off twice. Metrics can be computed per vertical agent, such as WIMO intent F1 or Expiry OCR accuracy, and horizontal agents can be evaluated on cross-cutting concerns like fraud precision, image reuse, and manipulation detection. Each piece can be measured in isolation and in combination.

The Framework in Action

When you first launch an AI agent into production, it feels like sending a brilliant but unpredictable intern out to represent your company. You give them instructions and hope for the best, but until they are under pressure, you are essentially flying blind.

Early on, we realized that traditional software monitoring is completely useless for AI. An agent can have perfect server uptime and zero errors while repeating the exact same wrong answer to a frustrated customer. To the engineers, the dashboard looks green. To the customer, it’s a disaster.

We knew we couldn't scale our AI on hope. We needed an evaluation framework that didn't just track if the AI was talking, but actually understood what it was saying and where it was failing. The following stories are the moments that framework earned its keep, proving that a good evaluation system unlocks entirely new product features.

Story 1: The ETA That Never Moved

A production issue left riders stuck in traffic while the agent kept replying "arriving in 10 mins" on a loop, because it was reading cached data. The customer asked where their order was, got the same line, asked again, and got the same line again.

The token-counter (monitoring token usage) and warning scorers caught the repetition and high escalation risk within 5 minutes, surfacing traces of stationary riders with unchanged ETAs. That triggered a rule change. If a rider stays stationary for more than 10 minutes, the agent now gives an honest update and proactively offers cancellation for a full refund, rather than repeating a stale promise.

One insight, caught by online evaluation, turned into a whole feature line: cancel on delay, a proactive pitch to cancel during rider shortages, auto-cancel if no rider is assigned within a set window, and no-questions-asked cancellation for high-value customers.

Story 2: Catching the Cancellation Regression Before Customers Did

When Zepto added cancellation handling to the WIMO agent, the model started confusing three very different intents: "where is my order," "I want to cancel," and "was my order cancelled."

Development-phase MLflow evaluation caught it immediately. Overall intent accuracy dropped from about 92.1 percent to 87.4 percent, with poor F1 on the new WIMO_CANCEL and WIMO_CANCEL_STATUS intents. Because the regression showed up against the golden dataset, no customer ever saw it. Prompt optimization and dataset updates restored overall accuracy to about 94.2 percent, better than the original baseline, with near-perfect tool-calling F1 on the cancellation APIs. The feature went live with zero rollbacks.

Story 3: Calibrating Multimodal Agents Against Human Judgment

Produce quality is genuinely hard to score, and humans do not always agree. The same image of mushrooms might get a 2 out of 5 from one rater and a 3 out of 5 from another. We measured that disagreement with Cohen's Kappa and treated it as our reliability ceiling, since no model can be more consistent than the humans it learns from.

We also found that the AI played it safe. Left alone, it piled up scores at 3 to avoid making a hard call, while human scores peaked at 4 and 5. So we did not just minimize error against the average. We matched the shape of the human score distribution. Online evaluation also surfaced cases the system was not built for, like curdled milk that is shelved as a packaged good but needs to be judged like fresh produce, and taste or smell complaints that a photo simply cannot show, which were routed to a separate path.

This calibrated baseline lets us decide which models to use per product type, how to iterate prompts against human judgment, and how to tune refund policy by customer segment based on real agent performance.

Story 4: Closing the Abuse Backdoors

Refund abuse attempts used catalog images, edited photos, and images reused across claims. The multimodal evaluation pipeline ran images through preprocessing checks for blur, brightness, and resolution, validated them with OCR, and then used a jury of three vision models with consensus rules to decide between auto-approval and human review. Layered on top were blur detection, screenshot detection, duplicate detection, image-versus-SKU matching, image-versus-stated-reason checks, and proof-of-delivery validation.

What We Learned Running This at Scale

Operating this framework at scale taught us a handful of principles that generalize beyond quick commerce.

• Never optimize a single metric. We once chased intent accuracy in isolation and watched CSAT drop. Optimizing intent alone gained 5 points of intent accuracy but raised latency by 133 percent and cost 0.4 points of CSAT. A composite, multi-objective score gained 3 points of intent at only 17 percent more latency and added 0.2 points of CSAT. Multiple perspectives, LLM judges, rule-based checks, human labels, and production metrics, create truth together. That redundancy is insurance, not waste.
• Golden datasets are the foundation. Invest early to reach thousands of diverse, high-quality examples. Track the gap between development and production accuracy and tune until it converges. The dev-prod gap is a dataset quality signal, not a mystery.
• LLM judges need calibration. Treat judges as models with their own evaluation. Measure their agreement with humans, use ensembles for high-stakes cases, and recalibrate as base models change.
• Sampling strategy matters more than sampling rate. Naive uniform sampling is cheap but blind. Stratified sampling focused on high-risk flows gives an order of magnitude better issue detection per dollar.
• Automate the feedback loop on day one. Detect, label, retrain, deploy, and monitor should all be automated, with every failure auto-enriching the next training round. For us this saved about 155 hours a month, the equivalent of two full-time engineers redirected to feature work.
• Treat evaluation as a product surface. Dashboards and metrics are used by support, product, fraud, and operations teams. They have to be interpretable and actionable, not just technically correct.

Recommendations for Agent Builders

For organizations building agentic systems on Databricks, the Zepto experience suggests the following roadmap:

• Start from traces, not just models: enforce a shared trace schema and log everything into Delta
• Make MLflow evaluations a mandatory gate in CI/CD: no deployment without beating baseline on agreed metrics
• Build golden datasets as an asset, with owned KPIs (size, dev–prod gap, edge-case coverage)
• Use LLM-as-judge for what humans judge today (quality, empathy, groundedness), but calibrate and limit to the right segments
• Implement stratified sampling and real-time alerts as early as possible; retrofitting operational visibility later is expensive
• Architect agents can be vertical (specialized) plus horizontal (oversight), so you can evaluate them in isolation and in composition

Conclusion

In the transition from experimental demos to mission-critical infrastructure, the primary constraint has shifted from raw model capability to system assurance. Zepto's journey demonstrates that by establishing evaluation as the fundamental development primitive, organizations can reliably scale agents to manage tens of thousands of complex daily interactions across multimodal inputs, backed by rigorous guarantees on quality, cost, and risk mitigation.

Databricks and MLflow serve as the essential substrate for this evolution, providing trace-centric data infrastructure on Unity Catalog and Delta, alongside scalable evaluation, automated prompt optimization, and seamless CI/CD integration. This composable stack, combined with model optionality, empowers teams to fine-tune the balance between performance and expenditure for every specific task.

Ultimately, the competitive advantage lies in the strategic decision to treat evaluation not as a final check, but as core AI infrastructure. The blueprint for operating production-scale agents is no longer a mystery; Zepto and Databricks have provided the answer. The challenge now is the speed of adoption. In the rapidly evolving AI landscape, the leaders will not be those who wait for perfect certainty, but those who engineer for reliability from day one.

For builders developing agents that must earn trust in production environments, these same foundational building blocks are ready on Databricks and MLflow 3. The organizations that define the next frontier will be the ones that begin their evaluation-first journey today.

Build Agents on Databricks Get started with MLflow evaluation and monitoring