Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
This talk highlights the challenges of debugging and evaluating AI agents, especially the lack of robust metrics and deterministic outcomes. It introduces a systematic eight-step loop for identifying and resolving agent failures, emphasizing the critical role of traces in understanding agent logic and behavior. The speaker demonstrates how to use a 'coding agent' to automate the debugging process, allowing engineers to quickly move from identifying issues to verifying fixes by leveraging detailed traces, evaluations, and integrated development environments.
read more
The speaker, Ankur, begins by acknowledging a common pain point in AI agent development: the reliance on intuition or 'vibes' rather than concrete metrics for debugging and shipping agents. This leads to the core problem that engineers know when an agent is 'off,' but struggle to quantify how to evaluate or fix the failure. Unlike traditional software development, where code explicitly defines logic, AI agents—particularly those driven by Large Language Models (LLMs)—have opaque internal workings, making direct code-level debugging insufficient. Traces are presented as the new source of truth for understanding and debugging AI agent behavior.
Ankur introduces an eight-step loop designed to transform the debugging process from days to minutes, allowing for rapid iteration and improvement of AI agents:
1. Production failure: Identify when an agent fails in a real-world scenario. 2. Trace evidence: Collect and analyze the detailed execution traces of the failing agent, which visualize its thought process, tool calls, and LLM interactions. 3. Failure cluster: Group similar failures from traces to identify common patterns and root causes. 4. Root-cause hypothesis: Formulate a hypothesis about why the agent failed, based on trace analysis. 5. Evaluator: Develop or refine evaluation metrics (evals) that specifically test the hypothesized failure modes. These can be LLM-as-a-judge evals or code-based evals. 6. Code/prompt fix: Implement changes to the agent's code, prompts, or tools to address the root cause. 7. Experiment: Run experiments against a dataset of failing traces using the new fixes. 8. Verified improvement: Confirm that the fixes resolve the identified failures and continuously monitor for regressions.
This loop is designed to be iterative, leading to fewer failures with each pass. The speaker emphasizes that the key to accelerating this loop is to centralize and automate the debugging signals through a coding agent. This agent integrates various data sources:
Traces: Detailed execution logs of agent interactions. Evaluations: Quantitative metrics from tests against specific failure modes. Annotations: Human feedback or labeling on agent responses. Production telemetry: Real-time performance data from deployed agents. Codebase: Access to the agent's source code and related libraries. APM data: Application Performance Monitoring data for system health. Engineering tickets: Integration with issue tracking systems like Jira. Repo history: Version control history for context.
The demonstration showcases using a 'coding agent' (specifically, a tool like Claude Code within a development environment) and integrating it with Arize AI's platform to streamline the debugging process. The workflow involves:
Setting up the CLI and skills: Installing necessary command-line interface tools and configuring skills that grant the coding agent access to different functionalities, such as pulling traces, running experiments, and creating evaluations. Spotting the issue: An example of a financial agent providing financial advice when it should not, or fabricating answers after tool failures. By inspecting traces within the Arize platform, the user can visually identify points of failure (e.g., incorrect tool selection, silent tool failures, hallucination). Automating analysis and fixes: The coding agent is prompted to pull down traces from the Arize platform, analyze them for issues, and suggest potential fixes. It can then generate a comprehensive analysis of existing evals, failure rates, and insights into root causes (e.g., 'bad summarization,' 'agent hallucination,' 'redundant tool calls'). Building and validating fixes: The coding agent can propose fixes (e.g., updating prompts, adding guardrails) and automatically build new evals to continuously monitor for these specific issues. It can run experiments to compare the performance of the baseline agent against the candidate agent with the proposed fixes. The visual analytics within Arize demonstrate the improvement in passing rates over multiple iterations.
The speaker highlights the importance of customizable evaluations that are tailored to specific use cases, rather than relying on generic benchmarks. These evaluations can be configured with different LLMs (or even local open-source models) and specific prompts to judge agent behavior. The Arize platform provides full trace history of these evaluations, including cost analysis, allowing engineers to make informed decisions about model choice and iteration efficiency.
In essence, the agent doesn't magically fix everything, but by following this iterative, data-driven loop powered by integrated coding agents and observability platforms like Arize, engineers can systematically and efficiently improve their AI agents, moving from an intuitive understanding of 'something broke' to a verified, data-backed 'here's the fix.' This approach aims to reduce debugging time from days to minutes, enabling faster shipping of better AI.