This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Agent Platform eval methodology and the Quality Flywheel. Covers ev

Agent Evaluation Guide Requires: agents cli ( uv tool install google agents cli ) — [install uv](https://docs.astral.sh/uv/getting started/installation/index.md) first if needed. Scaffolded project? If you used /google agents cli scaffold , you already have agents cli eval run (chains generate + grade ), tests/eval/datasets/ , and tests/eval/eval config.yaml . Start with executing eval run and iterate from there. Reference Files File Contents references/dataset schema.md Canonical EvaluationDataset schema — all field types, JSON examples for single turn / multi turn / multi agent, common mistakes references/metrics guide.md Complete metrics reference — all built in metrics, match types, custom metrics, judge model config references/user simulation.md Dynamic conversation testing — eval dataset synthesize flags, what scenarios are, compatible metrics references/builtin tools eval.md google search and model internal tools — trajectory behavior, metric compatibility references/advanced commands.md Opt in commands: eval analyze , eval optimize , eval submit / eval results references/multimodal eval.md Multimodal inputs — eval dataset schema, built in metric limitations, custom evaluator pattern The Quality Flywheel Improving agent quality is iterative. The 4 stages below describe the loop. Each stage has a Default path (you, the coding agent, do the work directly) and an Opt in CLI command that delegates to the Agent Platform Eval Service for better quality and scale. 1. Prepare Data Default: Use or edit the scaffolded tests/eval/datasets/basic dataset.json to define single turn eval inputs. Start with 1–2 cases. Opt in (ADK projects): agents cli eval dataset synthesize : user simulate multi turn datasets when you lack data; its output already includes traces, so Stage 2 collapses to agents cli eval grade alone. See Eval Commands and references/user simulation.md . 2. Run the Eval (always run) Default: agents cli eval run runs the agent over the dataset and grades the traces, writing results <ts .{json,html} to artifacts/grade results/ . Decoupled form: eval generate then eval grade , for a custom traces location, re grading without re running the agent, or traces from synthesize ( eval grade alone). 3. Analyze Failures Default: Open the latest artifacts/grade results/results <ts .html (or .json ) and identify failed metrics — see What to fix when scores fail below for the fix table. Opt in: agents cli eval analyze , LLM based failure clustering; prefer when you have 10+ failing cases and want categorized failure modes. See references/advanced commands.md . 4. Optimize & Code Fix Default: Edit the agent — adjust prompts, tool descriptions, instructions, or eval dataset based on the failure analysis. See What to fix when scores fail below for the failure → fix mapping. Opt in (ADK projects): agents cli eval optimize runs ADK GEPA prompt optimization against a target metric (see references/advanced commands.md ). Suitable for prompt only failures. The optimized prompt appears in the command output; capture it and apply it to the agent. For the full per iteration trace, set print detailed results: true in your optimization config file. Long running and expensive. GEPA optimization makes many LLM calls and can take a long time. Do not run it unless the user explicitly asks for prompt optimization. When you do run it, iterate as far as possible with manual fixes first, then run a single final eval optimize — never loop on this command. Running the loop Iterate stages 2 → 3 → 4 → 2 (with synthesize , re run Stage 1 each pass, then eval grade ). After each fix, run agents cli eval compare <prev results .json <new results .json to confirm the target metric improved without regressing others. Expect 5–10+ iterations per case before it passes, which is normal. Only after a case passes should you expand coverage with more eval cases. When doing 5+ iterations, maintain a task list of which cases are fixed, which are still failing, and what fixes you've tried. Prevents re attempting the same fix. Hold cases back. Keep a slice of cases out of the loop and grade them only when you think you're done — otherwise you can't tell a fix that generalizes from one fitted to the cases you iterated against. Shortcuts That Waste Time Recognize these rationalizations and push back — they always cost more time than they save: Shortcut Why it fails "I'll lower the bar so it passes" Lowering the bar hides real failures. If the agent can't meet the bar, fix the agent, don't move the bar. "This eval case is flaky, I'll skip it" Flaky evals reveal non determinism in your agent. Fix with temperature=0 , rubric based metrics, or more specific instructions — don't delete the signal. "I just need to fix the eval dataset, not the agent" If you're always adjusting expected outputs, your agent has a behavior problem. Fix the instructions or tool logic first. "I'll iterate until every case I have passes" Nothing is left to detect overfitting to your own cases. See Hold cases back above. Choosing the Right Metrics Pick built in metrics by what you want to measure. Only multi turn task success , multi turn trajectory quality , and multi turn tool use quality accept multi turn traces; every other built in 400s on one. When no built in fits, write a custom metric (see Evaluation Configuration Schema below). Goal Recommended built in metrics Did the agent achieve the user's goal? (catch all for multi turn agents) multi turn task success Was the agent's reasoning path logical and efficient? multi turn trajectory quality Quality of tool / function calling across turns multi turn tool use quality Final response quality (no ground truth reference needed) final response quality Factual grounding (catch hallucinated claims, e.g., RAG agents) hallucination , or grounding when the case carries a context field Safety policy compliance safety Match against a golden answer final response match (needs reference on the case) Different pass/fail criteria per case Put them on the case as rubric groups and grade with a managed rubric metric. See references/dataset schema.md ( Per Case Rubrics ). Domain specific check no built in covers Write a custom LLMMetric (LLM judge) or CodeExecutionMetric (deterministic Python). See Evaluation Configuration Schema below. Run agents cli eval metric list to see all available built ins. For full metric definitions and rubric details, see the [Agent Platform metric docs](https://cloud.google.com/gemini enterprise agent platform/optimize/evaluation/manage metrics) and references/metrics guide.md . What to fix when scores fail After agents cli eval run completes, inspect the latest artifacts/grade results/results <timestamp .json (or open the .html file) for per case scores and judge rationales, the input to every fix decision below. Failure What to change multi turn task success low The agent isn't completing the user's goal — fix orchestration, missing tool calls, premature termination, or wrong tool selection multi turn trajectory quality low The agent reaches the goal inefficiently or takes wrong steps — refine planning prompts, tighten instruction order, or remove redundant tool calls multi turn tool use quality low Fix tool descriptions, parameter docstrings, or agent instructions for tool selection final response quality low Read the auto generated rubric verdicts; refine agent instructions to address the worst scoring criterion (often clarity, completeness, or instruction following) hallucination low Tighten agent instructions to stay grounded in tool output; verify the tool actually returned the data the agent claimed safety low Add safety guardrails to instructions; review the violating content category in the rubric verdict Agent calls wrong tools Fix tool descriptions, agent instructions, or the model's tool choice config ( ADK: tool config ) Agent calls extra tools Add strict stop instructions, or switch to multi turn tool use quality After applying a fix, rerun agents cli eval run and use agents cli eval compare <prev results .json <new results .json to confirm the fix improved the target metric without regressing others. Eval Commands agents cli eval <subcommand help is the authoritative flag list; the examples below are the common invocations. eval run (default) Runs the agent over the dataset and grades the traces in one command. eval generate Runs an agent over an evaluation dataset and writes traces to disk. By default, runs the agent locally and records a trace per evaluation case. You can generate traces from an already running agent by passing its HTTP endpoint and app name to url and app name . ADK projects. The built in generator serves the agent over HTTP (the project's fast api app.py if it exists, else adk api server ) and drives it over ADK's /apps/... and /run sse routes — the same shape url / app name expect. Extensions for other frameworks replace eval generate with their own generator, which may not serve HTTP at all; url and app name are then unsupported. eval grade Scores traces (from eval generate , eval dataset synthesize , or hand authored) against built in or custom metrics. Writes timestamped results <YYYYMMDD HHMMSS .json (consumed by eval compare ) and .html (open in a browser) into the output dir, and prints a summary table to the console. See Evaluation Configuration Schema below for the config file format. eval compare Diffs two results .json files from an eval run. Run it after a fix to confirm the target metric improved without regressing others. eval dataset synthesize ADK projects. It loads and runs the agent through ADK, so it is unavailable on other frameworks. Generates user scenarios from your agent's tools and instructions, plays each against an LLM backed user simulator, and writes graded ready traces to artifacts/traces/ (feed straight to eval grade , skip eval generate ). Invocations, flags, and compatible metrics: references/user simulation.md . Advanced commands eval analyze (cluster failure modes), eval optimize (GEPA prompt tuning), and eval submit / eval results (managed cloud side runs for CI or large datasets) are documented in references/advanced commands.md . Evaluation Dataset Format An EvaluationDataset is a JSON file with an eval cases array. Cases come in two shapes depending on how they're used: Inference input (what you give to eval generate ) — a user prompt or a partial conversation ending in a user prompt. The agent runs and produces traces. Grading input (what you give to eval grade ) — a complete trace including the agent's responses and tool calls. Normally produced by eval generate or eval dataset synthesize ; you don't write these by hand. See references/dataset schema.md for the full canonical schema, all field types, and common mistakes. Inference input format Two shapes are supported. (a) Simple single turn prompt — what the scaffolded tests/eval/datasets/basic dataset.json uses. The agent runs from scratch. (b) Multi turn continuation via agent data — a partial conversation whose last turn ends with a user message; the agent's next response is evaluated. See references/dataset schema.md ( Multi Turn / Multi Agent Dataset ) for the JSON shape. Grading input format (traces) A complete trace — agent responses plus function call / function response parts — normally produced by eval generate / eval dataset synthesize (y

google-agents-cli-eval