Patterns and techniques for evaluating and improving AI agent outputs. Use this skill when: - Implementing self-critique and reflection loops - Building evaluator-optimizer pipelines for quality-critical generation - Creating test-driven code refinement workflows - Designing rubric-based or LLM-as-j

Agentic Evaluation Patterns Patterns for self improvement through iterative evaluation and refinement. Overview Evaluation patterns enable agents to assess and improve their own outputs, moving beyond single shot generation to iterative refinement loops. When to Use Quality critical generation : Code, reports, analysis requiring high accuracy Tasks with clear evaluation criteria : Defined success metrics exist Content requiring specific standards : Style guides, compliance, formatting Pattern 1: Basic Reflection Agent evaluates and improves its own output through self critique. Key insight : Use structured JSON output for reliable parsing of critique results. Pattern 2: Evaluator Optimizer Separate generation and evaluation into distinct components for clearer responsibilities. Pattern 3: Code Specific Reflection Test driven refinement loop for code generation. Evaluation Strategies Outcome Based Evaluate whether output achieves the expected result. LLM as Judge Use LLM to compare and rank outputs. Rubric Based Score outputs against weighted dimensions. Best Practices Practice Rationale Clear criteria Define specific, measurable evaluation criteria upfront Iteration limits Set max iterations (3 5) to prevent infinite loops Convergence check Stop if output score isn't improving between iterations Log history Keep full trajectory for debugging and analysis Structured output Use JSON for reliable parsing of evaluation results Quick Start Checklist

agentic-eval