Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles. Use when a Claude Code workflow needs a formal eval before it is trusted or changed.

Eval Harness Skill A formal evaluation framework for Claude Code sessions, implementing eval driven development (EDD) principles. When to Activate Setting up eval driven development (EDD) for AI assisted workflows Defining pass/fail criteria for Claude Code task completion Measuring agent reliability with pass@k metrics Creating regression test suites for prompt or agent changes Benchmarking agent performance across model versions Philosophy Eval Driven Development treats evals as the "unit tests of AI development": Define expected behavior BEFORE implementation Run evals continuously during development Track regressions with each change Use pass@k metrics for reliability measurement Eval Types Capability Evals Test if Claude can do something it couldn't before: Regression Evals Ensure changes don't break existing functionality: Grader Types 1. Code Based Grader Deterministic checks using code: 2. Model Based Grader Use Claude to evaluate open ended outputs: 3. Human Grader Flag for manual review: Metrics pass@k "At least one success in k attempts" pass@1: First attempt success rate pass@3: Success within 3 attempts Typical target: pass@3 90% pass^k "All k trials succeed" Higher bar for reliability pass^3: 3 consecutive successes Use for critical paths Eval Workflow 1. Define (Before Coding) 2. Implement Write code to pass the defined evals. 3. Evaluate 4. Report Integration Patterns Pre Implementation Creates eval definition file at .claude/evals/feature name.md During Implementation Runs current evals and reports status Post Implementation Generates full eval report Eval Storage Store evals in project: Best Practices 1. Define evals BEFORE coding Forces clear thinking about success criteria 2. Run evals frequently Catch regressions early 3. Track pass@k over time Monitor reliability trends 4. Use code graders when possible Deterministic probabilistic 5. Human review for security Never fully automate security checks 6. Keep evals fast Slow evals don't get run 7. Version evals with code Evals are first class artifacts Example: Adding Authentication Local Framework Utilities The mechanical utilities ship in scripts/lib/eval harness/ : Capsule: hash linked journal with five lineages and local integrity checks. Inspection: source digests, validated variant paths, and syntactic warnings. Replay: declared tools and content addressed fixtures. Missing fixtures fail closed; SE3 and above are refused in replay. Record mode invokes the registered implementation, so only register trusted functions. Receipt: offline verification of capsule and artifact bytes, with named checks. Retrospective preparation: node scripts/eval harness.js capsule group <dir [<dir ...] groups 1 to 100 explicitly selected, verified local capsule snapshots from one task family by declared harness version. Repeated snapshots count once; conflicting identities or invalid capsules reject the whole report. This is read only record counting, with no new rollouts, scores or promotion. Use small, quiescent capsules. Payloads, directory arguments and raw run/capsule IDs are omitted, but task family/version labels are verbatim and digest references are linkable; review them before sharing. Operational validation remains pending. Candidate execution is disabled on every OS because no verified OS containment backend is implemented. gate run , runGate , runVariant , direct child launch, and the retired effect preload refuse with gate.isolation required . No trust flag or caller supplied executor can bypass the refusal. The example records that refusal and inspects source without executing or scoring it. Do not present static warnings, a capsule receipt, or successful utility tests as candidate containment or promotion evidence. A future gate requires an independently reviewed OS boundary, protected checker and audit channels, and fatal baseline rejection. See docs/architecture/eval harness frameworks.md . Product Evals (v1.8) Use product evals when behavior quality cannot be captured by unit tests alone. Grader Types 1. Code grader (deterministic assertions) 2. Rule grader (regex/schema constraints) 3. Model grader (LLM as judge rubric) 4. Human grader (manual adjudication for ambiguous outputs) pass@k Guidance pass@1 : direct reliability pass@3 : practical reliability under controlled retries pass^3 : stability test (all 3 runs must pass) Recommended thresholds: Capability evals: pass@3 = 0.90 Regression evals: pass^3 = 1.00 for release critical paths Eval Anti Patterns Overfitting prompts to known eval examples Measuring only happy path outputs Ignoring cost and latency drift while chasing pass rates Allowing flaky graders in release gates Minimal Eval Artifact Layout .claude/evals/<feature .md definition .claude/evals/<feature .log run history docs/releases/<version /eval summary.md release snapshot

eval-harness