Phoenix Evals Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans. Quick Reference Task Files Setup [setup python](references/setup python.md), [setup typescript](references/setup typescript.md) Decide what to evaluate [evaluators overview](references/evaluators overview.md) Choose a judge model [fundamentals model selection](references/fundamentals model selection.md) Use pre built evaluators [evaluators pre built](references/evaluators pre built.md) Build code evaluator [evaluators code python](references/evaluators code python.md), [evaluators code typescript](references/evaluators code typescript.md) Build LLM evaluator [evaluators llm python](references/evaluators llm python.md), [evaluators llm typescript](references/evaluators llm typescript.md), [evaluators custom templates](references/evaluators custom templates.md) Batch evaluate DataFrame [evaluate dataframe python](references/evaluate dataframe python.md) Understand experiments [experiments overview](references/experiments overview.md) Run experiment [experiments running python](references/experiments running python.md), [experiments running typescript](references/experiments running typescript.md) Create dataset [experiments datasets python](references/experiments datasets python.md), [experiments datasets typescript](references/experiments datasets typescript.md) Generate synthetic data [experiments synthetic python](references/experiments synthetic python.md), [experiments synthetic typescript](references/experiments synthetic typescript.md) Validate evaluator accuracy [validation](references/validation.md), [validation evaluators python](references/validation evaluators python.md), [validation evaluators typescript](references/validation evaluators typescript.md) Sample traces for review [observe sampling python](references/observe sampling python.md), [observe sampling typescript](references/observe sampling typescript.md) Analyze errors [error analysis](references/error analysis.md), [error analysis multi turn](references/error analysis multi turn.md), [axial coding](references/axial coding.md) RAG evals [evaluators rag](references/evaluators rag.md) Avoid common mistakes [common mistakes python](references/common mistakes python.md), [fundamentals anti patterns](references/fundamentals anti patterns.md) Production [production overview](references/production overview.md), [production guardrails](references/production guardrails.md), [production continuous](references/production continuous.md) Workflows Starting Fresh: [observe tracing setup](references/observe tracing setup.md) → [error analysis](references/error analysis.md) → [axial coding](references/axial coding.md) → [evaluators overview](references/evaluators overview.md) Building Evaluator: [fundamentals](references/fundamentals.md) → [common mistakes python](references/common mistakes python.md) → evaluators {code llm} {python typescript} → validation evaluators {python typescript} RAG Systems: [evaluators rag](references/evaluators rag.md) → evaluators code (retrieval) → evaluators llm (faithfulness) Production: [production overview](references/production overview.md) → [production guardrails](references/production guardrails.md) → [production continuous](references/production continuous.md) Reference Categories Prefix Description fundamentals Types, scores, anti patterns observe Tracing, sampling error analysis Finding failures axial coding Categorizing failures evaluators Code, LLM, RAG evaluators experiments Datasets, running experiments validation Validating evaluator accuracy against human labels production CI/CD, monitoring Key Principles Principle Action Error analysis first Can't automate what you haven't observed Custom generic Build from your failures Code first Deterministic before LLM Validate judges 80% TPR/TNR Binary Likert Pass/fail, not 1 5

phoenix-evals