Handles LLM-as-judge and code evaluator workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, code evaluator, h

Arize Evaluator Skill SPACE — space flags accept a space name (e.g., my workspace ) or a base64 space ID (e.g., U3BhY2U6... ). Find yours with ax spaces list . This skill covers designing, creating, and running evaluators on Arize — both LLM as judge (template) evaluators and code evaluators (deterministic, no LLM required). An evaluator defines the judge; a task is how you run it against real data. Prerequisites Proceed directly with the task — run the ax command you need. Do NOT check versions, env vars, or profiles upfront. If an ax command fails, troubleshoot based on the error: command not found or version error → see [references/ax setup.md](references/ax setup.md) 401 Unauthorized / missing API key → run ax profiles show to inspect the current profile. If the profile is missing or the API key is wrong, follow [references/ax profiles.md](references/ax profiles.md) to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin API Keys Space unknown → run ax spaces list to pick by name, or ask the user LLM provider call fails (missing provider credentials) → run ax ai integrations list space SPACE to check for platform managed credentials. If none exist, use the arize ai provider integration skill — never ask the user to paste a provider key into chat. Security: Never read .env files or search the filesystem for credentials. Use ax profiles for Arize credentials and ax ai integrations for LLM provider keys. Never ask the user to paste secrets into chat. For missing credentials, see [references/ax profiles.md](references/ax profiles.md). CRITICAL — Never fabricate evaluation results: If an evaluation task fails, is cancelled, or produces no scores, report the failure clearly and explain what went wrong. Do NOT perform a "manual evaluation," invent quality scores, estimate percentages, or present any agent generated analysis as if it came from the Arize evaluation system. Instead suggest: (1) fix the identified issue and retry, (2) try running from the Arize UI, (3) verify integration credentials with ax ai integrations list , (4) contact support at https://arize.com/support Concepts What is an Evaluator? An evaluator is an LLM as judge definition. It contains: Field Description Template The judge prompt. Uses {{variable}} (double brace) placeholders (e.g. {{input}} , {{output}} , {{context}} ) that get filled in at run time via a task's column mappings. Classification choices The set of allowed output labels (e.g. factual / hallucinated ). Binary is the default and most common. Each choice can optionally carry a numeric score. AI Integration Stored LLM provider credentials (OpenAI, Anthropic, Bedrock, etc.) the evaluator uses to call the judge model. Model The specific judge model (e.g. gpt 4o , claude sonnet 4 5 ). Invocation params Optional JSON of model settings like {"temperature": 0} . Low temperature is recommended for reproducibility. Optimization direction Whether higher scores are better ( MAXIMIZE ) or worse ( MINIMIZE ). Sets how the UI renders trends. Data granularity Whether the evaluator runs at the span , trace , or session level. Most evaluators run at the span level. Evaluators are versioned — every prompt or model change creates a new immutable version. The most recent version is active. Code evaluators are the deterministic alternative — no AI integration or model, just Python. They run as a class subclassing CodeEvaluator , not a bare function, and have their own strict import path and evaluate() signature contract. Getting either wrong makes a run cancel silently at 0/0/0 . See "Custom Python code evaluators" in [references/cli reference.md](references/cli reference.md) before writing one. What is a Task? A task is how you run one or more evaluators against real data. Tasks are attached to a project (live traces/spans) or a dataset (experiment runs). A task contains: Field Description Evaluators List of evaluators to run. You can run multiple in one task. Column mappings Maps each evaluator's template variables to actual field paths on spans or experiment runs (e.g. "input" → "attributes.input.value" ). This is what makes evaluators portable across projects and experiments. Query filter SQL style expression to select which spans/runs to evaluate (e.g. "span kind = 'LLM'" ). Optional but important for precision. Continuous For project tasks: whether to automatically score new spans as they arrive. Sampling rate For continuous project tasks: fraction of new spans to evaluate (0–1). Data Granularity Set data granularity when creating the evaluator ( ax evaluators create template evaluator / create code evaluator ), not on the task — it controls what unit of data that evaluator scores whenever it runs against a project task (not dataset/experiment tasks — those evaluate experiment runs directly). It defaults to span . Level What it evaluates Use for Result column prefix span (default) Individual spans Q&A correctness, hallucination, relevance eval.{name}.label / .score / .explanation trace All spans in a trace, grouped by context.trace id Agent trajectory, task correctness — anything that needs the full call chain trace eval.{name}.label / .score / .explanation session All traces in a session, grouped by attributes.session.id and ordered by start time Multi turn coherence, overall tone, conversation quality session eval.{name}.label / .score / .explanation How trace and session aggregation works For trace granularity, spans sharing the same context.trace id are grouped together. Column values used by the evaluator template are comma joined into a single string (each value truncated to 100K characters) before being passed to the judge model. For session granularity, the same trace level grouping happens first, then traces are ordered by start time and grouped by attributes.session.id . Session level values are capped at 100K characters total. The {{conversation}} template variable At session granularity, {{conversation}} is a special template variable that renders as a JSON array of {input, output} turns across all traces in the session, built from attributes.input.value / attributes.llm.input messages (input side) and attributes.output.value / attributes.llm.output messages (output side). At span or trace granularity, {{conversation}} is treated as a regular template variable and resolved via column mappings like any other. Note: For {{conversation}} to work, spans must carry attributes.session.id . See the arize instrumentation skill for how to emit session.id from application code, including the force flush() pattern required for Jupyter notebooks and short lived scripts. Multi evaluator tasks A task can contain evaluators at different granularities. At runtime the system uses the highest granularity (session trace span) for data fetching and automatically splits into one child run per evaluator . Per evaluator query filter in the task's evaluators JSON further narrows which spans are included (e.g., only tool call spans within a session). Basic CRUD Full command reference for AI integrations , evaluators (template and code), and tasks — every flag, with examples — is in [references/cli reference.md](references/cli reference.md). The workflows below include the commands you need inline. Workflow A: Create an evaluator for a project Use this when the user says something like "create an evaluator for my Playground Traces project" . Step 1: Confirm the project name ax spans export accepts a project name directly — no ID lookup needed. If you don't know the project name, list available projects: Find the entry whose "name" matches (case insensitive) and use that name as PROJECT in subsequent commands. If you later hit a validation error with a name, fall back to using the project's "id" (a base64 string) instead. Step 2: Understand what to evaluate If the user specified the evaluator type (hallucination, correctness, relevance, etc.) → skip to Step 3. If not, sample recent spans to base the evaluator on actual data: Inspect attributes.input , attributes.output , span kinds, and any existing annotations. Identify failure modes (e.g. hallucinated facts, off topic answers, missing context) and propose 1–3 concrete evaluator ideas . Let the user pick. Each suggestion must include: the evaluator name (bold), a one sentence description of what it judges, and the binary label pair in parentheses. Format each like: 1. Name — Description of what is being judged. ( label a / label b ) Example: 1. Response Correctness — Does the agent's response correctly address the user's financial query? ( correct / incorrect ) 2. Hallucination — Does the response fabricate facts not grounded in retrieved context? ( factual / hallucinated ) Step 3: Confirm or create an AI integration If a suitable integration exists, note its ID. If not, create one using the arize ai provider integration skill. Ask the user which provider/model they want for the judge. Step 4: Create the evaluator Use the template design best practices below. Keep the evaluator name and variables generic — the task (Step 6) handles project specific wiring via column mappings . Step 5: Ask — backfill, continuous, or both? Recommended approach: Always start with a small backfill (~100 historical spans) to validate the evaluator before turning on continuous monitoring. This lets you catch column mapping errors, wrong span kinds, and template issues on known data before scoring all future production spans. Only enable continuous after a backfill confirms correct scoring. Before creating the task, ask: "Would you like to: (a) Run a backfill on historical spans (one time)? (b) Set up continuous evaluation on new spans going forward? (c) Both — backfill first to validate, then keep scoring new spans automatically? (recommended)" Step 6: Determine column mappings from real span data Do not guess paths. Pull a sample and inspect what fields are actually present: For each template variable ( {{input}} , {{output}} , {{context}} ), find the matching JSON path. Common starting points — always verify on your actual data before using : Template var LLM span CHAIN span input attributes.input.value attributes.input.value output attributes.llm.output messages.0.message.content attributes.output.value context attributes.retrieval.documents.contents — tool output attributes.input.value (fallback) attributes.output.value Validate span kind alignment: If the evaluator prompt assumes LLM final text but the task targets CHAIN spans (or vice versa), runs can cancel or score the wrong text. Make sure the query filter on the task matches the span kind you mapped. query filter only works on indexed attributes: The query filter in the evaluators JSON is evaluated against the eval index, not the raw span store. Attributes under attributes.metadata. or custom keys may not be indexed and will silently match nothing. Use well known indexed attributes like span kind or attributes.llm.model name for filtering. If a filter returns 0 spans despite data existing, try removing the filter as a diagnostic step. Full example evaluators JSON: Include a mapping for every variable the template references. Omitting one causes runs to produce no valid scores. Step 7: Create the task Backfill only (a): Continuous only (b): Both (c): Use is continuous on create, then also trigger a backfill run in Step 8. Step 8: Trig

arize-evaluator