Creates, runs, and analyzes Arize experiments for evaluating and comparing model performance. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. Use when the user mentions create experiment, run experiment, compare models, model performance, evaluat

Arize Experiment Skill SPACE — space flags accept a space name (e.g., my workspace ) or a base64 space ID (e.g., U3BhY2U6... ). Find yours with ax spaces list . Concepts Experiment = a named evaluation run against a specific dataset version, containing one run per example Experiment Run = the result of processing one dataset example includes the model output, optional evaluations, and optional metadata Dataset = a versioned collection of examples; every experiment is tied to a dataset and a specific dataset version Evaluation = a named metric attached to a run (e.g., correctness , relevance ), with optional label, score, and explanation The typical flow: export a dataset → process each example → collect outputs and evaluations → create an experiment with the runs. Prerequisites Proceed directly with the task — run the ax command you need. Do NOT check versions, env vars, or profiles upfront. If an ax command fails, troubleshoot based on the error: command not found or version error → see [references/ax setup.md](references/ax setup.md) 401 Unauthorized / missing API key → run ax profiles show to inspect the current profile. If the profile is missing or the API key is wrong, follow [references/ax profiles.md](references/ax profiles.md) to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin API Keys Space unknown → run ax spaces list to pick by name, or ask the user Project unclear → ask the user, or run ax projects list o json limit 100 and present as selectable options Security: Never read .env files or search the filesystem for credentials. Use ax profiles for Arize credentials and ax ai integrations for LLM provider keys. Never ask the user to paste secrets into chat. For missing credentials, see [references/ax profiles.md](references/ax profiles.md). CRITICAL — Never fabricate outputs: When running an experiment, you MUST call the real model API specified by the user for every dataset example. Never fabricate, simulate, or hardcode model outputs, latencies, or evaluation scores. If you cannot call the API (missing SDK, missing credentials, network error), stop and tell the user what is needed before proceeding. List Experiments: ax experiments list Browse experiments, optionally filtered by dataset. Output goes to stdout. Flags: see [references/experiments cli.md list](references/experiments cli.md list). Get Experiment: ax experiments get Quick metadata lookup returns experiment name, linked dataset/version, and timestamps. Flags: see [references/experiments cli.md get](references/experiments cli.md get). Response fields Field Type Description id string Experiment ID name string Experiment name dataset id string Linked dataset ID dataset version id string Specific dataset version used experiment traces project id string Project where experiment traces are stored created at datetime When the experiment was created updated at datetime Last modification time Export Experiment: ax experiments export Download all runs to a file. By default uses the REST API; pass all to use Arrow Flight for bulk transfer. Flags: see [references/experiments cli.md export](references/experiments cli.md export). REST vs Flight ( all ) REST (default): Lower friction no Arrow/Flight dependency, standard HTTPS ports, works through any corporate proxy or firewall. Limited to 500 runs per page. Flight ( all ): Required for experiments with more than 500 runs. Uses gRPC+TLS on a separate host/port which some corporate networks may block. The active ax profile supplies the regional endpoint; see [profile setup](references/ax profiles.md). Agent auto escalation rule: If a REST export returns exactly 500 runs, the result is likely truncated. Re run with all to get the full dataset. Output is a JSON array of run objects: Create Experiment: ax experiments create Create a new experiment with runs from a data file. Flags: see [references/experiments cli.md create](references/experiments cli.md create). dataset is optional — omit it to create a standalone experiment with no linked dataset (then space is required instead). Passing data via stdin Use file to pipe data directly — no temp file needed: Required columns in the runs file Column Type Required Description example id string yes The dataset example's top level id from ax datasets export output string yes The model/system output for this example Additional columns are passed through as additionalProperties on the run. example id must be the Arize row id — the top level id field on each exported dataset example ( ex["id"] ). Do not use a value nested inside the example's input fields or additional properties ; a wrong value fails silently or attaches the run to the wrong example. Export the dataset and inspect the top level id field before creating runs. ⚠️ Inline evaluations in the create file do NOT attach as scores. create only reads example id and output ; every other column — including an evaluations object — is stored as a passthrough additional field, not as an experiment evaluation, and will not appear as a score in the UI. This fails silently (no error). To attach scores/labels, create the experiment first, then run ax experiments annotate runs . The evaluations object in the schemas below is the export (read) shape returned once annotations exist — it is not an input to create . Run a Task Locally: ax experiments run Unlike create (needs a pre computed outputs file), run loads a Python task function, executes it against every dataset row, and uploads the results as an experiment. task.py must define a top level task(dataset row) function returning a JSON serializable value: dry run tests against the first 10 examples without uploading, to validate the task before a full run. Flags: see [references/experiments cli.md run](references/experiments cli.md run). Choose the run path based on where the logic lives. Use ax experiments run when there's a local Python task to execute — it runs task.py on this machine and uploads the results; no AI integration is required. Use ax tasks create run experiment when the run should be hosted and recurring — it registers a platform side run experiment task that Arize executes on a schedule or on demand, driven by a JSON run configuration (model + messages + AI integration) instead of local code. Default to ax experiments run for local/ad hoc runs and custom logic; use the task path for recurring, hosted runs — see the arize evaluator skill for that route. List Runs: ax experiments list runs Paginated terminal view of an experiment's runs (vs. export , which downloads them to a file). Flags: see [references/experiments cli.md list runs](references/experiments cli.md list runs). Delete Experiment: ax experiments delete Flags: see [references/experiments cli.md delete](references/experiments cli.md delete). Annotate Runs: ax experiments annotate runs This is the required step to attach evaluation scores/labels to an experiment and make them show up in the UI. Evaluations cannot be attached through create ; see the warning under Create Experiment. You write them here, after the experiment exists. Upsert semantics — resubmitting the same annotation name for the same run overwrites the previous value. Up to 1000 runs per request; unmatched record IDs are silently ignored. Annotation file schema A JSON array; each item annotates one run: Field Type Required Description record id string yes The experiment run ID (the run's id from ax experiments export ) — not the example id values array yes One or more annotation dicts, each with a name plus at least one of score , label , or text values[].name string yes Annotation/evaluation name (e.g., correctness ) — becomes the score column in the UI values[].score number no Numeric score (e.g., 0.0 – 1.0 ) values[].label string no Categorical label (e.g., correct , incorrect ) values[].text string no Freeform explanation record id keys on the run id, which only exists after create . So the order is always: create → export (to read each run's id ) → build annotations → annotate runs . Flags: see [references/experiments cli.md annotate runs](references/experiments cli.md annotate runs). Experiment Run Schema Each run corresponds to one dataset example. On create , only example id and output are consumed — evaluations shown here is the shape export returns after you attach scores via annotate runs ; it is not an input to create . Evaluation fields Field Type Required Description label string no Categorical classification (e.g., correct , incorrect , partial ) score number no Numeric quality score (e.g., 0.0 1.0) explanation string no Freeform reasoning for the evaluation At least one of label , score , or explanation should be present per evaluation. Workflows Run an experiment against a dataset 1. Find or create a dataset: 2. Export the dataset examples: 3. Call the real model API for each example and collect outputs. Use ax datasets export stdout to pipe examples directly into an inference script: Write infer.py to read examples from stdin, call the target model, and write runs JSON to stdout. Start from the template at references/inference template.py — copy it, inspect the exported dataset JSON to confirm the input field name, then uncomment the provider block the user wants. Before running: install the SDK, set the API key env var. If the API isn't reachable, stop and tell the user. 4. Verify the runs file: Each run must have example id (the dataset row's top level id ) and output . metadata is optional. Do not put evaluations here — create ignores them; scores are attached in steps 7–9 below. 5. Create the experiment: 6. Verify: ax experiments get "gpt 4o baseline" dataset DATASET NAME space SPACE Attach evaluation scores (required for scores to show in the UI). Evaluations do not come from the create file — you attach them with annotate runs , which keys on each run's id (assigned at create time), so you must export first to learn those IDs. 7. Export the experiment to structured data so you can read each run's id alongside its example id . Confirm that the exported run records include both fields. 8. Build the annotation file with structured JSON handling, keyed by record id (the run id ). Score/label each run via an LLM as judge, a code check, or human review; never fabricate scores. Emit this shape: 9. Attach the scores with ax experiments annotate runs ... file annotations.json , then export or inspect the experiment to confirm the evaluations are attached. The scores now render in the experiment view in the Arize UI. Compare two experiments 1. Export both experiments: 2. Average correctness score (swap a.json for b.json to check the other experiment): 3. Find examples where results differ: 4. Score distribution per evaluator (pass/fail/partial counts; swap files for the other experiment): 5. Find regressions (examples that passed in A but fail in B): Statistical significance note: reliable with ≥ 30 examples per evaluator; with fewer, treat the delta as directional only — a 5% difference on n=10 may be noise. Report sample size alongside scores: jq 'length' a.json . Download experiment results for analysis 1. ax experiments list dataset DATASET NAME space SPACE find ex

arize-experiment