Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond bette

Arize Prompt Optimization Skill SPACE — All space flags and the ARIZE SPACE env var accept a space name (e.g., my workspace ) or a base64 space ID (e.g., U3BhY2U6... ). Find yours with ax spaces list . Concepts Where Prompts Live in Trace Data LLM applications emit spans following OpenInference semantic conventions. Prompts are stored in different span attributes depending on the span kind and instrumentation: Column What it contains When to use attributes.llm.input messages Structured chat messages (system, user, assistant, tool) in role based format Primary source for chat based LLM prompts attributes.llm.input messages.roles Array of roles: system , user , assistant , tool Extract individual message roles attributes.llm.input messages.contents Array of message content strings Extract message text attributes.input.value Serialized prompt or user question (generic, all span kinds) Fallback when structured messages are not available attributes.llm.prompt template.template Template with {variable} placeholders (e.g., "Answer {question} using {context}" ) When the app uses prompt templates attributes.llm.prompt template.variables Template variable values (JSON object) See what values were substituted into the template attributes.output.value Model response text See what the LLM produced attributes.llm.output messages Structured model output (including tool calls) Inspect tool calling responses Finding Prompts by Span Kind LLM span ( attributes.openinference.span.kind = 'LLM' ): Check attributes.llm.input messages for structured chat messages, OR attributes.input.value for a serialized prompt. Check attributes.llm.prompt template.template for the template. Chain/Agent span : attributes.input.value contains the user's question. The actual LLM prompt lives on child LLM spans navigate down the trace tree. Tool span : attributes.input.value has tool input, attributes.output.value has tool result. Not typically where prompts live. Performance Signal Columns These columns carry the feedback data used for optimization: Column pattern Source What it tells you annotation.<name .label Human reviewers Categorical grade (e.g., correct , incorrect , partial ) annotation.<name .score Human reviewers Numeric quality score (e.g., 0.0 1.0) annotation.<name .text Human reviewers Freeform explanation of the grade eval.<name .label LLM as judge evals Automated categorical assessment eval.<name .score LLM as judge evals Automated numeric score eval.<name .explanation LLM as judge evals Why the eval gave that score most valuable for optimization attributes.input.value Trace data What went into the LLM attributes.output.value Trace data What the LLM produced {experiment name}.output Experiment runs Output from a specific experiment Prerequisites Proceed directly with the task — run the ax command you need. Do NOT check versions, env vars, or profiles upfront. If an ax command fails, troubleshoot based on the error: command not found or version error → see references/ax setup.md 401 Unauthorized / missing API key → run ax profiles show to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin API Keys Space unknown → run ax spaces list to pick by name, or ask the user Project unclear → ask the user, or run ax projects list o json limit 100 and present as selectable options LLM provider call fails (missing OPENAI API KEY / ANTHROPIC API KEY) → run ax ai integrations list space SPACE to check for platform managed credentials. If none exist, ask the user to provide the key or create an integration via the arize ai provider integration skill Security: Never read .env files or search the filesystem for credentials. Use ax profiles for Arize credentials and ax ai integrations for LLM provider keys. If credentials are not available through these channels, ask the user. Phase 1: Extract the Current Prompt Find LLM spans containing prompts Export a trace to inspect prompt structure Extract prompts from exported JSON Reconstruct the prompt as messages Once you have the span data, reconstruct the prompt as a messages array: If the span has attributes.llm.prompt template.template , the prompt uses variables. Preserve these placeholders ( {variable} or {{variable}} ) they are substituted at runtime. Phase 2: Gather Performance Data From traces (production feedback) From datasets and experiments Merge dataset + experiment for analysis Join the two files by example id to see inputs alongside outputs and evaluations: Identify what to optimize Look for patterns across failures: 1. Compare outputs to ground truth : Where does the LLM output differ from expected? 2. Read eval explanations : eval. .explanation tells you WHY something failed 3. Check annotation text : Human feedback describes specific issues 4. Look for verbosity mismatches : If outputs are too long/short vs ground truth 5. Check format compliance : Are outputs in the expected format? Phase 3: Optimize the Prompt The Optimization Meta Prompt Use this template to generate an improved version of the prompt. Fill in the three placeholders and send it to your LLM (GPT 4o, Claude, etc.): Preparing the performance data Format the records as a JSON array before pasting into the template: Applying the revised prompt After the LLM returns the revised messages array: 1. Compare the original and revised prompts side by side 2. Verify all template variables are preserved 3. Check that format instructions are intact 4. Test on a few examples before full deployment Phase 4: Iterate The optimization loop Measure improvement A/B compare two prompts 1. Create two experiments against the same dataset, each using a different prompt version 2. Export both: ax experiments export EXP A and ax experiments export EXP B 3. Compare average scores, failure rates, and specific example flips 4. Check for regressions examples that passed with prompt A but fail with prompt B Prompt Engineering Best Practices Apply these when writing or revising prompts: Technique When to apply Example Clear, detailed instructions Output is vague or off topic "Classify the sentiment as exactly one of: positive, negative, neutral" Instructions at the beginning Model ignores later instructions Put the task description before examples Step by step breakdowns Complex multi step processes "First extract entities, then classify each, then summarize" Specific personas Need consistent style/tone "You are a senior financial analyst writing for institutional investors" Delimiter tokens Sections blend together Use , , or XML tags to separate input from instructions Few shot examples Output format needs clarification Show 2 3 synthetic input/output pairs Output length specifications Responses are too long or short "Respond in exactly 2 3 sentences" Reasoning instructions Accuracy is critical "Think step by step before answering" "I don't know" guidelines Hallucination is a risk "If the answer is not in the provided context, say 'I don't have enough information'" Variable preservation When optimizing prompts that use template variables: Single braces ( {variable} ): Python f string / Jinja style. Most common in Arize. Double braces ( {{variable}} ): Mustache style. Used when the framework requires it. Never add or remove variable placeholders during optimization Never rename variables the runtime substitution depends on exact names If adding few shot examples, use literal values, not variable placeholders Workflows Optimize a prompt from a failing trace 1. Find failing traces: 2. Export the trace: 3. Extract the prompt from the LLM span: 4. Identify what failed from the error message or output 5. Fill in the optimization meta prompt (Phase 3) with the prompt and error context 6. Apply the revised prompt Optimize using a dataset and experiment 1. Find the dataset and experiment: 2. Export both: 3. Prepare the joined data for the meta prompt 4. Run the optimization meta prompt 5. Create a new experiment with the revised prompt to measure improvement Debug a prompt that produces wrong format 1. Export spans where the output format is wrong: 2. Look at what the LLM is producing vs what was expected 3. Add explicit format instructions to the prompt (JSON schema, examples, delimiters) 4. Common fix: add a few shot example showing the exact desired output format Reduce hallucination in a RAG prompt 1. Find traces where the model hallucinated: 2. Export and inspect the retriever + LLM spans together: 3. Check if the retrieved context actually contained the answer 4. Add grounding instructions to the system prompt: "Only use information from the provided context. If the answer is not in the context, say so." Troubleshooting Problem Solution ax: command not found See references/ax setup.md No profile found No profile is configured. See references/ax profiles.md to create one. No input messages on span Check span kind Chain/Agent spans store prompts on child LLM spans, not on themselves Prompt template is null Not all instrumentations emit prompt template . Use input messages or input.value instead Variables lost after optimization Verify the revised prompt preserves all {var} placeholders from the original Optimization makes things worse Check for overfitting the meta prompt may have memorized test data. Ensure few shot examples are synthetic No eval/annotation columns Run evaluations first (via Arize UI or SDK), then re export Experiment output column not found The column name is {experiment name}.output check exact experiment name via ax experiments get jq errors on span JSON Ensure you're targeting the correct file path (e.g., trace /spans.json )

arize-prompt-optimization