Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like "promp

Promptfoo Evaluation Overview This skill provides guidance for configuring and running LLM evaluations using [Promptfoo](https://www.promptfoo.dev/), an open source CLI tool for testing and comparing LLM outputs. Quick Start Configuration Structure A typical Promptfoo project structure: Core Configuration (promptfooconfig.yaml) Prompt Formats Text Prompt (system.md) Chat Format (chat.json) Few Shot Pattern Embed examples directly in prompt or use chat format with assistant messages: Test Cases (tests/cases.yaml) Python Custom Assertions Create a Python file for custom assertions (e.g., scripts/metrics.py ): Key points: Default function name is get assert Specify function with file://path.py:function name Return bool , float (score), or dict with pass/score/reason Access variables via context['vars'] LLM as Judge (llm rubric) When using a relay/proxy API , each llm rubric assertion needs its own provider config with apiBaseUrl . Otherwise the grader falls back to the default Anthropic/OpenAI endpoint and gets 401 errors: Best practices: Provide clear scoring criteria Use threshold to set minimum passing score Default grader uses available API keys (OpenAI → Anthropic → Google) When using relay/proxy : every llm rubric must have its own provider with apiBaseUrl — the main provider's apiBaseUrl is NOT inherited Common Assertion Types Type Usage Example contains Check substring value: "hello" icontains Case insensitive value: "HELLO" equals Exact match value: "42" regex Pattern match value: "\\d{4}" python Custom logic value: file://script.py llm rubric LLM grading value: "Is professional" latency Response time threshold: 1000 File References All file:// paths are resolved relative to promptfooconfig.yaml location (NOT the YAML file containing the reference). This is a common gotcha when tests: references a separate YAML file — the file:// paths inside that test file still resolve from the config root. Running Evaluations Relay / Proxy API Configuration When using an API relay or proxy instead of direct Anthropic/OpenAI endpoints: Key rules: apiBaseUrl goes in providers[].config — Promptfoo appends /v1/messages automatically maxConcurrency must be under commandLineOptions: — placing it at top level is silently ignored When using relay with LLM as judge, set maxConcurrency: 1 to avoid concurrent request limits (generation + grading share the same pool) Pass relay token as ANTHROPIC API KEY env var Troubleshooting Python not found: Large outputs truncated: Outputs over 30000 characters are truncated. Use head limit in assertions. File not found errors: All file:// paths resolve relative to promptfooconfig.yaml location. maxConcurrency ignored (shows "up to N at a time"): maxConcurrency must be under commandLineOptions: , not at the YAML top level. This is a common mistake. LLM as judge returns 401 with relay API: Each llm rubric assertion must have its own provider with apiBaseUrl . The main provider config is not inherited by grader assertions. HTML tags in model output inflating metrics: Models may output <br , <b , etc. in structured content. Strip HTML in Python assertions before measuring: Echo Provider (Preview Mode) Use the echo provider to preview rendered prompts without making API calls: Use cases: Preview prompt rendering before expensive API calls Verify Few shot examples are loaded correctly Debug variable substitution issues Validate prompt structure Cost: Free no API tokens consumed. Advanced Few Shot Implementation Multi turn Conversation Pattern For complex few shot learning with full examples: Test case configuration: Best practices: Use 1 3 few shot examples (more may dilute effectiveness) Ensure examples match the task format exactly Load examples from files for better maintainability Use echo provider first to verify structure Long Text Handling For Chinese/long form content evaluations (10k+ characters): Configuration: Python assertion for text metrics: Real World Example Project: Chinese short video content curation from long transcripts Structure: See: ./tiaogaoren/ (example project root) for full implementation. Resources For detailed API reference and advanced patterns, see [references/promptfoo api.md](references/promptfoo api.md).

promptfoo-evaluation