cli-eval
Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.
By diegosouzapw · 409 installs
npx skills add diegosouzapw/omniroute --skill cli-eval
Source repository · Upstream listing
<! generated by src/lib/agentSkills/generator.ts; manual edits will be overwritten
Overview
Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.
Quick install
Subcommands
eval
Example:
eval suites
Example:
eval list
Example:
eval get <suiteId
Example:
eval create
Flags:
file <path
Example:
eval run <suiteId
Flags:
m, model <id
combo <name
concurrency <n
tag <tag
watch
Example:
eval list
Flags:
suite <id
status <s
since <ts
limit <n
Example:
eval get <runId
Example:
eval results <runId
Flags:
failed
Example:
eval cancel <runId
Flags:
yes
Example:
eval scorecard <runId
Example:
simulate [prompt]
Flags:
file <path
m, model <id
combo <name
reasoning effort <level
thinking budget <n
explain
Example:
<! skill:custom start
<! Migrated from skills/omniroute cli eval/SKILL.md (preserved curated content)
OmniRoute — CLI Evals
Requires the omniroute CLI. See [CLI entry point skill](https://raw.githubusercontent.com/diegosouzapw/OmniRoute/main/skills/omniroute cli/SKILL.md) for install + global flags.
What are evals?
Evals are automated test suites that score LLM outputs against expected answers or rubrics. OmniRoute stores suites and run results in its local database.
Eval suites
Create a suite
Rubric options: exact match , contains , llm judge , regex .
samples file format (one JSON object per line):
Run an eval
The run is asynchronous. Use watch for a live terminal dashboard or poll manually:
Manage runs
Scorecard output
Response fields per sample:
Comparing models
Run the same suite against multiple models and compare:
CI integration
Errors
suites create fails with invalid rubric → use one of: exact match , contains , llm judge , regex
suites run returns model not found → verify model ID with omniroute models search <name
eval get shows status: failed → check omniroute logs search eval for error details
scorecard returns empty results → the run may still be running ; poll omniroute eval get <runId until status is completed
<! skill:custom end