Write Harbor task verifiers using Reward Kit. Use when creating or editing a task's tests/ directory, adding grading criteria, setting up LLM/agent judges, or designing verifiers that produce a reward score.

Help the user write task verifiers with Reward Kit. Reward Kit is a lightweight Python package that turns a directory of criteria files into a reward score. Each criterion is a Python function call or a TOML judge file; folders become separate rewards. Setup in a Harbor task Put criteria alongside test.sh in the task's tests/ directory: tests/test.sh : This runs all criteria in /tests/ against the workspace at /app and writes /logs/verifier/reward.json . Defaults match Harbor's conventions — no extra config needed. If judge criteria need API keys, pass them through task.toml : Ask whether Reward Kit should run in the agent's shared environment or in a separate verifier environment. Prefer a separate verifier environment when judge prompts, grading dependencies, API keys, or clean room checks should not be available to the agent: In shared mode, the verifier runs in the agent container and inherits [environment].network mode . Put [verifier].network mode only when verify() needs different network access than the agent phase (a phase override, not a baseline). If agent and verifier need different baselines without runtime switching, use environment mode = "separate" and set [verifier.environment].network mode . Judge criteria that call external APIs need a public baseline or allowlist on the verifier environment. Programmatic checks that only read local files can use no network . In separate mode, tests/ is the verifier image build context and must provide /tests/test.sh at runtime; Harbor does not upload tests/ into the running verifier container. Programmatic criteria Call built ins from any .py file in tests/ : All criteria accept weight (default 1.0 ) and isolated (default False , runs in overlayfs so side effects don't leak). Available built ins Files : file exists , file not exists , file contains , file contains regex , file matches , files equal , diff ratio Commands : command succeeds , command output contains , command output matches , command output matches regex (30s default timeout, optional cwd ) Data : json key equals , json path equals , csv cell equals , xlsx cell equals (needs [office] extra), sqlite query equals HTTP : http status equals , http response contains Images : image similarity , image size equals (needs [image] extra) Trajectory : trajectory tool used , trajectory tool not used , trajectory turn count For extras, install with uv tool install harbor rewardkit[all] . Custom criteria Use the @criterion decorator. First parameter is always workspace: Path . Returns bool or float : Zero parameter criteria auto register. Criteria with extra args must be called via rk : For criteria shared across reward subdirs, define with shared=True in a root level file and call from subdirs. Judge criteria (LLM or agent as a judge) For subjective checks (quality, readability, edge cases), create a TOML file: Criterion types: binary — yes/no → 1.0 or 0.0 likert — 1..points, normalized to [0, 1] numeric — min..max, normalized to [0, 1] Agent judges Agent judges shell out to a CLI and can explore the filesystem: Slower and more expensive than LLM judges, but they can run commands and inspect files. Useful [judge] options timeout (default 300), reasoning effort ( low medium high ), reference (path to reference solution), atif trajectory (evaluate the agent's trajectory), weight , prompt template (custom prompt with {criteria} placeholder). Scoring aggregation (within one judge TOML) Only affects how this file's own criteria combine. To aggregate across dimensions, see [Aggregating dimensions]( aggregating dimensions). Scoring config for a dimension Bare .py criteria in a directory form a single group at weight 1.0, combined by weighted mean. To change that group's weight or aggregation, add a reward.toml to the same directory: weight sets how much the deterministic checks count against a judge in the same directory. [scoring] takes the same values as a judge toml. Unknown keys raise, and so does a reward.toml declaring weight / [scoring] in a directory with no .py files. Directories may be nested recursively. A non root directory can aggregate its local .py bucket, local judges, and immediate child directories with one unnamed [[reward]] table: Membership is implicit. Child directories have weight 1.0 unless overridden; use $checks for the local .py bucket and judge TOML filename stems for local judges. Without [[reward]] , the directory defaults to weighted mean. Multi reward tasks Put criteria in subdirectories — each becomes a separate reward: Judge TOMLs may also sit directly at the tests root alongside reward subdirectories. Each is exposed as a top level reward named after its filename stem and can be referenced by a root aggregation. Produces: Aggregating dimensions To add aggregated scores on top of the per dimension keys, add a root level tests/reward.toml with one or more [[reward]] tables. Each adds one key to reward.json , aggregating the dimensions with the same modes as [scoring] : The per dimension scores stay; aggregated keys are added alongside them (a name may not collide with a dimension). Top level dimensions have equal weight unless that aggregation's inline map overrides them; reward details.json keeps the full recursive breakdown. Output files /logs/verifier/reward.json — per reward scores /logs/verifier/reward details.json — per criterion results, judge reasoning, errors Multi step tasks In a multi step task, each step has its own tests/ under steps/{name}/tests/ , and the verifier runs once per step. Reward Kit behaves the same as in a single step task: for each step it reads /tests , runs the criteria against /app , and writes /logs/verifier/reward.json for that step. Harbor then aggregates per step results into a trial level reward via multi step reward strategy in task.toml — aggregation happens outside Reward Kit, so don't try to encode cross step logic in your criteria. A task level tests/ directory (at the task root) is uploaded to /tests first, then the step's own tests/ is layered on top (same name files win). Put shared helpers (common checks.py functions with shared=True , fixture files, a fallback test.sh ) at the task level, and step specific criteria under each step. Multi reward subdirectories still work within a step: steps/foo/tests/ can contain correctness/ , structure/ , quality/ — each produces a separate reward key for that step, and multi step reward strategy = "mean" averages each key across steps. Use "final" when the last step is an end to end check whose rewards already represent the full task. When to reach for what Use built ins for file existence, string matches, command output, JSON/CSV checks, HTTP probes. Use @criterion when logic is task specific but still programmatic. Use LLM judges for subjective quality dimensions (readability, correctness of prose). Use agent judges when the rubric requires exploring the filesystem or running code (e.g. "does the test suite actually pass?"). Use subdirectories when you want separate scores (correctness vs structure vs quality) rather than one blended number. Use isolated=True for any criterion that runs mutating commands, so it doesn't corrupt the workspace for other criteria. Working example See examples/tasks/reward kit example/ in the Harbor repo.