Write Harbor task verifiers using Reward Kit. Use when creating or editing a task's tests/ directory, adding grading criteria, setting up LLM/agent judges, or designing verifiers that produce a reward score.

Help the user write task verifiers with Reward Kit. Reward Kit is a lightweight Python package that turns a directory of criteria files into a reward score. Each criterion is a Python function call or a TOML judge file; folders become separate rewards. Setup in a Harbor task Put criteria alongside test.sh in the task's tests/ directory: tests/test.sh : This runs all criteria in /tests/ against the workspace at /app and writes /logs/verifier/reward.json . Defaults match Harbor's conventions — no extra config needed. If judge criteria need API keys, pass them through task.toml : Ask whether Reward Kit should run in the agent's shared environment or in a separate verifier environment. Prefer a separate verifier environment when judge prompts, grading dependencies, API keys, or clean room checks should not be available to the agent: In shared mode, the verifier runs in the agent container and inherits [environment].network mode . Put [verifier].network mode only when verify() needs different network access than the agent phase (a phase override, not a baseline). If agent and verifier need different baselines without runtime switching, use environment mode = "separate" and set [verifier.environment].network mode . Judge criteria that call external APIs need a public baseline or allowlist on the verifier environment. Programmatic checks that only read local files can use no network . In separate mode, tests/ is the verifier image build context and must provide /tests/test.sh at runtime; Harbor does not upload tests/ into the running verifier container. Programmatic criteria Call built ins from any .py file in tests/ : All criteria accept weight (default 1.0 ) and isolated (default False , runs in overlayfs so side effects don't leak). Available built ins Files : file exists , file not exists , file contains , file contains regex , file matches , files equal , diff ratio Commands : command succeeds , command output contains , command output matches , command output matches regex (30s default timeout, optional cwd ) Data : json key equals , json path equals , csv cell equals , xlsx cell equals (needs [office] extra), sqlite query equals HTTP : http status equals , http response contains Images : image similarity , image size equals (needs [image] extra) Trajectory : trajectory tool used , trajectory tool not used , trajectory turn count For extras, install with uv tool install harbor rewardkit[all] . Custom criteria Use the @criterion decorator. First parameter is always workspace: Path . Returns bool , float , or a dict with score plus optional reasoning , confidence , and model : Zero parameter criteria auto register. Criteria with extra args must be called via rk : For criteria shared across reward subdirs, define with shared=True in a root level file and call from subdirs. Judge criteria (LLM or agent as a judge) For subjective checks (quality, readability, edge cases), create a TOML file: Criterion types: binary — yes/no → 1.0 or 0.0 likert — 1..points, normalized to [0, 1] numeric — min..max, normalized to [0, 1] rubric — 2 to 10 described levels forming a scale from worst to best; position sets the score Agent judges Agent judges shell out to a CLI and can explore the filesystem: Slower and more expensive than LLM judges, but they can run commands and inspect files. JEV judge JEV is a new type of language model from TypeSafe. It answers each criterion with a probability or a rubric score and returns no reasoning, so it is fast and cheap. It needs the jev extra ( harbor rewardkit[jev] ) and TYPESAFE API KEY . Binary criteria pass at a probability of 0.5 or higher. Only binary and rubric criteria and text files are supported; atif trajectory and prompt template are not. The task image needs CA certificates ( ca certificates on Debian and Ubuntu), because the TypeSafe SDK verifies TLS against the system trust store. Useful [judge] options timeout (default 300), reasoning effort ( low medium high for every LLM and agent judge, other levels depend on the judge; jev accepts none), reference (path to reference solution), atif trajectory (evaluate the agent's trajectory), weight , prompt template (custom prompt with {criteria} placeholder), samples (run the judge N times and score each criterion with its median sample), guard (protects the judge from instructions the agent wrote into the graded files; "flag" reports them under guard in the details, "penalize" also scores a flagged submission 0), isolated (agent judges only: read only overlayfs copy of the workspace). More than one agent run in a task (several agent judges, samples above 1, or individual mode with several criteria) requires isolated = true on each agent judge, which needs a separate verifier with fuse overlayfs and mount privileges; yolo skips the check. Scoring aggregation (within one judge TOML) all pass , any pass , and required pass count a score as passing only when it is 1.0, so a Likert or numeric criterion needs full marks. Only affects how this file's own criteria combine. To aggregate across dimensions, see [Aggregating dimensions]( aggregating dimensions). Scoring config for programmatic files Each .py file that registers criteria is an equal weighted scoring component named after its filename stem. Files that only provide imports or shared criterion factories and register no checks are ignored. To change how criteria within a file combine, use [scoring.<stem ] in the same directory's reward.toml : Each entry takes the same aggregation values as a judge TOML. Unknown keys and stems that do not resolve to a criterion bearing Python file raise. Directories may be nested recursively. A non root directory can aggregate its local Python files, local judges, and immediate child directories with one unnamed [[reward]] table: Membership is implicit. Child directories have weight 1.0 unless overridden; use filename stems for local Python files and judge TOMLs, and directory names for child groups. Without [[reward]] , the directory defaults to weighted mean. Multi reward tasks Put criteria in subdirectories — each becomes a separate reward: Judge TOMLs may also sit directly at the tests root alongside reward subdirectories. Each is exposed as a top level reward named after its filename stem and can be referenced by a root aggregation. Criterion bearing Python files at the tests root are also top level dimensions named after their stems. Root support files that register no criteria are ignored. Produces: Aggregating dimensions To add aggregated scores on top of the per dimension keys, add a root level tests/reward.toml with one or more [[reward]] tables. Each adds one key to reward.json , aggregating the dimensions with the same modes as [scoring] : The per dimension scores stay; aggregated keys are added alongside them (a name may not collide with a dimension). Top level dimensions have equal weight unless that aggregation's inline map overrides them; reward details.json keeps the full recursive breakdown. Output files /logs/verifier/reward.json — per reward scores /logs/verifier/reward details.json — per criterion results, judge reasoning, errors Multi step tasks In a multi step task, each step has its own tests/ under steps/{name}/tests/ , and the verifier runs once per step. Reward Kit behaves the same as in a single step task: for each step it reads /tests , runs the criteria against /app , and writes /logs/verifier/reward.json for that step. Harbor then aggregates per step results into a trial level reward via multi step reward strategy in task.toml — aggregation happens outside Reward Kit, so don't try to encode cross step logic in your criteria. A task level tests/ directory (at the task root) is uploaded to /tests first, then the step's own tests/ is layered on top (same name files win). Put shared helpers (common checks.py functions with shared=True , fixture files, a fallback test.sh ) at the task level, and step specific criteria under each step. Multi reward subdirectories still work within a step: steps/foo/tests/ can contain correctness/ , structure/ , quality/ — each produces a separate reward key for that step, and multi step reward strategy = "mean" averages each key across steps. Use "final" when the last step is an end to end check whose rewards already represent the full task. When to reach for what Use built ins for file existence, string matches, command output, JSON/CSV checks, HTTP probes. Use @criterion when logic is task specific but still programmatic. Use LLM judges for subjective quality dimensions (readability, correctness of prose). Use agent judges when the rubric requires exploring the filesystem or running code (e.g. "does the test suite actually pass?"). Use subdirectories when you want separate scores (correctness vs structure vs quality) rather than one blended number. Use isolated=True for any criterion that runs mutating commands, so it doesn't corrupt the workspace for other criteria. Working example See examples/tasks/reward kit example/ in the Harbor repo.