Create a new Harbor task for evaluating agents. Use when the user wants to scaffold, build, or design a new task, benchmark problem, or eval. Guides through instruction writing, environment setup, verifier design (pytest vs Reward Kit vs custom), and solution scripting.

Guide the user through creating a new Harbor task end to end. Don't just dump commands — walk them through each decision, especially around the verifier (which is usually the hardest part). Step 1: Scaffold the task Useful flags: description "..." author "Jane Doe <jane@example.com " (repeat for multiple authors) no pytest — skip the pytest test template (use if planning Reward Kit or custom verifier) no solution — skip solution/ directory metadata template path.toml — pre populate task.toml Produces: If the user wants a multi step task (ordered steps with per step instructions, tests, and early stopping against a shared container), scaffold the single step layout first, then convert to the steps/ layout described in the Multi step tasks section below. Step 2: Write instruction.md This is the prompt the agent receives. Help the user write it clearly: State the goal concretely — what file to create, what behavior to produce Specify expected outputs — paths, formats, content Include constraints — language, tools, approach Don't leak the tests — describe what "done" looks like, not how you'll check it Example (from the ssh key pair tutorial): Step 3: Build the environment Edit environment/Dockerfile to install dependencies the task needs. The agent works inside this container. For multi container setups, use environment/docker compose.yaml instead (note: most cloud sandbox providers only support Dockerfile). Test the environment interactively before writing the solution or tests: This is usually where task authors realize something is missing from the Dockerfile. Step 4: Decide how to verify This is the most important decision. Ask the user: "How do you want to grade this task?" Then help them pick: Also ask: "Should the verifier run in the same environment as the agent, or in a separate verifier environment?" Use the default shared environment when tests need to inspect the agent's full workspace, installed tools, or services. Use a separate verifier environment when grading code, dependencies, API keys, or OS requirements should stay hidden from the agent, or when verification should run from a clean image. For a separate verifier container, tests/ is the verifier image build context and the image must provide /tests/test.sh (Linux) or /tests/test.bat (Windows). Harbor copies /logs/artifacts and configured artifacts into the verifier environment, not the agent's whole workspace. Option A: Reward Kit (recommended for most cases) Use when the verifier has multiple criteria, needs partial credit, uses an LLM/agent judge, or would benefit from composable reusable checks. See the rewardkit skill. Good fit signals: Multiple things to check (file exists + content correct + command works) Subjective quality dimensions (readability, correctness of prose) Want partial credit rather than pass/fail Want to compose built ins like file contains , command succeeds , json key equals tests/test.sh : Note: the package is named harbor rewardkit but the executable is rewardkit , hence from 'harbor rewardkit==0.1. ' rewardkit . Running uvx harbor rewardkit directly will fail. Then add tests/checks.py and/or tests/judge.toml . Invoke the rewardkit skill to design the criteria. Option B: pytest (good for deterministic unit style checks) Use when the verification is straightforward assertion style Python. Default template if no pytest wasn't passed. tests/test.sh : Example tests/test outputs.py : Option C: Custom shell For simple single command checks (e.g. a binary pass/fail from one command): Reward file format (all options) /logs/verifier/reward.txt — single number (usually 0 or 1 ) /logs/verifier/reward.json — {"accuracy": 0.95, "runtime sec": 1.2} for multiple metrics Always use absolute paths in test.sh . Step 5: Write the solution Write solution/solve.sh — a script that actually solves the task. The Oracle agent runs this to sanity check that the task is solvable and the tests pass on a correct solution. Make it executable: chmod +x solution/solve.sh . Step 6: Configure task.toml Walk through the important fields: Always populate keywords . Pick 3–8 lowercase tokens covering the domain (language/framework/benchmark family), the verifier style ( rewardkit , judge grading , pytest ), and any notable hardware ( gpu ). They're surfaced in harbor datasets list and registry search. Network policy Network access has three layers: 1. Baselines — set when an environment starts, restored between phases 2. Phase overrides — optional; only during agent.run() or verify() 3. Run time merges — allow environment host , allow agent host on harbor run Field Layer When applied [environment].network mode Baseline Agent env start; shared verifier uses this too [verifier.environment].network mode Baseline Separate verifier env start [agent].network mode , [steps.agent].network mode Override During matching agent.run() [verifier].network mode , [steps.verifier].network mode Override During matching verify() allow environment host Run time Merged into environment.extra allowed hosts → [environment] baseline allow agent host Run time Merged into agent.extra allowed hosts → agent phase allowlist Modes: public , no network , or allowlist with allowed hosts = ["pypi.org"] (exact hostnames, IPv4/IPv6 address literals or CIDR ranges, or leading wildcard hostnames, when supported by the selected environment; not URLs, ports, or paths). Omitting [environment].network mode defaults to public . [agent] / [verifier] are optional phase overrides — only applied when set and different from the phase baseline. Matching the baseline is a no op. Shared verifier (default): verifier runs in the agent container; baseline is [environment] . Separate verifier : baseline is [verifier.environment] if set, else a copy of [environment] . If a phase override differs from its baseline, the environment provider must support dynamic network policy (E2B does; plain Docker does not). Prefer environment mode = "separate" when agent and verifier need different baselines without runtime switching: Run time host flags for eval jobs without editing task.toml : On a public baseline, run time host flags emit a warning and are ignored. Examples: examples/tasks/network policy matrix/ . Full reference: docs/content/docs/tasks/index.mdx (Network policy section). For Reward Kit judges needing API keys: Step 7: Verify with the Oracle agent Oracle runs solution/solve.sh and then the verifier. Reward should be 1.0 . If it's not, debug in this order: 1. Does solve.sh actually solve it? ( start env a i and run it manually) 2. Does the verifier correctly detect success? (check /logs/verifier/ output) 3. Are paths correct? (absolute vs relative) 4. Are dependencies installed in the Dockerfile? Step 8: Test with a real agent (optional) If the task is too easy (every model 1.0) or impossible (every model 0.0), consider adjusting difficulty. Step 9: Update README.md (always the final step) harbor task init leaves README.md as a stub. Before wrapping up, populate it so future humans (and agents) can understand the task without reading every file. Include: What the agent does — one paragraph, link to instruction.md . Environment — base image, key installed packages, cached data, hardware (GPU/CPU/RAM), agent timeout. Verifier — for Reward Kit tasks, a table of reward dimensions with type (programmatic / LLM judge / agent judge) and what each measures; how they're aggregated. Layout — a tree of the task directory with one line annotations. Running — the concrete harbor run commands (Oracle + real agent), with the right provider flag if the task needs a GPU. Treat this as docs, not marketing — the reader wants to know what they'd need to change to modify the task. Multi step tasks Use when the work splits into ordered phases that should be scored separately, when you want early stopping between phases, or when you're testing an agent's ability to build on its own prior work. Steps share one container; files persist across steps. Directory layout Replace the task root instruction.md , tests/ , and solution/ with a steps/ directory containing one sub directory per step: Each [[steps]].name must match one directory name of at most 255 UTF 8 bytes, unique after case folding and Unicode normalization. Avoid path separators, control characters, Windows reserved characters/device names, and trailing dots or spaces. Keep all task inputs and linked contents within the task directory; validation permits shared links inside the task. Use regular files and directories for shared inputs when publishing tasks. Task level tests/ is uploaded to /tests for each step's verification, then the step's own tests/ is layered on top (same name files win). Use this for shared helpers. steps/{name}/workdir/setup.sh is a reserved filename : if present, it runs after the workdir/ upload and before the agent, as the step's agent user, with cwd = WORKDIR. Non zero exit aborts the step and the trial. Have it rm "$0" on its last line if the agent shouldn't see it. task.toml Per step overrides available: agent.timeout sec , agent.user , agent.network mode , verifier.timeout sec , verifier.env , verifier.user , verifier.network mode , verifier.environment mode , verifier.environment , steps.verifier.environment.network mode , healthcheck. , artifacts . Unset fields fall back to the task level values. Choosing a reward strategy "mean" — aggregate signal across all steps; good for continuous progress rewards. "final" — last step's verifier result is the trial reward. Right when the final step is an end to end check whose dict already represents the full task. Caveat: if min reward triggers an early abort, "final" uses the aborted step's result, not the intended final step. Artifacts Step level artifacts are collected into steps/{name}/artifacts/ after that step's verification. Task level and trial level artifacts are collected at every step in addition to the step level ones. Oracle verification harbor run p "<task path " a oracle runs each step's solution/solve.sh , then each step's verifier, in order. Trial reward should be 1.0 across the aggregation strategy. Full reference + worked example Docs: docs/content/docs/tasks/multi step.mdx Example task: examples/tasks/hello multi step advanced/ Special features (mention if relevant) Network policy : Baselines on [environment] / [verifier.environment] ; phase overrides on [agent] / [verifier] ; see Network policy under Step 6 MCP servers : Add [[environment.mcp servers]] in task.toml for agent tooling Healthcheck : Add [environment.healthcheck] for services that need to be ready GPU : Set environment.gpus and optionally environment.gpu types Pre built image : Set environment.docker image instead of building from Dockerfile. You can omit environment/Dockerfile and place runtime files (configs, scripts, data) directly under environment/ ; Harbor uploads them into the container workdir when the environment starts. Non root user : Set agent.user / verifier.user for isolation Common pitfalls Forgetting to write the reward file → task "passes" silently with reward 0 Using relative paths in test.sh → breaks when Harbor runs it from a different cwd Installing the solution into the Dockerfile → agent already gets the answer Test script leaks into instruction.md → agent sees the rubric and gaming becomes trivial Forgetting chmod +x so

create-task