customize
Adapt this C/C++ ASAN vulnerability pipeline to a different vulnerability class, target shape, language, or detection mechanism. Use when the user wants to port, migrate, retarget, customize, or fork the pipeline for something other than C/C++ memory-safety bugs — web apps, smart contracts, deserial
By anthropics · 950 installs
npx skills add anthropics/defending-code-reference-harness --skill customize
Source repository · Upstream listing
Customize the vuln pipeline
This pipeline ships as an opinionated C/C++ + AddressSanitizer demo. Its real shape is more general: an agent crafts an input, runs a target in a sandbox, a detector fires, a second agent verifies, a third agent analyzes exploitability. Every noun in that sentence can be swapped. Your job is to interview the user, figure out which nouns they want to swap, and rewrite the relevant files.
The existing C/C++ code is the worked example. You don't need a playbook for each domain — read what's there, understand what's generic vs. ASAN specific, and adapt.
STEP 1 — Read the pipeline (do this BEFORE asking anything)
Skim these files so your questions are grounded:
README.md — pipeline overview (recon → find → grade → judge → report)
harness/cli.py — orchestration; shows how stages wire together and what lands on disk
harness/find.py , harness/grade.py , harness/report.py — the three container agent loops; mostly generic plumbing
harness/prompts/find prompt.py , harness/prompts/grade prompt.py — the C/C++ specific parts ; bug taxonomy, quality tiers, grading rubric
harness/prompts/report prompt.py , harness/prompts/report grader prompt.py — also C/C++ specific ; exploitability sections (primitive, heap layout, escalation path) and the rubric that scores them
harness/prompts/judge prompt.py — triage prompt; keys on ASAN excerpts and memory safety crash classes
harness/prompts/system prompt.py — authorization block; hard codes "C/C++ target" and "sanitizer output"
harness/asan.py — stack trace parser for dedup/judge signatures; ASAN specific regex
harness/artifacts.py — CrashArtifact , GraderVerdict , JudgeVerdict , ReportVerdict data contracts
harness/config.py , targets/drlibs/config.yaml — target config schema
targets/README.md — how a target directory is structured (Dockerfile + config.yaml + entry wrapper)
You don't need agent.py , docker ops.py , recon.py , judge.py , or novelty.py in detail — they're generic plumbing (judge/novelty domain specificity lives in the prompts and the asan parser, not the flow). One caution: part of harness/ doubles as shared infrastructure — dnr harness/ (the detection and response pipeline) imports agent.py , sandbox.py , docker ops.py , agent image.py 's build / ensure base , auth.py , cli.py 's resolve target dir / terminate subprocesses , and prompts/system prompt.py 's load engagement context . A domain port should touch only the domain specific files (the prompts, asan.py , artifacts.py , target Dockerfiles); treat any edit to the shared files as a change to both pipelines.
STEP 2 — Interview the user
Use AskUserQuestion to gather requirements. Start with broad context, then narrow to technical specifics based on what they say.
Round 1 — Context (always ask these first, together)
Two open ended questions to understand who you're talking to and what they're after. Expect most answers to come via Other as free text — the options are there to prompt thinking, not to constrain.
Question A — Operating context
header: Context
question: What's your operating environment? Who will run this pipeline and why?
options: a few archetypes as inspiration — e.g. "Pentesting firm — client engagements, need reportable findings", "Internal appsec team — scan our own services in CI", "Smart contract auditor — pre deployment reviews", "Security researcher — hunting novel bug classes". These tell you what output format, grading rigor, and workflow integration matter.
Question B — Goal
header: Goal
question: Describe in your own words what you want this pipeline to find. What kind of target, what kind of bugs?
options: 2–3 concrete examples (e.g. "Web vulnerabilities like SQLi/XSS in HTTP services", "Reentrancy and access control bugs in Solidity contracts", "Deserialization RCE in Java microservices").
The context answer calibrates your follow ups: a pentesting firm probably cares about CVSS scoring and SARIF output; a researcher may want differential testing and novel detection signals; an internal team likely wants CI integration and low false positive rates.
Round 2 — Technical follow ups (adaptive — derive from round 1 answers)
Parse their round 1 answers against the axes of variation below. For each axis left ambiguous, ask a targeted follow up. Batch up to 4 questions per AskUserQuestion call. Common follow ups:
Detection signal — "How will the pipeline know it found something?" (crash, exception, canary file appears, DNS callback, differential mismatch, invariant violation)
PoC shape — "What does a proof of concept look like?" (single file, HTTP request sequence, transaction list, test pipeline code)
Isolation — "Where does the target run?" (Docker, VM, testnet, remote sandbox, or no execution — static only)
Grading criteria — "What makes a finding high quality vs. low quality in this domain?"
Exploitability analysis — "What sections should a report contain?" The C/C++ report has primitive · reachability · heap layout · escalation path · constraints. A web vuln report might want injection vector · auth bypass · data exposure · chaining potential. Ask what they need, or whether they want the report stage at all.
Novelty/upstream check — "Should the pipeline check if a finding is already fixed upstream?" The C/C++ version shallow clones the target's GitHub and checks git log <commit ..HEAD <crash file . Only applies if targets have a canonical upstream and a sensible "crashing file" to key on — many domains won't.
Scope — "Replace the C/C++ support entirely, or keep it alongside the new domain via a profile system?"
Keep going until you can fill in every row of the architecture map in STEP 3. If an answer is vague, ask a narrower follow up rather than guessing.
Background — axes of variation (context for formulating follow ups)
These are the dimensions along which customers might want to deviate from the C/C++ demo. Use this list to spot gaps in the user's description and generate follow up questions — do not present it as a menu.
Vulnerability class: memory safety · web/API (SQLi, XSS, SSRF, XXE, path traversal, IDOR) · deserialization RCE · logic/race (TOCTOU, privilege escalation) · crypto (weak RNG, timing, nonce reuse) · DoS (ReDoS, hash flooding) · smart contracts (reentrancy, access control, front running) · ML/AI (prompt injection, jailbreaks, data extraction) · protocol parsing
Target shape: CLI binary + file · HTTP service · library via test harness · network daemon · smart contract · browser extension · mobile app
Detection mechanism: crash/abort · uncaught exception · sanitizer hooks (Jazzer/Atheris) · outcome based (canary file, DNS callback, shell spawn) · differential testing · invariant violation · taint tracking
Input modality: single file · HTTP request chain · multi file archive · stdin stream · args + env + config combo · transaction sequence
Isolation boundary: Docker container · full VM · remote sandbox · local testnet · none (static analysis)
Dedup signature: (crash type, top frame) · (vuln type, endpoint, param) · (function, state transition) · (component, precondition)
Report structure: primitive/heap/escalation (memory safety) · vector/auth/exposure (web) · invariant/path/impact (contracts) · or drop the report stage entirely if find+grade is the deliverable
Output format: result.json + poc.bin · SARIF · Nuclei template · prose report
Patch verification signal: ASAN clean exit · uncaught exception free · sanitizer hook silent (Jazzer/Atheris) · canary file untouched · invariant assertion holds · differential output matches reference. This is what t1 passes() in patch grade.py encodes — "the bug is gone" for the new domain.
Background — architecture map (what changes vs. what stays)
File C/C++ specific? What it does
harness/prompts/find prompt.py Yes — rewrite Bug taxonomy, quality tiers, ASAN output format, exit code examples
harness/prompts/grade prompt.py Yes — rewrite 5 criterion rubric assumes ASAN traces and Unix signal exit codes
harness/prompts/report prompt.py Yes — rewrite Exploitability sections: primitive, heap layout, escalation path — memory safety specific
harness/prompts/report grader prompt.py Yes — rewrite Scores the above sections; rubric is tied to the section set
harness/prompts/judge prompt.py Yes — rewrite Triage keys on ASAN excerpts and crash class taxonomy
harness/prompts/patch prompt.py Yes — rewrite Asks for git diff ' .c' ' .h' , assumes ASAN trace, memcpy style root cause guidance
harness/prompts/system prompt.py Yes — rewrite Authorization block says "C/C++ target", "sanitizer output"
harness/asan.py Yes — rewrite Regex for N 0xHEX in func /path:line frames; feeds dedup, judge, novelty
targets/README.md + Dockerfile template Yes — rewrite gcc fsanitize=address , entry.c wrapper pattern
harness/patch grade.py Light edit t1 passes() checks AddressSanitizer: substring; rest of the verification ladder is generic
harness/report.py Light edit SECTIONS tuple and token lists need to match the new report structure; flow is generic
harness/novelty.py Light edit crash file from frame() is ASAN specific; git log logic is generic. Drop entirely if no upstream.
harness/config.py Light edit May need new fields ( profile , run command instead of binary path ); attack surface likely stays
harness/artifacts.py Light edit crash type / exit code semantics may shift; ReportVerdict.section scores keys must match new sections
harness/dedup.py Light edit Signature function needs the new parser; grouping logic is generic
harness/prompts/recon prompt.py Light edit Mostly language agnostic; scrub C idioms
harness/cli.py Unchanged Orchestration is domain neutral
harness/agent.py Unchanged Agent runner is generic
harness/docker ops.py Unchanged Container plumbing is generic (may need changes if isolation ≠ Docker)
harness/find.py , grade.py , recon.py , judge.py , patch.py Unchanged Flow is generic; only injected prompts change
STEP 3 — Present a plan and get confirmation
Before editing anything, summarize back to the user:
1. What you understood — restate their goal in one sentence
2. What will change — list each file you'll edit with a one line rationale
3. What stays — reassure them the orchestration core is untouched; call
out explicitly if the plan does reach into the shared files listed above
(they also serve dnr harness/ ) and why
4. Open questions — anything you're still unsure about
Wait for explicit approval. If they adjust the plan, incorporate and re confirm.
STEP 4 — Execute
Edit the files per the approved plan. Work through them in dependency order: prompts and parser first (they're standalone), then config/artifacts, then the target template, then README. Commit incrementally if the user wants checkpoints.
STEP 5 — Validate
1. Add a canary target under targets/<domain canary/ with 2–3 planted bugs of the new class
2. Run: bin/vp sandboxed run <domain canary model <model id runs 3 parallel stream max turns 50 (use Claude Opus unless the user specifies a different model). Run ./scripts/setup sandbox.sh once first if the sandbox isn't already set up.
3. Confirm all planted bugs are found and graded PASS
4. Confirm judge triage worked: cat results/<domain canary/<ts /reports/judge log.jsonl — expect one NEW per distinct bug, DUP SKIP for repeats
5. Confirm reports landed: ls results/<domain canary/<ts /reports/bug /report.json and spot check section scores
6. Run vuln pipeline dedup results/<domain canary/ and confirm signatures group correctly