Consolidates raw security findings to eliminate redundant reports. Use when raw findings have been generated by the researcher and need consolidation before review. Don't use for initial code auditing or patch generation.

Deduplicator (/mantis dedupe) System Goal Duplicate Finding Merger. Evaluates lists of raw findings to cluster and consolidate identical or highly overlapping issues into singular, descriptive records. Command Definition Command: /mantis dedupe Description: Consolidates raw security findings to eliminate redundant reports. Arguments (optional; supplied by the orchestrator, consumed by Block A): snapshot root / snapshot id / state root . All absent MODE OFF/legacy mode (behaves as today; snapshot gating disabled). Input/Output Contract Reads : workspace/findings/ (raw finding JSON files, ignoring .trash/ ). workspace/archive/findings pass / .json and workspace/archive/loop findings/ .json (to skip findings already evaluated and triaged in previous passes). workspace/.mantis state.json (to track current loop pass). Writes : Moves duplicate findings to workspace/findings/.trash/ after setting "status": "DUPLICATE" and "duplicate of" . Sets "possible duplicate of" (soft, non terminal) on NOT MATCHED matches and stamps "discovery commit" on current findings that lack it. Reads active snapshot / snapshot pinned from .mantis state.json . Appends transaction logs to workspace/.tx log.jsonl . Generates/executes merging script workspace/helpers/merge findings.py . Updates primary finding workspace/findings/<primary id .json (merges fields and history). Preconditions : workspace/findings/ must exist and contain finding files. Idempotency Guarantee : Cross references against archived findings in workspace/archive/ to filter out any findings already processed in previous passes of this run. Snapshot gated: a resolved archived finding on a differing snapshot is flagged as POSSIBLE REGRESSION (never silently filtered). Logs transactions to workspace/.tx log.jsonl to support tracking and potential rollbacks. Deterministic merging rules implemented in merge findings.py . Instructions Step 0: Locator Resolution (run first) [!NOTE] CURRENT PASS CHECK (defensive; the binding guarantee is on the harness per mantis pipeline adapter Scenario 2): if active snapshot is present AND active snapshot.pass != state.pass number , treat the snapshot as STALE for this pass — STOP "stale active snapshot: pass mismatch" or degrade as HALT ( snapshot pinned effectively false: no authoritative verdicts, Block B NOT MATCHED, reproduce not attempted ). This catches a custom harness that preserved active snapshot across the Stage 15 pass increment without re pinning. The reference meta agent re pins every pass, so this check never fires there. Block B itself cannot detect this (it is snapshot id only, not pass aware). Notes: workspace/findings/ , workspace/archive/ , workspace/.tx log.jsonl , workspace/helpers/ and .mantis state.json are STATE RELATIVE (under state root). Any code snippet you inspect for a finding is SNAPSHOT RELATIVE (under CODE ROOT). Never write under CODE ROOT. Review a list of security findings and merge duplicate findings that refer to the exact same security flaw or adjacent code paths. Execute your task as follows: 1. Load Raw Findings & Archived Findings Queue: List the contents of the directory and read the files in workspace/findings/ . If the directory is empty or does not exist, notify the user and exit. Important: Ignore hidden files and directories (such as the .trash/ subdirectory) when listing or processing findings. Locate and load all archived finding JSON files from previous loop passes, if they exist, under workspace/archive/findings pass / .json and workspace/archive/loop findings/ .json . These files represent vulnerabilities that have already been fully evaluated, triaged, and potentially patched in previous passes. Important: Do NOT read or deduplicate against workspace/historical learnings.jsonl (VCS history), as we want to catch regressions if old bugs were reintroduced. 2. Filter Loop Duplicates (snapshot gated). First, stamp discovery commit on any CURRENT finding that lacks it, using active snapshot.snapshot id from .mantis state.json (skip when unpinned). Then, for each current finding that matches an archived finding (by code paths + title similarity), run: Signature based candidate matching (Phase 3) — TIGHTENS, never replaces: signature may only PROMOTE a pair to "candidate for the pairwise snapshot check"; it may NEVER by itself cause a hard DUPLICATE/trash. A pair is a candidate for the Pairwise Snapshot Match Check below ONLY if it satisfies BOTH: it matches under today's code paths + title similarity, comparing code paths entries line inclusively (WITH their trailing :line ); AND (when both findings have a signature ) their signature fields are equal. A signature match WITHOUT the code paths + title agreement is NOT a duplicate — at most a soft possible duplicate of (keep the finding ACTIVE), never a trash. Rationale: signature strips the line number and all but first code paths entry, so two DISTINCT bugs in the same file (e.g. parser.c:100 vs parser.c:900 ) with the same title+CWE share one signature ; trashing on signature alone would silently delete a real finding. If EITHER lacks signature , use today's code paths + title similarity matching unchanged. PAIRWISE SNAPSHOT MATCH CHECK (decides MATCHED vs NOT MATCHED) — compares the CURRENT finding's discovery commit against the ARCHIVED finding's discovery commit for this pair (NOT against SNAPSHOT ID ): 1. If snapshot pinned is false AND there is NO active snapshot in state (MODE OFF) NOT MATCHED. Stop. (In HALT — active snapshot present but snapshot pinned=false — do NOT short circuit here; fall through to the pairwise comparison below, which will be NOT MATCHED because the current finding's discovery commit is a live: id that will not equal the archived one.) 2. Read the CURRENT finding's discovery commit and the ARCHIVED finding's discovery commit : If EITHER is missing, empty, or the literal "MIXED" NOT MATCHED. If they are NOT byte for byte equal to each other NOT MATCHED. If they ARE byte for byte equal to each other (both present, non MIXED) MATCHED. There is no other route to MATCHED; never fuzzy compare. The global "default the field and proceed" backward compat rule does NOT apply to discovery commit : absent = NOT MATCHED. (There is NO separate "dirty" gate: a dirty tree's SNAPSHOT ID already embeds the working tree content hash, so within pass findings MATCH and cross pass bare commit findings do not.) Note: this is a PAIRWISE check (current vs archived), NOT a check against the global SNAPSHOT ID — dedupe stamps the current finding's discovery commit to SNAPSHOT ID in Step 2 above, so a check against SNAPSHOT ID would always be MATCHED and would trash reintroduced/regression bugs as DUPLICATE. Then, using the idempotency rule (Input/Output Contract → Idempotency Guarantee) to avoid double writes, decide mechanically: MATCHED (both present and equal): soft delete the current finding as a loop duplicate exactly as before — set "status": "DUPLICATE" and "duplicate of": "<archived uuid " , clear possible duplicate of if present, ensure mkdir p workspace/findings/.trash/ , move it there, and log a loop filter transaction in workspace/.tx log.jsonl . If the current finding lacks lineage id but the archived finding has one, inherit the archived finding's lineage id onto the current finding before moving it (so the lineage chain is preserved across the merge). NOT MATCHED (differ, or either absent): do NOT set DUPLICATE and do NOT move to trash. Keep the current finding ACTIVE and set "possible duplicate of": "<archived uuid " (a soft, non terminal hint). If the current finding lacks lineage id but the archived finding has one, inherit the archived finding's lineage id onto the current finding (so the lineage chain is preserved for report folding even when the findings are on different snapshots). [!IMPORTANT] STATUS & DUPLICATE INVARIANTS: A finding MUST NOT carry both duplicate of and possible duplicate of pointing to the same target UUID. status = "DUPLICATE" MUST NOT coexist with possible duplicate of pointing to the same target UUID. Under NOT MATCHED (outside the MODE OFF fallback exception), the finding's status MUST remain active (e.g. VALID , PROVISIONALLY VALID , NEEDS RESEARCH ), duplicate of MUST NOT be set, and the finding MUST NOT be moved to .trash/ . Setting possible duplicate of is a non terminal hint only. POSSIBLE REGRESSION: if the archived match has a RESOLVED status ( patch status in { VERIFIED SECURE , MITIGATION PROPOSED } OR status == FALSE POSITIVE OR production viability == NON VIABLE ) AND the pair is NOT MATCHED, keep the current finding ACTIVE, add a history note "POSSIBLE REGRESSION vs \<archived uuid ", and do NOT filter it. (A reverted fix re discovered on new code must never be trashed.) EXACT UUID retry exception (unchanged): if the current finding has the EXACT SAME UUID as the archived one, it was intentionally copied back for a retry — do NOT filter it (keep as is). Permanently unpinned exception (MODE OFF only — no active snapshot ): if there is NO active snapshot in state (MODE OFF = today's default; a target with no snapshot boundary, e.g. a live endpoint), snapshot pinned is false at the PASS level ( active snapshot.snapshot pinned — it is NOT a per finding field) and Block B is uninformative — fall back to today's dedup by signature if present, else stable key = normalized title + first code paths entry including its trailing :line (line inclusive, same as Step 2). Rationale: stripping :line would collapse two DISTINCT bugs in the same file (e.g. parser.c:100 vs parser.c:900 ) with the same title+CWE into one — silently deleting a real finding. Note: signature itself strips :line by design (it is a coarse identity for cross pass lineage, not a dedup key); this fallback therefore prefers signature only when stable key 's line inclusive match ALSO agrees, never on signature alone. This preserves dedup for targets that can never MATCH. When active snapshot IS present but unpinned (HALT mode), this exception does NOT fire: keep the snapshot gated behavior above (NOT MATCHED → keep ACTIVE + possible duplicate of , never DUPLICATE ). POSSIBLE REGRESSION takes precedence over this fallback: a resolved archived finding paired with a NOT MATCHED current finding is ALWAYS kept active (never trashed), regardless of mode. 3. 3 Tier Deduplication Ladder (Deterministic & Semantic Matching): When resolving finding lineages and evaluating deduplication candidates, the deduplicator uses a hierarchical 3 tier ladder designed for sub millisecond fast path execution with semantic RAG fallback: Tier 1 (Fast Path Exact Heuristic Anchors — < 1ms, 0 tokens): 1. Exact content identity signature match: hashlib.sha256(canonical fp canonical cwe target symbol) (invariant to line shifts, backticks, and title paraphrasing). 2. Exact canonical filepath + normalized CWE + target symbol match. 3. Strict line proximity window

mantis-dedupe