Builds a content-addressed semantic-unit index from source code for structural context. Use when a pinned or live codebase is available and structural cross-reference data would improve research quality. Don't use for findings analysis, patching, or reporting.

Structural Code Index Builder This is an optional first class stage in the Pass Lifecycle Contract. It runs immediately after the snapshot is pinned (Block D), before the first code reading analysis stage (summarize/architecture). It only needs CODE ROOT \+ SNAPSHOT ID and must not depend on architecture/KB. System Goal Structural Code Index Builder. Builds a content addressed semantic unit index from source code using capability based per partition backend selection, degrading gracefully to grep. Provides find callers(symbol) , get function boundary(file, line) , and call site awareness to improve LLM reasoning quality during discovery — supplementing (never replacing) grep based call site discovery with structural data. The index is composed from independent semantic compilation units, persisted as a manifest + SQLite catalog, and exposed through a bounded query helper. Command Definition Command: /mantis structural index Description: Build a content addressed semantic unit index from source code under CODE ROOT . Arguments (optional; supplied by the orchestrator, consumed by Block A): snapshot root / snapshot id / state root . All absent → MODE OFF/legacy mode (reads source from the current directory, writes index to ./workspace/kb/structural index/ ). Input/Output Contract Reads : workspace/.mantis state.json (to read active snapshot for provenance checking and snapshot aware rebuild logic). workspace/kb/structural index/manifest.json (to check snapshot id for reuse on match idempotency — primary check). workspace/kb/structural index.jsonl (backward compat provenance check if manifest absent). workspace/kb/structural index/units/ (content addressed cache for incremental unit reuse). workspace/kb/structural index/native/ and sidecar provenance.json files (prebuilt index attachments and metadata manifests). CODE ROOT source files (via generated helper script — the script parses all source files under CODE ROOT ). Writes : workspace/kb/structural index/manifest.json (STATE RELATIVE — atomic commit point, written LAST). workspace/kb/structural index/catalog.sqlite (STATE RELATIVE — query optimized serving store). workspace/kb/structural index/units/ (STATE RELATIVE — content addressed immutable unit outputs). workspace/kb/structural index/shards/ (STATE RELATIVE — partitioned serving data for large corpora). workspace/kb/structural index/native/ (STATE RELATIVE — prebuilt index attachments: SCIP, Kythe, LSIF). workspace/kb/structural index/tmp/ (STATE RELATIVE — temporary objects during build). workspace/kb/structural index.jsonl (STATE RELATIVE — compatibility pointer; full export below threshold). workspace/helpers/build structural index.py (STATE RELATIVE — the builder helper script). workspace/helpers/query structural index.py (STATE RELATIVE — the query helper script). Preconditions : Source files must exist under CODE ROOT . If CODE ROOT is not resolved (MODE OFF and no readable active snapshot ), build against the current directory with snapshot id set to "unknown" . Do NOT skip — this is the standalone efficiency case. Inert until wired: This skill returns an empty index until a caller invokes it (the harness, mantis plan , or mantis researcher ). It never fails — it simply returns an empty index if tools are unavailable or source cannot be parsed. Idempotency Guarantee : Read only on CODE ROOT . Writes only to STATE RELATIVE paths. Re running with the same CODE ROOT and SNAPSHOT ID reuses the existing index (manifest snapshot id match) rather than rebuilding — except in MODE OFF, where it always rebuilds. Individual semantic units are reused across snapshots when their content addressed cache keys match (incremental rebuild). Instructions Step 0: Locator Resolution (run first) This is a CODE READING stage — it reads target source files under CODE ROOT via the helper script. Block A step 0's findings only skip does NOT apply. Step 1: Idempotency / Freshness Check (MANDATORY FIRST) MANDATORY FIRST STEP: Before writing any build scripts, probing backends, or extracting symbols, check for MODE OFF or an existing manifest: 1. MODE OFF check (FIRST): In MODE OFF ( SNAPSHOT ID is "unknown" or absent) → always rebuild . Do NOT reuse a previous "unknown" index, because the live tree is mutable and "unknown" is a constant (not a freshness signal). Skip directly to Step 2 (individual units whose content has not changed may still hit the content addressed cache). 2. If workspace/kb/structural index/manifest.json exists, read its snapshot id field. 3. If snapshot id matches the current SNAPSHOT ID (and SNAPSHOT ID is NOT "unknown" ) → reuse the index immediately and STOP . Do NOT invoke Step 2, probe backends, or write build scripts. This bounds cost across retries/crash resume. 4. If manifest.json is absent but workspace/kb/structural index.jsonl exists (backward compat), read its provenance header ( provenance , snapshot id keys). If snapshot id matches (and is NOT "unknown" ) → reuse and STOP . Otherwise proceed to rebuild. 5. If SNAPSHOT ID differs → proceed to rebuild (Steps 2–5). For incremental reuse: before rebuilding a semantic unit, check units/ for a content addressed cache hit (see Content Addressed Cache Key below). A cache hit reuses the unit output without re extraction. Step 2: Select Backend per Partition Partition the codebase into semantic units (see Per Language Semantic Units below). For each unit, select the most precise backend available in this environment using a capability based, per partition decision — not a single global ladder. The following are examples, not an exhaustive enum . The helper probes each tier per partition and selects the highest available: 1. Snapshot matched SCIP / LSIF / Kythe / clangd static / remote index (precision: semantic ) — if a pre built index matching the current snapshot id or root fingerprint is available. Most precise: full type aware cross reference, call hierarchy, and hover/signature data. 2. Compiler / typechecker backed extractor (precision: typecheck ) — if compile commands.json , build context, or typechecker is available. Type accurate symbol resolution and call edges. 3. Language aware AST extraction (precision: ast ) — if tree sitter , ast grep , or a language specific parser is available. Full AST parsing: function boundaries, call expressions, signatures. 4. Symbol only extraction (precision: symbol only ) — if ctags or equivalent is on PATH . Symbol table only (function definitions, locations — no call graph). Call site extraction uses a lightweight regex pass within known function boundaries. 5. Heuristic fallback (precision: heuristic ) — Python stdlib regex pass over source files. Identifies function definitions and call patterns using language agnostic heuristics. Less precise but zero dependency. 6. Coverage only manifest + lexical fallback (precision: coverage only ) — grep; no structural index is written. Manifest records status: "empty" . Consumers fall back to grep based discovery (today's behavior byte for byte). Native Index Probing & Resolution Rules : Probe instruction : Before evaluating per partition backends, probe workspace/kb/structural index/native/ and subdirectories native/{scip,lsif,kythe}/ for prebuilt index files. Snapshot declaration convention : Because native formats (SCIP, LSIF, Kythe) do not embed snapshot identity directly in their binary payload, prebuilt indexes MUST declare their target snapshot using a sidecar provenance.json manifest located at workspace/kb/structural index/native/provenance.json or native/<kind /provenance.json . The manifest contains an array of attachments: [{"kind": "scip lsif kythe", "path": "...", "snapshot id": "...", "root fingerprint": "...", "language": "...", "indexer": "...", "precision": "semantic", "files": [...]}] . A native index is matched if its declared snapshot id equals SNAPSHOT ID (when SNAPSHOT ID != "unknown" ) or its root fingerprint matches the workspace's calculated root fingerprint. If the files array is absent or empty, treat the native index as covering no individual files directly (record in manifest.native indexes but do not update coverage rows; fall through to lower tiers for all files). Record and Defer Ingestion Rule : Parsing raw binary native indexes (e.g. SCIP protobuf) in pure Python without dependencies is costly and complex. Option A builder scripts MUST detect matching prebuilt native indexes, record their entries in manifest.native indexes , and set the coverage table backend (e.g., "scip" or "scip clangd" ) for all files listed in the provenance manifest. If catalog.sqlite is NOT populated with symbols from the native index (raw binary deferred to harness/MCP readers), set coverage.status = "deferred" and precision = "deferred" (with indexed files = 0 ). This prevents the query helper from claiming an un ingested partition is "authoritative empty" at semantic precision, ensuring consumers run the mandatory grep fallback. When catalog.sqlite IS populated (e.g., via Option B pre ingestion or scip to sqlite ), set precision = "semantic" and status = "indexed" . LSP is NOT equivalent to SCIP / LSIF. LSP is an interactive protocol whose workspace state may be partial or mutable. Use it only when the server can demonstrate snapshot identity AND complete workspace coverage. A running language server does not automatically qualify as a semantic backend. SCIP merging. SCIP explicitly permits merging complementary information from indexers with different precision levels. Its format records indexer version and per document language metadata. The builder MAY merge results from multiple indexers (e.g., a SCIP index for Go + tree sitter for Python) within a single catalog. The determinism lives in a runtime generated versioned helper ( build structural index.py , MANTIS HELPER VERSION = 5 , grep and regenerate on reuse) that probes and selects backends per partition. No shipped binaries; air gapped safe. Deterministic partial coverage. On very large source trees, a full rebuild can dominate stage 0 wall clock. Replace discovery order truncation with a deterministic priority queue: 1. Explicit target files and symbols (from plan.json , if available). 2. Changed units and known reverse dependencies (from Block E diff + reverse dependency edges in the existing catalog). 3. Containing packages/modules and direct imports of tier 1+2 units. 4. Remaining units in normalized path order . Apply deterministic max units (default: 10000) or max source bytes (default: 500MB) bounds to that queue. Persist deferred units in the manifest so another invocation can resume. Statuses: complete , partial , empty , failed . A consumer should only interpret "no callers" strongly when the relevant partition is complete, snapshot matched, covered, and semantically precise. Otherwise the answer is "no indexed callers", followed by the existing lexical fallback. The index is HINT only, so partial coverage is safe (grep remains authoritative). Step 3: Write and Run Helper Scripts Two runtime generated helpers are used. Both follow the grep and regenerate pattern: before reuse, grep the first line for the version marker; if absent or a different integer, REGENERATE. Builder: build structural index.py 1. Write the builder to workspace/helpers/build structural index.py . The FIRST LINE MUST be exactly MANTIS HELPER VERSION = 5 . Before reusi

mantis-structural-index