After writing code, detect and clean up duplication at three levels — copy-paste blocks, cross-package shared code, unnecessary wrappers, and concept-level SoT violations. Detect with indexion, fix, and verify.

indexion refactor — Codebase Refactoring Detect and eliminate duplication at three levels — textual, structural, and conceptual — using indexion's analysis commands, then verify SoT is enforced. When to Use After adding a new abstraction (type, module, API layer) After introducing a new file format or I/O boundary When a fix required touching 3+ files for the same reason When a "guard" or "skip" was added to work around a structural problem When opendir , ENOENT , or similar filesystem errors appear from unexpected paths When extracting shared code across packages When cleaning up after a refactor (removing trivial wrapper functions) Periodic SoT health check on a codebase Three Levels of Duplication Level What it is Tool Example Textual Copy pasted code blocks, identical functions plan refactor is whitespace copied across 5 modules Structural Same logic structure with different names plan solid , plan unwrap cross package extraction candidates, trivial wrappers Conceptual Same domain concept implemented independently explore + manual analysis Three modules each deciding "is this file an archive?" Textual duplication is easy to find and fix. Conceptual duplication is the hardest and most dangerous — it produces no copy paste matches but means changing one concept requires updating every scattered implementation. Workflow Phase 1: Clear textual duplication ( plan refactor ) Start with high confidence matches and work down. Read the output in three sections: Section What it finds Action Similar Files Files with high overall similarity Investigate for structural consolidation Duplicate Code Blocks Line level identical code between files Extract to @common or shared module Function Level Duplicates Structurally similar functions (TF IDF on bodies) Unify into single SoT function Same file duplicates (functions within one file at 90%+) are the highest value targets — easiest to fix, clearest wins. Example: get global data dir and get global cache dir share 95% structure, extracted into resolve os dir . plan refactor options: Option Default Description threshold=FLOAT 0.7 Minimum similarity threshold strategy=NAME hybrid Similarity: hybrid, tfidf, bm25, jsd, ncd fdr=FLOAT 0 FDR correction (0=disabled) style=STYLE raw Output: raw, structured format=FORMAT md Output: md, json, text, github issue name=NAME Project name (for structured style) include=PATTERN Include pattern (repeatable) exclude=PATTERN Exclude pattern (repeatable) o, output=FILE stdout Output file path specs dir=DIR kgfs KGF specs directory What remains after cleanup (stop signals): Platform stubs ( native.mbt / stub.mbt ) — intentional platform branching Type method similarity ( to string on different types) — different types, same pattern CLI command boilerplate ( command() functions) — @argparse API pattern, not duplication Semantic but different functions ( is disqualifying keyword vs is skip token ) — different purpose Phase 2: Extract cross package shared code ( plan solid ) After cleaning within each directory, find code that should be shared across packages. plan solid differs from plan refactor : plan refactor plan solid Scope Internal duplication within directories Cross directory overlap Goal Consolidate within a codebase Extract shared code into a new package Input <path from=dirA,dirB plan solid options: Option Default Description from=DIRS (required) Source directories (comma separated or repeatable) to=DIR Target directory for extraction rules=FILE Rules file (.solidrc) rule=RULE Inline rule (repeatable) threshold=FLOAT 0.9 Minimum similarity threshold strategy=NAME tfidf Similarity: tfidf, apted, tsed include=PATTERN Include pattern (repeatable) exclude=PATTERN Exclude pattern (repeatable) format=FORMAT md Output: md, json, github issue o, output=FILE stdout Output file path specs dir=DIR kgfs KGF specs directory Workflow: 1. Run plan refactor on each directory individually first to clean internal duplication 2. Run plan solid from=dirA,dirB to find cross directory extraction candidates 3. Extract shared code following the plan's recommendations 4. Use indexion grep "TypeIdent:SharedType" to verify all references are updated Phase 3: Remove unnecessary wrappers ( plan unwrap ) After consolidation, clean up trivial delegation functions that add indirection without value. What gets detected: Functions whose body is a single function call with all arguments forwarded as simple identifiers — no control flow, no transforms. plan unwrap modes: Mode Flag Description Report (default) List wrappers found Preview dry run Show all edits without modifying files Fix fix Apply edits to files plan unwrap options: Option Default Description dry run Preview edits fix Apply edits all Include self delegation and bare constructor wrappers include self Include self.field.method patterns include bare Include bare constructor wrappers include=PATTERN Include pattern (repeatable) exclude=PATTERN Exclude pattern (repeatable) format=FORMAT md Output: md, json, text o, output=FILE stdout Output file path specs dir=DIR kgfs KGF specs directory Review before removing: Platform wrappers (FFI, @osenv path ) are abstraction layers, not accidental indirection Public API wrappers used by external packages — removing them is a breaking change Always dry run first Phase 4: Detect concept level duplication ( explore + analysis) This is the hardest level. Textual and structural tools won't find it because the code is different — but the concept is the same. Files at 40 60% similarity without structural duplication are concept neighbors — they use the same terms because they deal with the same domain. For each high similarity pair, ask: "What concept do they share, and who owns it?" Common patterns of concept leakage: Symptom Concept leaked Fix Both files call is X(spec) then Y::from spec(spec) "Determine if X and configure Y" Extract try do X(path, spec) into the module that owns X Both files @fs.read file to string(path) when content is already loaded "Read file content" Pass content as argument, don't re read Both files parent dir(path) then @fs.read dir(dir) "List sibling files" Centralize directory walking into pipeline Multiple if is virtual path(x) { skip } guards "Real vs virtual path" Make the type system prevent virtual paths from reaching here Both files buf.write string("\n"); buf.write string(x) "Join text entries" Extract join text entries() into the owning module Phase 5: Consolidate into SoT The module that defines the concept should be the only one that implements the logic . Rules: 1. One concept, one module, one function. If "extract text from archive" appears in vfs.mbt , discover.mbt , and args.mbt , it belongs in vfs.mbt only. 2. Callers receive results, not ingredients. Don't export is archive spec + ArchiveSpec::from spec + expand archive separately. Export try extract archive text(path, spec) String? . 3. Guards are symptoms, not fixes. if is virtual path(x) { skip } means virtual paths shouldn't reach here at all. Fix the source, not the sink. 4. Re reading from disk what's already in memory is a concept leak. If SupportedFile.content holds the text, no downstream code should call @fs.read file to string(file.path) . Phase 6: Verify After SoT consolidation: Textual similarity between the concept owner and its callers drops Callers become shorter (one API call instead of multi step logic) The concept owner may grow, but it's the single place to change Phase 7: Prove non recurrence with tests Write a test that structurally prevents the old pattern from recurring: The test doesn't check behavior — it checks the SoT invariant . Red Flags "I need to add a guard here" If you're adding if is special case(x) { skip } to a function that shouldn't receive special cases, the problem is upstream . The function's caller should never pass that value. "It works but prints errors to stderr" Stderr messages from C runtime ( opendir: No such file or directory ) mean invalid data reached a system call. catch absorbs the error, but perror() already printed. The only fix is preventing invalid data from reaching the call. "I'll fix it in each command separately" If the same fix is needed in explore, search, grep, reconcile, plan documentation... the fix belongs in the shared pipeline, not in each command. "The similarity is just shared vocabulary, not real duplication" 40 60% TF IDF similarity between modules that aren't supposed to share concepts is a warning. The vocabulary match IS the signal. Quick Reference: Which Command When Question Command "What files are similar?" explore format=list "What exactly is duplicated?" plan refactor threshold=0.9 "What code overlaps between packages A and B?" plan solid from=A,B "Which functions are trivial wrappers?" plan unwrap or grep semantic=proxy "What concept do these files share?" explore file a file b threshold=0 strategy=apted "Has the duplication been fixed?" Re run plan refactor with same threshold

indexion-refactor