Audit a whole regression suite and prune/restructure it with evidence: per-test coverage fingerprinting, AST near-duplicate clustering, CI-history mining for never-failing and flaky tests, prune decision rules (redundant/obsolete/low-value/keep), smoke/core/extended tiering by risk and defect-detect

<objective Test A and Test B cover the exact same lines, so a tired engineer deletes B — and three weeks later a production defect slips through because B was the only test that asserted the rounding was correct. Coverage equality is not redundancy. This skill audits an entire regression suite as a corpus (the redundancy analysis no human does by hand), prunes it on evidence rather than vibes, and treats every deletion as a destructive change that requires a quarantine grace period, human sign off, and a record you can defend to an auditor. </objective Quick Route You want to... Go to Find which lines/branches each test uniquely covers [Coverage Fingerprinting]( 1 coverage fingerprinting per test) Decide if "same coverage" means "delete one" [The Coverage Equality Trap]( 2 the coverage equality trap the load bearing rule) Surface copy pasted near duplicate tests [Near Duplicate Clustering]( 3 near duplicate clustering) Find never failing and always flaky tests [Mining CI History]( 4 mining ci history) Decide redundant vs obsolete vs low value vs keep [Prune Decision Rules]( 5 prune decision rules) Split a flat suite into smoke/core/extended [Tiering]( 6 tiering smokecoreextended) Safely delete the tests you flagged [Destructive Safety]( 7 destructive safety the grace period) Produce the "what we deleted and why" record [The Audit Record]( 8 the audit record) Discovery Questions First, check .agents/qa project context.md in the project root and skip anything it already answers. Then clarify: Which language/runner? pytest+coverage.py, Jest/Vitest+Istanbul, Go, JUnit — the per test context mechanism differs per stack (and so does the mutation tool). Is there CI test result history, and how far back? No JUnit/Datadog/Trunk history means you cannot mine never failing or flaky signals — you only have coverage and clustering. What is the suite size and current wall clock time? This sizes the tiering target (e.g. smoke under 5 min) and whether per test coverage is feasible in one run or must be sharded. What is the business risk map / critical paths? Tiering and the "keep" disposition both depend on it. If absent, run risk based testing first. Who signs off on deletions, and is there a CODEOWNERS file? Deletion is destructive; you need a named approver before this skill removes anything. What is the acceptable observation window? How long the team will run quarantined tests as skipped before permanent removal (default: 2 sprints / 2 releases). Core Principles 1. Coverage equality is not redundancy. Two tests hitting the same lines can assert completely different things — different oracles, inputs, edge cases. Line/branch coverage tells you what code ran , never what was checked . The only evidence that one test subsumes another is that the survivor catches the same faults, which you prove with mutation testing, not a coverage diff. 2. The agent's edge is whole corpus analysis, not deletion authority. An agent can fingerprint 4,000 tests, cluster near duplicates, and cross reference CI history in minutes — work no human does by hand. That is the entire value. But the agent proposes ; a human approves . Never let the corpus scale analysis become corpus scale auto deletion. 3. Deletion is destructive and must be reversible in practice. "It's in git history" is not a recovery plan. Quarantine first (skip/xfail, or move to a deprecated suite), observe for a defined window, watch for escaped defects, then delete with sign off. The grace period is the safety net, not the commit log. 4. Every disposition is differentiated. Redundant, obsolete, and low value are three different states with three different actions. Collapsing them into one "delete" bucket is how you lose real coverage. A test that never failed is not the same as a test that cannot fail. 5. Evidence over intuition, recorded per test. Each removal carries its own row: category, the test that supersedes it, the coverage delta, who approved, and how to restore. If you cannot fill the row, you cannot delete the test. 1. Coverage Fingerprinting (per test) The wrong answer is a single combined cov report that tells you per file percentages. That cannot tell you which test covered which line, so it cannot tell you which tests overlap. You need per test (dynamic) contexts . pytest / coverage.py — record which test hit each line with dynamic contexts, and turn on branch coverage: cov context=test makes coverage.py call switch context() around each test, tagging every measured line with the test that ran it. Add cov branch so a test that takes the if and one that takes the else are not treated as covering "the same line." The result is written into the .coverage SQLite database, in the context and line bits / arc tables. Then read the contexts table out of the .coverage SQLite DB to build a per test fingerprint: for each test, the exact set of (file, line) and (file, branch arc) pairs it covered. From those sets you compute: Uniquely covered lines — lines/branches that only one test covers. Lose that test and you lose that coverage outright. These tests are pulling their weight; protect them. Subsumption — test A's covered set is a superset of test B's. A candidate for redundancy (but see §2 — it is not proof). See references/coverage fingerprinting.md for the SQL to pull contexts from .coverage , the Python that builds per test line/branch sets and computes uniquely covered and subsumption relations, and the JS/Vitest+Istanbul coverage equivalent ( coverage final.json with per test reporters). Do not rank tests by line count per test file, and do not delete tests merely for having a low overall coverage percentage — a one line test can be the only thing guarding a critical branch. 2. The Coverage Equality Trap (the load bearing rule) This is the single most important rule in the skill. When per test data shows Test A and Test B cover exactly the same lines , the naive conclusion is "redundant, delete one." That is wrong, and here is the gotcha: Coverage equality does not prove the tests have the same assertions, the same inputs, or the same oracle. Two tests can cover identical lines while one asserts the HTTP status and the other asserts the response body, or while they pass different edge case inputs. One covers same lines but asserts different values; the other covers same lines but checks different state. Coverage measures execution, not verification. To find out whether B is actually redundant — whether A truly subsumes B's fault detection — run mutation testing : mutmut (3.6.0+) or cosmic ray for Python, StrykerJS (9.x) for JS/TS, PIT for Java/JVM, cargo mutants for Rust. Mutation testing injects faults (mutants) into the covered code. A test "kills" a mutant if it fails on the mutated code. If A kills every mutant that B kills, A genuinely subsumes B's fault detection and B is a defensible delete candidate. If B kills a mutant A misses, B catches a defect class A does not — keep B , even though coverage was identical. Decision: identical coverage → flag as a candidate → confirm with mutation testing → only then propose deletion. When assertions differ and mutation results differ, retain both, do not delete . See references/mutation confirmation.md for the mutmut/Stryker config that scopes mutation runs to the suspect lines and the kill set comparison. 3. Near Duplicate Clustering Goal: surface copy pasted tests without flagging every test in the same file. Grouping tests by filename is not clustering — it tells you nothing about similarity. Two defensible signals, combined: 1. AST (abstract syntax tree) similarity. Parse each test into an AST, normalize away identifier names and literals, then compare structure. Use a token/tree similarity metric (Jaccard over normalized token shingles, cosine over AST n grams, or tree edit distance). AST based comparison ignores formatting and variable name noise that defeats exact string matching or raw diff . Never use raw line numbers as a similarity signal. 2. Coverage profile signature. From §1, each test already has a covered line/branch set — its execution profile . Tests with near identical coverage signatures and near identical ASTs are strong near duplicate candidates; either signal alone is weak. Cluster with a tunable similarity threshold (e.g. agglomerative/hierarchical clustering, cut at a configurable cutoff — start ~0.85, tune to your false positive tolerance). Output clusters, never deletions. Every cluster is routed to human review. The agent does not delete a whole cluster automatically — copy paste tests frequently diverge in one assertion that matters. Present each cluster with its members, the pairwise similarity, and the coverage profile overlap, and let a human confirm which (if any) collapse. See references/clustering.md for the AST normalization, the shingle/Jaccard and tree edit similarity functions, and the agglomerative clustering with the tunable threshold. 4. Mining CI History Parse your test result history — JUnit XML archives, or a platform that already stores it: Datadog Test Optimization , Trunk Flaky Tests , BuildPulse , CircleCI test insights . For each test compute pass rate / fail rate and the flip rate (how often consecutive runs transition pass↔fail). Two findings, two very different meanings: Never failing tests (zero failures / 100% pass over the window). The naive move is to delete any test that has never once failed. Wrong — never failing does not mean delete or useless . A test most often never fails because it guards low churn, low risk, stable code — exactly the code nobody touches, so the test never trips. That is low defect detection signal in this window, not zero value. Disposition: investigate, do not delete — check churn and risk of the code under test. If it covers a critical path that simply has not regressed, it stays. Always flaky tests. Flakiness is not decided by a single run. The real definition is different results on the same SHA / same commit — the same code produced a pass and a fail. Detect that by grouping runs by commit SHA and finding tests with both outcomes on one SHA (Trunk and BuildPulse do this natively). The naive move is "delete flaky tests to clean up CI." Wrong: a flaky test may still be your only coverage of a real path. Disposition: quarantine and fix the flake, never delete to clean up CI . Quarantine de noises CI immediately; the root cause still gets fixed. See test reliability for runtime quarantine and self healing of an individual flaky test. See references/ci history mining.md for the JUnit XML aggregation script, the same SHA flake query, and the Datadog/Trunk API pulls. 5. Prune Decision Rules Stop deleting everything that "looks redundant." The three failure categories are distinct states, and each gets a different disposition : Category Definition (the test is...) Disposition Redundant subsumed by another test — covers the same lines AND the survivor kills the same mutants (§2) merge or delete — but only after the mutation kill check confirms subsumption ; quarantine first Obsolete testing a feature that was removed / dead code / a path that no longer exists delete — the code it tested is gone; verify the target truly no longer exists, then remove Low value never failed AND trivial (a getter, a no op, no meaningful assertion) quarantine / route to review — low value is not zero value; confirm before removal Keep covers something uniquely, catches defects (positive defect detection history), or guards a high risk path keep — protected regardless of coverage overlap The discipline: a different action per category. redundant = merge/dele

test-suite-curation