quality-playbook

Run a complete quality engineering audit on any codebase. Derives behavioral requirements from the code, generates spec-traced functional tests, runs a three-pass code review with regression tests, executes a multi-model spec audit (Council of Three), and produces a consolidated bug report with TDD-

By github · 1,584 installs

npx skills add github/awesome-copilot --skill quality-playbook

Source repository · Upstream listing

Quality Playbook Generator Plan Overview — read this first, then explain it to the user Before reading any other section of this skill, understand the plan and its dependencies. Each phase produces artifacts that the next phase depends on. Skipping or rushing a phase means every downstream phase works from incomplete information. Phase 0 (Prior Run Analysis): If previous quality runs exist, load their findings as seed data. This is automatic and only applies to re runs. Phase 1 (Explore): Run the v1.5.3 documentation intake first ( python m bin.reference docs ingest <target to walk reference docs/ — cite/ files produce quality/formal docs manifest.json records; top level files are loaded as Tier 4 context via reference docs ingest.load tier4 context(<target ) ). Then explore the codebase in three stages: open exploration driven by domain knowledge, domain knowledge risk analysis, and selected structured exploration patterns. Write all findings to quality/EXPLORATION.md . This file is the foundation — Phase 2 reads it as its primary input. Phase 2 (Generate): Read EXPLORATION.md and produce the quality artifacts: requirements, constitution, functional tests, code review protocol, integration tests, spec audit protocol, TDD protocol. ( AGENTS.md at the target's repo root is generated by the orchestrator AFTER Phase 6, not by you in Phase 2 — see "File 6" below for the contract.) Phase 3 (Code Review): Run the three pass code review against HEAD. Write regression tests for every confirmed bug. Generate patches. Phase 4 (Spec Audit): Three independent AI auditors review the code against requirements. Triage with verification probes. After triage, the same Council runs the v1.5.3 Layer 2 semantic citation check — one prompt per reviewer, structured per REQ verdicts for every Tier 1/2 citation, output to quality/citation semantic check.json . Write regression tests for net new findings. Phase 5 (Reconciliation): Close the loop — every bug from code review and spec audit is tracked, regression tested or explicitly exempted. Run TDD red green cycle. Finalize the completeness report. Phase 6 (Verify): Run self check benchmarks against all generated artifacts. Check for internal consistency, version stamp correctness, and convergence. Phase 7 (Present, Explore, Improve): Present results to the user with a scannable summary table, offer drill down on any artifact, and provide a menu of improvement paths (iteration strategies, requirement refinement, integration test tuning). This is the interactive phase where the user takes ownership of the quality system. Every bug found traces back to a requirement, and every requirement traces back to an exploration finding. The critical dependency chain: Exploration findings → EXPLORATION.md → Requirements → Code review + Spec audit → Bug discovery. A shallow exploration produces abstract requirements. Abstract requirements miss bugs. The exploration phase is where bugs are won or lost. MANDATORY FIRST ACTION: After reading and understanding the plan above, print the following message to the user, then explain the plan in your own words — what you'll do, what each phase produces, and why the exploration phase matters most. Emphasize that exploration starts with open ended domain driven investigation, followed by domain knowledge risk analysis that reasons about what goes wrong in systems like this, then supplemented by selected structured patterns. Do not copy the plan verbatim; paraphrase it to demonstrate understanding. Quality Playbook v1.5.6 — by Andrew Stellman https://github.com/andrewstellman/quality playbook Generate a complete quality system tailored to a specific codebase. Unlike test stub generators that work mechanically from source code, this skill explores the project first — understanding its domain, architecture, specifications, and failure history — then produces a quality playbook grounded in what it finds. How to run this — v1.5.4 self encoded invocation contract If the operator hands you this skill (or points you at any QPB installed target) and says "Run the Quality Playbook" — possibly with a hint like "this is a bootstrap run" or "run on itself" or "self audit" — this section tells you exactly what to do. The operator should not need to provide additional instructions; the canonical invocation, the defaults, the guardrails, and the output contract all live here. Pick your execution mode QPB ships in two execution shapes. Pick the one that matches your runtime — the wrong choice produces the codex on codex indirection pathology surfaced by the 2026 04 30 bootstrap test. Mode When this is you What you do A. Skill direct (UI context) You are a coding agent (Claude Code, Cursor, Copilot, Codex desktop, etc.) handed this skill in your own chat. Your runtime IS the reasoning loop — you read files, you write files, you decide. Walk through Phase 1 → Phase 6 yourself using the externalized phase prompts in phase prompts/ . Write artifacts into the target's quality/ directory directly. No subprocess, no runner. B. Runner driven (CLI automation) The operator is invoking python3 m bin.run playbook deliberately — to batch across multiple targets, drive a headless CI run, or fan out per phase work to a different model than the one reading this prose. The orchestrator spawns a CLI agent ( claude , copilot , codex , or cursor ) per phase. You (or whoever is reading this) are the operator side control loop, not the per phase reasoner. Both modes use the same phase prompt content — the phase prompts/ .md files at the repo root are the single source of truth, loaded by bin/run playbook.py:: load phase prompt and read directly by Mode A walkthroughs. The only thing the two modes differ on is WHO drives — you (Mode A) or the orchestrator subprocess spawning a CLI agent (Mode B). When in doubt, default to Mode A. If the operator wanted runner driven invocation they would have run the runner themselves; if they pasted "Run the Quality Playbook" into your chat, they want you to drive. The Mode B section below tells you what to do if the operator explicitly invokes the runner. Mode A — skill direct walkthrough (UI context) The operator's prompt is just "Run the Quality Playbook" (or "run on itself", "self audit", etc.). You drive every phase inline. For each phase 1..6, in order: 1. Load the phase prompt. Read phase prompts/phaseN.md (resolve via the same install location fallback list documented for references/ below). For phase1.md , substitute {seed instruction} (the prelude that says "skip Phase 0/0b" — empty string when seeds are allowed) and {role taxonomy} (the taxonomy block rendered from the role taxonomy below). For phase2.md through phase6.md , the file is pure literal — read it verbatim. 2. Execute the phase per the prompt. Read the inputs the prompt names, do the analysis, write outputs into the target's quality/ directory. 3. STOP at the end of phase boundary. Every phase prompt ends with an "IMPORTANT: Do NOT proceed to Phase N+1" instruction. Honor it. The operator advances to the next phase by saying so. You are responsible — without the orchestrator's structural backstop — for the same source unchanged invariant the runner enforces: do NOT modify any file outside the target's quality/ directory . In Mode B the gate would catch this; in Mode A you are the gate. The 2026 04 30 bootstrap test specifically failed on a Phase 2 LLM modifying the target's root AGENTS.md — the same failure mode applies in Mode A. For the bootstrap run (self audit) variant of Mode A, see "Bootstrap mode" below — the only delta is that the target IS the QPB repo, so cite the same phase prompts/ files you read from. Mode A scope — what's covered, what's Mode B only Council 2026 04 30 P1 3: the per phase walkthrough above scopes Mode A to phases 1..6 . The following surfaces are deliberately Mode B only — if the operator wants them, point them at the runner instead of trying to drive them yourself: Phase 0 / Phase 0b (seed injection from prior runs). The orchestrator handles seed discovery, prior run scanning, and seed prompt injection. In Mode A, treat every run as no seeds (skip Phase 0/0b entirely, start at Phase 1). If the operator explicitly asks for seed driven exploration, hand off to Mode B ( python3 m bin.run playbook with seeds <target ). Phase 7 (interactive Present / Explore / Improve). This phase is a back and forth dialogue with the operator about the generated artifacts; it has no pre baked prompt in phase prompts/ . After Phase 6 in Mode A, present the artifact summary table inline (see "What this run produces" below for the file list) and let the operator drive what to explore next conversationally — that IS Phase 7. There is no orchestrator subprocess to spawn. Iteration strategies (gap / unfiltered / parity / adversarial). Iterations re enter the playbook with a strategy specific addendum. In Mode A, after Phase 6 completes cleanly, hand off to Mode B for iterations: python3 m bin.run playbook next iteration strategy <name <target . The iteration prompts ( phase prompts/iteration.md ) ARE single source of truth, but the iteration orchestration loop (rotating through gap → unfiltered → parity → adversarial) is the runner's job. A Mode A operator who wants iterations after Phase 6 should be told: "Phase 6 is done; run python3 m bin.run playbook full run <target to get all four iteration strategies, or pick one strategy explicitly with next iteration strategy gap ." If the operator asks for one of these surfaces in Mode A and the request is ambiguous (e.g., "also do the iterations"), surface the mode handoff explicitly rather than improvising — improvisation is how the prompt content drifts away from the runner's canonical loop. Mode B — runner driven invocation (CLI automation) The operator runs python3 m bin.run playbook themselves (typically because they want batching, headless CI, or to route per phase work to a different model). The orchestrator at bin/run playbook.py spawns a CLI agent per phase, feeds it the externalized phase prompt, and aggregates the result. Canonical invocation The orchestrator is the entry point. Always invoke it as a Python module: Never invoke it script style ( python bin/run playbook.py ... ). The runtime guard exits with EX USAGE=64 because relative imports require packaged execution. <target is the path to the project to audit. For a bootstrap run (target IS the QPB repo), pass . from the repo root. For any other target, pass the path to that target's repo root. Default behavior (no flags) Bare invocation triggers a full run : all 6 phases (Explore → Generate → Code Review → Spec Audit → Reconciliation → Verify) followed by all 4 iteration strategies (gap → unfiltered → parity → adversarial), executed synchronously in the same session. Any prior quality/ directory is auto archived to quality/previous runs/<TIMESTAMP / before the new run starts. This is the canonical operator path. Don't ask permission to add flags; the defaults are the answer. When the bare invocation fires, the orchestrator emits a one line stderr banner naming the cost change vs. v1.5.3 (~5–10× the legacy "Phase 1 only" default). That banner is informational; let it scroll. Common overrides Use only when the operator asks for something specific: Need Flag Effect Run a single phase phase N (where N ∈ 1..6) Recovers the v1.5.3 "explore only" pattern with phase 1 . Skip iteration strategies omit iterations and pass phase 1,2,3,4,5,6 Phases run; iterations don't. Specific iteration strategy <name next iteration Iterates on an existing quality/ run with a chosen strategy. Multi target pass several positional targets Each runs independently. Per phase CLI age