ci-cd-integration — an agent skill by petrkindlmann

Design CI/CD pipelines that run test suites. Covers GitHub Actions and GitLab CI templates, parallelism and sharding, artifact management, flaky-test quarantine, test-result publishing, coverage quality gates, OIDC keyless deploy, and copy-paste workflows for Playwright, Jest, and multi-stage pipeli

<objective A 20 minute serial suite on every push destroys developer velocity; a green pipeline that retries flaky tests three times hides the race condition until it ships. This skill produces CI/CD pipelines that run the right tests at the right trigger, shard them across runners, store traces and reports as evidence, quarantine flaky tests instead of masking them, and gate merges on real coverage numbers. Use this skill when the question is about running tests in a pipeline, not writing them. </objective Discovery Questions Check .agents/qa project context.md first — if it exists, use it and skip anything already answered there (especially team maturity and existing CI conventions). Then: 1. Which CI platform? GitHub Actions, GitLab CI, CircleCI, Jenkins? This skill ships templates for GitHub Actions and GitLab CI. 2. What test types need to run? Unit, integration, E2E, visual, performance? Each has different resource and timing needs. 3. What is the current CI duration? Over 10 minutes means parallelism and sharding are mandatory, not optional. 4. How many developers push per day? High frequency teams need aggressive concurrency cancellation and caching. 5. What triggers should run which tests? Not every push needs a full E2E suite — map triggers to suites before writing YAML. Calibrate to team maturity Set team maturity in .agents/qa project context.md ; pick the matching pipeline shape: startup — one job: lint + unit + one E2E smoke on PR. Fast feedback over completeness. growing — separate jobs for unit, integration, E2E. Parallelization, artifact uploads, result publishing, flaky quarantine. established — full matrix: sharded E2E, multi environment promotion gates, perf and security scans, deploy gated checks, SLA backed pipelines. Core Principles 1. Fast feedback: right tests at the right time. Unit tests on every push (under 2 min). E2E on PRs (under 10 min). Full suite on merge and nightly. The trigger to suite map below is the contract. 2. Parallel first: shard tests across workers. A 20 minute serial suite becomes 5 minutes across 4 shards. Always worth the runner cost. 3. Artifacts are evidence. Every run stores traces, screenshots, coverage, and HTML reports. Without artifacts, a CI failure is an undebuggable "reproduce locally" cycle. 4. Flaky tests need quarantine, not retries. Retrying hides the problem — the test passes on retry, the report is green, the race condition persists. Move flaky tests to a non blocking job, track them, fix the root cause. 5. Quality gates get stricter toward production. Define what must pass at each stage; PR gate is fast and cheap, deploy gate is comprehensive. 6. Read thresholds from config, not from bash. Let the test runner enforce coverage via its own coverageThreshold / thresholds and exit non zero. Scraping percentages out of stdout with regex is fragile across runner versions. Pipeline Architecture What runs when Trigger Tests Max duration Push to branch lint, type check, unit 2 min PR opened/updated + integration, E2E smoke 10 min Merge to main + full E2E, visual, perf budget 15 min Nightly schedule full suite, security, a11y, flaky quarantine 30 min Release tag full suite, smoke against staging 20 min GitHub Actions For complete copy paste workflow files (unit, sharded Playwright E2E, full pipeline, nightly, PR gate), see references/github actions templates.md . Action versions (June 2026) Pin to the current major and let Dependabot bump them. The actions/ family runs on the Node 24 runner; Node 20 is deprecated on GH hosted runners. Action Current major Notes actions/checkout @v6 actions/setup node @v6 v5+ auto caches only when packageManager is set; use cache: npm to be explicit actions/cache @v5 new cache service v2 backend actions/upload artifact @v7 v7 can upload unzipped ( archive: false ) actions/download artifact @v7 pair with upload artifact major dorny/test reporter @v3 v3 requires Node 24 runner; reporter keys unchanged dorny/paths filter @v3 marocchino/sticky pull request comment @v3 slackapi/slack github action @v2 floating major; see notification note before adopting v3 For supply chain sensitive pipelines, pin third party actions (dorny, marocchino, slackapi, knapsack) to a full length commit SHA with a version comment, and let Dependabot update the SHA: uses: dorny/test reporter@<40 char sha v3.0.0 . First party actions/ are lower risk; tags are acceptable there. Key concepts Concurrency groups cancel wasted runs when a branch gets multiple pushes: Matrix sharding across runners: Caching browsers so they aren't re downloaded every run: Artifacts for reports and traces, and merging sharded reports into one HTML report — see references/github actions templates.md (E2E workflow). The merge job uses actions/download artifact@v7 with pattern: test results then npx playwright merge reports reporter=html . Smarter sharding at scale Past 10–15 shards, naïve hash based splitting wastes runner time on uneven shards. Use a timing aware balancer: knapsack pro — timing data based, supports Playwright/Jest/Cypress/RSpec; distributes by historical duration. CloudBees Smart Tests (formerly Launchable ) — ML prioritization + Test Impact Analysis; runs only the tests likely to fail for the diff. Datadog Test Optimization — TIA + flake management; shard balancing by historical time. Trunk Flaky Tests — flake aware quarantine + retry budgeting. Before reaching for a paid balancer: Playwright's shard already distributes by file and balances on duration from prior runs. To inspect or feed custom timing data, dump it yourself — npx playwright test reporter=json jq '[.suites[].specs[] {file: .file, duration: .tests[].results[].duration}]' . For Jest, jest slow test reporter surfaces the slowest specs so you can split or fix them. For self hosted runners on Kubernetes, use Actions Runner Controller ( arc runner set / gha runner scale set ) — Helm installed, auto scales runner pods per workflow. Replaces the deprecated runner deployment CRD. Required status checks Protect main in Settings → Branches → Branch protection rules: enable "Require status checks to pass before merging," add lint , unit tests , and e2e (all shards) as required checks, and enable "Require branches to be up to date." GitLab CI For the full pipeline, see references/gitlab ci template.md . Key points: Stages [validate, test, e2e, deploy] ; node:22 alpine for lint/unit, mcr.microsoft.com/playwright:v1.60.0 noble for E2E (keep this pinned to your installed @playwright/test minor). Parallel sharding: parallel: 4 exposes CI NODE INDEX / CI NODE TOTAL ; run npx playwright test shard=$CI NODE INDEX/$CI NODE TOTAL . Coverage: emit a cobertura coverage report artifact and a junit report; GitLab reads the percentage and test results from those. The legacy coverage: stdout regex is a fragile fallback across Jest versions — prefer the cobertura report. Advanced Patterns Test result publishing to PR comments For the sticky coverage PR comment ( marocchino/sticky pull request comment@v3 ), see references/github actions templates.md (PR Quality Gate). Conditional test execution Only test what changed. Use dorny/paths filter@v3 to set outputs, then gate steps on them — see references/github actions templates.md (Conditional execution). Flaky test quarantine Separate flaky tests into a non blocking job so they run in CI but don't block merges: Tag flaky tests at the source so the grep splits them: If a quarantined test passes 10 consecutive runs, remove the @flaky tag. For runtime self healing of a single flaky test (selector recovery, auto retry policy), use test reliability . Cache strategies Layer Path Cache key Node modules (handled by setup node cache: npm ) automatic Playwright browsers ~/.cache/ms playwright pw {os} {hash(package lock.json)} Build cache (Next.js) .next/cache nextjs {os} {hash(lockfile)} {hash(src)} Test fixtures e2e/fixtures/.cache test data {hash(seed.sql)} Use actions/cache@v5 for layers 2–4; add restore keys on build caches for partial matches. OIDC keyless deploy Don't store a long lived DEPLOY TOKEN . Use GitHub Actions OIDC to assume a cloud role for short lived credentials — nothing static to leak or rotate: The IAM role's trust policy pins the sub claim to your repo and branch. GCP ( google github actions/auth ) and Azure ( azure/login ) have equivalent OIDC flows. Slack/Teams notification on failure Use slackapi/slack github action@v2 with webhook type: incoming webhook , gated on if: failure() && github.ref == 'refs/heads/main' so only main branch failures notify. Before moving to @v3 , note v3 changed payload handling for workflow trigger webhooks (no longer flattened/stringified) — verify your payload against the v3 docs first. Full example in references/github actions templates.md (Nightly Full Suite). Quality Gates Gate When Required checks Blocking? PR Gate PR opened/updated lint, type check, unit, coverage threshold Yes Merge Gate Before merge to main + E2E smoke suite Yes Deploy Gate Before production deploy + full E2E, visual, perf budget Yes Nightly Gate Scheduled 2am daily full suite, npm audit, axe a11y Alert only PR Gate (under 3 minutes) Enforce the coverage floor in the test runner's config, not in bash. In jest.config.js (or vitest.config.ts coverage.thresholds ): Then jest coverage exits non zero when coverage drops, so the job fails with no extra script. If you must read the number in CI (e.g. to print it), have Jest emit json summary and read the file — there is no coverage summary CLI: ( json summary reporter writes coverage/coverage summary.json . For nyc/c8 projects, nyc report reporter=text summary . The standalone istanbul CLI is deprecated — don't use istanbul report .) Merge Gate (under 10 minutes) PR Gate + E2E smoke. Configure as required status checks in branch protection. Deploy Gate (under 15 minutes) Needs [unit tests, e2e tests, visual tests] , then a perf budget check ( npx lhci autorun / lhci assert config=lighthouserc.json ) before the OIDC deploy step above. Nightly Gate (up to 30 minutes) Full E2E across all browsers, security scan, a11y audit, flaky quarantine. Wire the security and a11y steps as real jobs, not just prose: Where the @a11y tagged specs use @axe core/playwright : Results go to Slack, not as blocking checks. Anti Patterns 1. Running all tests on every commit A 20 minute full suite on every push destroys velocity. Use the trigger to suite map: fast tests on push, comprehensive on PR and merge. 2. No artifact storage Without traces, screenshots, and logs, every CI failure becomes a "reproduce locally" cycle that wastes hours. Upload artifacts on if: ${{ !cancelled() }} . 3. Retrying flaky tests without tracking them retries: 3 hides flakiness — the report is green but the race condition persists. Quarantine, track, fix the root cause. 4. CI only failures without local reproduction If a test only fails in CI, document why (timezone, missing env var, screen resolution) and add a script that replicates CI locally with the same Playwright image you run in CI — don't pin a stale image. See references/github actions templates.md (Local repro). 5. Shared state between CI jobs Jobs that read files from sibling jobs without artifacts or needs . Each job starts fresh; pass data via upload artifact / download artifact . 6. No concurrency co