test-environments

Design environment strategy for testing across dev, CI, preview, staging, and production — Docker Compose test infrastructure, multi-stage Dockerfiles, seed-data lifecycle, per-PR preview environments, production parity, and external-dependency stubbing at the HTTP boundary. Use when: "set up test e

By petrkindlmann · 620 installs

npx skills add petrkindlmann/qa-skills --skill test-environments

Source repository · Upstream listing

<objective Staging on SQLite passes tests that break on prod Postgres; a shared staging box becomes a queue where one broken deploy blocks the whole team; an unmocked Stripe call flakes CI at random. This skill prevents those by designing environment tiers that mirror production where it matters, isolate per PR, and stub external dependencies at the HTTP boundary. It delivers a working docker compose up local/CI stack, a parity checklist, and a stubbing strategy keyed to dependency type. </objective Discovery Questions Check .agents/qa project context.md first — if it exists, use it and skip anything already answered there. Then: 1. How many environments exist today? Local dev, CI, staging, preview, production? Map what you have before designing what you need. 2. Is the app containerized? Check for Dockerfile , docker compose.yml , or compose.yaml . If yes, multi stage targets and compose come for free; if not, that is the first deliverable. 3. How is test data seeded? Manual SQL, migration based, factory libraries, or production snapshots? This decides whether seed scripts are a quick win or a rewrite. 4. How close is staging to production? Same DB engine, queue, cache, auth provider, orchestration? Each mismatch is a class of bugs staging can never catch. 5. External dependencies: How many third party APIs does the system call, and are they stubbed in non prod? Unstubbed third parties are the top source of CI flake. Core Principles 1. Staging must mirror production where bugs hide. If staging uses SQLite and production uses PostgreSQL, staging tests prove nothing about prod behavior. Match the database engine and version , the queue system, the cache layer, and the auth provider — those are where environment specific bugs live. 2. Ephemeral environments beat long lived ones. A shared staging environment becomes a bottleneck where one broken deploy blocks the entire team. Per PR preview environments give isolation and parallel testing; keep staging only for final pre release validation. 3. Deterministic seed data, not production copies. Production snapshots carry PII, stale references, and non reproducible state. Build seed data from factories that generate consistent, valid, minimal datasets. (For factory patterns, see test data management .) 4. Stub external dependencies at the boundary, not deep inside. Third party APIs are unreliable, rate limited, and expensive. Stub them at the HTTP boundary with MSW or WireMock — never by mocking internal service classes, which hides integration bugs between your own code. 5. Environment config is code. Every environment difference (URLs, flags, credentials, resource limits) must be version controlled and reviewable. No manual setup that cannot be reproduced from the repo. Environment Strategy Environment Tiers Environment Purpose Data External Deps Lifecycle Local dev Fast inner loop Seeded fixtures, minimal Stubbed (MSW/WireMock) Developer managed CI Automated validation Seeded per run, ephemeral Stubbed or containerized Created/destroyed per pipeline Preview PR level review & E2E Seeded from factories Stubbed or sandbox Created on PR, destroyed on close Staging Pre production validation Anonymized production like Real integrations (sandbox accounts) Long lived, regularly reset Production Live users Real Real Permanent Local Development Fast feedback, zero shared state. Developers must be able to run the full stack locally in under two minutes: Use Docker Compose for infrastructure deps (database, cache, queue) but run the application natively for fast reload. External APIs are stubbed with MSW handlers loaded in dev mode. CI Environment Fully containerized, created fresh per pipeline run, destroyed after. The block below is the services: fragment of a job — nest it under jobs.<id .services alongside runs on and steps ; on its own it is not a valid workflow file. Docker Compose vs Testcontainers Two ways to give tests real infrastructure. Pick by where the lifecycle should live: Docker Compose — declarative stack you bring up before the suite ( docker compose up wait ) and tear down after, usually via a trap guarded script. Best for local dev, a shared CI stack, and E2E where many tests share one set of services. Testcontainers (Node / JVM / Python / Go) — containers spun up from test code and auto torn down per suite or per test, with no compose file or trap to maintain. Best for integration tests that need isolated, programmatic infra (a throwaway Postgres per test class). The 2026 default for "ephemeral infra owned by the test," and a strong alternative to hand rolled compose + trap scripts. Reach for Compose when humans and many tests share the stack; reach for Testcontainers when each test (or suite) wants its own disposable copy. Preview Environments (Per PR) Each pull request gets its own isolated environment; reviewers click a link and test the exact changes without interfering with other PRs. Hosting options (2026), pick by stack: Vercel preview deployments — Next.js / static / serverless; per PR URL automatically. Cloudflare Pages preview — git integrated, generous free tier. Render / Railway preview environments — full stack including databases. Northflank , Qovery , Bunnyshell , Uffizzi — full ephemeral environment platforms (Kubernetes backed) when previews need the whole stack, not just a frontend. For each preview, pair the env lifecycle with a database branch (Neon, Supabase, PlanetScale style): create a branch on PR open, drop it on close. That gives every preview a cheap, instant, isolated DB copy instead of a shared staging DB. (See test data management .) For local dev parity with CI: Devcontainers ( .devcontainer/devcontainer.json ) — VS Code, Codespaces, JetBrains. The standard for "everyone gets the same Docker backed dev env." Tilt ( Tiltfile ) — Kubernetes first local dev with hot reload and multi service orchestration. Pick when staging itself is K8s. A frontend preview with E2E against the generated URL is a few lines: A custom Docker preview keyed to a per PR namespace, auto torn down on close: Staging Long lived environment that mirrors production infrastructure. Reset weekly or on demand to prevent drift: Caveat: DROP SCHEMA public CASCADE also drops the schema's default privileges and any installed extensions ( uuid ossp , pgcrypto , …). Your migration pipeline must recreate them ( CREATE EXTENSION IF NOT EXISTS … , re grant defaults) or the migrate step fails. Don't assume a bare CREATE SCHEMA public restores the prior grants — it does not. Docker Compose for Testing A production quality docker compose.test.yml spins up the full stack (app, Postgres, Redis, a one shot seed container, Mailpit) for integration and E2E tests. Two details that matter: Health checks gate depends on . Without a healthcheck + condition: service healthy , depends on only waits for the container to start , not for the service to accept connections — tests then race the database and fail with connection errors. Seed is a one shot container, not a long running service. It uses depends on: condition: service completed successfully , so the app starts only after seeding exits 0 . Teams that model seed as a long running service get a race where the app boots mid seed. See references/docker compose.md for the full docker compose.test.yml , the trap guarded integration test runner, the multi stage Dockerfile (with the production target), and the MinIO block. Multi Stage Dockerfile One base layer installs deps once; development , test , and seed stages reuse it; and a slim production stage runs prod deps only ( npm ci omit=dev ) with build artifacts copied from the test stage. The split keeps test dependencies and source out of the shipped image while giving each environment its own entrypoint. Use npm ci include=dev in base — the modern flag; production=false is legacy omit / include syntax. Full Dockerfile in references/docker compose.md . External Dependency Management Stubbing Strategy by Dependency Type Dependency Type Local/CI Strategy Staging Strategy Payment (Stripe) MSW handler returning mock responses Stripe test mode with sk test keys Email (SendGrid) Mailpit capturing SMTP (web UI on :8025, SMTP on :1025) SendGrid sandbox mode Auth (Auth0) Local JWT issuer with test keys Auth0 dev tenant Storage (S3) MinIO container (S3 compatible) Dedicated test bucket with lifecycle policy Search (Elasticsearch) Testcontainers Elasticsearch Dedicated test index with reset script SMS (Twilio) MSW handler Twilio test credentials Avoid: MailHog — unmaintained, last release 2020. Use Mailpit ( axllent/mailpit ); it is a drop in on the same ports (1025 SMTP / 8025 UI). MSW for HTTP Stubs Stub external APIs at the HTTP boundary with MSW 2.x: http + HttpResponse from msw , setupServer from msw/node , lifecycle wired through beforeAll / afterEach / afterAll . Set onUnhandledRequest: "error" so an unmocked external call fails the test loudly instead of leaking a real network request. See references/stubbing.md for the Stripe/SendGrid/geocoding handlers and the server lifecycle. MinIO as an S3 Substitute Run S3 compatible storage in a container instead of hitting real AWS in local/CI tests. Point the AWS SDK S3Client at it with endpoint , env var credentials, and forcePathStyle: true (required for MinIO). Compose service + client config in references/docker compose.md . Contract Testing as Stub Validation Stubs drift from reality. Pair every stub with a contract test that verifies the stub matches the real API shape. For details, see contract testing . Environment Parity Checklist Run this when setting up or auditing a non production environment. Dimension Question Red Flag Database engine Same engine and version as production? SQLite in test, PostgreSQL in prod Database schema Same migration pipeline applied? Manual schema changes in staging Data shape Seed data covers all entity states? Only "happy path" records, no edge cases Infrastructure Same container orchestration? Docker Compose in CI, Kubernetes in prod Network Same internal service topology? Monolith in test, microservices in prod Config Env vars documented and version controlled? Undocumented env vars, manual setup Auth Same auth provider/flow? Bypassed auth in test with hardcoded tokens Feature flags Same flag evaluation engine? Hardcoded flags in test, LaunchDarkly in prod TLS/HTTPS Same certificate handling? HTTP in staging, HTTPS in prod Timeouts/Limits Same rate limits, pools, timeouts? Infinite timeouts in test hide perf issues For factory based seed data patterns, see test data management . Anti Patterns Shared staging as the only test environment. One developer's broken deploy blocks everyone. Use ephemeral per PR environments for isolation and keep staging for final pre release validation only. Production database copies for test data. PII risk, non reproducible state, massive datasets that slow tests. Build minimal seed data from factories with deterministic values. Environment specific code paths. if (process.env.NODE ENV === "test") { skipAuth(); } means you are not testing the real auth flow. Swap implementations via dependency injection or config, not environment conditionals. Manual environment setup. If setup needs a 15 step wiki page, it will be wrong within a week. Script everything: docker compose up d && npm run db:seed should be the only steps. Stubbing internal services instead of ex