Safe-release techniques DURING rollout: feature flags, progressive rollouts, canary analysis, guardrail metrics, production smoke tests, and synthetic users. Bridges QA and SRE practices. Use when: "feature flag testing," "canary deploy," "progressive rollout," "guardrail metrics," "dark launch," "s

<objective Production is the only environment that is production. Every other environment is an approximation — staging never replicates real data volume, traffic, or third party quirks. This skill covers how to validate quality in production safely: controlled blast radius, automated rollback, guardrail metrics, and smoke tests that catch problems before users do. The recurring failure it prevents: shipping to 100% of users with no flag to flip, no baseline to compare against, and no tested way back. </objective Quick Route Situation Go to Shipping a feature behind a flag Feature Flag Testing Ramping traffic 1% → 100% with gates Progressive Rollout + references/rollout policy.md Need post deploy checks on every release Production Smoke Tests + references/patterns.md Deciding what numbers gate the rollout Guardrail Metrics New code path with no user visible change yet Dark Launches Proving the rollback actually works Verification Discovery Questions Check .agents/qa project context.md first. If it exists, use it as context and skip questions already answered there. Feature flag system: Do you have a feature flag platform? (LaunchDarkly, Statsig — now part of OpenAI, GrowthBook, Unleash, Flagsmith, Harness FME — formerly Split, custom, none) How are flags managed? (Dashboard, config file, environment variables) Can flags target specific users, percentages, or segments? How many active flags exist today? Is there a cleanup process? Rollout capability: Can you deploy to a subset of traffic? (Canary infrastructure, weighted routing, feature flags) How long does a deployment take? How long does a rollback take? Do you have blue green or rolling deployments? Can you route traffic by region, user cohort, or percentage? Monitoring maturity: What observability is in place? (APM, logging, error tracking, metrics) Do you have dashboards for error rate, latency, and business metrics? Are alerts configured with appropriate thresholds? Can you compare metrics between canary and baseline in real time? Production access and safety: Who has production access? Is there an approval process? Are there dedicated test accounts in production? Can you run operations in production without affecting real user data? Is there a production incident response process? Core Principles 1. Production is the final test environment Staging approximates production. It does not replicate production's data volume, traffic patterns, third party integrations, infrastructure quirks, or user behavior. Testing in production is not reckless — it is realistic. The question is not whether to test in production, but how to do it safely. 2. Safety through blast radius control Every production test must answer: "If this goes wrong, how many users are affected?" The answer must be as small as possible. Feature flags, canary deploys, and traffic splitting exist to shrink the blast radius from 100% to 1% or less. 3. Always have a tested rollback plan Before any production test begins, the rollback mechanism must be identified, tested, and fast. "Disable the flag" is a good rollback plan. "Redeploy the previous version" is acceptable. "We'll figure it out" is not a plan. A rollback you have never fired is a hypothesis, not a plan — see [Verification]( verification) for how to prove it works. 4. Monitoring is a prerequisite, not a nice to have You cannot test in production without monitoring. If you cannot measure error rates, latency, and business metrics in real time, you cannot detect problems. Fix monitoring gaps before adding production tests. 5. Production tests must be non destructive Production tests must never corrupt real user data, send real notifications to real users, charge real payment methods, or create side effects that require manual cleanup. Synthetic accounts, test flags, and isolated resources are mandatory. Feature Flag Testing Feature flags are the safest mechanism for production testing. They decouple deployment from release and provide instant rollback. Test with flags ON and OFF Every flagged feature needs tests in both states. The flag off path is the rollback path and must work flawlessly — use setFeatureFlag(name, true false, { userId: TEST USER ID }) to drive both. See references/patterns.md for the full ON/OFF test pair. Flag lifecycle testing Flags are not just on or off. They transition through states, and each transition must be validated. Stale flag cleanup Flags left in code become technical debt. Run a weekly CI job that queries the flag provider for flags that are 100% rolled out and older than 14 days . These are candidates for code cleanup — remove the flag branching logic and retain only the enabled path. Removing the flag without removing the dead code is half a cleanup. Flag combination testing When multiple flags interact, test the combinations that matter. Do not test all 2^N combinations — focus on flags that affect the same user flow (e.g. new checkout , express pay , discount engine v2 all touch checkout). Pick the critical combinations: all new, a representative mixed state, and all legacy. See references/patterns.md for the combination test loop. Progressive Rollout Vendor native canary analysis. Before hand rolling the rollout policy YAML below, check whether your platform already does it: LaunchDarkly Guarded Rollouts (auto monitored progressive rollouts with metric based auto rollback; uses a frequentist sequential testing analysis model since early 2026), Statsig Auto tune , Argo Rollouts AnalysisRun , Flagger , Harness Continuous Verification . If you have one, prefer it — the integration with your metrics and rollback mechanics is cheaper than maintaining a custom analysis loop. AI feature rollout is its own pattern: model variant + prompt as a flag value, with cost guardrails and a kill switch. LaunchDarkly AI Configs / AgentControl (AI Configs rebranded under the AgentControl umbrella, announced May 2026) is the documented path for shipping LLM features behind progressive rollout. See release readiness for the full pattern. Canary stages: 1% to 100% A structured rollout with explicit promotion criteria at each stage. Stage Traffic Hold Time Key Checks Canary 1% 15 30 min Error rate, crash rate, exceptions Early adopters 10% 1 2 hours Latency P95, conversion rate Partial 50% 2 4 hours All guardrails, business metrics Full 100% 24 hours monitoring Long tail issues, batch job compatibility Automated promotion and rollback Define machine checkable conditions for advancing between stages — hold duration plus metric conditions ( error rate 5xx < 0.5% , latency p95 < 500ms , crash rate == 0 ). Automatic rollback fires when guardrails are breached, with no human approval needed : error rate 5xx 2x baseline for 5m , latency p99 3x baseline for 5m , crash rate 0.1% for 2m , each notifying on call. Gate on error budget burn rate , not only raw multipliers, so slow burns that still blow the SLO are caught. See references/rollout policy.md for the full promotion YAML, rollback triggers, and SLO gate config. Verify the rollback fired correctly Auto rollback firing is not the same as the incident being resolved. After a rollback triggers, do not declare "recovered" until you have confirmed the system is actually back: 1. Re run the health check smoke test against production ( GET /api/health returns healthy and the previous version ). 2. Confirm the flag/deploy state reverted — query the flag platform that the flag is off (or the deploy that the previous build is serving), don't assume. 3. Confirm guardrail metrics returned to baseline — error rate, P99 latency, and crash rate back within their pre deploy windows. 4. Confirm the rollback notification reached on call (Slack/PagerDuty) so the incident is owned. Only when all four hold is the rollback verified. See [Verification]( verification) for the staging dry run that proves this chain before first production use. Production Smoke Tests Post deploy critical path tests Run immediately after every deployment as a pipeline stage (not only in pre deploy CI). These verify core functionality works with production configuration, data, and infrastructure: a /api/health check, the authentication flow with synthetic credentials, core data loading, and search. Configure retries: 1 and a timeout so a flaky post deploy run doesn't block the pipeline on the first blip. See references/patterns.md for the full production smoke.spec.ts . Synthetic user accounts Production test accounts must be clearly distinguishable from real users: a reserved email pattern ( smoke test+{env}@yourcompany.com ), an is synthetic = true flag, and exclusion from analytics, billing, and email campaigns. Create them via admin API, and prefer short lived OIDC / workload identity tokens over long lived passwords. See references/patterns.md for the full conventions. Non destructive assertions Production smoke tests must read, not write. When writes are unavoidable, clean up in fixture teardown so cleanup runs whether the test passes or fails. Do not call test.afterEach() inside a test() body — Playwright registers hooks at describe/file scope, so a hook registered mid test never schedules teardown and throws test.afterEach() can only be called in a describe block . The data leaks. Use an auto cleanup fixture that records created resource IDs and deletes them on teardown (or try/finally for a one off script). See references/patterns.md for the fixture based create verify cleanup pattern. Guardrail Metrics What to monitor during rollout Category Metric Comparison Method Alert Threshold Errors HTTP 5xx rate vs. pre deploy baseline 2x baseline for 5 min Errors Unhandled exception count vs. pre deploy baseline Any new exception type Latency P50 response time vs. pre deploy baseline 1.5x baseline Latency P95 response time vs. pre deploy baseline 2x baseline Latency P99 response time vs. pre deploy baseline 3x baseline Business Conversion rate vs. 7 day average Drop 5% Business Revenue per session vs. 7 day average Drop 10% Client Crash rate (mobile) vs. previous release 0.1% increase Client JavaScript error rate vs. pre deploy baseline 2x baseline Infra CPU utilization absolute 80% sustained Infra Memory utilization absolute 85% sustained Baseline comparison Compare canary metrics against a control group running the previous version, not against historical data alone. Statistical significance For business metrics (conversion, revenue), small sample sizes produce noisy results. Wait for statistical significance before drawing conclusions. Dark Launches Dark launches deploy new functionality to production but hide it from users. Real production traffic exercises the new code path without user visible impact. Traffic shadowing Duplicate incoming requests to the new service. Compare responses without returning the new response to the user. Parallel execution For migrations (new database, new algorithm, new service), run both the old and new path in production. The old path returns the result to the user; the new path runs asynchronously, logs differences, and discards its result. Track the match rate over time — target 99%+ match before cutting over. Anti Patterns Testing in production without monitoring Running production tests without dashboards and alerts is flying blind. You will not know if your tests caused an issue until a user reports it. Fix: Monitoring is a prerequisite. Before adding any production test, verify you can see error rates, latency

testing-in-production