Scheduled probes that run CONTINUOUSLY after release. Covers probe design for critical user journeys, alerting integration, SLA validation, multi-region monitoring, and the boundary between QA and SRE. Use when: "synthetic monitoring," "uptime testing," "scheduled probes," "SLA validation," "availab

<objective Synthetic monitoring runs scripted tests against production on a schedule, 24/7. It catches outages, performance degradation, and broken flows before real users report them — at 3 AM when traffic is zero, probes are the only thing checking your app works. A login page that always returns 200 but never authenticates passes a naive uptime check; a synthetic probe that asserts the dashboard loads catches it. This skill covers probe design, alerting integration, SLA validation, multi region execution, and the runbook discipline that keeps a 3 AM page actionable. </objective Quick Route Situation Go to Picking a platform (Checkly, Datadog, Grafana, CloudWatch…) Platform Options Writing a probe (login, API health, search) Probe Design → references/probe implementations.md Probe runs but users still report outages Failure Modes Alerts too noisy or missing real outages Alerting Integration Calculating downtime budget for an SLA SLA Validation Probe failed at 3 AM and on call is lost runbook template in references/platforms and ci.md Discovery Questions Check .agents/qa project context.md first. If it exists, use it as context and skip questions already answered there. Each answer changes the probe set, the alerting config, or the SLA math. Critical flows (decides which probes you write): What are the 5 10 most important user journeys (login, search, checkout, signup, core workflow)? These become your probe list. Which flows, if broken, cause immediate revenue loss or churn? These get the shortest frequency and page on call. Are there flows that break silently (data sync, background jobs, webhooks)? Silent failures need probes most — no user reports them. Do you have documented SLAs/SLOs for availability and response time? They set the downtime budget and the alert thresholds. Current monitoring (decides where the gaps are): What exists today (uptime checks, APM, error tracking, dashboards)? Avoid duplicating; find the gap. Are there gaps between what monitoring covers and what users experience? That gap is where probes earn their keep. How do you learn about production issues today (alerts, user reports, social media)? If it's user reports, detection time is your first metric to fix. What was the last outage, and how long before it was detected? Sets the detection time target probes must beat. Infrastructure (decides regions and CDN assertions): Is the app served from multiple regions or one? Multi region serving needs multi region probing. Are there CDN, caching, or edge layers that could mask origin failures? If yes, probes must assert on x cache /origin headers. Do third party dependencies (payment, auth, email) have their own monitoring? Their status pages feed your runbooks. What alerting systems exist (PagerDuty, OpsGenie, Slack, email)? Decides the routing config. Test accounts (decides data safety): Do dedicated synthetic test accounts exist in production? Without them, probes pollute real data and analytics. Can test accounts be excluded from analytics, billing, and email campaigns? If not, probe traffic skews every downstream number. Is there API access for programmatic account setup and data cleanup? Decides whether create then delete probes are viable. Core Principles 1. Synthetic tests validate the user experience continuously RUM tells you what happened; synthetic tells you what is happening right now, whether or not real users are active. At 3 AM when traffic is zero, synthetic probes are the only thing checking the app works. Complement RUM, never replace it: synthetic covers known paths with predictable inputs, RUM discovers the creative ways real users break things. 2. Keep probes simple and fast A probe that takes 2 minutes and touches 15 pages is not a probe — it is an E2E suite running in production. Probes are short (per probe wall clock budget under 30 seconds), focused (one critical path each), and stable ( retries: 0 , zero flakiness tolerance). The 30 second ceiling is a hard budget: a slow probe that "still passes" is masking a degradation users feel. 3. Alert on trends, not single failures A single probe failure is noise — network blips and DNS hiccups cause them constantly. Two consecutive failures are a signal; failures from 2+ regions confirm it is not local. Configure consecutive failure and multi region thresholds, or alert fatigue teaches the team to ignore the pager. 4. Probes must not affect production data Probes run every few minutes, 24/7. Even small side effects (a created record, an incremented counter) compound. Probes must be non destructive, created data cleaned up immediately, and synthetic traffic excluded from analytics, billing, and — critically — from SLO/error budget math itself, or probes inflate your own reliability numbers. 5. Assert on the goal, not the status code A login page that returns 200 but never authenticates is broken. A search page that returns 200 with zero results is broken. Probes assert that the user can accomplish their goal — data loads, auth succeeds, results appear — not merely that the page returns a 2xx. Probe Design Design probes around critical user journeys, not infrastructure components. Users do not care if your load balancer is healthy — they care if they can log in and use the product. Probe What It Validates Frequency Timeout Homepage load DNS, CDN, server, basic rendering 1 min 10s Login flow Authentication service, session management 5 min 15s Core workflow Primary value delivering action (create document, run report) 5 min 20s API health Backend services, database connectivity 1 min 5s Search Search index, query processing, result rendering 5 min 15s Checkout (if applicable) Payment integration (sandbox mode), cart, order creation 10 min 25s Third party integrations OAuth providers, email delivery, file storage 10 min 15s Probe implementations Three probe shapes cover most needs: a login flow (Playwright browser), an API health check (status + auth + latency budget), and a search probe (fill query → submit → assert on results, not just status). Keep each to one critical path, a tight timeout, and retries: 0 . See references/probe implementations.md for runnable login flow, API health, and search probes plus the environment aware config. Non destructive probe patterns Dedicated test accounts Platform Options Platform Strengths When to Use Checkly Playwright native, code first, Git integration; Rocky AI agent (GA 2026) now does automated root cause analysis across Playwright/API/Multistep/TCP/DNS/ICMP checks; CLI access from any AI agent; MCP server Teams already using Playwright for E2E Datadog Synthetic Deep APM integration, browser and API tests Teams on the Datadog platform Grafana Synthetic Monitoring Open source, integrates with Grafana dashboards; pairs with k6 2.x; pin a version channel (v1.x/v2.x) for reproducibility Teams using the Grafana stack AWS CloudWatch Synthetics Blueprints (heartbeat, API, broken link, visual diff); Python or Node Puppeteer canaries Teams already on AWS New Relic Synthetics Full stack observability integration Teams on the New Relic platform Better Stack Lightweight uptime + status pages + on call SMB friendly, fast setup Uptime Kuma OSS, self hosted, lightweight Self hosting requirement, small surface area Custom (Playwright / k6 / Puppeteer + cron) Full control, no vendor lock in Budget constrained or custom requirements Probes can be authored in Playwright (TS/JS), Puppeteer, k6 (JS — k6 2.0 shipped May 2026 with AI assisted test authoring and a clearer Assertions API; first class for synthetic), or Python (CloudWatch Synthetics, Checkly). Pick what your team already maintains. Avoid: Pingdom for new setups — it is a legacy uptime tool; prefer code first alternatives (Checkly, Grafana, custom Playwright) that version control probes alongside your app. Custom and managed implementations Self managed: schedule Playwright probes with a GitHub Actions schedule cron (every 5 minutes), inject prod credentials via secrets, and report results to a monitoring webhook. Managed: Checkly is Playwright native and runs from multiple locations on a fixed frequency. Either way, make probes environment aware so the same code runs against staging and production with different base URLs and thresholds. See references/platforms and ci.md for the GitHub Actions workflow, the Checkly config, the alert routing rules, and the runbook template. Alerting Integration Not every probe failure is an incident. Configure rules that cut noise while catching real problems. Routing. Critical failures on revenue paths (login, checkout, api health) page on call (PagerDuty + Slack incidents) with a short repeat interval; warnings on secondary probes go to a Slack monitoring channel; info level events use a long repeat interval. The probe must tag each result with a severity and probe label for these routes to match — see the routing note in references/platforms and ci.md for the full alerting rules.yaml and the tagging step. Suppress synthetic alerts during planned maintenance. A maintenance window should silence synthetic paging (the probes will fail by design) and exclude that window from error budget math, or scheduled work burns budget and pages on call for nothing. Alert message format — include enough context to start investigating immediately: The {link to runbook} points at a per probe runbook (six lines: what it tests, first checks, manual repro, escalation, dashboard, owner). See the template in references/platforms and ci.md . SLA Validation Availability calculation These are common availability targets, not prescriptive tiers. Pick targets from a user impact analysis, not by tier name. Modern practice (Google SRE Workbook, OpenSLO) favors explicit SLO + error budget policies over labelled tiers — define what user visible failure looks like, set the budget user impact tolerates, and let the SLO follow. References: https://sre.google/workbook/ ; https://openslo.com/ Response time percentiles Track percentiles, not averages — averages hide the worst experiences. Error budget tracking Error budget connects SLA targets to engineering decisions. Exclude synthetic probe downtime caused by your own maintenance windows from this calculation, or planned work shows as budget burn. Multi Region Monitoring Run probes from regions where your users are. A service that works from us east 1 but is broken from ap southeast 1 is broken for APAC users. Track regional latency separately — a global average hides regional degradation. CDN validation. Synthetic probes verify CDN caching by checking response headers ( x cache , cf cache status ) for HIT and confirming the server header matches the expected provider. This catches CDN misconfigurations — and origin failures masked by a stale cache — before users hit slow uncached responses. Anti Patterns Complex probes that break often A probe that navigates 10 pages, fills 5 forms, and asserts on 20 elements is an E2E test, not a synthetic probe. When it breaks, you cannot tell if the app is down or the probe is flaky. Fix: one critical path per probe, under 30 seconds, under 5 assertions. A failure should make it immediately clear what is broken. Alerting on every single failure Network blips, DNS hiccups, and transient cloud issues cause occasional failures. Alerting on every one produces noise that teaches the team to ignore alerts. Fix: require 2 3 consecutive failures and failures from 2+ regions before paging. Escalating severity: first failure logs, second warns, third pages. No test account isolation Probes sh

synthetic-monitoring