Name: aws-observability
Author: aws

Builds, configures, debugs, and optimizes AWS observability with CloudWatch (Log Insights, Metrics, Alarms, Dashboards, EMF), X-Ray, CloudTrail, and ADOT (AWS Distro for OpenTelemetry), AND enables/onboards services to Application Signals using ADOT auto-instrumentation SDKs. Covers Log Insights que

AWS Observability Overview Domain expertise for AWS observability across metrics, logs, and traces, for two products that share the CloudWatch name but are separate services with separate control planes, data models, and APIs: CloudWatch CloudWatch Omni What it is Log groups, metric namespaces, alarms, Log Insights, X Ray, Application Signals Application Observability / Agent Observability. A Space per account per Region is the access boundary over the account's CloudWatch Dataset (OpenTelemetry logs, traces, and metrics); the Dataset is a CloudWatch resource the Space reads, not something the Space contains Control plane aws cloudwatch , aws logs , aws xray , aws application signals aws cloudwatchomni (endpoint prefix cloudwatch omni , signing name cloudwatch ) Query Log Insights query language; GetMetricData SQL over logs.default / traces.default ; PromQL over metrics; named views Notify Alarms (metric, composite, anomaly) Alerts (SQL/PromQL rule, contributors, OK/WARNING/CRITICAL/NODATA) Topology Application Signals service map Context graph (GetContextGraph) Access IAM only Domain → Space → access grants and access profiles (set up in setting up cloudwatch observability ) Only here Dynamic Instrumentation, Synthetics canaries, CloudTrail auditing, EMF Agent quality evaluation, views, context graph First time setup Application Signals onboarding — ADOT auto instrumentation, the amazon cloudwatch observability add on, ServiceEvents, CI/CD metadata Domain → Space → grants → telemetry in → plain ADOT instrumentation References references/cloudwatch/ references/cloudwatch omni/ This skill covers using both products, never setting either one up. Both first time setup paths above live in setting up cloudwatch observability (same two folder split); route there rather than describing an onboarding procedure from memory. Enabling Omni does not replace CloudWatch; log groups, metrics, and alarms keep working, and most customers use both. Works best with the [AWS MCP server](https://docs.aws.amazon.com/aws mcp/) — enables running CLI commands, querying CloudWatch, and validating configurations directly. All guidance also works with standard AWS CLI access. Note: Reference files contain specific runtime versions, quota values, and feature matrices that may change. When precision matters (e.g. deploying to production, choosing a runtime, or checking a quota), confirm values against current AWS documentation rather than relying solely on the values in these files. Scope guard — is this an observability request? This skill owns the scope decision for every request routed to it, and this section is its sole home. If a request is NOT about AWS observability with CloudWatch or CloudWatch Omni (weather, trivia, general chit chat, non observability coding), decline it in one sentence: state plainly that it is out of scope, never fabricate an answer, and never claim a false capability limitation (no "no internet access", no "no weather data") — the reason is simply that it is out of scope. Then redirect by naming what this skill does cover: querying CloudWatch and CloudWatch Omni logs, traces, and metrics, building dashboards, configuring alerts, investigating a service, or agent evaluation. Do not attempt the off topic task, and do not call a tool or run a query in pursuit of it. Keep it brief: no lecture, no long refusal. Step 0 — CloudWatch or CloudWatch Omni? Decide this before routing. The natural wording ("set up an alert for high latency", "build a dashboard", "query my logs") does not say which product the customer means. 1. The customer names the product — "Omni", "Application Observability", "Agent Observability", a Space, Domain, Dataset, access grant, spaceId , cloudwatch omni , Omni SQL, PromQL, views, context graph, evaluators, agent evaluation / scoring traces / online or continuous evaluation / gen ai.evaluation , and OTel span vocabulary ( traces.default , logs.default , spans, durationNano , status.code , resource attributes, service.name ) → Omni . "Log Insights", "log group", "metric namespace", "CloudWatch alarm", "metric alarm", "composite alarm", "anomaly alarm", "X Ray", "Application Signals", "canary", "CloudTrail", "Dynamic Instrumentation" → CloudWatch . A "log group" named inside an agent evaluation request is the online evaluation data source to verify, not a CloudWatch signal. A bare "alarm" (or "alert") with no other product signal is ambiguous — fall through to rule 3 and probe: a Space → Omni alert ([alerts.md](references/cloudwatch omni/alerts.md)); no Space → CloudWatch alarm ([cloudwatch/alarms.md](references/cloudwatch/alarms.md)). Exception: a "PromQL alarm " is a CloudWatch alarm on OTel metrics ([cloudwatch/alarms.md](references/cloudwatch/alarms.md)) — "alarm" wins over "PromQL". 2. Knowledge or how to question ("what is an Omni alert", "does Omni have an API", "how do alerts differ from alarms") → answer from the reference files directly. Do not probe the account, and do not divert to the other product. Whether a capability exists is a fact about the product, not the account. [concepts.md](references/cloudwatch omni/concepts.md) carries the full feature equivalence matrix. 2a. Authoring / how to alert request — "create / set up / write an Omni alert that fires when X", "how should I alert on Y" — with no Space or Region supplied and no go ahead to actually create it, is a HOW TO request. Deliver the authoring guidance (the alert row of "Must state checklists" below, then [alerts.md](references/cloudwatch omni/alerts.md)) FIRST , from the reference files. Do NOT stall on a Region/Space clarifying question and do NOT fall back to a CloudWatch alarm for a request that explicitly says "Omni alert". Probe the account only once the user supplies a Space/Region or asks you to create it. 2b. A telemetry object is already in the prompt — the user pastes, or the UI passes as context, a span, trace, log record, or query result and asks what it means, what errors it has, or how long it took. This is not a live data request: do not probe the account, run a query, or ask the user to fetch it again. Read it in place per the "Reading a span or trace you already have" section of [query/sql logs traces.md](references/cloudwatch omni/query/sql logs traces.md) and state every item in its must state list. 2c. A query request whose live result is empty or unreachable ("show me the slowest spans", "which traces failed") is still answered with the methodology — the exact query and the field rules that make it correct (see the must state callouts in [query/sql logs traces.md](references/cloudwatch omni/query/sql logs traces.md)). Zero rows, a wrong Region Space, or an unreachable endpoint is reported as a finding alongside the query, never as the whole answer. 3. Request that must act on live data, and the wording is ambiguous (not a Step 0.5 content question, which is answered from the catalog) → probe the target Region first: list spaces is account global (the region flag only selects the endpoint), so filter its result to the target Region as shown rather than reading a non empty list as proof; a Space is one per account per Region. See references/cloudwatch omni/concepts.md for the full Region probe rationale. A Domain and a Space in that Region (a non empty filtered list) → Omni . No Space in that Region (an empty filtered list) → CloudWatch : alarms → [cloudwatch/alarms.md](references/cloudwatch/alarms.md), dashboards → [cloudwatch/dashboards.md](references/cloudwatch/dashboards.md), queries → [cloudwatch/log insights.md](references/cloudwatch/log insights.md), metrics → [cloudwatch/metrics.md](references/cloudwatch/metrics.md). If the customer explicitly asked for Omni and has no Space, that is a first time setup — see step 4. The probe itself errors ("not yet supported", unknown service, endpoint does not resolve) → the CLI/SDK model in use lacks cloudwatchomni . That is not evidence Omni is absent and must not be reported as "Omni is unavailable". The customer's installed AWS CLI/SDK most likely predates the service. If the request carried any Omni signal, give the customer the upgrade command to run (AWS CLI v2 reinstall or brew upgrade awscli ; pip install U boto3 botocore ) and have them re run aws cloudwatchomni list domains — exact steps in [programmatic access.md](references/cloudwatch omni/programmatic access.md). Never run a package manager upgrade or installer on the host yourself ( brew , pip install U , .pkg /MSI); it mutates the customer's machine beyond the request and can break unrelated tooling — hand over the command and continue with the guidance. Never substitute a CloudWatch or X Ray command for an Omni request. If the request carried no Omni signal, do not block on the upgrade. When you are already going back to the customer for missing inputs (for example an alert's threshold and period), ask which product they want in that same question: say the Space probe could not run on this CLI, describe both options, and build neither until they answer. When the request is fully specified or says not to ask, proceed on the CloudWatch path (the pre Omni default) and mention the upgrade only in passing. Still inconclusive → ask the customer. 4. First time setup, on EITHER product → STOP and route to the setting up cloudwatch observability skill. This skill covers a service or Space that already reports and holds no onboarding procedure for either product, so there is nothing here to fall back on: not Omni setup (creating a Domain or Space, granting access, ingestion, forwarding, Slack, instrumenting an app or AI agent for a Space) and not CloudWatch setup ( onboarding a service to Application Signals — ADOT auto instrumentation, the amazon cloudwatch observability add on, monitored service, ServiceEvents, CI/CD metadata, the per platform enablement guides). Any instrumentation / ADOT / collector request is a setup request either way; the setup skill owns the product decision too, so hand the whole request over rather than probing list spaces yourself. Application Signals once it is reporting — service map, alarms on its metrics, Dynamic Instrumentation debugging — belongs here. Under specified alert requests: When an alert or alarm request names what to watch (a symptom or a service) but not the inputs it needs — the threshold value and the evaluation period — ask for those rather than inventing them. Notifications are optional (per alerts.md), so ask for a notification destination only if the user wants to be notified. This holds on both paths (an Omni alert or a CloudWatch alarm). Under specified dashboard requests: When a dashboard request names what to show but not which metrics, panels, or layout, ground those against the data and confirm the panel set rather than inventing panels; a dashboard has no threshold, period, or notification. For ANY dashboard authoring/save request, also open [dashboards.md](references/cloudwatch omni/dashboards.md) and surface its "Facts you MUST surface when building or saving an Omni dashboard" checklist (see "Must state checklists" below). Which signals or panels a named resource type needs is a Step 0.5 catalog question, not a dashboards file question. Step 0.5 — Service health investigation (routing) A large share of real questions are phrased as an operator symptom, not as a tool: "is my <service throttled / slow / erroring / unhealthy," "which of my <service s are <symptom ," "what