This skill should be used when the user wants to "set up tracing", "monitor my agent", "configure logging", "add observability", "debug production traffic", or needs guidance on monitoring deployed agents, including ADK (Agent Development Kit) agents. Covers Cloud Trace, prompt-response logging, Big

Observability Guide Cloud Trace works out of the box — no infrastructure needed. Prompt response logging and BigQuery Agent Analytics require Terraform provisioned infrastructure (service account, GCS bucket, BigQuery dataset). Run agents cli infra single project project PROJECT ID to provision these resources. See references/cloud trace and logging.md for details, env vars, and verification commands. If your project isn't scaffolded yet, see /google agents cli scaffold first. Order of operations for agent runtime deployments For deployment target = agent runtime , run agents cli infra single project before the first agents cli deploy . The Terraform module owns the entire Reasoning Engine resource (service account, deployment spec, env vars), so applying it after an SDK based deploy creates a state mismatch Terraform can't reconcile without taking ownership of the whole resource. Already ran agents cli deploy ? Two options: 1. Switch to Terraform managed — delete the SDK deployed Reasoning Engine, then run agents cli infra single project and agents cli deploy (sessions and in flight state are lost). 2. Keep the SDK deployed instance — skip infra single project and set the observability env vars by re running agents cli deploy update env vars "KEY=VALUE,..." ; deploy matches the existing Reasoning Engine by display name and updates it in place, preserving env vars set outside the deploy. You must also grant its service account the telemetry IAM roles the Terraform module would otherwise provision: roles/storage.admin (write completions to the logs bucket), roles/logging.logWriter , roles/cloudtrace.agent , plus roles/bigquery.dataOwner + roles/bigquery.jobUser when scaffolded with bq analytics . The full set lives in deployment/terraform/single project/iam.tf (from app sa roles ) and telemetry.tf . Terraform managed env vars aren't available in this mode. Reference Files File Contents references/cloud trace and logging.md Scaffolded project details — Terraform provisioned resources, environment variables, verification commands, enabling/disabling locally references/bigquery agent analytics.md BQ Agent Analytics plugin — enabling, key features, GCS offloading, tool provenance references/adk docs.md ADK: adk.dev pages to fetch for detail beyond this skill references/feedback mechanism.md Adding a user feedback endpoint — request model, structured logging, log sink → BigQuery Observability Tiers Choose the right level of observability based on your needs: Tier What It Does Scope Default State Best For Cloud Trace Distributed tracing — execution flow, latency, errors via OpenTelemetry spans All templates, all environments Always enabled Debugging latency, understanding agent execution flow Prompt Response Logging GenAI interactions exported to GCS, BigQuery, and Cloud Logging Scaffolded projects Disabled locally, enabled when deployed Auditing LLM interactions, compliance BigQuery Agent Analytics Structured agent events (LLM calls, tool use, outcomes) to BigQuery ADK agents with the plugin enabled Opt in ( bq analytics at scaffold time) Conversational analytics, custom dashboards, LLM as judge evals Third Party Integrations External observability platforms (AgentOps, Phoenix, MLflow, etc.) Any OpenTelemetry instrumented agent Opt in, per provider setup Team collaboration, specialized visualization, prompt management Ask the user which tier(s) they need — they can be combined. Cloud Trace is always on; the others are additive. Cloud Trace Scaffolded agents use OpenTelemetry to emit distributed traces. Every agent invocation produces spans that track the full execution flow. Span Hierarchy ADK projects. These are ADK's span names; other frameworks emit their own ( generate content comes from the shared google genai instrumentor either way). Setup by Deployment Type Deployment Setup Agent Runtime Automatic — exporters wired at startup, gated on GOOGLE CLOUD AGENT ENGINE ENABLE TELEMETRY (set by deploy); exports to Cloud Trace/Logging + Agent Engine console Cloud Run / GKE (scaffolded) Automatic — exporters wired at startup, exports to Cloud Trace/Logging Cloud Run / GKE (manual) Configure OpenTelemetry exporter in your app Local dev Works with agents cli playground ; traces visible in Cloud Console ADK: the wiring is get fast api app(otel to cloud=True) in app/fast api app.py . Other templates call their own setup at startup (e.g. app/app utils/telemetry.py ). View traces: Cloud Console → Trace → Trace explorer ADK: for detailed setup instructions (Agent Runtime CLI/SDK, Cloud Run, custom deployments), fetch https://adk.dev/integrations/cloud trace/index.md . Prompt Response Logging Captures GenAI interactions and exports to GCS (JSONL) and BigQuery (via log sinks + external tables). Content is governed by two independent tiers ; the net Terraform deploy default is full content in GCS/BigQuery, none in traces : Tier Captures Controlled by Default (Terraform deploy) GCS/BigQuery completions Full prompts/responses (the prompt response logging feature) OTEL INSTRUMENTATION GENAI COMPLETION HOOK=upload + LOGS BUCKET NAME On — full content Trace spans / Cloud Logging events Span/event content OTEL INSTRUMENTATION GENAI CAPTURE MESSAGE CONTENT (plus ADK CAPTURE MESSAGE CONTENT IN SPANS=false , ADK only ) Off — NO CONTENT The tiers are independent: GCS/BigQuery uploads capture full content whenever their upload vars are set and do not honor OTEL INSTRUMENTATION GENAI CAPTURE MESSAGE CONTENT , which governs the traces/events tier only. Its valid (experimental semconv) values: NO CONTENT — no content in spans/events (scaffolded default) EVENT ONLY — content in Cloud Logging events SPAN ONLY / SPAN AND EVENT — content in trace spans true / false — invalid ; fall back to NO CONTENT For the full mechanics (semconv opt in, declarative Terraform config, env var table, enabling/disabling, verification commands), see references/cloud trace and logging.md . For ADK logging docs (log levels, configuration, debugging), fetch https://adk.dev/observability/logging/index.md . BigQuery Agent Analytics Plugin ADK projects. Optional ADK plugin that logs structured agent events to BigQuery. Enable with bq analytics at scaffold time. See references/bigquery agent analytics.md for details. Third Party Integrations Many third party observability platforms can ingest agent telemetry (via OpenTelemetry or custom instrumentation). The table below covers common ones; the full list is larger (see the pointer below it). Platform Key Differentiator Setup Complexity Self Hosted Option AgentOps Session replays, 2 line setup, replaces native telemetry Minimal No (SaaS) Arize AX Commercial platform, production monitoring, evaluation dashboards Low No (SaaS) Phoenix Open source, custom evaluators, experiment testing Low Yes MLflow OTel traces to MLflow Tracking Server, span tree visualization Medium (needs SQL backend) Yes Monocle 1 call setup, VS Code Gantt chart visualizer Minimal Yes (local files) Weave W&B platform, team collaboration, timeline views Low No (SaaS) Freeplay Prompt management + evals + observability in one platform Low No (SaaS) Ask the user which platform they prefer — present the trade offs and let them choose. ADK: fetch a platform's setup page at https://adk.dev/integrations/<slug /index.md (slugs for the table above: agentops , arize ax , phoenix , mlflow tracing , monocle , weave , freeplay ); ADK has more observability integrations (Datadog, Galileo, LangWatch, Latitude, Future AGI, Respan, Zespan, …) — browse the complete, current list at https://adk.dev/integrations/ (observability topic). On other frameworks the OpenTelemetry based platforms still work, but follow the platform's own setup docs. Troubleshooting Issue Solution No traces in Cloud Trace Verify telemetry setup runs at startup ( ADK: fast api app.py uses get fast api app(otel to cloud=True) ; Agent Runtime gates it on GOOGLE CLOUD AGENT ENGINE ENABLE TELEMETRY ) and the SA has the cloudtrace.agent role Prompt response data not appearing Check LOGS BUCKET NAME is set; verify SA has storage.objectCreator on the bucket; check app logs for telemetry setup warnings Content in traces/events (unwanted) OTEL INSTRUMENTATION GENAI CAPTURE MESSAGE CONTENT=NO CONTENT keeps content out of spans/events. NOTE: GCS/BigQuery completions still capture full content — to stop that, remove LOGS BUCKET NAME / OTEL INSTRUMENTATION GENAI COMPLETION HOOK (drop the upload block in service.tf ) BigQuery Analytics not logging ADK: verify the plugin is configured in app/agent.py ; check BQ ANALYTICS DATASET ID env var is set Third party integration not capturing spans Check provider specific env vars (API keys, endpoints); some providers (AgentOps) replace native telemetry Traces missing tool spans ADK: tool execution spans appear under execute tool (other frameworks use their own span names) — check trace explorer filters High telemetry costs Switch to NO CONTENT mode; reduce BigQuery retention; disable unused tiers Related Skills /google agents cli deploy — Deployment targets, CI/CD pipelines, and production workflows /google agents cli workflow — Development workflow, coding guidelines, and operational rules /google agents cli adk code — ADK Python API quick reference for writing agent code

google-agents-cli-observability