Distributed traces, spans, service dependencies, and request flow analysis. Use when investigating span-level details, failures, performance bottlenecks, or trace correlation. Trigger: "trace analysis", "slow requests", "failed spans", "service dependencies", "distributed trace", "span details", "HT

Application Tracing Skill Overview Distributed traces in Dynatrace consist of spans building blocks representing units of work. With Traces in Grail, every span is accessible via DQL with full text searchability on all attributes. This skill covers trace fundamentals, common analysis patterns, and span type specific queries. Use Cases 1. Investigate Slow Requests Goal: Find and diagnose requests exceeding a latency threshold Trigger: "slow requests", "high latency", "p99 response time", "find traces over 5 seconds" Done: List of slow traces with duration, endpoint, service, and trace IDs for drilldown 2. Analyze Request Failures Goal: Identify failed requests, failure reasons, and exception patterns Trigger: "failed spans", "HTTP 500 errors", "exception analysis", "failure rate by service" Done: Failure breakdown by reason (HTTP code, exception, gRPC status) with exemplar traces 3. Map Service Dependencies Goal: Understand service to service communication patterns and external API calls Trigger: "service dependencies", "what services does X call", "outgoing HTTP calls" Done: Dependency map showing call counts, latency, and error rates between services Core Concepts Understanding Traces and Spans Spans represent logical units of work in distributed traces: HTTP requests, RPC calls, database operations Messaging system interactions Internal function invocations Custom instrumentation points Span kinds : span.kind: server Incoming call to a service span.kind: client Outgoing call from a service span.kind: consumer Incoming message consumption call to a service span.kind: producer Outgoing message production call from a service span.kind: internal Internal operation within a service Root spans : A request root span ( request.is root span == true ) represents an incoming call to a service. Use this to analyze end to end request performance. Key Trace Attributes Essential attributes for trace analysis: Attribute Description trace.id Unique trace identifier span.id Unique span identifier span.parent id Parent span ID (null for root spans) request.is root span Boolean, true for request entry points request.is failed Boolean, true if request failed duration Span duration in nanoseconds span.timing.cpu Overall CPU time of the span (stable) span.timing.cpu self CPU time excluding child spans (stable) dt.smartscape.service Service Smartscape node ID dt.service.name Dynatrace service name derived from service detection rules. It is equal to the Smartscape service node name. endpoint.name Endpoint/route name Service Context Spans reference services via Smartscape node IDs and the detected service name dt.service.name which is also present on every span. Node functions : getNodeName(dt.smartscape.service) Adds dt.smartscape.service.name field with the human readable service name getNodeField(dt.smartscape.service, "attribute name") Access specific node attributes 📖 Learn more : See [Entity Lookups](references/entity lookups.md) for advanced entity selectors, infrastructure correlation, and hardware analysis. Sampling and Extrapolation One span can represent multiple real operations due to: Aggregation : Multiple operations in one span ( aggregation.count ) ATM (Adaptive Traffic Management) : Head based sampling by agent ALR (Adaptive Load Reduction) : Server side sampling Read Sampling : Query time sampling via samplingRatio parameter When to extrapolate : Always extrapolate when counting actual operations (not just spans). Use the multiplicity factor: 📖 Learn more : See [Sampling and Extrapolation](references/sampling extrapolation.md) for detailed formulas and examples. Common Query Patterns Basic Span Access Fetch spans and explore by type: Explore spans by function and type: Request Root Filtering List request root spans (incoming service calls): Service Performance Summary Analyze service performance with error rates: Trace ID Lookup Find all spans in a specific trace: Performance Analysis Response Time Percentiles Calculate percentiles by endpoint: 💡 Best practice : Use percentiles (p95, p99) over averages for performance insights. Slow Trace Detection Find requests exceeding a threshold: Duration Buckets with Exemplars Performance Timeseries Extract response time as timeseries: 📖 Learn more : See [Performance Analysis](references/performance analysis.md) for advanced patterns and timeseries techniques. Failure Investigation Failed Request Summary Summarize failures by service: Failure Reason Analysis Breakdown by failure detection reason: Failure reasons : http code HTTP response code triggered failure grpc code gRPC status code triggered failure exception Exception caused failure span status Span status indicated failure custom rule Custom failure detection rule matched HTTP Code Failures Find failures by HTTP status code: Recent Failed Requests List recent failures with details: 📖 Learn more : See [Failure Detection](references/failure detection.md) for exception analysis and custom rule investigation. Service Dependencies Service to Service Analysis Analyze service communication patterns: Outgoing HTTP Calls Identify external API dependencies: Trace Aggregation Complete Trace Analysis Aggregate all spans in a trace to understand full request flow: Root detection strategy : Use takeMin(record(...)) with a detection helper to reliably find the root request: 1. Priority 1: Spans with request.is root span == true 2. Priority 2: Spans without parent (root spans) 3. Priority 3: All other spans Multi Service Traces Find traces spanning multiple services: Request Level Analysis Request Attributes Access custom request attributes captured by OneAgent on request root spans: Field patterns : request attribute.<name , captured attribute.<name (always arrays) → [Request Attributes](references/request attributes.md) — full patterns for request attributes, captured attributes, and request ID aggregation Span Types Span Type Detection Key Fields Reference HTTP server (incoming) span.kind == "server" and isNotNull(http.request.method) http.route , http.request.method , http.response.status code [http spans.md](references/http spans.md) HTTP client (outgoing) span.kind == "client" and isNotNull(http.request.method) server.address , server.port [http spans.md](references/http spans.md) Database span.kind == "client" and isNotNull(db.system) db.system , db.namespace , db.statement [database spans.md](references/database spans.md) Messaging isNotNull(messaging.system) messaging.system , messaging.destination.name , messaging.operation.type [messaging spans.md](references/messaging spans.md) RPC / gRPC isNotNull(rpc.system) rpc.system , rpc.service , rpc.method , rpc.grpc.status code [rpc spans.md](references/rpc spans.md) Serverless / FaaS isNotNull(faas.name) and span.kind == "server" faas.name , faas.trigger.type , cloud.provider [serverless spans.md](references/serverless spans.md) ⚠️ Database spans : Can be aggregated (one span = multiple calls). Always use aggregation.count extrapolation for accurate operation counts. 📖 Detailed patterns per span type: See the reference files above. Advanced Topics Exception Analysis Exceptions are stored as span.events within spans: 💡 Tip : Use iAny() to check conditions within span event arrays. → [Logs Correlation](references/logs correlation.md) — joining logs and traces, filtering traces by log content → [Network Analysis](references/networking analysis.md) — client IPs, DNS resolution, subnet analysis Best Practices Area Rule Filtering Apply request.is root span == true and endpoint filters first Sampling Use samplingRatio (e.g., 100 = read 1%) for performance Percentiles Use p95/p99 over averages for performance analysis Root spans Use request.is root span == true for end to end analysis Trace grouping Group by trace.id for complete trace metrics Request grouping Group by request.id for OneAgent only request metrics Extrapolation Always apply multiplicity for accurate operation counts Exemplars Use takeAny(record(start time, trace.id)) to enable UI drilldown Troubleshooting Problem Cause Solution Duration values seem wrong (too large) duration is in nanoseconds, not milliseconds Divide by 1000000 or compare with 5s (DQL duration literal) Span counts don't match expected request volume Sampling or aggregation not accounted for Use multiplicity extrapolation — see Sampling and Extrapolation reference getNodeName(dt.smartscape.service) returns null Service not yet resolved or OneAgent not monitoring Verify OneAgent monitors the service; entity resolution may have a short delay request.is root span filter returns nothing Querying OpenTelemetry only traces without OneAgent Use isNull(span.parent id) as fallback for root span detection trace.id filter returns no results Trace ID not converted to UID format Use filter trace.id == toUid("abc123...") for string based trace IDs Database span counts are too low Database spans are aggregated (one span = N calls) Always use aggregation.count extrapolation for database operation counts Related Skills dt dql essentials — Core DQL syntax for querying trace data dt app dashboards — Embed trace queries in dashboards dt migration — Smartscape entity model and relationship navigation References Detailed documentation for specific topics: [Performance Analysis](references/performance analysis.md) Advanced timeseries, duration buckets, endpoint ranking [Failure Detection](references/failure detection.md) Failure reasons, exception investigation, custom rules [Sampling and Extrapolation](references/sampling extrapolation.md) Multiplicity calculation, database extrapolation [Request Attributes](references/request attributes.md) Request attributes, captured attributes, request ID aggregation [Entity Lookups](references/entity lookups.md) Advanced node lookups, infrastructure correlation, hardware analysis [HTTP Span Analysis](references/http spans.md) Status codes, payload analysis, client IPs [Database Span Analysis](references/database spans.md) Extrapolated counts, slow queries, statement analysis [Messaging Span Analysis](references/messaging spans.md) Kafka, RabbitMQ, SQS throughput and latency [RPC Span Analysis](references/rpc spans.md) gRPC, SOAP, service dependencies [Serverless Span Analysis](references/serverless spans.md) Lambda, Azure Functions, cold start analysis [Logs Correlation](references/logs correlation.md) Joining logs and traces, correlation patterns [Network Analysis](references/networking analysis.md) IP addresses, DNS resolution, communication mapping

dt-obs-tracing