Name: dt-obs-kubernetes
Author: dynatrace

Kubernetes cluster, pod, node, and workload monitoring. Use when analyzing K8s health, resource optimization, pod failures, OOMKills, scheduling, or security posture. Also use for Kubernetes operational events like pod restarts, OOM events, evictions, and cluster event history. Trigger: "Kubernetes

Infrastructure Kubernetes Monitor and analyze Kubernetes infrastructure using Dynatrace DQL. Query cluster resources, monitor workload health, analyze pod placement, optimize costs, and assess security posture. When to Use This Skill Monitoring Kubernetes cluster health and capacity Analyzing pod and container resource utilization Investigating pod failures, OOMKills, evictions, or crash loops Debugging degraded deployments, stuck rollouts, or node pressure Optimizing Kubernetes resource costs Assessing security posture and compliance Troubleshooting workload scheduling and placement Auditing ingress routing and network policies Knowledge Base Structure Core Monitoring (Start Here) 1. Cluster Inventory → references/cluster inventory.md Clusters, namespaces, resource distribution 2. Node Monitoring Node capacity, CPU/memory usage, pod density 3. Pod Monitoring Pod CPU, memory, lifecycle events 4. Workload Monitoring Deployment, StatefulSet, DaemonSet resources Advanced Topics 1. Configuration Analysis → references/labels annotations.md Parse k8s.object, labels, annotations 2. Scheduling & Placement → references/pod node placement.md Node selectors, affinity, taints, HA 3. Cost Optimization Right sizing, waste detection, efficiency scoring 4. Security & Compliance Privileged containers, security contexts Key Concepts Entity Types Workloads: K8S DEPLOYMENT , K8S STATEFULSET , K8S DAEMONSET , K8S JOB , K8S CRONJOB , K8S HORIZONTALPODAUTOSCALER Infrastructure: K8S CLUSTER , K8S NAMESPACE , K8S NODE , K8S POD Configuration: K8S SERVICE , K8S CONFIGMAP , K8S SECRET , K8S PERSISTENTVOLUMECLAIM , K8S PERSISTENTVOLUME , K8S INGRESS , K8S NETWORKPOLICY Note: HPA, Job, CronJob, and configuration entity types are listed for inventory queries only. For HPA scaling analysis see references/workload health.md ; for PVC/PV see references/pv pvc.md ; for Ingress/NetworkPolicy see the corresponding reference files. Query Types smartscapeNodes Query K8s entities (current state, no time range needed): timeseries Monitor metrics over time (always specify a from: range): timeseries returns each metric as a time bucket array. To collapse it to a scalar in a downstream fieldsAdd , use arrayAvg(series) or arraySum(series) — calling avg() or sum() on a series field outside the timeseries {} block is not supported. fetch logs Analyze log events: Core Fields k8s.cluster.name , k8s.namespace.name , k8s.pod.name , k8s.node.name k8s.workload.name , k8s.workload.kind , k8s.container.name k8s.object Full JSON configuration for deep inspection tags[label] Access labels and annotations k8s.workload.kind values (always lowercase in DT — not Pascal case as in the K8s API): "deployment" , "statefulset" , "daemonset" , "replicaset" , "job" , "cronjob" k8s.object availability: Entity type k8s.object available? K8S POD Yes K8S NAMESPACE Yes K8S NODE Yes Workload types ( K8S DEPLOYMENT , etc.) Yes K8S CLUSTER No Available Metrics CPU: dt.kubernetes.container.cpu usage , cpu throttled , limits cpu , requests cpu Memory: dt.kubernetes.container.memory working set , limits memory , requests memory Operations: dt.kubernetes.container.restarts , oom kills Node: dt.kubernetes.node.pods allocatable , cpu allocatable , memory allocatable , dt.kubernetes.pods Aggregation rule for requests/limits: Always use sum() — never avg() — when aggregating requests cpu , limits cpu , requests memory , or limits memory . avg() returns a per container average and silently undercounts multi container pods and multi replica workloads. For DaemonSets (one pod per node), this error is proportional to cluster size. Entity Disambiguation K8S POD vs CONTAINER : these are different entity types in Dynatrace. K8S POD — K8s native entities with k8s.object JSON, scheduling state, conditions, and K8s metrics. Use this skill. CONTAINER — Host level container inventory (image, lifetime, host assignment). Use dt obs hosts skill instead. The smartscape edge is CONTAINER (is part of) K8S POD . To reach containers from a pod, traverse backward: Service → K8S POD Correlation No direct smartscape edge exists between SERVICE and K8S POD . The correlation key is the shared dimension k8s.workload.name . See [Service → Pod Drill Down](references/pod debugging.md service pod drill down) in references/pod debugging.md for the full two step pattern. Common Workflows 1. Cluster Health Check List all clusters: Check node capacity: Identify pods in non Running state: Succeeded is a healthy terminal phase for completed Job pods — excluding it avoids false positives. Adjust if you specifically want to audit completed Jobs. 2. Resource Optimization Pod level — find over provisioned pods (usage < 30%): Workload level — aggregate across all replicas (required for correct DaemonSet accounting): Use sum() for both cpu usage and cpu requests at every aggregation level. A DaemonSet running on 50 nodes has 50× the per pod request total; avg() would report 1/50th of the true reserved capacity. Identify containers without limits: 3. Troubleshooting Pod Issues Pod troubleshooting benefits from combining metrics (timeseries) with Kubernetes events (event stream) for a complete picture. Metrics Based Troubleshooting Find pods with OOMKills: Analyze pod restart patterns: Event Based Troubleshooting For operational events (pod restarts, OOM kills, evictions, scheduling failures), Kubernetes events provide richer context than metrics alone — including event reasons, messages, and timestamps. When to use Kubernetes events over metrics: User asks about recent operational events ("show me pod restart events") User wants event details like reasons and messages User asks about events in a specific time window ("last 48 hours") User wants to correlate events with root causes Kubernetes events are available through the get events for kubernetes cluster tool (a Dynatrace MCP tool). Call it with findAllK8Events: true to get events for all clusters, or findAllK8Events: false with either clusterId ( k8s.cluster.uid ) or kubernetesEntityId ( dt.entity.kubernetes cluster ) to scope to one cluster. Use history to set the lookback window (e.g. "1h" , "24h" , "7d" ; max "60d" ). Prefer this tool when the user asks about OOM events, pod restarts, evictions, or cluster wide event history. Fall back to the fetch events DQL pattern below if the tool is unavailable. Important: distinguish event types when filtering results. Kubernetes events cover many categories. When the user asks about a specific event type, filter the results accordingly — do not report unrelated events: User Asks About Relevant Event Reasons NOT Related Pod restarts BackOff , CrashLoopBackOff , Killing Readiness probe failures, CPU throttling OOM events OOMKilling , OOMKilled Memory pressure warnings Evictions Evicted , Preempting Node pressure Scheduling failures FailedScheduling , Unschedulable Resource quotas For a complete answer , combine both approaches: 1. Use the events tool to get the event details (what happened, when, why) 2. Use timeseries metrics to show the quantitative impact (how many restarts, OOM kill counts over time) Fetch Kubernetes Events via DQL Pod restart and operational events can also be queried via DQL from the events table. Check which providers are active first. Not all clusters ingest all event types. Confirm with: Three providers may appear — each covers different signals and uses different field names: Provider What it covers Key fields KUBERNETES EVENT Raw K8s API events forwarded to DT dt.kubernetes.event.reason , dt.kubernetes.event.message , dt.kubernetes.event.important KUBERNETES INFERRED EVENT Pod/workload lifecycle events inferred by DT (OOMKills, spec changes) event.name , event.description , k8s.workload.name , k8s.container.name KUBERNETES ANOMALY DETECTION Davis anomaly findings on K8s entities (stuck deployments, memory pressure, throttling) event.name , event.description , event.type If none appear, use the get events for kubernetes cluster tool or the metrics based queries above instead. KUBERNETES EVENT — raw K8s API events All Warning type events (those flagged type: Warning in the Kubernetes API): Filter for specific event reasons: KUBERNETES INFERRED EVENT — DT inferred pod lifecycle events OOMKilled containers — prefer this over KUBERNETES EVENT for OOM analysis because it includes workload and container context: Workload spec changes (deployment, StatefulSet, DaemonSet rollouts): KUBERNETES ANOMALY DETECTION — Davis anomaly findings Error severity anomalies (no pod ready, deployment stuck, pod evictions, backoff): All anomalies including warnings (memory pressure, CPU throttling, pending pods): Field names differ by provider. dt.kubernetes.event.reason , dt.kubernetes.event.message , and dt.kubernetes.event.important only exist on KUBERNETES EVENT records — they are null on the other two providers. Use event.name and event.description for KUBERNETES INFERRED EVENT and KUBERNETES ANOMALY DETECTION instead. 4. Security Assessment Identify privileged containers: Find containers running as root: 5. Scheduling Analysis Verify pod distribution (HA compliance) for Deployments and StatefulSets: DaemonSets are intentionally excluded — each pod runs on exactly one node by design, so single node placement is not an HA violation for them. 6. DAVIS Problems affecting K8s Entities Find active DAVIS problems affecting K8s entities: smartscape.affected entities is a record array; each record has id , type , and name . Use [][id] to get the array of Smartscape IDs to look up the affected entity, or [id] after expand smartscape.affected entities . Without a preceding expand , [id] returns null silently. A filter cannot take a bare iterative expression, so wrap it in iAny(...) . Best Practices Choosing the Right Data Source User Question Best Approach Why "Show me OOM events" Events tool + metrics Events give reasons/messages; metrics show trends "Show me pod restart events" Events tool + timeseries metrics Events reveal the reason (BackOff, Killing, CrashLoopBackOff); dt.kubernetes.container.restarts metric gives the actual restart counts "How many pod restarts?" Timeseries metrics Quantitative data over time "What happened to my pods in the last 48h?" Events tool Operational event history with context "Which pods are using the most CPU?" Timeseries metrics Resource utilization analysis "List all clusters/namespaces" smartscapeNodes Entity discovery and inventory "Are there scheduling failures?" Events tool Event reasons explain why "Which workloads are over provisioned?" Timeseries metrics, workload level Must use sum() for requests; group by k8s.workload.name Time Ranges smartscapeNodes — no time range; always returns current entity state. timeseries — always add from: now() <window . Default window is system determined and often too short for trend analysis. Recommended defaults: now() 1h for recent spikes, now() 24h for daily patterns, now() 7d for weekly trends. fetch events — adding from: is recommended for reproducible results; without it the system default window applies. fetch dt.davis.problems — use from: now() 2h for active problems; e