Kubernetes cluster, pod, node, and workload monitoring. Use when analyzing K8s health, resource optimization, pod failures, OOMKills, scheduling, or security posture. Also use for Kubernetes operational events like pod restarts, OOM events, evictions, and cluster event history. Trigger: "Kubernetes

Infrastructure Kubernetes Monitor and analyze Kubernetes infrastructure using Dynatrace DQL. Query cluster resources, monitor workload health, analyze pod placement, optimize costs, and assess security posture. When to Use This Skill Monitoring Kubernetes cluster health and capacity Analyzing pod and container resource utilization Investigating pod failures, OOMKills, evictions, or crash loops Debugging degraded deployments, stuck rollouts, or node pressure Optimizing Kubernetes resource costs Assessing security posture and compliance Troubleshooting workload scheduling and placement Auditing ingress routing and network policies Knowledge Base Structure Core Monitoring (Start Here) 1. Cluster Inventory → references/cluster inventory.md Clusters, namespaces, resource distribution 2. Node Monitoring Node capacity, CPU/memory usage, pod density 3. Pod Monitoring Pod CPU, memory, lifecycle events 4. Workload Monitoring Deployment, StatefulSet, DaemonSet resources Advanced Topics 1. Configuration Analysis → references/labels annotations.md Parse k8s.object, labels, annotations 2. Scheduling & Placement → references/pod node placement.md Node selectors, affinity, taints, HA 3. Cost Optimization Right sizing, waste detection, efficiency scoring 4. Security & Compliance Privileged containers, security contexts Key Concepts Entity Types Workloads: K8S DEPLOYMENT , K8S STATEFULSET , K8S DAEMONSET , K8S JOB , K8S CRONJOB , K8S HORIZONTALPODAUTOSCALER Infrastructure: K8S CLUSTER , K8S NAMESPACE , K8S NODE , K8S POD Configuration: K8S SERVICE , K8S CONFIGMAP , K8S SECRET , K8S PERSISTENTVOLUMECLAIM , K8S PERSISTENTVOLUME , K8S INGRESS , K8S NETWORKPOLICY Query Types smartscapeNodes Query K8s entities: timeseries Monitor metrics over time: fetch logs Analyze log events: Core Fields k8s.cluster.name , k8s.namespace.name , k8s.pod.name , k8s.node.name k8s.workload.name , k8s.workload.kind , k8s.container.name k8s.object Full JSON configuration for deep inspection tags[label] Access labels and annotations Available Metrics CPU: dt.kubernetes.container.cpu usage , cpu throttled , limits cpu , requests cpu Memory: dt.kubernetes.container.memory working set , limits memory , requests memory Operations: dt.kubernetes.container.restarts , oom kills Node: dt.kubernetes.node.pods allocatable , cpu allocatable , memory allocatable , dt.kubernetes.pods Entity Disambiguation K8S POD vs CONTAINER : these are different entity types in Dynatrace. K8S POD — K8s native entities with k8s.object JSON, scheduling state, conditions, and K8s metrics. Use this skill. CONTAINER — Host level container inventory (image, lifetime, host assignment). Use dt obs hosts skill instead. The smartscape edge is CONTAINER (is part of) K8S POD . To reach containers from a pod, traverse backward: Service → K8S POD Correlation No direct smartscape edge exists between SERVICE and K8S POD . The correlation key is the shared dimension k8s.workload.name . See [Service → Pod Drill Down](references/pod debugging.md service pod drill down) in references/pod debugging.md for the full two step pattern. Common Workflows 1. Cluster Health Check List all clusters: Check node capacity: Identify pods in non Running state: 2. Resource Optimization Find over provisioned pods (usage < 30%): Identify containers without limits: 3. Troubleshooting Pod Issues Pod troubleshooting benefits from combining metrics (timeseries) with Kubernetes events (event stream) for a complete picture. Metrics Based Troubleshooting Find pods with OOMKills: Analyze pod restart patterns: Event Based Troubleshooting For operational events (pod restarts, OOM kills, evictions, scheduling failures), Kubernetes events provide richer context than metrics alone — including event reasons, messages, and timestamps. When to use Kubernetes events over metrics: User asks about recent operational events ("show me pod restart events") User wants event details like reasons and messages User asks about events in a specific time window ("last 48 hours") User wants to correlate events with root causes Kubernetes events are available through the get events for kubernetes cluster tool. Prefer this tool when the user asks about OOM events, pod restarts, evictions, or cluster wide event history. Important: distinguish event types when filtering results. Kubernetes events cover many categories. When the user asks about a specific event type, filter the results accordingly — do not report unrelated events: User Asks About Relevant Event Reasons NOT Related Pod restarts BackOff , CrashLoopBackOff , Killing Readiness probe failures, CPU throttling OOM events OOMKilling , OOMKilled Memory pressure warnings Evictions Evicted , Preempting Node pressure Scheduling failures FailedScheduling , Unschedulable Resource quotas For a complete answer , combine both approaches: 1. Use the events tool to get the event details (what happened, when, why) 2. Use timeseries metrics to show the quantitative impact (how many restarts, OOM kill counts over time) Fetch Kubernetes Events via DQL Pod restart and operational events can also be queried via DQL from the events table: Filter for specific event reasons: Field names in fetch events : Use event.reason and event.message — not dt.kubernetes.event.reason . The dt.kubernetes. prefix is for timeseries metrics, not the events table. Queries using the wrong prefix return zero results. 4. Security Assessment Identify privileged containers: Find containers running as root: 5. Scheduling Analysis Verify pod distribution (HA compliance): 6. DAVIS Problems affecting K8s Entities Find active DAVIS problems affecting K8s entities: smartscape.affected entities is a record array; each record has id , type , and name . Use [][id] to get the array of Smartscape IDs to look up the affected entity, or [id] after expand smartscape.affected entities . Without a preceding expand , [id] returns null silently. A filter cannot take a bare iterative expression, so wrap it in iAny(...) . Best Practices Choosing the Right Data Source User Question Best Approach Why "Show me OOM events" Events tool + metrics Events give reasons/messages; metrics show trends "Show me pod restart events" Events tool + timeseries metrics Events reveal the reason (BackOff, Killing, CrashLoopBackOff); dt.kubernetes.container.restarts metric gives the actual restart counts "How many pod restarts?" Timeseries metrics Quantitative data over time "What happened to my pods in the last 48h?" Events tool Operational event history with context "Which pods are using the most CPU?" Timeseries metrics Resource utilization analysis "List all clusters/namespaces" smartscapeNodes Entity discovery and inventory "Are there scheduling failures?" Events tool Event reasons explain why Query Performance 1. Filter early Apply cluster/namespace filters immediately 2. Use specific entity types Avoid wildcards 3. Limit result sets Use limit for exploration 4. Cache cluster lists Store in variables Monitoring Recommendations 1. Set resource limits on all containers 2. Monitor OOMKills and adjust memory limits 3. Track CPU throttling and adjust CPU limits 4. Review resource efficiency regularly (target 70 80%) 5. Implement security best practices (non root, read only filesystem) 6. Use specific image tags (avoid :latest) Configuration Standards 1. Use labels for organization (app, environment, team) 2. Set resource requests and limits 3. Configure health checks (liveness/readiness probes) 4. Use TLS for all ingress resources 5. Document with annotations Troubleshooting Problem Cause Solution No pod data returned Wrong entity type or missing cluster filter Use K8S POD (not POD ); add k8s.cluster.name filter k8s.object parsing errors Complex JSON structure Use parse k8s.object, "JSON:config" then access nested fields Pod network metrics unavailable Not available in Grail Use service mesh metrics or host level network metrics Large result sets No time range or cluster filter Add time range and filter by cluster/namespace early Missing labels in output Labels accessed incorrectly Use tags[label name] to access labels Limitations Unavailable Metrics: Pod network metrics (rx bytes, tx bytes) are NOT available in Grail Workaround: Use service mesh metrics or host level network metrics Query Considerations: Minimize result set size: Do not include the k8s.object field if not necessary Keep result set as simple as possible: Parsing k8s.object increases query complexity Large clusters may require pagination or time range limits Some K8s status fields update asynchronously When to Load References Load cluster inventory.md when: Performing cluster, namespace, or resource distribution analysis Auditing workload counts across clusters → [references/cluster inventory.md](references/cluster inventory.md) Load labels annotations.md when: Filtering by labels or annotations Parsing k8s.object for detailed configuration inspection → [references/labels annotations.md](references/labels annotations.md) Load pod node placement.md when: Analyzing scheduling constraints (affinity, taints, tolerations) Verifying HA compliance and pod distribution → [references/pod node placement.md](references/pod node placement.md) Load pod debugging.md when: Investigating pod exit codes, crash loops, or init container failures Diagnosing image pull errors or service to pod connectivity issues Drilling down from a service problem to pod level details → [references/pod debugging.md](references/pod debugging.md) Load workload health.md when: Investigating degraded deployments or stuck rollouts Checking node conditions, CPU throttling, or HPA scaling Analyzing StatefulSet ordering or DaemonSet coverage → [references/workload health.md](references/workload health.md) Load pv pvc.md when: Working with persistent storage (PVC/PV lifecycle, orphaned volumes) Checking StorageClass configurations → [references/pv pvc.md](references/pv pvc.md) Load ingress.md when: Analyzing ingress routing rules or TLS certificates Auditing ingress controller configurations → [references/ingress.md](references/ingress.md) Load network policies.md when: Listing or auditing network policies Checking namespace isolation configurations → [references/network policies.md](references/network policies.md) References [cluster inventory.md](references/cluster inventory.md) — Cluster, namespace, and resource distribution analysis [labels annotations.md](references/labels annotations.md) — Label/annotation filtering and k8s.object parsing [pod node placement.md](references/pod node placement.md) — Scheduling, affinity, taints, and HA patterns [pod debugging.md](references/pod debugging.md) — Exit codes, pod conditions, init containers, image pull errors, logs, service to pod drill down [workload health.md](references/workload health.md) — Degraded deployments, stuck rollouts, node conditions, CPU throttling, HPA, StatefulSet ordering [pv pvc.md](references/pv pvc.md) — PVC/PV lifecycle, phase reference, orphaned volumes, StorageClass [ingress.md](references/ingress.md) — Routing rule parsing, TLS audit [network policies.md](references/network policies.md) — Policy listing, namespace isolation audit Related Skills dt obs problems — For problems associated with Kubernetes cluster

dt-obs-kubernetes