Kubernetes Troubleshooting: Guide and Cheat Sheet

What is Kubernetes troubleshooting?

Kubernetes troubleshooting, or debugging, means finding why a cluster or a workload misbehaves, then fixing the cause. Start from the symptom and narrow the scope from the cluster to a node, a workload, a pod and a container. At each layer, read the status, events and logs. kubectl covers most checks, and kubectl debug reaches containers without a shell.

Most failures surface as a pod status, such as Pending or CrashLoopBackOff, or as a node marked NotReady. This guide maps each symptom to a first command, then covers pods, exit codes, kubectl debug, Services, nodes, the control plane, storage and tools. It reflects Kubernetes 1.37, the current release, and sources checked in October 2026.

How to troubleshoot Kubernetes issues step by step

Work from the outside in, and stop at the first layer that looks wrong. The six steps double as a troubleshooting flowchart.

1. Find the scope. Check whether the problem covers the cluster, one node or one workload.

kubectl get nodes
kubectl get pods -A -o wide
kubectl events -A --types=Warning

-o wide shows each pod's node, so failures that share a node point at it. If kubectl itself times out, start with the control plane. kubectl events sorts events by time and has been stable since kubectl 1.28.

2. Describe the object. kubectl describe pod <pod> -n <namespace> shows container states, last states, exit codes, restart counts, conditions and recent events (Debug Running Pods).

3. Read the logs, including the last crash.

kubectl logs <pod> -c <container>
kubectl logs <pod> -c <container> \
  --previous

--previous prints the logs of the previous instance, the one that crashed.

4. Read the events in order. kubectl events --for pod/<pod> lists the events of one object. The API server keeps events for one hour by default (--event-ttl), so save them early.

5. Get inside the container. kubectl exec -it <pod> -- sh needs a shell in the image. Without one, or when the container keeps crashing, use kubectl debug.

6. Check the layer below Kubernetes. Disks, networks and clouds fail on their own too. If pods on two nodes can't reach each other, test the same path between the two hosts directly.

kubectl get all lists only the all category, such as pods, Services, Deployments and Jobs (pod storage). It skips ConfigMaps, Secrets, PersistentVolumeClaims and Ingresses, as kubectl api-resources --categories=all shows. Our kubectl guide covers the basic commands.

Kubernetes troubleshooting cheat sheet

The table lists common Kubernetes issues by symptom. Run the first command, then read the matching section below.

If a cluster is down or an upgrade went wrong, our Kubernetes consulting team steps in, gets it running again and tells you what happened.

How to troubleshoot Kubernetes pods

The STATUS column of kubectl get pods is a summary that kubectl prints. The pod's phase is a separate field with five values, Pending, Running, Succeeded, Failed and Unknown (pod lifecycle). Each container is Waiting, Running or Terminated, and the first and last carry a reason (container states).

Native sidecars, which are init containers with restartPolicy: Always, have been stable since 1.33. They count toward the READY column (table printer). A status that starts with Init: points at an init container (init container statuses).

Pod stuck in Pending

A Pending pod isn't scheduled yet, or its images are still downloading (pod phase). kubectl describe pod shows a FailedScheduling event that starts with 0/N nodes are available (scheduler). The reasons follow:

• Insufficient cpu or Insufficient memory. No node has enough unrequested capacity (fit.go). Lower the requests or add nodes.
• node(s) had untolerated taint(s). The pod has no toleration for the nodes' taints (taint_toleration.go). Before 1.35, it named the taint, as in node(s) had untolerated taint {<key>: <value>} (1.34 source).
• node(s) didn't match Pod's node affinity/selector. No node has the labels the pod asks for (node_affinity.go).
• pod has unbound immediate PersistentVolumeClaims. The claim isn't bound yet (volume_binding.go). See PVC stuck in Pending.

With node autoscaling, a FailedScheduling event is normal at first. Look for the autoscaler's response after it, as our autoscaling guide and Karpenter guide explain.

Pod stuck in ContainerCreating

The kubelet sets up the pod's sandbox, network and volumes before any container starts. The PodReadyToStartContainers condition, stable since 1.37, shows which half is stuck. False points at that setup, and True at image pulls or container creation. The events name the step:

• FailedMount, "Unable to attach or mount volumes" (kubelet.go). A ConfigMap or Secret volume may point at a missing object, or the CSI driver isn't running on the node.
• FailedCreatePodSandBox, "Failed to create pod sandbox" (kuberuntime_manager.go). The runtime couldn't create the sandbox or its network, so check the CNI plugin's pods on that node.
• FailedAttachVolume. The volume couldn't be attached to the node. AttachVolume.Attach failed carries the storage driver's error (operation_generator.go). Waiting for detach or Multi-Attach error means the volume is still attached to another node, as in Multi-Attach.

ImagePullBackOff and ErrImagePull

ErrImagePull means a pull failed, and ImagePullBackOff means the kubelet is waiting to retry (kubelet image errors). The wait grows up to 300 seconds (images). The event message carries the registry's error. Check three things:

• the image name and tag, for typos;
• imagePullSecrets, which must be Secrets in the pod's namespace (pull secrets);
• the node's access to the registry, which crictl pull <image> on the node tests directly (crictl).

CreateContainerConfigError

The kubelet couldn't build the container's configuration (kuberuntime_container.go). Look for one of three causes:

• A missing ConfigMap or Secret in env or envFrom. A reference that isn't optional keeps the pod from starting (ConfigMaps, Secrets).
• A missing key, reported as couldn't find key <key> in ConfigMap <namespace>/<name> (kubelet_pods.go).
• A root image under runAsNonRoot: true, reported as container has runAsNonRoot and image will run as root (security_context_others.go).

Create the object or key, mark the reference optional: true, or set a non-zero runAsUser.

CrashLoopBackOff

CrashLoopBackOff means a container keeps exiting, and the kubelet waits longer before each restart, up to 300 seconds (restart policy). The image was pulled and the container ran, which sets it apart from ImagePullBackOff. To find the cause:

1. kubectl logs <pod> --previous shows why the last run ended, such as a database the app can't reach.
2. kubectl describe pod shows the last state's reason and exit code, and any Unhealthy events from failed probes (prober.go).

Our Kubernetes CrashLoopBackOff guide maps each reason and exit code to its fix. It also covers crashes with no logs, init containers and system pods. To test a fix, run a copy of the pod with a shell.

OOMKilled and exit code 137

A container that exceeds its memory limit gets killed. Its last state then records reason OOMKilled and exit code 137 (memory task), which kubectl describe pod prints under Last State. kubectl top pod <pod> --containers shows each container's memory use to compare with its limit. It needs Metrics Server or another Metrics API provider. Our performance guide explains memory limits and what to check before you raise one.

If the whole node runs out of memory first, the kernel's OOM killer scores each container by its memory use plus an oom_score_adj. The kubelet sets that value from the pod's QoS class, so BestEffort pods are the most likely to die (node OOM behavior).

Pod running but not ready

READY counts containers that passed their last readiness probe (Debug Running Pods). So STATUS Running with READY 0/1 means the probe fails, and Services stop sending the pod traffic (readiness probes). kubectl describe pod shows Unhealthy events with the probe's error, so check the probe's path, port and timeout. A native sidecar's readiness probe counts too (sidecar containers).

Pod stuck in Terminating

Deleting a pod starts a grace period, 30 seconds by default. The kubelet runs any preStop hook, sends SIGTERM, then SIGKILL when the period ends (pod termination). If a pod stays Terminating much longer, check two causes:

• A finalizer. kubectl get pod <pod> -o jsonpath='{.metadata.finalizers}' lists them. The pod stays until each owning controller removes its entry (finalizers).
• A dead node. Its kubelet can't confirm the shutdown, so the pod waits (StatefulSet pod deletion).

kubectl delete pod <pod> --grace-period=0 --force removes the pod from the API without waiting. Use it only when you know the containers have stopped. kubectl warns that force-deleted processes can keep running and corrupt shared data (kubectl delete). For StatefulSet pods, the docs tie it to split brain and data loss. They also advise against removing finalizers by hand (finalizers). For a node that is shut down or powered off, use the out-of-service taint.

Evicted pods

When a node runs short of memory, disk or process IDs, the kubelet evicts pods. They get phase Failed, reason Evicted and a message such as The node was low on resource: memory (node-pressure eviction, eviction helpers). The kubelet ignores PodDisruptionBudgets, and at hard thresholds it uses no grace period. The Linux defaults are memory.available<100Mi, nodefs.available<10%, imagefs.available<15% and 5% free inodes (hard eviction thresholds). Pods that use more than they request go first.

Evicted pods stay until someone deletes them, or until the cluster holds 12,500 terminated pods, the PodGC default (pod garbage collection, kube-controller-manager). This deletes every failed pod in a namespace, failed Job pods included:

kubectl delete pods -n <namespace> \
  --field-selector=status.phase=Failed

Deployment not rolling out

kubectl rollout status deployment/<name> waits for a rollout to finish. It exits with an error once the Deployment passes its progress deadline, 600 seconds by default. kubectl describe deployment then shows the Progressing condition as False, with reason ProgressDeadlineExceeded (failed Deployment). The docs name quota, readiness probes, image pulls, permissions, limit ranges and app misconfiguration as causes, so check the new pods for the statuses above. Kubernetes doesn't roll back by itself. If the new version is at fault, kubectl rollout undo deployment/<name> returns to the previous revision.

If a Deployment has no pods at all, run kubectl describe rs on its ReplicaSet. A FailedCreate event gives the reason (controller_utils.go), such as exceeded quota (quota admission).

Kubernetes exit codes and what they mean

When a container stops, its state records an exit code. Codes above 128 follow the shell rule of 128 plus the signal number (Bash manual). Signal numbers come from signal(7).

containerd also sets the reason, such as Completed, Error, OOMKilled or StartError (containerd). So on containerd, a missing entrypoint shows StartError with 128, while a shell script calling a missing command exits with 127.

How to use kubectl debug

kubectl debug adds tools without changing your images. It adds an ephemeral container to a running pod, creates a modified copy of a pod, or starts a pod on a node.

Debug a running pod with an ephemeral container

Ephemeral containers have been stable since Kubernetes 1.25. They join a running pod, never restart and can't be removed once added.

kubectl debug -it <pod> \
  --image=busybox:1.37 \
  --target=<container>

--target joins the app container's process namespace, if the runtime supports it (ephemeral debug containers). The default general profile also grants SYS_PTRACE for process tracing (profiles.go). Users need patch on pods/ephemeralcontainers, because kubectl adds the container with a PATCH request (debug.go, request verbs). With -it, they also need create on pods/attach (feature gates).

When the image has no shell

Distroless images have no shell, so kubectl exec can't start one (ephemeral containers). Attach an ephemeral container with --target and find the app's PID with ps. The app's files are then under /proc/<pid>/root (process namespace). On the node, crictl and nsenter still work, as our networking guide shows.

Debug a crashing pod with a copy

A container that exits at once leaves nothing to exec into. Copy the pod with a shell in place of the app, then run the app by hand (copy with a new command):

kubectl debug <pod> -it \
  --copy-to=<pod>-debug \
  --container=<container> -- sh

--set-image swaps images in the copy, for example to a debug build. The general profile drops the copy's labels, annotations and probes unless you pass flags such as --keep-labels (profiles.go). Delete the copy afterwards.

kubectl debug node

kubectl debug node/<node> -it \
  --image=busybox:1.37

The pod shares the node's network, PID and IPC namespaces and mounts its root filesystem at /host. It isn't privileged, so chroot /host may fail. Add --profile=sysadmin for a privileged pod (node debugging). The command also fails on a node that is down. Delete the node-debugger-... pod afterwards.

kubectl debug profiles

Since kubectl 1.36, the default profile is general (1.36 change, kubectl debug). The old legacy default is deprecated, with removal planned for 1.39 (debug.go). Some kubernetes.io task pages still call legacy the default.

The settings come from profiles.go and the profile docs. --custom adds a partial container spec on top, stable since kubectl 1.32.

How to troubleshoot Kubernetes Services

Our networking guide has the full walk-through, with packet captures and MTU checks. The short version runs from the Service down.

No endpoints. List the pods behind the Service:

kubectl get endpointslices \
  -l kubernetes.io/service-name=<service>

<unset> under ENDPOINTS means no pod that matches the selector has an IP (table printer, EndpointSlice code). Usually the selector matches no pods at all. A label typo is a common cause. Listed pods may still be unready, marked ready: false, and get no traffic (EndpointSlice code). Also check that targetPort matches the container's port.

DNS. nslookup kubernetes.default from a test pod should answer. If it doesn't, check the pods labeled k8s-app=kube-dns in kube-system and their logs (DNS debugging). Failures in one namespace only point at a default-deny egress policy without a DNS rule.

Network policies. kubectl get networkpolicy -n <namespace> lists them. They work only if your network plugin implements NetworkPolicy.

kube-proxy. Check its pods and their logs. kubeadm labels them k8s-app=kube-proxy (kubeadm manifest). kube-proxy's ipvs mode has been deprecated since 1.35 (KEP-5495). In that mode, kube-proxy logs a warning at startup and records an IPVSDeprecation event on the node (server_linux.go). Since 1.37, the warning says ipvs will be off by default in 1.40 and removed in 1.43 (v1.37 release). Cilium can replace kube-proxy, so use its tools there.

Ingress. Our Ingress guide covers 404, 503 and TLS errors. The community ingress-nginx controller was retired in March 2026 and gets no more fixes.

How to troubleshoot Kubernetes nodes

Node NotReady

NotReady means the node's Ready condition is False or Unknown. Unknown means the kubelet hasn't reported within node-monitor-grace-period, 50 seconds by default (node status). The node then gets a not-ready or unreachable taint, which pods tolerate for 300 seconds before eviction (taint-based evictions). kubectl describe node <node> shows the message on Ready:

• "Kubelet stopped posting node status." The kubelet is down, or the node can't reach the API server (Troubleshooting Clusters).
• "container runtime network not ready" with "cni plugin not initialized". The CNI plugin isn't running or configured on the node (kubelet, containerd, go-cni).
• "container runtime is down". containerd or CRI-O isn't answering the kubelet (runtime.go).

Since 1.35, the kubelet won't start on a cgroup v1 node by default, so an old OS can drop out after an upgrade (cgroups). On the node, check the services and the runtime:

systemctl status kubelet containerd
journalctl -u kubelet --since "30 min ago"
crictl ps -a

crictl talks to the container runtime directly. A node missing from kubectl get nodes never registered, or its Node object was deleted (nodes, cloud controller). Read its kubelet log for errors reaching the API server.

Node pressure and evictions

The MemoryPressure, DiskPressure and PIDPressure conditions add matching NoSchedule taints, so new pods avoid the node (taints by condition). kubectl describe node shows them, along with every pod's requests and limits (Troubleshooting Clusters). For live usage, run kubectl top node and kubectl top pod -A --sort-by=memory (kubectl top). Under disk pressure, the kubelet removes dead containers and unused images before it evicts pods (reclaiming resources).

Read kubelet and containerd logs

On systemd nodes, the kubelet and the container runtime write to journald (log location). With SSH, run journalctl -u kubelet or journalctl -u containerd. Without SSH, read the host's journal from a privileged debug pod:

kubectl debug node/<node> -it \
  --profile=sysadmin \
  --image=busybox:1.37
# then, in the debug pod's shell:
chroot /host journalctl -u kubelet -n 100

AWS documents kubectl debug with the sysadmin profile for reading kubelet logs on EKS Auto Mode nodes, which allow no SSH.

How to troubleshoot a Kubernetes cluster and its control plane

On EKS, GKE and AKS, the provider runs the control plane, so start with its status page. kubeadm runs it as static pods from /etc/kubernetes/manifests (kubeadm constants), labeled tier=control-plane (static pod labels).

The API server is unreachable

kubectl cluster-info
kubectl get --raw='/readyz?verbose'
kubectl get nodes -v=6

/readyz?verbose lists each readiness check, etcd included (health endpoints). -v=6 logs each request kubectl makes (verbosity levels). If kubectl can't connect at all, run crictl ps -a | grep kube-apiserver on a control plane node, then crictl logs on that container.

etcd problems

The API server keeps all cluster state in etcd, so an etcd failure looks like an API failure (etcd backups). On a control plane node, run these with the certificate flags from the etcd task page:

etcdctl endpoint status --write-out=table
etcdctl alarm list

Over its space quota, etcd raises a NOSPACE alarm and accepts only reads and deletes. Writes fail with mvcc: database space exceeded. Compact, defragment, then disarm the alarm (etcd maintenance). Take an etcdctl snapshot save first, and encrypt it, since it holds all cluster state.

Expired certificates

kubeadm's client certificates expire after one year, and its CA after ten (kubeadm certificates). Expired ones break kubectl and the control plane with x509: certificate has expired or is not yet valid (Go crypto/x509). Check and renew them on each control plane node:

kubeadm certs check-expiration
kubeadm certs renew all

Then restart the control plane static pods, because not every component reloads certificates. kubectl can't restart static pods, so move each manifest out of /etc/kubernetes/manifests, wait 20 seconds, and move it back (manual renewal). If your kubeconfig is a copy of admin.conf, copy it again (admin certificate). kubeadm also renews all certificates during each control plane upgrade, so upgrading at least once a year keeps them current (automatic renewal).

Problems after an upgrade

Check four things:

• Version skew. With a 1.37 API server, the skew policy allows kubelets and kube-proxy from 1.34 to 1.37, never newer. kube-controller-manager and kube-scheduler may run 1.36 or 1.37, and kubectl 1.36 to 1.38. kubeadm doesn't support skipping minor versions (kubeadm upgrade).
• Removed APIs. Manifests that use an API version the cluster no longer serves fail to apply. The deprecated API migration guide lists each removal, and the kubectl convert plugin rewrites old manifests.
• Changed behavior. Since 1.34, the kubelet rejects static pods that reference Secrets, ConfigMaps or other API objects by default (1.34 feature gate). In 1.37, the PreventStaticPodAPIReferences gate that turned this off is gone (v1.37 release). Read the release notes before each minor upgrade.
• Backups. Take an etcd snapshot first. kubeadm also writes etcd and manifest backups under /etc/kubernetes/tmp during an upgrade (kubeadm upgrade).

We moved a long-term client's big-data platform onto on-prem Kubernetes, and we still run those clusters, including their version upgrades.

How to troubleshoot Kubernetes storage

PVC stuck in Pending

kubectl describe pvc <pvc> shows why a claim waits:

• waiting for first consumer to be created before binding. The StorageClass uses WaitForFirstConsumer, which is normal until a pod uses the claim (pv_controller.go). A pod that sets nodeName skips the scheduler and leaves such a claim pending (volume binding mode).
• no persistent volumes available for this claim and no storage class is set. The claim names no class, and there is no default StorageClass (pv_controller.go, default StorageClass). kubectl get storageclass marks the default with (default) (table printer).
• ProvisioningFailed. The provisioner couldn't create the volume, and the event carries its error (pv_controller.go).

Multi-Attach and volumes stuck on another node

A ReadWriteOnce volume can be mounted read-write by a single node (access modes). If a pod moves while its volume is still attached elsewhere, the new pod waits with a FailedAttachVolume event. Since 1.37, and in patch releases 1.36.4 and 1.35.9, the message starts with Waiting for detach for volume (reconciler.go, 1.36 backport, 1.35 backport). Older releases start it with Multi-Attach error.

When pods on another node still use the volume, the message ends with Volume is already used by pod(s) and their names. The Kubernetes source names the typical cause, a ReplicaSet whose replicas share one claim (reconciler.go). Give each replica its own volume with a StatefulSet, or use a ReadWriteMany volume, such as CephFS. If the message ends with Volume is already exclusively attached to one node, the old pod is gone but the volume hasn't detached yet (reconciler.go). The other case is a node that died before its volumes could detach. If a pod's deletion hasn't finished after 6 minutes and its node is unhealthy, Kubernetes force-detaches the volume, which the docs warn can corrupt data (forced detach). Once you have confirmed the node is off, the out-of-service taint is safer:

kubectl taint nodes <node> \
  node.kubernetes.io/out-of-service:NoExecute

Kubernetes then force-deletes the node's pods and detaches their volumes. Remove the taint once the pods have moved and the node has recovered (non-graceful node shutdown).

Mount failures

A FailedMount event with MountVolume.SetUp failed means the kubelet couldn't mount a volume (operation_generator.go). Check the CSI driver's pods on that node, kubectl get volumeattachments, and any ConfigMap or Secret the volume references.

Kubernetes troubleshooting tools

kubectl is the main tool, and the commands above cover most checks. Three more built-ins matter:

• Events expire after one hour by default (--event-ttl), so export them during an incident with kubectl events -A -o yaml.
• Metrics Server serves kubectl top, though its maintainers say it is meant only for autoscaling. Keep history in a monitoring system such as Prometheus, as our Kubernetes monitoring guide describes.
• crictl and journalctl show what the runtime and the kubelet see on a node.

Open-source helpers add views that kubectl lacks. The right one depends on how your team works.

---

We have run production systems since 2009 and have managed more than 1,350 production servers. If you'd rather not debug a cluster alone, our Kubernetes consulting team starts with read-only access and a written list of what to fix first. We can also run your clusters afterwards. On-call is by agreement, with up to 24-hour coverage under an SLA, and post-incident reviews are part of on-call work.

Frequently Asked Questions

What are the basic Kubernetes troubleshooting commands?

Start with six kubectl commands. kubectl get pods -A -o wide lists pods with their status and node. kubectl describe pod shows states, exit codes and events. kubectl logs --previous reads the crashed instance's logs. kubectl events lists events in time order. kubectl exec opens a shell, and kubectl debug adds tools when the image has none.

What is the command to view Kubernetes pod logs?

Run kubectl logs <pod>. Add -c <container> in multi-container pods, --previous for the instance that crashed, -f to stream, and -l app=<label> to read several pods. The logs are what the containers wrote to standard output and standard error. An app that logs only to a file shows nothing, so the docs suggest writing to /dev/stdout instead.

How do you delete failed pods in Kubernetes?

Use a field selector: kubectl delete pods -n <namespace> --field-selector=status.phase=Failed, or -A for every namespace. Evicted pods are included, because eviction sets their phase to Failed. Otherwise failed pods stay until someone removes them, or until the cluster passes 12,500 terminated pods, the default garbage collection threshold.

Is there a GUI for Kubernetes?

Yes. The original Kubernetes Dashboard is archived and unmaintained, and the Kubernetes docs point new users to Headlamp instead. Headlamp is a web UI under Kubernetes SIG UI, with logs, exec and a resource editor. It runs in the cluster or as a desktop app. If you prefer the terminal, k9s provides a terminal UI for your clusters.

Related Reading

• Kubernetes Networking Explained: A Deep Dive
• Kubernetes Performance Tuning & Optimization (2026)
• Kubernetes Autoscaling: HPA, VPA, Karpenter and KEDA
• Kubernetes Ingress in 2026: Examples and Controllers