Why a Kubernetes GPU pod gets stuck Pending, and how to fix it

Kubernetes only knows a node has GPUs if a device plugin DaemonSet is running on that node and registers the nvidia.com/gpu extended resource with the kubelet; without it, the scheduler sees zero allocatable GPUs on every node, and any pod requesting nvidia.com/gpu in its resource limits will never find a fit, no matter how many physical GPUs exist.

Fix: Verify the nvidia-device-plugin or GPU Operator DaemonSet pods are Running on your GPU nodes, and check node capacity for the nvidia.com/gpu entry; reinstall the plugin or operator if it is missing or crashlooping.

kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds
kubectl describe node <gpu-node> | grep -A5 Capacity
kubectl get pods -n gpu-operator

GPU requests in Kubernetes are not divisible or schedulable across nodes; a pod asking for nvidia.com/gpu: 4 can only land on a node with at least 4 free GPUs advertised, so on a cluster of 2-GPU nodes that pod stays Pending forever even though the cluster's total GPU count is far higher.

Fix: Reduce the pod's GPU request to fit within a single node's capacity, or move to a multi-node job that uses gang scheduling or an MPI operator to span nodes instead of one pod claiming all GPUs.

kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu

GPU node pools are commonly tainted so only GPU-aware workloads land on expensive hardware; if the pod spec has no matching toleration, or a required nodeSelector or affinity label is missing, the scheduler correctly refuses to place it and it stays Pending indefinitely.

Fix: Add the matching toleration block and nodeSelector or affinity to the pod spec so it targets the tainted GPU pool, confirmed against the actual taint and label keys shown by describing the node.

kubectl describe node <gpu-node> | grep -A3 Taints
kubectl get nodes --show-labels | grep gpu

A pod can request a GPU that is available but also request more CPU or memory than remains free on that specific node after other pods have been scheduled, and Kubernetes will not partially satisfy a pod's resource requests, so it stays Pending on a resource dimension that has nothing to do with the GPU at all.

Fix: Check the full describe pod events for the actual reason string, not just the GPU count, and either lower CPU or memory requests or free capacity on the target node.

kubectl describe pod <pod-name> | grep -A10 Events