Why nvidia-container-cli throws initialization errors, and how to fix them
nvidia-container-cli: initialization error: driver error: failed to process request
Also appears as
- Error response from daemon: OCI runtime create failed: ... nvidia-container-cli: mount error
- nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.4
Short answer
nvidia-container-cli initialization errors mean the host's NVIDIA kernel module failed to load or the driver's supported CUDA version does not meet the minimum your container image requires. Check nvidia-smi on the bare host first; if it fails there, fix the kernel module or driver before touching Docker. If the host is healthy, compare its CUDA support against your image's requirement and either upgrade the driver or use an older image tag.
Affects: Any host running NVIDIA Container Toolkit 1.13 or later with Docker or containerd, most common after a driver or kernel update
Fastest path to a working GPU container
- 1Run nvidia-smi directly on the host (not in a container) to rule out a broken driver first.
- 2Check lsmod | grep nvidia; if empty, run sudo modprobe nvidia or rebuild via DKMS.
- 3Note the CUDA version shown in nvidia-smi's header, that is your driver's ceiling.
- 4Compare it to your image's required CUDA version and switch to a compatible tag if needed.
- 5Re-run sudo nvidia-ctk runtime configure --runtime=docker and restart Docker.
- 6Retest with docker run --rm --gpus all <your-image> nvidia-smi.
How to confirm this is your problem
- container fails at 'nvidia-container-cli: initialization error'
- error mentions 'unsatisfied condition: cuda>=X.X'
- GPU containers worked yesterday and fail after a routine host update
- OCI runtime create failed mentioning nvidia-container-cli
Root causes and fixes
Host kernel module not loaded or driver/kernel version mismatch after an update
nvidia-container-cli talks directly to the host's NVIDIA kernel driver to set up device nodes; if the driver kernel module failed to load, often after an unattended kernel upgrade that did not rebuild the module via DKMS, the initialization call fails before it ever reaches your container, so every single container GPU launch fails identically regardless of image.
Fix: Check lsmod for nvidia, and if missing, reload it or reinstall the driver via DKMS so it matches the running kernel; reboot if a new kernel is active and the module cannot load without one.
lsmod | grep nvidia sudo modprobe nvidia dkms status sudo apt-get install --reinstall nvidia-driver-535
Unsatisfied CUDA version condition between the image and the host driver
CUDA base images embed a minimum driver requirement label such as cuda>=12.4; nvidia-container-cli checks the host driver's supported CUDA version against that label before it hands the container anything, and refuses to start if your driver is older than what the image demands, which is a safety check, not a bug.
Fix: Upgrade the host driver to a version supporting the required CUDA release, or switch to an older, compatible CUDA base image tag; nvidia-smi's header shows the maximum CUDA version your current driver supports.
nvidia-smi docker run --rm --gpus all nvidia/cuda:12.1.1-base-ubuntu22.04 nvidia-smi
Toolkit and Docker/containerd version incompatibility after an upgrade
Docker, containerd, and the NVIDIA Container Toolkit are updated independently by the OS package manager, and a toolkit release can ship a config schema or CLI interface that an older containerd shim does not understand, producing OCI runtime create failures that look like GPU errors but are actually a plumbing mismatch.
Fix: Update all three components together (docker-ce, containerd.io, nvidia-container-toolkit) from their official repositories in the same maintenance window, then re-run nvidia-ctk runtime configure.
apt list --installed | grep -E 'docker|containerd|nvidia-container' sudo apt-get update && sudo apt-get install --only-upgrade docker-ce containerd.io nvidia-container-toolkit
Corrupted or partial toolkit install leaving stale config
An interrupted apt install or a manual package removal can leave the NVIDIA container runtime config half-written or referencing binaries that no longer exist, causing nvidia-container-cli to fail in ways unrelated to actual driver or CUDA compatibility.
Fix: Purge and reinstall the toolkit package cleanly, then regenerate config with nvidia-ctk, rather than hand-editing the existing config file.
sudo apt-get purge nvidia-container-toolkit sudo apt-get install nvidia-container-toolkit sudo nvidia-ctk runtime configure --runtime=docker
Diagnostic commands
Check the kernel module is loaded
lsmod | grep nvidia
No output means the driver kernel module never loaded; that has to be fixed before any container-level troubleshooting matters.
Check host CUDA capability
nvidia-smi
The top-right corner shows the CUDA version your driver supports; compare it against your base image's required version.
Check toolkit config integrity
cat /etc/nvidia-container-runtime/config.toml
Look for missing or malformed paths; a truncated file usually means a failed or interrupted package install.
Run the CLI directly for a verbose error
nvidia-container-cli -k -d /dev/tty info
Surfaces the exact initialization failure reason instead of Docker's wrapped, less specific message.
Stopping it from happening again
- Pin nvidia-container-toolkit, docker-ce, and containerd.io to tested versions instead of unattended upgrades
- Rebuild and verify the DKMS driver module after every kernel upgrade before assuming GPU hosts are healthy
- Add a post-update health check that runs nvidia-smi and a --gpus all test container before marking a node ready
- Track host driver CUDA support against the CUDA base image versions used across your fleet
When this becomes an architecture problem
If this keeps recurring after unattended kernel or driver upgrades on a fleet of GPU nodes, it is worth locking down update policy and building an automated post-patch GPU health check rather than manually diagnosing the same failure on every node.
Frequently asked questions
What does 'unsatisfied condition: cuda>=12.4' actually mean?
Every CUDA base image carries a label declaring the minimum driver-supported CUDA version it needs to run correctly. nvidia-container-cli checks this label against what your installed host driver actually supports before allowing the container to start. If your driver only supports an older CUDA version than the image requires, the container is refused rather than allowed to start and fail unpredictably later. Check the supported version in the header of nvidia-smi's output and either upgrade the driver or use an older image tag.
Why did GPU containers break after a routine apt upgrade?
Unattended kernel upgrades are the most common cause. When the kernel changes, the NVIDIA driver's kernel module needs to be rebuilt against the new kernel headers via DKMS; if that rebuild fails silently or the DKMS package was not installed, the module never loads for the new kernel, and every GPU container fails at initialization even though the driver package itself still shows as installed.
Is nvidia-container-cli part of Docker or NVIDIA's toolkit?
It ships as part of the NVIDIA Container Toolkit, not Docker itself. Docker calls into it through the registered nvidia OCI runtime whenever a container requests GPU access. Because it operates below Docker, its errors reference driver and CUDA compatibility details rather than Docker concepts, which is why the messages can look unfamiliar even to experienced Docker users.
Should I reinstall the driver or just the toolkit?
Start by checking whether nvidia-smi works on the host at all outside of any container. If it fails there too, the driver or kernel module is the problem and needs reinstalling or a DKMS rebuild. If nvidia-smi works fine on the host but containers fail, the issue is isolated to the toolkit or its configuration, and reinstalling nvidia-container-toolkit and re-running nvidia-ctk runtime configure is the right next step.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
On-Prem AI Deployment Checklist
A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Related problems
GPU not visible inside a Docker container
Docker containers cannot see a host GPU unless the NVIDIA Container Toolkit is installed and the nvidia runtime is registered with the daemon, since containers are isolated from host devices by default. The fix is almost always to install nvidia-container-toolkit, run nvidia-ctk runtime configure, restart Docker, and launch with --gpus all. If nvidia-smi already fails on the host itself, the problem is the driver, not Docker.
CUDA version mismatch between PyTorch and the system driver
PyTorch ships its own bundled CUDA runtime inside the wheel, so it never uses your system's CUDA toolkit (the one nvcc reports). The only number that matters is the driver's maximum supported CUDA version, shown top right in nvidia-smi output. Fix the mismatch by installing a torch wheel built for a CUDA version at or below that number, not by touching nvcc or the toolkit.
NVIDIA GPU Operator pods stuck installing or crashlooping
GPU Operator installation problems almost always come from a preinstalled host driver conflicting with the Operator's own driver container, or from Node Feature Discovery never labeling GPU nodes so downstream components stay unscheduled. Check kubectl get pods -n gpu-operator first to see which subsystem is failing, then confirm whether the node has a preexisting driver and whether NFD applied the expected NVIDIA labels.
NVIDIA driver installation fails on Ubuntu
The single most common reason NVIDIA driver installation fails on Ubuntu is Secure Boot rejecting the unsigned or self-signed kernel module at load time, since most machines now ship with Secure Boot enabled out of the box. The fix is enrolling the MOK key the installer generates, or disabling Secure Boot in the BIOS, then clearing any lingering nouveau or mixed-install conflicts before rebooting.
GuideOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
GuideOn-Prem GPU Cluster Design: Node Sizing, Networking, and Storage
Design an on-prem GPU cluster: node sizing for H100/H200/B200, InfiniBand vs RoCE networking, storage throughput, and rack power for enterprise AI workloads.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.