Why your model loads slowly inside a container every time it restarts, and how to fix it
Loading checkpoint shards: 100%|... [took 8+ minutes]
Also appears as
- INFO vllm: Loading model weights took 420.35 seconds
- pod restart causes multi-minute delay before readiness probe passes
Short answer
A model that reloads slowly on every container restart almost always means the weight directory is not backed by a persistent volume, so each restart re-downloads or cold-reads the full checkpoint instead of hitting a warm cache. Mount a persistent volume for the model cache path, set HF_HUB_OFFLINE=1 once weights are local, and if load time is still slow, benchmark your storage backend's raw throughput since network storage is a common hidden bottleneck.
Affects: Any containerized LLM serving deployment, especially on Kubernetes with pod autoscaling or frequent restarts
Fastest path to fast restarts
- 1Check whether the model cache path is mounted from a persistent volume with kubectl get pod -o jsonpath.
- 2If it is emptyDir or unmounted, add a PersistentVolumeClaim or hostPath mount at that path.
- 3Set HF_HOME to the mounted path and HF_HUB_OFFLINE=1 once weights are cached there.
- 4Benchmark raw read throughput on the storage backend with dd to confirm it is not the bottleneck.
- 5For frequent restarts, add an init container that prefetches weights before the main container starts.
How to confirm this is your problem
- every pod restart re-triggers a multi-minute model load
- readiness or startup probes time out specifically after a restart, not on first deploy
- load time is the same whether the model was just loaded minutes ago or not at all
- logs show weight download progress on restarts, not just at first launch
Root causes and fixes
Weights are re-downloaded or re-pulled from a cold cache on every container restart
If the Hugging Face cache directory or model volume is ephemeral, part of the container's writable layer rather than a persistent mount, every restart, pod reschedule, or autoscale event starts from zero: the full multi-gigabyte weight download happens again before inference can start, even though nothing about the model changed.
Fix: Mount a persistent volume or node-local cache directory for HF_HOME so weights survive container restarts, and set HF_HUB_OFFLINE=1 once weights are cached to skip hub metadata checks entirely.
export HF_HOME=/mnt/model-cache export HF_HUB_OFFLINE=1 docker run -v model-cache:/mnt/model-cache myimage
Page cache is lost on container restart, forcing a cold read from disk or network storage
The Linux page cache that made a previous load fast lives in host RAM, not in the container; when a pod is rescheduled to a different node, common with Kubernetes autoscaling or node drains, that node has never read these weight files before, so even a fast local disk read still has to traverse the actual storage backend cold, and if that backend is slow network storage, load time balloons further.
Fix: For workloads that restart frequently, prefer node-local NVMe caching or a warm pool of pinned nodes for large models, and consider a prefetch init container that primes the cache before the main container starts serving traffic.
Network storage throughput is the actual bottleneck, not the application
A 70B parameter model in FP16 is on the order of 140GB; reading that from an NFS or generic network block store with modest throughput can take several minutes purely on I/O, regardless of CPU, GPU, or how efficient the loading code is, and this often gets misdiagnosed as an application or framework problem.
Fix: Benchmark raw read throughput from the actual storage backend independent of model loading, and if it is the bottleneck, move to a storage class or backend rated for high sequential throughput, or use a local NVMe cache in front of it.
dd if=/mnt/models/testfile of=/dev/null bs=1M status=progress
No volume cache at all, model directory lives inside the container's ephemeral filesystem
Some Dockerfiles or deployment manifests never define a volume for the model cache path at all, meaning weights are downloaded into the container's own writable layer each run; this is functionally identical to having no cache, and it also means disk usage accumulates against the container's own storage quota rather than a shared, cleanable volume.
Fix: Explicitly define a PersistentVolumeClaim or named volume mounted at the model cache path in every deployment manifest, never leaving it as an implicit ephemeral path.
Diagnostic commands
Time the actual load phase
grep -i 'loading' pod-logs.txt
Frameworks like vLLM and transformers log an explicit load duration; compare it across restarts to see if it is consistently slow, storage-bound, or only slow on cold nodes, cache-bound.
Check whether the cache volume is actually persistent
kubectl get pod <pod> -o jsonpath='{.spec.volumes}'If the model path maps to emptyDir or no volume at all rather than a PVC or hostPath, the cache never survives a restart by design.
Benchmark raw storage throughput
dd if=/mnt/models/<large-file> of=/dev/null bs=1M status=progress
Sub-500MB/s sustained reads on a large model directory point to storage, not application code, as the real bottleneck.
Check if offline mode is set
echo $HF_HUB_OFFLINE
If unset, the loader may spend extra time on hub metadata or version checks even when weights are already cached locally.
Stopping it from happening again
- Always mount model weight directories from a persistent volume, never the container's writable layer
- Set HF_HUB_OFFLINE=1 and HF_HUB_DISABLE_TELEMETRY=1 once weights are locally cached to remove network round trips from the load path
- Benchmark your storage backend's throughput against your model size before choosing it, not after latency complaints
- Use an init container to prefetch and warm weights onto local node storage before the main serving container starts
When this becomes an architecture problem
If model load time is dominated by storage throughput rather than caching gaps, and multiple models or replicas need fast cold starts simultaneously, this becomes a storage architecture decision, local NVMe tiering, a dedicated model artifact store, or a warm pod pool, rather than something a volume mount alone will fix.
Frequently asked questions
Why is my model slow to load only after a Kubernetes pod restart, not the first time?
This is almost always a caching problem, not a code problem. If the model directory is not backed by a persistent volume, every restart starts the download from scratch. Even with a persistent volume, if the pod lands on a different node than before, that node's page cache and any local disk cache are cold, so the read has to go all the way back to the underlying storage backend, which is slower than a warm local read.
Does HF_HUB_OFFLINE actually make loading faster, or just prevent network errors?
Both. With HF_HUB_OFFLINE=1, the Hugging Face libraries skip contacting the hub to check for newer file versions or metadata, which removes network round trips from the load path entirely and also prevents failures when the hub is unreachable in an air-gapped or restricted network. It only helps once weights are already present locally; it does not download anything itself.
What is the fastest way to make repeated container restarts load a large model quickly?
Mount the model cache directory from a persistent volume so weights survive restarts, prefer node-local NVMe-backed storage classes over generic network storage for that volume, set HF_HUB_OFFLINE=1, and where restarts are frequent, consider pinning replicas to a warm pool of nodes that already have the weights in local page cache rather than letting the scheduler place them on cold nodes.
Is it worth keeping the model loaded in a separate long-running process instead of reloading per container?
For very frequently restarted or scaled services, yes; some teams run a persistent model server process on the node and have short-lived containers proxy to it, avoiding reload entirely. For most production serving setups, though, a well-cached persistent volume plus offline mode gets load time down to a level where a dedicated always-on process is unnecessary complexity.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Document Ingestion Pipeline Estimator
Estimate total pipeline time from document count, OCR share, and embedding throughput, so ingestion timelines stop being a guess in the project plan.
Free ToolOn-Prem LLM Total Cost of Ownership Calculator
Model the full multi-year cost of running LLMs on your own hardware, including GPU capex, power, cooling, support contracts, and operations staffing.
Free ToolKubernetes Cluster Cost Calculator
Estimate the true monthly and annual cost of a Kubernetes cluster, including compute, control plane fees, managed service premiums, and utilization waste.
Related problems
LLM container image is tens of gigabytes and slow to pull
LLM container images balloon past ten or twenty gigabytes almost always because model weights were copied directly into a layer instead of mounted at runtime, or because a devel CUDA base image and unstaged build tools shipped into production by mistake. Remove weights from the Dockerfile, switch to a runtime base image and a multi-stage build, and image size typically drops by an order of magnitude without any change to the serving code.
Kubernetes PersistentVolumeClaim errors when serving model weights
PersistentVolumeClaim errors serving model weights almost always come from using a ReadWriteOnce volume with more than one replica, since that access mode only allows a single node to mount it at a time. Switch to a ReadOnlyMany-capable storage class, mount weights read-only, and set volumeBindingMode to WaitForFirstConsumer to avoid zone mismatches; if the volume mounts fine but loading is still slow, the real problem is storage throughput, not access mode.
HuggingFace model download is extremely slow or stalls partway through
Slow or stalled HuggingFace downloads are usually caused by huggingface_hub's default transfer path not using parallel chunked downloads, a corporate proxy or firewall throttling or dropping long-lived connections, or genuinely insufficient bandwidth for a hundreds-of-gigabytes model. Enable hf_transfer for a much faster Rust-based parallel downloader, rely on the client's built-in resume behavior rather than restarting from zero, and for regulated or air-gapped sites, download once and mirror internally instead of pulling repeatedly over the internet.
Kubernetes readiness probe fails while the model is still loading
LLM service pods get killed or marked unready during startup because default Kubernetes readiness and liveness probes assume a service starts in seconds, while loading multi-gigabyte weights into GPU memory can take minutes. Add a startupProbe sized with a failureThreshold times periodSeconds budget that comfortably exceeds your worst-case load time; Kubernetes suppresses readiness and liveness checks entirely until the startup probe succeeds, which stops premature restarts without needing a fragile fixed initialDelaySeconds guess.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideOn-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
GuideOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.