How to set up a container registry for air-gapped Kubernetes deployments
Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: no such host
Also appears as
- ImagePullBackOff: rpc error: code = Unknown desc = failed to pull and unpack image
- x509: certificate signed by unknown authority (private registry)
Short answer
Air-gapped Kubernetes clusters cannot reach public registries or the Hugging Face Hub, so image pulls fail at DNS resolution unless a local registry mirror is stood up ahead of time and populated from a connected staging environment. Mirror container images and model weights as two distinct pipeline steps, distribute the internal registry's CA certificate to every node, and sign mirrored artifacts so provenance, not just reachability, is auditable, which is a mandatory control in ITAR and CMMC environments.
Affects: Any air-gapped or network-restricted Kubernetes or Docker environment, mandatory in ITAR and CMMC-regulated deployments
Get a working air-gapped pull path
- 1Stand up an internal registry (Harbor or a private Docker Registry) reachable from every node inside the secure enclave.
- 2From a connected staging environment, pull, tag, and push every required container image into the internal registry.
- 3Separately, download required model weights from the Hugging Face Hub and sync them into internal object storage or an OCI artifact store.
- 4Distribute the internal registry's CA certificate to every node and add it to the container runtime's trust store.
- 5Repoint deployment manifests to the internal registry address and set HF_HUB_OFFLINE=1 for weight loading.
- 6Sign mirrored images with cosign and enforce signature verification at admission before allowing them to deploy.
How to confirm this is your problem
- dial tcp: lookup registry-1.docker.io: no such host on every pull attempt
- ImagePullBackOff on all nodes for any public image reference
- x509: certificate signed by unknown authority when pulling from the internal registry
- containers start but fail trying to reach huggingface.co for model weights
Root causes and fixes
No local registry mirror exists, and nodes have no path to the public internet at all
Air-gapped networks by design have no route to public registries like Docker Hub, ghcr.io, or the Hugging Face Hub, so any image pull or model download that assumes internet access fails at DNS resolution before it ever reaches an authentication or compatibility problem; this is expected behavior, not a misconfiguration, and the deployment needs its own internal registry populated ahead of time.
Fix: Stand up an internal registry, such as Harbor or a private Docker Registry mirrored into the secure enclave, mirror every required image and model artifact into it from a connected staging environment, and repoint all deployment manifests to pull from the internal registry address.
docker pull myimage:tag docker tag myimage:tag internal-registry.local/myimage:tag docker push internal-registry.local/myimage:tag
Model weights were never mirrored alongside container images, only the images themselves
Teams frequently mirror container images into an internal registry but forget that model weights are a separate artifact entirely, typically downloaded from the Hugging Face Hub at container startup; without a plan to mirror those weight files into internal object storage as well, the image pulls fine but the container then fails trying to reach a hub it cannot route to.
Fix: Treat model weights as a first-class artifact in your mirroring pipeline, downloading them once in a connected staging environment and pushing them into internal object storage or a registry-compatible OCI artifact store, then configuring containers to load from that internal source with HF_HUB_OFFLINE=1.
huggingface-cli download <model-id> --local-dir /staging/models/<model-id> aws s3 sync /staging/models/<model-id> s3://internal-bucket/models/<model-id>
Private registry uses a certificate the container runtime does not trust
Internal registries commonly use certificates signed by an internal certificate authority rather than a public one, and unless that CA's certificate is explicitly added to the trust store of every node's container runtime, pulls fail with an x509 unknown authority error even though the registry itself is reachable and otherwise healthy.
Fix: Distribute the internal CA certificate to every node and configure the container runtime to trust it, either by adding it to the OS trust store or via runtime-specific registry certificate configuration.
sudo cp internal-ca.crt /usr/local/share/ca-certificates/ sudo update-ca-certificates sudo systemctl restart containerd
No pull-through cache or signing policy, so image provenance cannot be verified after mirroring
Simply copying images into an internal registry without a signing and verification policy means there is no cryptographic guarantee that what got mirrored matches the original upstream artifact, which is a specific problem in ITAR and CMMC environments where supply chain integrity of every deployed artifact must be demonstrable, not just its availability.
Fix: Sign images during the mirroring process with a tool like cosign, and configure admission control, such as a Kubernetes admission webhook, to reject unsigned images at deploy time, closing the provenance gap rather than only closing the network gap.
cosign sign --key cosign.key internal-registry.local/myimage:tag cosign verify --key cosign.pub internal-registry.local/myimage:tag
Diagnostic commands
Confirm nodes truly have no external route
curl -v https://registry-1.docker.io/v2/
A DNS resolution or connection timeout confirms genuine air-gap; a certificate or auth error instead means there is a route but a trust or credential problem.
Check whether the internal registry is reachable and healthy
curl -v https://internal-registry.local/v2/_catalog
A 200 response with a repository list confirms the registry itself works; failures here point to the registry, not the air-gap.
Check the container runtime's configured registry mirrors
cat /etc/containerd/config.toml | grep -A5 registry
Confirms whether nodes are actually configured to redirect pulls to the internal registry rather than the public default.
Verify the internal CA is trusted by the runtime
openssl s_client -connect internal-registry.local:443 -showcerts
Certificate verify errors here confirm the CA trust chain is the specific blocker, not network reachability.
Stopping it from happening again
- Build the internal registry and mirroring pipeline before any application deployment begins, not reactively after the first failed pull
- Mirror model weights as a distinct pipeline step alongside container images, never assume they travel together
- Distribute internal CA certificates to every node's container runtime trust store as part of node provisioning
- Sign every mirrored artifact and enforce signature verification at admission, since air-gapping solves reachability but not provenance
When this becomes an architecture problem
Air-gapped registry and artifact mirroring is not a one-time script; in ITAR, CMMC, or similar regulated environments it is a mandatory, auditable pipeline covering images, model weights, and their provenance, and designing that pipeline correctly from the start is exactly the kind of architecture work worth bringing in outside expertise for rather than improvising per deployment.
Frequently asked questions
Why is a local registry mandatory in ITAR or CMMC environments, not just convenient?
ITAR and CMMC compliance require that controlled technical data and the software processing it never traverse networks or infrastructure outside the approved boundary, and pulling images or model weights directly from public registries or the Hugging Face Hub would mean data flows in and out of that boundary in ways that cannot be audited or controlled. A local, mirrored registry inside the secure enclave is how you maintain a fully auditable, air-gapped supply chain for every artifact the deployment depends on.
Do I need to mirror model weights separately from container images?
Yes. Container images and model weights are different artifact types with different typical sources; images usually come from registries like Docker Hub or a private registry, while weights are commonly pulled from the Hugging Face Hub at runtime. An air-gapped pipeline needs to explicitly mirror both into the secure enclave, since mirroring only the container images still leaves the application trying to reach the public Hugging Face Hub for weights and failing.
What does the x509 certificate signed by unknown authority error mean for a private registry?
It means your container runtime does not trust the certificate authority that signed your internal registry's TLS certificate, which is common when organizations use an internal CA rather than a public one. The registry itself is reachable and working; the fix is distributing the internal CA's certificate to every node and adding it to the container runtime's trust store, not changing anything about the registry configuration itself.
Is image signing actually necessary if the registry is already air-gapped?
Yes, they solve different problems. Air-gapping controls where an artifact can be reached from, but it says nothing about whether the artifact that got mirrored into your internal registry is actually the exact, untampered artifact you intended to mirror. Signing images with a tool like cosign and verifying signatures at deployment time closes that provenance gap, which is specifically what frameworks like CMMC expect you to demonstrate for supply chain integrity.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Air-Gapped LLM Deployment Checklist
A practical control checklist for deploying and maintaining large language models in a fully air-gapped environment, from initial staging through ongoing patching and drift detection.
Free ToolOn-Prem AI Deployment Checklist
A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.
Free ToolOn-Prem LLM Total Cost of Ownership Calculator
Model the full multi-year cost of running LLMs on your own hardware, including GPU capex, power, cooling, support contracts, and operations staffing.
Related problems
Model loading fails offline or in an air-gapped environment despite having local files
Passing a local path to from_pretrained does not guarantee an offline load, because transformers and related libraries (tokenizers, some model configs, auto-mapping code) can still issue background network calls to check for updates, fetch a referenced remote component, or resolve auto_map entries that point back at the original HuggingFace repo. The fix is to set HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 explicitly, use a complete local snapshot directory (not just the weights file), and verify no config field still references a remote repo ID.
LLM container image is tens of gigabytes and slow to pull
LLM container images balloon past ten or twenty gigabytes almost always because model weights were copied directly into a layer instead of mounted at runtime, or because a devel CUDA base image and unstaged build tools shipped into production by mistake. Remove weights from the Dockerfile, switch to a runtime base image and a multi-stage build, and image size typically drops by an order of magnitude without any change to the serving code.
401/403 Unauthorized pulling a gated model from HuggingFace
A 401 or 403 on a gated HuggingFace repo means the request reached the Hub but was rejected for authorization, not because the model does not exist. The three real causes are: you have not accepted the model's license on the web page with the account tied to your token, the token exists but was created without read access to gated repos, or the token is valid but was never actually passed to the download call (no HF_TOKEN in the environment, no login run). Fix by accepting the license, generating a token with the right scope, and exporting it where the client library will find it.
Kubernetes PersistentVolumeClaim errors when serving model weights
PersistentVolumeClaim errors serving model weights almost always come from using a ReadWriteOnce volume with more than one replica, since that access mode only allows a single node to mount it at a time. Switch to a ReadOnlyMany-capable storage class, mount weights read-only, and set volumeBindingMode to WaitForFirstConsumer to avoid zone mismatches; if the volume mounts fine but loading is still slow, the real problem is storage throughput, not access mode.
GuideAir-Gapped LLM Deployment Patterns That Actually Work
Air-gapped LLM deployment patterns that work: offline model transfer, update workflows, monitoring without telemetry, and CMMC-ready architectures.
GuideSecuring Model Weights in the Enterprise
Secure model weights end to end: custody controls, encryption at rest, access policies, and exfiltration prevention for regulated AI deployments.
GuideAir-Gapped Model Updates: A Patching Guide
Air-gapped model updates for enterprise AI: secure transfer procedures, hash verification, and staged rollout so patches never introduce risk.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.