Air-Gapped LLM Deployment Patterns That Actually Work
An air-gapped LLM deployment runs language models on a network with no connection to the internet - the standard posture for classified programs, ITAR enclaves, and the most conservative CMMC environments. The models themselves run fine offline; what breaks are the assumptions baked into modern AI tooling: package registries, model hubs, license phone-homes, and telemetry endpoints that all silently expect connectivity. Successful air-gapped deployments are therefore less about GPUs and more about disciplined artifact transfer, offline-first software choices, and monitoring that works without any external service. This guide covers the patterns that survive contact with a real disconnected enclave.
Build a Complete Offline Artifact Pipeline
Everything the stack needs must cross the air gap as verified artifacts: model weights, container images, Python wheels, CUDA drivers, and OS packages. The working pattern is a connected staging environment that assembles a versioned bundle - model safetensors with SHA-256 manifests, a private container registry export (docker save or a Harbor replication bundle), a pip wheelhouse built with pip download for the exact Python and CUDA versions, and offline apt/dnf mirrors. The bundle is scanned (AV plus checksum verification against publisher hashes), burned to write-once media or approved transfer drives, and imported under a two-person rule with a signed transfer log. Teams that skip the wheelhouse step discover on day one that pip install vllm cannot run inside the enclave - build the entire dependency closure outside, always.
- Bundle model weights as safetensors with SHA-256 manifests verified against publisher hashes on both sides
- Export container images to a private registry bundle; never assume docker pull works inside the enclave
- Build a complete pip wheelhouse and OS package mirror for exact Python/CUDA/driver versions
- Transfer on approved media under a two-person rule with signed logs - this procedure belongs in your SSP
Choose Software That Is Genuinely Offline-First
Many AI tools fail closed networks in subtle ways: Hugging Face libraries attempt hub connections unless HF_HUB_OFFLINE=1 is set, some orchestration frameworks fetch prompt templates remotely, and commercial tools with license servers simply stop working. Standardize on components proven offline: vLLM or llama.cpp for serving, Qdrant or pgvector for retrieval, and open-source orchestration you can vendor entirely. Set explicit offline flags (HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE, DO_NOT_TRACK) and then verify with a packet capture during a full workload cycle - the deny-all firewall will block leaks, but you want zero connection attempts in the logs, because every blocked attempt is a finding conversation with your security team. Avoid any component whose license requires periodic online activation unless the vendor provides a documented offline licensing path.
Model Updates and Patch Cadence Without a Connection
Air-gapped does not mean frozen. Establish a quarterly update train: each cycle, the staging environment assembles a new bundle - updated model versions, patched containers rebuilt against current CVE scans, and refreshed vector-store software - which crosses the gap through the same verified transfer procedure. Critical security patches (a CUDA driver CVE, a container escape) get an out-of-band emergency train with the same controls compressed to days. Version everything as immutable, atomically-switchable releases: the enclave runs release N while N+1 is staged, with rollback being a symlink or Kubernetes deployment flip. Keep the previous two releases on enclave storage. This cadence discipline is what assessors and program security officers actually probe - an air-gapped system with no documented patch pathway fails RA.L2 and SI.L2 expectations regardless of how well it is isolated.
Monitoring and Evaluation Without Telemetry
You lose every SaaS observability crutch inside the gap, so build monitoring on self-hosted components: Prometheus and Grafana for GPU utilization (via DCGM exporter), vLLM's built-in metrics endpoint for token throughput and queue depth, and Loki for logs. Model quality monitoring matters more offline because you cannot A/B against a cloud baseline: maintain a frozen eval set of 200-500 domain prompts inside the enclave, run it after every release train, and track scores release-over-release. Prompt and completion logs go to append-only storage with retention aligned to program requirements. All of this runs on one modest VM alongside the inference nodes - the point is that nothing in the observability path ever needs to resolve an external hostname.
- Self-host Prometheus, Grafana, and DCGM exporter for GPU and serving metrics - no SaaS agents
- Keep a frozen 200-500 prompt eval set in the enclave and score every release train against it
- Ship prompts and completions to append-only local storage mapped to AU.L2 retention requirements
- Verify zero external connection attempts via packet capture during acceptance testing of each release
How Netray Delivers Air-Gapped AI Enclaves
Netray has packaged the entire pattern: a staged artifact pipeline with signed manifests, an offline-verified serving stack (vLLM, Qdrant, Prometheus/Grafana) delivered as a transferable release bundle, quarterly update trains, and transfer procedures written directly into SSP-ready documentation. Our AI agents run fully inside the enclave against local ERP extracts, automating documentation, extraction, and classification work without any external dependency. Clients typically stand up a functioning air-gapped LLM enclave in 8-12 weeks including security review, and one defense program went from zero to a scored, eval-baselined offline RAG system across two update cycles with no security findings - because every transfer, hash, and version was documented from the first bundle.
Frequently Asked Questions
Can LLMs run completely offline without internet access?
Yes. Open-weight models like Llama 3.3 and Qwen 2.5 run entirely offline once the weights, serving software, and all dependencies are transferred inside the network. Nothing about inference requires connectivity. The practical challenges are logistical: building complete offline package repositories, disabling hub and telemetry callbacks (HF_HUB_OFFLINE=1), and establishing a verified media-transfer procedure for models and patches. Properly prepared, an air-gapped LLM stack is operationally identical to a connected one.
How do you update AI models in an air-gapped environment?
Through a versioned release train: a connected staging environment assembles a bundle - new model weights with SHA-256 manifests, rebuilt container images, updated packages - which is scanned, transferred on approved media under a two-person rule, and imported as an immutable release. Quarterly cadence is typical, with an emergency path for critical CVEs. The enclave keeps the previous two releases staged for instant rollback, and every transfer is logged for assessment evidence.
Is an air gap required for CMMC compliance?
No. CMMC 2.0 Level 2 requires the 110 NIST SP 800-171 controls, which can be met on connected networks with proper boundary protection, and Level 3 adds enhanced controls - still without mandating disconnection. Air gaps appear as program-specific requirements: classified work, certain ITAR enclaves, or customer flow-downs. Many manufacturers instead run a connected but egress-denied AI enclave, which achieves most isolation benefits while keeping updates and monitoring far simpler.
Key Takeaways
- 1Build a Complete Offline Artifact Pipeline: Everything the stack needs must cross the air gap as verified artifacts: model weights, container images, Python wheels, CUDA drivers, and OS packages. The working pattern is a connected staging environment that assembles a versioned bundle - model safetensors with SHA-256 manifests, a private container registry export (docker save or a Harbor replication bundle), a pip wheelhouse built with pip download for the exact Python and CUDA versions, and offline apt/dnf mirrors.
- 2Choose Software That Is Genuinely Offline-First: Many AI tools fail closed networks in subtle ways: Hugging Face libraries attempt hub connections unless HF_HUB_OFFLINE=1 is set, some orchestration frameworks fetch prompt templates remotely, and commercial tools with license servers simply stop working. Standardize on components proven offline: vLLM or llama.cpp for serving, Qdrant or pgvector for retrieval, and open-source orchestration you can vendor entirely.
- 3Model Updates and Patch Cadence Without a Connection: Air-gapped does not mean frozen. Establish a quarterly update train: each cycle, the staging environment assembles a new bundle - updated model versions, patched containers rebuilt against current CVE scans, and refreshed vector-store software - which crosses the gap through the same verified transfer procedure.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
Air-Gapped AI Readiness Assessment
A 10-question assessment that scores how prepared your organization is to deploy and operate LLMs inside an air-gapped or classified enclave.
Free ToolOn-Prem AI Security Hardening Checklist
A practical control checklist for securing self-hosted language models, covering model provenance, network isolation, data governance, host hardening, and audit readiness.
Free ToolSovereign AI Readiness Assessment
Score your organization across eleven dimensions of sovereign AI readiness, from data residency and model provenance to cleared personnel and air-gapped operations.
Terms used in this article
If your program requires AI behind an air gap, start with Netray's transfer-pipeline blueprint instead of learning the failure modes yourself.
Related Resources
On-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
AI & AutomationSecurity Hardening for On-Prem AI Stacks
Security hardening for on-prem AI stacks: container isolation, model supply chain, prompt injection defenses, and audit logging for defense contractors.
AI & AutomationA Data Governance Framework for Private AI
A practical data governance framework for private AI: classification, access controls, retention, and audit patterns that satisfy CMMC 2.0 and ITAR.