AI & Automation4 min readNetray Engineering TeamUpdated March 4, 2025

Running LLMs in Classified and Air-Gapped Environments: A Field Guide

Running LLMs in classified and air-gapped environments means deploying open-weight or government-authorized models on hardware with no external network path, where every input, weight file, and update crosses an accredited transfer process. It is entirely feasible in 2026: open-weight models like Llama 3.1 70B run on commodity GPU servers, and the DoD has authorized cloud AI up to IL6 in Microsoft Azure Government Secret regions. The hard problems are not model quality but logistics: getting weights across the air gap, sizing hardware without cloud elasticity, and satisfying the accreditation authority. This guide covers the patterns that work.

The Deployment Landscape: On-Prem, IL5, IL6, and True Air Gap

There are three distinct scenarios that people conflate. First, connected-but-controlled: CUI enclaves at defense contractors, governed by CMMC and NIST SP 800-171, where on-prem inference with tightly restricted egress suffices. Second, classified cloud: DoD Impact Level 5 (CUI and national security systems) and IL6 (Secret) regions, where Azure OpenAI variants and AWS Secret Region services now hold authorizations, useful for programs already operating there. Third, true air gap: SCIFs and closed research networks with zero external connectivity, where everything, model weights, serving software, GPU drivers, embedding models, must arrive via approved media transfer under the facility's cross-domain and media control procedures. Architecture and cost differ sharply across the three; scoping the wrong one wastes months.

Hardware Sizing Without Cloud Elasticity

Air-gapped deployments cannot burst to the cloud, so size for peak concurrency up front. Rules of thumb from production deployments: a 4-bit-quantized 70B model needs roughly 48-64 GB of VRAM and serves 5-15 concurrent users acceptably on 2x NVIDIA L40S; an 8B model runs on a single 24 GB card and handles extraction and classification at high throughput.

Pilot tier: one server, 2x L40S (96 GB VRAM total), 70B quantized model, 5-15 users, roughly $35,000-$50,000
Production tier: 2 nodes with 4x L40S or 2x H100 each for redundancy and 30-60 users, $120,000-$250,000
Add 10-20 percent VRAM headroom for embedding models, rerankers, and KV-cache growth at long context
Plan physical logistics early: power (2-4 kW per GPU node), cooling, and rack space inside the accredited facility

Getting Models and Updates Across the Air Gap

Model weights are large (a 70B model is 40-140 GB depending on quantization) and update cadence matters, so define the transfer pipeline as a first-class process, not an afterthought. The defensible pattern mirrors existing software supply chain controls: download on a connected staging system, verify checksums against publisher-signed hashes, scan per your media protection procedures, then transfer via approved one-way media or data diode.

Verify SHA-256 hashes of weight files against the publisher's signed manifest before and after transfer
Mirror the full dependency chain offline: vLLM, CUDA, drivers, and Python packages via an internal repository
Batch updates quarterly; chasing every model release across an air gap burns accreditation goodwill
Document the pipeline in your SSP or accreditation package as part of configuration and media management

Accreditation: What Your ISSM and AO Will Ask

Whether the authority is a C3PAO, an ISSM under NISPOM (32 CFR Part 117), or a government Authorizing Official running RMF, the questions converge. Where do prompts and outputs persist, and at what classification? How is the model prevented from becoming an uncontrolled aggregation point, since retrieval indexes concentrate data that was previously compartmented? Who audits prompt logs, and how are they marked? Answer these by design: enforce source-document permissions in the retrieval layer so the model cannot surface content beyond a user's authorization, store logs at the enclave's classification with standard marking, and disable any telemetry or phone-home behavior in serving software. Deployments that treat the LLM as just another accredited application, with an SBOM, baseline configuration, and audit story, get approved; those pitched as magic boxes stall.

How Netray Supports Air-Gapped AI Programs

Netray delivers turnkey air-gapped AI stacks for defense manufacturers and cleared facilities: GPU hardware specification, offline installation media with hash-verified model weights and a fully mirrored dependency chain, permission-aware retrieval over ERP and document data, and the accreditation artifacts, SBOM, configuration baseline, data flow diagrams, and audit logging design, your ISSM or AO needs. Our stacks run SyteLine, Infor LN, and document intelligence workloads with zero external connectivity and quarterly offline update packages. Clients have taken Netray-built systems through NISPOM and CMMC reviews without AI-related findings, while giving engineers natural-language search over decades of controlled technical data for the first time.

Frequently Asked Questions

Can you run a large language model completely offline?

Yes. Open-weight models such as Llama 3.1 70B or Qwen 2.5 72B run fully offline on local GPU servers using serving software like vLLM, with no internet dependency at inference time. You download weights once on a connected staging system, verify hashes, transfer via approved media, and mirror all software dependencies internally. A quantized 70B model serves 5-15 concurrent users on roughly $35,000-$50,000 of GPU hardware.

What DoD impact level is required for AI on classified data?

Secret-level workloads require IL6-authorized environments, such as Azure Government Secret or AWS Secret Region, or accredited on-prem systems within a cleared facility under RMF. CUI and national security systems below Secret map to IL5. For contractor-owned air-gapped networks, the governing framework is NISPOM (32 CFR Part 117) with the facility ISSM and the cognizant Authorizing Official approving the system.

How do you update AI models in an air-gapped environment?

Through a controlled offline transfer pipeline: download weights and dependencies on a connected staging system, verify SHA-256 hashes against publisher-signed manifests, scan per media protection procedures, then move them via approved one-way media or a data diode. Production teams batch updates quarterly rather than chasing each release, and document the pipeline in the SSP or accreditation package as part of configuration management.

Key Takeaways

1The Deployment Landscape: On-Prem, IL5, IL6, and True Air Gap: There are three distinct scenarios that people conflate. First, connected-but-controlled: CUI enclaves at defense contractors, governed by CMMC and NIST SP 800-171, where on-prem inference with tightly restricted egress suffices.
2Hardware Sizing Without Cloud Elasticity: Air-gapped deployments cannot burst to the cloud, so size for peak concurrency up front. Rules of thumb from production deployments: a 4-bit-quantized 70B model needs roughly 48-64 GB of VRAM and serves 5-15 concurrent users acceptably on 2x NVIDIA L40S; an 8B model runs on a single 24 GB card and handles extraction and classification at high throughput..
3Getting Models and Updates Across the Air Gap: Model weights are large (a 70B model is 40-140 GB depending on quantization) and update cadence matters, so define the transfer pipeline as a first-class process, not an afterthought. The defensible pattern mirrors existing software supply chain controls: download on a connected staging system, verify checksums against publisher-signed hashes, scan per your media protection procedures, then transfer via approved one-way media or data diode..