The IT Director Playbook for On-Prem AI: Hardware, Models, Security, Rollout
Running on-prem AI as an IT director means standing up four layers: GPU inference hardware, a serving stack, a model portfolio, and the security controls that make auditors comfortable. The good news is the stack has standardized. A single server with two NVIDIA L40S or RTX 6000 Ada GPUs, vLLM serving a quantized Llama 3.3 70B, an OpenAI-compatible API endpoint, and SSO-gated access covers the majority of enterprise workloads for a 200-2,000 employee manufacturer. This playbook covers sizing, model choice, hardening, and a rollout sequence that avoids the common failure of infrastructure without adoption.
Hardware Sizing: What to Actually Buy
Size for concurrent users and model class, not peak theoretical demand. For most mid-market manufacturers, one inference server handles company-wide chat, RAG, and agent workloads. A dual L40S configuration (96GB combined VRAM) serves a 4-bit quantized 70B model with continuous batching to 100-300 concurrent users at acceptable latency. Smaller task models - Llama 3.1 8B or Qwen 2.5 14B for classification and extraction - run alongside on spare VRAM or a second modest GPU.
- Entry tier ($15K-$25K): single RTX 6000 Ada 48GB - runs 8B-32B models, good for pilots and agents
- Standard tier ($35K-$50K): dual L40S server - quantized 70B models for company-wide inference
- Performance tier ($250K-plus): H100 or H200 nodes - only justified for fine-tuning or very high concurrency
- Always spec 2x NVMe storage headroom for model weights, embeddings, and vector indexes
The Serving Stack and Model Portfolio
Standardize on vLLM as the inference server exposing an OpenAI-compatible API - it gives you continuous batching, paged attention, and drop-in compatibility with virtually every AI framework and client library. Ollama is fine for developer laptops but not for shared production serving. Run a small portfolio rather than one model: a 70B generalist for chat and drafting, an 8B-14B workhorse for high-volume extraction and routing, and a dedicated embedding model such as BGE-M3 for retrieval. Quantize with AWQ or GPTQ to 4-bit for the large model; quality loss is under 2 percent on enterprise tasks while VRAM drops by roughly three quarters. Version-pin everything and maintain an evaluation set of 100-300 real prompts from your business so model upgrades are gated by measured regression testing, not release-note optimism.
Security Hardening for Compliance Environments
Treat the inference endpoint like any tier-1 internal application, then add AI-specific controls. For defense contractors, on-prem inference keeps CUI inside the CMMC 2.0 assessed boundary - but only if the surrounding controls hold up to NIST SP 800-171 scrutiny.
- Gate access via SSO/OIDC with group-based entitlements; no shared API keys in scripts
- Log every prompt, retrieval, and agent action to your SIEM with user identity attached
- Network-segment the GPU server; block all outbound internet so models cannot exfiltrate even if compromised
- Apply role-based filtering at the retrieval layer so RAG never surfaces documents a user cannot already open
Rollout Sequence and Where Netray Fits
The standard failure mode is infrastructure without adoption: a GPU server that hums at 5 percent utilization because nothing useful was built on it. Sequence rollout in 30-day phases - internal chat and document Q&A first, RAG over controlled document sets second, ERP-connected agents third - each phase with named users and measured usage targets. Netray delivers this playbook as a service: we spec and stage the hardware, deploy the hardened vLLM stack, integrate SSO and SIEM logging, and build the first ERP-connected agents against SyteLine, LN, or M3. Typical engagements reach production in 6-8 weeks with 60-plus percent weekly active usage by month three, and your team receives full runbooks and admin training so you own the stack, not us.
Frequently Asked Questions
What hardware do you need to run an LLM on-premises?
For a mid-market company, a single server with two NVIDIA L40S GPUs (96GB combined VRAM, roughly $35K-$50K installed) serves a 4-bit quantized 70B model to 100-300 concurrent users via vLLM. A single RTX 6000 Ada at $15K-$25K handles 8B-32B models for pilots and agent workloads. H100-class nodes at $250K-plus are only necessary for fine-tuning at scale or very high concurrency.
What is the best software stack for on-prem AI?
The 2026 standard is vLLM serving quantized open-weight models (Llama 3.3 70B, Qwen 2.5) behind an OpenAI-compatible API, with SSO-gated access, SIEM logging, and a vector database for retrieval. Use AWQ or GPTQ 4-bit quantization to fit large models on mid-range GPUs. Keep Ollama for developer experimentation only. Maintain an evaluation set of real business prompts so model upgrades are regression-tested before deployment.
Is on-prem AI compliant with CMMC 2.0?
On-prem AI is the most defensible architecture for CMMC 2.0 because prompts and data never leave your assessed boundary. Compliance still depends on surrounding controls mapped to NIST SP 800-171: SSO-based access control, full audit logging of prompts and agent actions, network segmentation with blocked outbound traffic from the GPU server, and role-based retrieval filtering so CUI is only surfaced to authorized users.
Key Takeaways
- 1Hardware Sizing: What to Actually Buy: Size for concurrent users and model class, not peak theoretical demand. For most mid-market manufacturers, one inference server handles company-wide chat, RAG, and agent workloads.
- 2The Serving Stack and Model Portfolio: Standardize on vLLM as the inference server exposing an OpenAI-compatible API - it gives you continuous batching, paged attention, and drop-in compatibility with virtually every AI framework and client library. Ollama is fine for developer laptops but not for shared production serving.
- 3Security Hardening for Compliance Environments: Treat the inference endpoint like any tier-1 internal application, then add AI-specific controls. For defense contractors, on-prem inference keeps CUI inside the CMMC 2.0 assessed boundary - but only if the surrounding controls hold up to NIST SP 800-171 scrutiny..
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
On-Prem AI Security Hardening Checklist
A practical control checklist for securing self-hosted language models, covering model provenance, network isolation, data governance, host hardening, and audit readiness.
Free ToolOn-Prem vs Cloud AI Cost Comparison Calculator
Compare your projected cloud LLM API bill against an amortized on-prem GPU deployment and find the breakeven month.
Free ToolSelf-Hosted LLM Hardware Estimator
Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.
Terms used in this article
Download nothing, guess nothing - book a Netray on-prem AI architecture session and get a sized bill of materials and rollout plan for your environment.
Related Resources
Why On-Prem AI Is Back in 2026
On-prem AI is back in 2026 as data sovereignty, CMMC 2.0, and GPU economics shift the math. Why manufacturers are moving LLMs behind the firewall.
AI & AutomationDefense CIO AI Briefing: 2026 Edition
Defense CIO AI briefing for 2026: CMMC 2.0, ITAR, and DFARS constraints on AI, plus how defense contractors deploy LLMs without risking CUI exposure.
AI & AutomationAn AI Governance Framework for Manufacturers
An AI governance framework for manufacturers: policies, model risk controls, and audit trails that satisfy AS9100D, CMMC 2.0, and customer flow-downs.