Self-Hosted LLM Hardware Estimator: VRAM, GPUs, and Budget
This estimator computes the hardware you need to self-host an open-weight LLM - total VRAM, GPU count, and a realistic server budget - for platform teams planning an on-prem deployment. The most common sizing mistake is budgeting for model weights alone and forgetting the KV cache, the per-session attention memory that grows linearly with concurrent users and context length and frequently exceeds the weights themselves. Enter your model, precision, concurrency, and context window, and the tool produces a defensible hardware spec you can take to procurement.
Your numbers
Parameter count of the open-weight model you plan to host.
Bytes per parameter; INT8/FP8 is the enterprise default in 2026.
Simultaneous active conversations the server must hold in memory.
Longer contexts inflate KV cache memory linearly.
Your results
Estimates only. KV cache varies with attention architecture (GQA/MLA), and vendor pricing moves quarterly. Confirm with a proof-of-concept deployment before purchasing.
Get your full hardware sizing report
We will email a personalized hardware specification with current vendor pricing for your model and workload, and a Netray specialist will follow up to walk through procurement options.
No spam. Your results stay private. Unsubscribe anytime.
The methodology: weights plus KV cache plus overhead
Weights memory is parameters times bytes per parameter: a 32B model at INT8 needs 32 GB. KV cache is where deployments blow their budgets - each active session holds attention keys and values for every token in its context, and the estimator models this at roughly 2 MB per 1K tokens per billion parameters for modern grouped-query-attention (GQA) models. At the defaults - 25 concurrent sessions at 32K context on a 32B model - that is about 51 GB of cache, larger than the weights. We add 10% for the serving runtime and CUDA overhead, then divide by usable VRAM (90% of each 80GB card) to get GPU count.
Benchmark assumptions behind the numbers
The defaults reflect 2025-2026 open-weight architectures and street pricing. Older non-GQA models cache roughly 4x more per token, and newer MLA-style attention caches substantially less - so treat the estimate as a mid-range planning figure.
- GQA KV cache: ~2 MB per 1K context tokens per billion parameters at FP16 cache precision
- Usable VRAM per card: 90% of nominal, to avoid fragmentation OOMs
- 80GB-class GPU street price: ~$32K including vendor margin
- Base server platform (CPU, RAM, NVMe, NIC): ~$18K per node
How to shrink the spec if the number shocks you
Three levers cut hardware requirements fastest. First, quantize: INT8 serving is effectively lossless for most enterprise tasks and halves weight memory versus FP16; INT4 halves it again with a small, measurable quality cost. Second, cap the context window - most chat sessions never exceed 8K tokens, so provisioning every session for 128K wastes enormous cache memory; serving stacks can enforce per-request limits. Third, question concurrency: sessions idle between turns can have their cache evicted and recomputed, so the number that matters is sessions actively generating, which is usually far below open connections.
From estimate to running hardware with Netray
Netray specifies, procures, and deploys self-hosted LLM servers for aerospace, defense, and discrete manufacturers - from single-node pilots to multi-node air-gapped clusters. We validate this estimate against your real traffic with a short proof of concept, negotiate hardware through our vendor relationships, and deliver a hardened serving stack with quantized models, monitoring, and failover configured. Clients typically go from estimate to production inference in six to ten weeks.
Frequently Asked Questions
Why is the KV cache sometimes bigger than the model itself?
Because it scales with users and context, not just model size. Weights are fixed, but every concurrent session stores keys and values for each token in its context window. Twenty-five sessions at 32K tokens is 800K cached token positions - on a 32B model that is roughly 50 GB even with grouped-query attention. This is why long-context enterprise deployments need far more VRAM than a quick weights-only calculation suggests.
Can I use cheaper 24GB or 48GB GPUs instead of 80GB cards?
Often, yes - especially for models at 32B and below with INT4/INT8 quantization. Four 48GB L40S cards cost roughly half of two H100s while offering similar total VRAM, at the price of lower memory bandwidth and slower token generation. For latency-tolerant internal workloads that trade is frequently worth it. Divide the estimator's total VRAM figure by 90% of your candidate card's memory to get the alternative count.
Does this estimate cover fine-tuning hardware?
No - it sizes inference serving only. LoRA-style fine-tuning can usually run on the same class of hardware with reduced concurrency, but full fine-tuning requires roughly 8-10x the weight memory for gradients and optimizer states. Most Netray clients fine-tune periodically on a separate schedule (or in a secure cloud burst where policy allows) and deploy the resulting adapters onto the inference hardware this tool estimates.
Get your hardware spec, then let Netray validate it with a proof of concept before you commit capex.
Related Tools
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
On-Prem AIOn-Prem LLM Total Cost of Ownership Calculator
Model the full multi-year cost of running LLMs on your own hardware, including GPU capex, power, cooling, support contracts, and operations staffing.
On-Prem AIOpen-Weight Model Selector
A 10-question assessment that matches your hardware budget, workload complexity, and operational maturity to the right open-weight model size class.
Go Deeper
Enterprise GPU Cluster Planning for AI Workloads
Plan an enterprise GPU cluster for AI workloads: H100 vs L40S sizing, networking, power, cooling, and cost models for on-prem LLM inference and training.
On-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.