RAG Infrastructure Sizing Calculator: Vectors, RAM, and GPUs
This calculator sizes the infrastructure behind an enterprise retrieval-augmented generation (RAG) deployment - vector index storage, node RAM, generation GPU count, and a monthly cost estimate - for teams planning to put an LLM over their document corpus. RAG sizing has two halves that scale independently: the retrieval side grows with corpus size, while the generation side grows with query volume. Enter your document counts, embedding model, and daily query load, and the tool produces a node-level specification you can take into architecture review.
Your numbers
Total documents to index: manuals, drawings metadata, SOPs, contracts, tickets.
A page is roughly 500 words; long manuals raise this quickly.
RAG questions asked across all users and automations daily.
Vector width of your embedding model; higher is more accurate but heavier.
Production deployments should run at least an HA pair.
Your results
Estimates only. Chunking strategy, quantized vectors, and disk-backed indexes can change these numbers significantly. Validate with a pilot index over a corpus sample.
Get your full RAG sizing report
We will email a personalized infrastructure specification with embedding and chunking recommendations for your corpus, and a Netray RAG specialist will follow up to plan a pilot index.
No spam. Your results stay private. Unsubscribe anytime.
The sizing methodology
Documents become chunks: at roughly 500 words per page and 400-500 token chunks with overlap, each page yields about two vectors. Each vector consumes embedding dimensions times four bytes as float32, and production HNSW indexes plus metadata add roughly 50% on top - so 250,000 eight-page documents at 1,024 dimensions produce about 4 million chunks and a 23 GB index. Because HNSW performs best fully in memory, node RAM is sized at 125% of the index plus a 16 GB base, with a 32 GB floor. Generation GPUs are sized from daily queries at about 800 output tokens each, a 3x peak-hour factor, and 600 tokens per second per GPU - which is why most mid-size corpora need exactly one generation GPU.
Benchmarks the defaults are built on
The defaults model a realistic mid-market manufacturing knowledge base: engineering specs, quality procedures, maintenance manuals, and contracts. Numbers worth knowing when you sanity-check your own inputs:
- Chunking yield: ~2 chunks per page at 400-500 tokens with 10-15% overlap
- Float32 vector cost: 4 KB per chunk at 1,024 dims before index overhead
- HNSW overhead: 40-60% above raw vector bytes for links and metadata
- A 4M-chunk index serves sub-100ms retrieval comfortably on a 64 GB node
How to interpret and refine the result
If the vector index lands under 50 GB, a simple HA pair of general-purpose nodes handles retrieval and your cost center is the generation GPU - which is the common case and good news, since GPU capacity is reusable across use cases. If the index exceeds a few hundred gigabytes, before buying bigger nodes consider scalar or binary quantization (2-4x smaller with minor recall loss), disk-backed indexes like DiskANN, or trimming the corpus - most enterprises find 20-40% of indexed documents are obsolete duplicates. Also stress-test query volume honestly: agentic workflows that fire multiple retrievals per user question can multiply effective load 3-10x.
How Netray builds production RAG on these numbers
Netray designs and deploys on-prem RAG systems for manufacturers - typically over engineering documents, quality records, and ERP data from SyteLine or LN environments. We validate this sizing with a pilot index over a corpus sample, choose the embedding model and chunking strategy against your actual retrieval accuracy targets, and deliver the full stack: ingestion pipelines, vector database, reranking, generation serving, and evaluation harness. Air-gapped variants for defense programs are a specialty.
Frequently Asked Questions
Why does the calculator assume vectors live in RAM?
Because HNSW, the dominant production index type, degrades sharply when its graph pages to disk - latency jumps from milliseconds to seconds under load. For corpora up to a few tens of millions of chunks, RAM-resident indexes are the reliable default and RAM is cheap relative to the GPU side. At larger scales, disk-backed designs like DiskANN or quantized indexes change the economics, which is exactly the kind of threshold decision worth an architecture review.
Does query volume really only need one GPU in most cases?
Usually, yes - and it surprises people. Two thousand queries a day at 800 output tokens averages under one token per second of sustained generation; even with a 3x peak factor, a single modern GPU serving a quantized 14B-32B model has ample headroom. Query volume starts demanding multiple GPUs around 20,000-50,000 daily queries, or much earlier if agentic pipelines chain several LLM calls per user question.
What drives RAG answer quality more: the corpus size or the model?
Neither - retrieval quality does. In Netray evaluations, chunking strategy, embedding model choice, metadata filtering, and reranking explain most of the gap between disappointing and excellent RAG systems, while swapping a 14B for a 70B generation model moves scores far less. That is why we size infrastructure with modest generation GPUs and invest the savings in ingestion quality, hybrid search, and a proper evaluation harness with a gold question set.
Size your RAG stack, then have Netray validate retrieval quality with a pilot index over a sample of your real corpus.
Related Tools
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
On-Prem AISelf-Hosted LLM Hardware Estimator
Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.
On-Prem AIOn-Prem AI Deployment Checklist
A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.
Go Deeper
On-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
Enterprise GPU Cluster Planning for AI Workloads
Plan an enterprise GPU cluster for AI workloads: H100 vs L40S sizing, networking, power, cooling, and cost models for on-prem LLM inference and training.