RAG Cost Optimization: Where the Money Actually Goes
Most RAG cost optimization efforts target the wrong stage, because teams intuitively assume the vector database or the embedding calls are the expensive part when generation is almost always the dominant line item. A RAG query typically involves one cheap embedding call for the query itself, a retrieval and rerank step, and then a generation call carrying the full retrieved context as input tokens, and that generation call, especially with 5 to 10 retrieved chunks stuffed into context, usually accounts for 60 percent or more of per-query cost. Real cost optimization starts by measuring where spend actually concentrates in your system rather than assuming, then applying the specific lever that matches the bottleneck, since the fix for expensive generation is completely different from the fix for expensive vector storage.
Where RAG Spend Actually Concentrates
Generation tokens dominate because context stuffing, sending more retrieved chunks than the query actually needs, directly multiplies input token cost on every single query, and it is the easiest place for cost to creep upward unnoticed as teams add chunks to improve accuracy without measuring whether the extra chunks help. Reranking calls, if using an API-based cross-encoder, add a per-query cost proportional to how many candidates get reranked. Embedding calls are cheapest at query time, one embedding per query, but can be significant at ingestion time for large corpora processed all at once. Vector database compute and storage is usually the smallest line item unless the corpus is genuinely enormous, which is why optimizing it first is rarely where the money is.
Cutting Generation Cost Without Cutting Accuracy
Better retrieval precision, achieved through reranking, is the highest-leverage generation cost lever, because sending 5 highly relevant chunks instead of 15 loosely relevant ones cuts input tokens by two-thirds while typically improving faithfulness, not hurting it. Prompt caching, supported by major model providers, caches the static portion of a prompt, system instructions and any consistently repeated context, so repeated queries against the same context pay a fraction of the input token cost on cache hits. Model routing, sending straightforward factual queries to a smaller, cheaper model and escalating genuinely complex queries to a larger one, can cut generation cost substantially on corpora where query difficulty varies widely, provided the routing decision itself is cheap and reliable.
- Better reranking cuts chunks sent to generation, often reducing input tokens by half or more
- Prompt caching on static system instructions and repeated context cuts cost on cache hits
- Model routing sends easy queries to a cheaper model, reserving the expensive model for hard ones
Embedding and Reranking Cost Levers
Matryoshka-style embedding models let you truncate a full-dimension vector down to a smaller size for cheap first-pass retrieval and reserve full precision for the reranking stage, cutting both storage and query-time compute without a separate model. Batch embedding at ingestion time, rather than one document at a time, reduces API overhead substantially for large corpora. Self-hosting an open embedding model through vLLM or a similar serving stack removes per-call API cost entirely in exchange for GPU infrastructure cost, which becomes the better economics past a certain query volume, covered in the breakeven discussion below. Caching query embeddings for frequently repeated queries, common in enterprise search where the same handful of questions recur constantly, avoids redundant embedding calls entirely.
Vector Database Cost: Storage, Replicas, and Index Choice
Vector storage cost scales with dimension count and vector count, so a truncated embedding dimension directly reduces storage cost alongside its retrieval speed benefit. Scalar or binary quantization compresses stored vectors further, at a small recall cost that is often negligible when paired with reranking to correct for the precision loss. Tiered storage, keeping frequently queried recent documents in a fast in-memory index and archiving rarely queried older documents in cheaper storage with a slower retrieval path, matches infrastructure cost to actual access patterns rather than treating every document as equally hot. Right-size replica count to actual query throughput and availability requirements rather than defaulting to a high replica count that adds cost without a corresponding traffic need.
Self-Hosting vs API: The Breakeven Math
For high query volume corpora, self-hosting embedding, reranking, and generation on owned GPU infrastructure removes per-token API spend entirely, and the breakeven point where owned hardware costs less than API spend typically lands somewhere between several hundred thousand and a few million queries per month depending on model size and GPU pricing, a calculation worth running against your own volume rather than assuming either direction. On-premises deployment also removes the recurring cost uncertainty of API pricing changes and adds the separate, often decisive, benefit of keeping sensitive documents off third-party infrastructure entirely for regulated manufacturers.
How Netray Optimizes RAG Cost for Clients
Netray's DataRay platform includes a cost dashboard breaking down spend by pipeline stage, embedding, retrieval, reranking, and generation, so clients see exactly where money goes rather than optimizing blind. For high-volume corpora we run the breakeven analysis explicitly and, when the numbers support it, deploy the full pipeline on-premises on customer-owned GPUs, which removes per-token API spend entirely for the highest-volume clients while also satisfying the data residency requirements many of our manufacturing and defense clients carry independently of cost.
Frequently Asked Questions
What is the biggest cost driver in a typical RAG system?
Generation, not retrieval or vector storage, is almost always the dominant cost, since the generation call carries the full retrieved context as input tokens on every query. Sending more chunks than necessary, context stuffing, directly multiplies this cost. Improving retrieval precision through reranking so fewer, more relevant chunks reach generation is typically the single highest-leverage cost reduction available.
Does self-hosting embedding models actually save money?
It depends on query volume. Self-hosting removes per-call API cost entirely in exchange for GPU infrastructure cost, and the breakeven point typically falls somewhere between several hundred thousand and a few million queries per month depending on model size and hardware pricing. Below that volume, API-based embedding is usually cheaper once GPU amortization, power, and operational overhead are counted honestly.
How does prompt caching reduce RAG generation cost?
Prompt caching stores the static portion of a prompt, typically system instructions and any consistently repeated context, so that when the same static content appears again, the model provider charges a reduced rate for that cached portion rather than full input token price. For RAG systems that repeat a stable system prompt across many queries, this can meaningfully cut generation cost on high-volume traffic.
When does on-premises RAG deployment become cheaper than API-based RAG?
Once query volume crosses roughly several hundred thousand to a few million queries per month, owned GPU infrastructure running open-weight embedding, reranking, and generation models typically costs less than equivalent API spend, though the exact crossover depends heavily on model size and hardware pricing at the time. For regulated manufacturers, on-premises deployment is often chosen for data residency requirements independent of the cost comparison.
Key Takeaways
- 1Where RAG Spend Actually Concentrates: Generation tokens dominate because context stuffing, sending more retrieved chunks than the query actually needs, directly multiplies input token cost on every single query, and it is the easiest place for cost to creep upward unnoticed as teams add chunks to improve accuracy without measuring whether the extra chunks help. Reranking calls, if using an API-based cross-encoder, add a per-query cost proportional to how many candidates get reranked.
- 2Cutting Generation Cost Without Cutting Accuracy: Better retrieval precision, achieved through reranking, is the highest-leverage generation cost lever, because sending 5 highly relevant chunks instead of 15 loosely relevant ones cuts input tokens by two-thirds while typically improving faithfulness, not hurting it. Prompt caching, supported by major model providers, caches the static portion of a prompt, system instructions and any consistently repeated context, so repeated queries against the same context pay a fraction of the input token cost on cache hits.
- 3Embedding and Reranking Cost Levers: Matryoshka-style embedding models let you truncate a full-dimension vector down to a smaller size for cheap first-pass retrieval and reserve full precision for the reranking stage, cutting both storage and query-time compute without a separate model. Batch embedding at ingestion time, rather than one document at a time, reduces API overhead substantially for large corpora.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
RAG Context Window Budget Calculator
Allocate your context window across system prompt, retrieved chunks, and conversation history, then see window utilization and the real cost of every RAG query.
Free ToolLLM Inference Cost Per Million Tokens Calculator
Convert GPU hourly cost, cluster throughput, and real-world utilization into a defensible self-hosted cost per million tokens, exposing what idle GPU time actually costs you.
Free ToolLLM Token Cost Calculator
Turn request volume, prompt length, and per-million token pricing into a defensible monthly and annual LLM budget, including the effect of prompt caching.
Terms used in this article
Not sure where your RAG spend is actually going? Netray will break down your pipeline cost by stage and show you which lever actually moves the number.
Related Resources
Enterprise RAG Architecture: The Full 2026 Blueprint
A practitioner's blueprint for enterprise RAG in 2026: ingestion, chunking, embedding, retrieval, rerank, generation, and the eval loop that keeps it honest.
AI & Automationpgvector vs Dedicated Vector Databases: An Honest Comparison
pgvector vs Milvus, Qdrant, and Weaviate for enterprise RAG: real tradeoffs on scale, latency, operational overhead, and when Postgres is genuinely enough.
AI & AutomationAgentic RAG Patterns for Production Systems
Agentic RAG patterns: query rewriting, iterative retrieval, self-correction, and tool-calling agents that decide when and what to retrieve, not just once.