Why your cost per token is too high, and the biggest levers to fix it
cost per million tokens on our own gpus is not much cheaper than a hosted api
Also appears as
- self-hosted inference is not saving money compared to an api provider
- gpu costs are high relative to the actual tokens being served
Short answer
Cost per token is dominated by GPU utilization far more than by hardware choice: an underutilized GPU serving one request at a time can cost more per token than a well-tuned smaller GPU serving at full continuous-batching concurrency. Before concluding self-hosting is not worth it, check utilization, whether the model is right-sized for the task, whether quantization and prefix/response caching are in use, and whether the comparison to an API is even apples-to-apples once amortization is accounted for.
Affects: Any organization running self-hosted LLM inference and comparing cost per token against a hosted API baseline
Attack the biggest lever first
- 1Measure current GPU utilization under real traffic; if it is well under 60-70 percent, fixing batching and concurrency is almost always the highest-leverage first step, cheaper than any hardware change.
- 2Confirm you are not running a much larger model than the task actually requires; check whether a smaller or distilled model meets your quality bar at a fraction of the cost per token.
- 3Apply quantization (FP8, AWQ, or GPTQ) if not already in use, which increases effective throughput per GPU by reducing memory bandwidth pressure.
- 4Enable prefix caching and, where applicable, response caching for repeated or templated queries, since a cache hit costs essentially nothing compared to a fresh generation.
- 5Recompute the cost comparison against a hosted API including full amortization (hardware, power, staff time, redundancy) over the hardware's realistic lifetime, not just a monthly cloud GPU rental rate.
How to confirm this is your problem
- Calculated cost per million tokens on owned or rented GPUs is close to or above API pricing
- GPU utilization metrics are low even though the cost analysis assumes full utilization
- The same task is being served by a much larger model than smaller alternatives that were never benchmarked
- No caching is in place for prompts or contexts that repeat frequently across requests
Root causes and fixes
Low GPU utilization inflates effective cost per token
A GPU costs roughly the same per hour whether it is serving one request or fifty concurrently, so cost per token is essentially (GPU cost per hour) divided by (tokens generated per hour); utilization is the direct multiplier on the denominator. A GPU running at 20 percent effective utilization has roughly five times the cost per token of the same GPU at full, well-batched utilization, independent of any other factor.
Fix: Fix continuous batching, concurrency, and max-num-seqs sizing (see the GPU utilization and batch size tuning pages) before considering any hardware or model change; this is almost always the single largest lever on cost per token.
Model is larger than the task requires
Cost scales strongly with model parameter count, both in memory bandwidth per token and in raw compute, so running a 70B model for a task a well-tuned 7B or 14B model could handle multiplies cost per token for no quality benefit actually being used.
Fix: Benchmark smaller or distilled models against your actual task and quality bar rather than defaulting to the largest available model, and right-size deliberately.
No quantization applied, leaving throughput per GPU below what the hardware supports
Full-precision or even FP16 weights consume more memory bandwidth per token than a quantized model, directly limiting how much concurrency and throughput a given GPU can sustain; quantization (FP8, AWQ, GPTQ) shrinks that footprint and typically raises achievable throughput per GPU meaningfully with limited quality cost when applied carefully.
Fix: Apply FP8 or a validated INT4 method (AWQ/GPTQ) and re-measure throughput per GPU, confirming quality holds using a task-specific eval, not just perplexity.
No caching for repeated prompts, system prompts, or templated queries
Workloads with a shared system prompt, RAG template, or frequently repeated queries pay full prefill and/or generation cost on every request even when a meaningful fraction of traffic is effectively duplicate or near-duplicate work, when prefix caching or a response cache could serve much of it far more cheaply.
Fix: Enable prefix caching for shared prompt structure, and add an explicit response cache keyed on normalized query content for genuinely repeated queries where fresh generation is not required.
Cost comparison against a hosted API does not account for full on-prem amortization
Comparing only a cloud GPU rental's hourly rate against API per-token pricing ignores that owned or reserved hardware amortizes differently over its lifetime, and also omits power, cooling, staff time, and redundancy costs that a fair comparison must include; conversely it can also omit that API pricing embeds a margin that scales with the same utilization dynamics.
Fix: Build the cost comparison using a full total-cost-of-ownership model over a realistic hardware lifetime and expected utilization, not a spot-rental hourly rate compared to a per-token API price in isolation.
Diagnostic commands
Compute effective cost per token from real utilization data
nvidia-smi --query-gpu=utilization.gpu --format=csv -l 60
Multiply GPU hourly cost by hours run, divide by actual tokens generated in that window; comparing this real number against a theoretical full-utilization estimate quantifies exactly how much utilization is costing you.
Check whether prefix or response caching is enabled
curl -s localhost:8000/v1/models | python -m json.tool
If caching is off and a meaningful share of traffic shares a system prompt or repeats queries, this is low-effort, high-leverage cost savings being left on the table.
Benchmark cost per token across model sizes for the actual task
python -c "print('compare tokens/sec and quality across model sizes on your eval set')"If a smaller model meets the task's quality bar at meaningfully lower cost per token, the largest lever may simply be right-sizing the model rather than any infrastructure change.
Stopping it from happening again
- Track cost per token as a dashboard metric derived from real utilization, not a theoretical full-utilization estimate
- Re-benchmark model size choices against task-specific evals periodically rather than assuming the largest model is always necessary
- Enable prefix and response caching by default for any workload with repeated structure
- Build TCO comparisons with full amortization whenever justifying self-hosting versus an API to stakeholders
When this becomes an architecture problem
If utilization, model right-sizing, quantization, and caching are all already optimized and cost per token is still not competitive with alternatives at your actual traffic volume, the remaining lever is fleet-level architecture: consolidating workloads onto fewer, better-utilized GPUs, renegotiating hardware acquisition strategy, or reconsidering whether self-hosting is the right choice for that specific workload's volume, which is a cost-architecture decision rather than a serving optimization.
Frequently asked questions
Is self-hosting always cheaper than an API per token?
Only above a certain utilization and volume threshold; at low utilization or low volume, the fixed cost of owned or reserved GPU capacity can exceed API pricing, since APIs effectively pool utilization across many customers. The crossover point depends heavily on your actual utilization, not just list GPU pricing.
What is the single biggest lever for reducing cost per token?
In the overwhelming majority of cases, GPU utilization. Fixing continuous batching and concurrency to move from a lightly-loaded server to a well-batched one has a bigger cost-per-token impact than switching hardware or even quantization, because it directly multiplies the tokens produced per GPU-hour.
Does quantization always reduce cost per token?
It usually does by increasing achievable throughput per GPU, but the actual savings should be validated against a task-specific quality eval, since a quantization method that degrades output quality enough to require retries or a larger fallback model can erase the cost benefit.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
LLM Inference Cost Per Million Tokens Calculator
Convert GPU hourly cost, cluster throughput, and real-world utilization into a defensible self-hosted cost per million tokens, exposing what idle GPU time actually costs you.
Free ToolLLM API vs Self-Hosted Cost Calculator
Model your API bill from requests and token mix, compare it against an all-in self-hosted monthly cost, and see monthly and annual savings.
Free ToolOn-Prem LLM Total Cost of Ownership Calculator
Model the full multi-year cost of running LLMs on your own hardware, including GPU capex, power, cooling, support contracts, and operations staffing.
Free ToolLLM Quantization Memory Savings Calculator
Compare FP16, FP8, and INT4 memory footprints for any model size and see how many fewer GPUs quantization requires to serve it.
Related problems
GPU utilization stays low during LLM inference even under load
Low GPU utilization during inference almost always means the GPU is waiting on something else: request concurrency is too low for the batching scheduler to fill, the client code is calling the server synchronously one request at a time, tokenization or network I/O is serialized in front of the GPU call, or max-num-seqs is set too low to admit enough concurrent sequences. Raising effective concurrency, either by fixing the client or the server's admission limits, is almost always the fix, not more GPU compute.
LLM inference is much slower in production than in benchmarks
Production inference is usually slower than a benchmark because real traffic exposes problems a single-request test never hits: full-precision weights instead of BF16/FP16, no continuous batching so requests queue one at a time, CPU-bound tokenization or post-processing sitting in front of the GPU call, or hardware whose memory bandwidth cannot keep up with the model size and concurrency you actually see. Fix the dtype and batching first, they account for most of the gap, then profile the request path for CPU-bound steps.
Model output quality dropped noticeably after quantization
Quality degradation after quantization usually comes from choosing too aggressive a quantization level for the model size and task, quantizing layers that are unusually sensitive to precision loss (often attention output projections and the final layers), or from trusting perplexity as the only quality signal when perplexity can look nearly unchanged while task-specific accuracy drops meaningfully. FP8 is close to lossless for most models and tasks, while INT4 methods carry real risk that must be validated with a task-specific eval set before shipping, not assumed safe from a perplexity number alone.
Not sure how to tune batch size for LLM inference throughput vs latency
Batch size is a direct tradeoff between throughput and per-request latency: larger batches keep the GPU busier and raise aggregate tokens-per-second, but each additional concurrent sequence adds contention for the same compute and memory, increasing the latency of every individual request. The right batch size is not the largest one that fits in memory, it is the point on that curve, the knee, where added throughput per unit of batch size starts costing more latency than your SLO allows, and it should be derived from measurement against your actual latency target, not a fixed default.
GuideThe LLM Inference Cost Optimization Playbook
Cut LLM inference costs with a practical playbook: quantization, batching, GPU right-sizing, caching, and the on-prem vs API breakeven math for 2026.
GuideLLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
GuideLLM Quantization: AWQ vs GPTQ vs FP8 vs GGUF
AWQ, GPTQ, FP8, and GGUF compared for production LLM serving: memory savings, throughput impact, quality loss, and which format fits which deployment.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.