The LLM Inference Cost Optimization Playbook

Step One: Right-Size the Model Before Optimizing Serving

The single biggest cost lever is using a smaller model that meets the task's actual quality bar, and it is the step most teams skip because a large frontier-class model is the safe default choice during prototyping. A 7B to 14B class model handling classification, extraction, or structured summarization at production scale, quantized and well-served, can cost an order of magnitude less per token than routing the same task to a 70B or larger model, often with no measurable quality difference on the specific task once you have evaluated it properly. Build a task-specific eval set, test model size down from your current default, and only pay for a bigger model where the eval set proves you need it. This single step frequently dwarfs every serving optimization combined.

• Build a task-specific eval set before assuming you need your current model size
• Test smaller models (7B-14B class) against the eval set for classification, extraction, and structured tasks specifically
• Route by task complexity rather than defaulting every request to your largest deployed model
• Revisit model choice quarterly; the smaller-model quality bar keeps rising as new releases land

Step Three: Batching, Prefix Caching, and Request Shaping

Continuous batching (on by default in vLLM and SGLang) already captures most of the batching benefit automatically, but concurrency limits still need tuning against your latency SLA. Prefix caching, covered in detail in our KV cache optimization guide, can meaningfully cut compute cost for workloads with shared system prompts or RAG templates, sometimes by 20 to 40 percent on the cache-hit portion of traffic, at zero cost beyond restructuring prompts to put static content first. On the application side, request shaping matters too: capping unnecessary max-token generation limits, using structured output schemas to avoid verbose free-text responses, and batching non-latency-sensitive background jobs into off-peak windows all reduce total token volume without touching the serving stack at all.

• Tune concurrency (--max-num-seqs) against measured latency SLA, not a throughput benchmark alone
• Restructure prompts static-content-first to capture prefix caching savings on repeated system prompts
• Cap max-token generation limits to actual task needs; unconstrained limits silently inflate cost on verbose responses
• Move non-interactive batch jobs (summarization backlogs, embedding regeneration) to off-peak scheduling

How Netray Runs Cost Optimization Engagements

Netray works this playbook in order for every on-prem inference engagement: model right-sizing against a client-specific eval set first, then quantization and vLLM config tuning, then prefix caching and request shaping, with speculative decoding evaluated last and only where the concurrency profile supports it. We build the on-prem versus API breakeven model against a client's actual measured token volume and utilization pattern rather than a generic industry rule of thumb, and we have seen the honest answer go either way: some clients are better served staying on API pricing at their current volume, and we say so. The clients who do move to on-prem inference typically see 60 to 85 percent cost-per-token reduction after the full optimization sequence, measured against their pre-optimization baseline.

Frequently Asked Questions

Put this into numbers

Free interactive tools for exactly this problem. No signup to use them.