Model Distillation for Production: Shrinking Models Without Losing Quality
Distillation trains a smaller student model to reproduce a larger teacher model's behavior on your specific task, and it is the most reliable path to cutting serving cost and latency once a large frontier or open-weight model has already proven a use case works. The practical version most enterprise teams use is sequence-level distillation: generate a large volume of teacher outputs on task-representative prompts, filter them for quality, and then run supervised fine-tuning (and often a DPO pass) on a much smaller student model using those outputs as training data. Done well, distillation can cut per-token serving cost 5 to 20 times while preserving 90 percent or more of task-specific quality, because the student only needs to match the teacher's behavior on your narrow task distribution, not its full general capability.
Why Distill Instead of Fine-Tuning the Small Model Directly
Fine-tuning a small model directly on your own labeled data works when you already have a large, high-quality labeled dataset, but most enterprise teams do not have that at project start, they have a working prompt against a large model and a business case to serve it cheaper. Distillation lets you generate the training data the small model needs using the large model itself as the labeling engine, which is both faster and often higher quality than what a team could hand-label at the same volume, especially for tasks requiring multi-step reasoning where the teacher's intermediate reasoning trace can be captured as part of the training example, not just the final answer.
Generating Distillation Data: Volume, Diversity, Filtering
Generate teacher outputs across a prompt set that mirrors your actual production input distribution, not a narrow set of clean examples, since the student inherits whatever gaps exist in the generation set. Sample the teacher at a moderate temperature (0.5 to 0.8) with multiple completions per prompt to introduce useful diversity, then filter aggressively: remove outputs where the teacher itself expressed low confidence or hedged, remove outputs that fail a rule-based or format check, and run a judge pass (often the teacher model itself or a comparably strong model) to score quality and discard the bottom tier. Target 10,000 to 100,000 generated examples depending on task complexity, with a strong bias toward filtering quality over raw volume.
- Sample from a prompt distribution matching real production traffic, not a curated clean subset
- Moderate temperature (0.5 to 0.8) with multiple completions per prompt for useful diversity
- Filter out low-confidence, hedged, and format-invalid teacher outputs before training
- Run a judge pass to score and discard the lowest-quality tier of generated examples
Sequence-Level vs Logit-Level Distillation
Sequence-level distillation, training the student on the teacher's final text output, works with any teacher including closed commercial APIs and is the practical default for enterprise use. Logit-level distillation, matching the student's output probability distribution to the teacher's at each token position, transfers more information per example and can produce a better student for the same data volume, but it requires white-box access to the teacher's logits, which limits it to open-weight teachers you control the inference stack for. For most enterprise projects distilling from a large open model like Llama or Qwen down to a smaller variant in the same family, logit-level distillation is worth the extra engineering; for projects distilling from a closed API, sequence-level is the only option available.
Measuring Distillation Success the Right Way
Judge the student against the teacher on your task-specific held-out evaluation set, not against general benchmarks like MMLU, since a distilled student will almost always score worse on broad general knowledge while matching or nearly matching the teacher on the narrow task it was distilled for, and general benchmark scores tell you nothing useful about that gap. Track the specific quality delta on your task metric alongside the latency and cost improvement, and set an explicit acceptance threshold before starting, such as within 5 percentage points of teacher accuracy at one-tenth the serving cost, so the project has a clear pass criterion rather than an open-ended optimization.
Real Serving Cost Savings by Model Size
Serving cost per million tokens scales roughly with model size and active parameters, so distilling a 70B teacher down to an 8B student typically cuts inference cost 5 to 10 times on the same hardware, and distilling down to a 3B student for a narrow, well-defined task can push savings past 15 times while running comfortably on hardware that could not serve the teacher at all. The comparison that matters for a business case is total cost of ownership at your actual production volume: a distilled 8B model serving 50 million tokens a day on a single mid-range GPU versus the same volume on a 70B model requiring multiple high-end GPUs is frequently the difference between a project that pays back in months and one that never clears its hosting bill.
Frequently Asked Questions
How much does distillation reduce LLM serving cost?
Distilling a large teacher model down to a much smaller student typically cuts per-token serving cost 5 to 20 times, depending on the size gap and how narrow the target task is. Distilling a 70B model down to 8B commonly saves 5 to 10 times on serving cost, while distilling to a 3B model for a well-defined narrow task can exceed 15 times, provided the task does not require the teacher's broad general knowledge.
How much training data do I need for distillation?
Most enterprise distillation projects generate 10,000 to 100,000 examples from the teacher model, with the right number depending on task complexity. Prioritize filtering quality over raw volume: aggressively remove low-confidence, hedged, or format-invalid teacher outputs before training the student, since a smaller cleaned set consistently outperforms a larger unfiltered one.
Should I compare a distilled model against general benchmarks?
No. Judge the distilled student against the teacher on your task-specific held-out evaluation set, not general benchmarks like MMLU. A distilled student will almost always score worse on broad general knowledge while matching the teacher closely on the narrow task it was distilled for, and general benchmark scores do not reflect that gap accurately.
What is the difference between sequence-level and logit-level distillation?
Sequence-level distillation trains the student on the teacher's final text output and works with any teacher, including closed commercial APIs. Logit-level distillation matches the student's token probability distribution to the teacher's at each position, transferring more information per example, but requires white-box access to teacher logits, limiting it to open-weight teachers you control the inference stack for.
Key Takeaways
- 1Why Distill Instead of Fine-Tuning the Small Model Directly: Fine-tuning a small model directly on your own labeled data works when you already have a large, high-quality labeled dataset, but most enterprise teams do not have that at project start, they have a working prompt against a large model and a business case to serve it cheaper. Distillation lets you generate the training data the small model needs using the large model itself as the labeling engine, which is both faster and often higher quality than what a team could hand-label at the same volume, especially for tasks requiring multi-step reasoning where the teacher's intermediate reasoning trace can be captured as part of the training example, not just the final answer..
- 2Generating Distillation Data: Volume, Diversity, Filtering: Generate teacher outputs across a prompt set that mirrors your actual production input distribution, not a narrow set of clean examples, since the student inherits whatever gaps exist in the generation set. Sample the teacher at a moderate temperature (0.5 to 0.8) with multiple completions per prompt to introduce useful diversity, then filter aggressively: remove outputs where the teacher itself expressed low confidence or hedged, remove outputs that fail a rule-based or format check, and run a judge pass (often the teacher model itself or a comparably strong model) to score quality and discard the bottom tier.
- 3Sequence-Level vs Logit-Level Distillation: Sequence-level distillation, training the student on the teacher's final text output, works with any teacher including closed commercial APIs and is the practical default for enterprise use. Logit-level distillation, matching the student's output probability distribution to the teacher's at each token position, transfers more information per example and can produce a better student for the same data volume, but it requires white-box access to the teacher's logits, which limits it to open-weight teachers you control the inference stack for.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
AI Model Selection Assessment
Score ten decision factors - data sensitivity, task complexity, volume, latency, and internal capability - to see whether a self-hosted open-weight model fits your workload.
Free ToolModel Distillation Savings Calculator
Compare the ongoing cost of calling a large teacher model API against the one-time cost of distilling and self-hosting a small student model.
Free ToolFrontier vs Open Model Gap Assessment
Score your use case across reasoning complexity, data control, cost pressure, and customization needs to see whether a large open model closes the gap to a frontier model for you.
Terms used in this article
Have a use case validated on a large model but priced out of production at that model's serving cost? Netray builds distillation pipelines that generate, filter, and train the student model entirely on your own infrastructure.
Related Resources
The Real Cost of LLM Fine-Tuning in 2026: A Full Breakdown
The real cost of LLM fine-tuning in 2026: GPU-hour pricing by method, data preparation labor, evaluation cost, and the hidden line items teams miss.
AI & AutomationSynthetic Data Generation for Fine-Tuning: A Practical Guide
Synthetic data generation for fine-tuning: self-instruct pipelines, judge-based filtering, model collapse risk, and cost versus human labeling.
AI & AutomationHow to Evaluate a Fine-Tuned Model Before Production
Evaluate a fine-tuned model before production: held-out eval sets, task-specific metrics, calibrated LLM-as-judge setups, and regression testing.