What Is Model Distillation?
Also known as: Knowledge Distillation, Teacher-Student Training
Definition
Model distillation is a training technique in which a small student model learns to reproduce the behavior of a larger teacher model, retaining most of the teacher's quality on a target task at a fraction of the size and serving cost.
Model Distillation Explained
Classical distillation trains the student on the teacher's full output probability distribution rather than only the single correct answer. That distribution carries more information than a hard label, because it reveals which alternatives the teacher considered plausible and by how much. The student learns not just what to answer but the shape of the teacher's uncertainty, which is why distillation transfers capability more efficiently than training the same small model on labeled data alone.
In practice with language models, a simpler variant dominates. The teacher generates responses across a large set of representative inputs, those responses are filtered for quality, and the student is fine-tuned on the resulting input-output pairs. This requires only access to the teacher's outputs rather than its internals, so it works with hosted models as well as local ones, subject to license terms. Many strong small open-weight models were produced this way.
The technique fits enterprise economics neatly because production tasks are narrow. You do not need a small model that matches a large one across all capabilities; you need one that matches it on classifying your service notes or extracting fields from your purchase orders. Distilling on your own task distribution routinely produces a student that meets or beats the teacher on that task while running on a fraction of the hardware.
Two constraints matter. First, license terms: several model licenses explicitly prohibit using outputs to train competing models, and that clause is directly aimed at distillation, so legal review of the teacher's license is not optional. Second, error inheritance: the student learns the teacher's mistakes and biases along with its strengths, so filtering the generated training data for quality, ideally with human review of a sample, materially affects the result.
Why It Matters
- Produces a task-specific small model that matches a much larger one, cutting serving hardware and latency together.
- Generates training data automatically from a teacher, avoiding the manual labeling effort that usually blocks fine-tuning projects.
- Teacher license terms frequently prohibit training competing models, making legal review a gate before any distillation run starts.
- The student inherits the teacher's errors and biases, so filtering the generated dataset is a quality control step, not an optimization.
In Practice
A workable recipe: run the large model over ten thousand real historical inputs, have a subject matter expert review a random sample of a few hundred outputs, discard the categories where quality is poor, and fine-tune a small model on what remains. The small model typically matches the large one on that task and runs on a single GPU.
Frequently Asked Questions
How is distillation different from fine-tuning?
Fine-tuning trains a model on human-created or existing labeled examples. Distillation trains a small student on examples generated by a larger teacher model. Mechanically the student's training step is fine-tuning; the distinction lies in where the training data comes from. Distillation removes the manual labeling bottleneck, which is often what makes a customization project feasible at all.
Is it legal to distill from a commercial AI model?
It depends entirely on the license or terms of service. Several major providers explicitly prohibit using their model outputs to train competing models, and some open-weight licenses carry similar clauses. Others permit it outright. Because the restriction is common and enforcement risk is real, review the teacher model's terms with counsel before generating any distillation dataset.
Related Terms
SLM (Small Language Model)
An SLM (Small Language Model) is a compact language model, typically between about 1 and 15 billion parameters, designed to run on a single modest GPU or even a CPU while still handling focused enterprise tasks well.
Fine-Tuning
Fine-tuning is the process of continuing to train an already pretrained model on a smaller set of your own examples, adjusting its weights so it reliably produces the formats, vocabulary, and behavior your task requires.
LLM (Large Language Model)
An LLM (Large Language Model) is a neural network with billions of parameters trained on huge volumes of text to predict the next token in a sequence, which gives it broad language understanding, reasoning, and generation ability.
Go Deeper
On-Prem LLM Total Cost of Ownership Calculator
Model the full multi-year cost of running LLMs on your own hardware, including GPU capex, power, cooling, support contracts, and operations staffing.
Small Language Models vs LLMs for Enterprise Workloads
Small language models vs LLMs for enterprise workloads: accuracy, cost, GPU footprint, and when a 7B model beats a frontier model for ERP automation.
On-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
Working with Model Distillation in a live environment? Our engineers do this every day - and our AI agents automate most of it.