Model Distillation Savings Calculator: Teacher API vs Distilled Student
This free model distillation savings calculator compares the ongoing cost of calling a large teacher model API against the one-time cost of distilling and self-hosting a small student model, and it is built for engineering leads deciding whether a narrow, high-volume AI workload justifies distillation. Enter current request volume and teacher pricing, the size of the distillation dataset you will generate, and your self-hosted student inference cost, and the tool returns current spend, one-time distillation cost, monthly savings, and a breakeven timeline. Distillation makes the most economic sense for narrow, well-defined tasks running at high volume, exactly the profile of most production enterprise AI workloads once the initial exploration phase ends.
Your numbers
Current or projected call volume this workload makes against a large frontier model API.
Combined input and output tokens for one typical request.
Blended input and output price for the frontier model you currently call.
Teacher-generated examples used to train the student, typically 10,000-100,000 for a solid distilled model.
Input plus output tokens the teacher model consumes to produce one training example.
Amortized GPU cost to serve the small distilled student model at your required capacity.
GPU cost to fine-tune the student model on the distilled dataset. Small models train cheaply.
Your results
Estimates only. Actual student model quality depends on task difficulty and distillation technique. Validate the distilled student against your evaluation set before decommissioning the teacher-based workflow.
Get your distillation feasibility report
We will email you a personalized breakeven analysis and student model sizing recommendation, and a Netray AI specialist will follow up with a distillation plan.
No spam. Your results stay private. Unsubscribe anytime.
How the distillation economics work
Distillation has two costs and one ongoing saving. First, you pay the teacher model to generate training examples, typically prompt-completion pairs covering the task's real input distribution. Second, you pay compute to fine-tune a small student model on that dataset, which is cheap since student models are usually 1B-8B parameters. From then on, every production request goes to the self-hosted student instead of the teacher API. With the defaults, a $4,000 monthly teacher bill drops to an $1,800 monthly student hosting cost, saving $2,200 per month; against a roughly $1,192 one-time distillation cost that pays back in well under a month, after which the savings compound indefinitely.
- Distillation data generation cost scales with the number of examples and the teacher's per-token price.
- Student training compute is typically small since distilled students are usually 1B-8B parameters.
- Ongoing savings equal the gap between teacher API spend and self-hosted student inference cost.
- Breakeven is usually measured in weeks, not months, for workloads already spending meaningfully on a teacher API.
When distillation is the right call
Distillation earns its keep on narrow tasks: classification, extraction, structured output generation, routing decisions, and similar patterns where the teacher model's broad general capability is mostly wasted. A student model does not need to match the teacher's full breadth, only its performance on your specific task distribution, which is why a well-distilled 3B model can match a much larger teacher on a narrow classification task while running ten to fifty times cheaper per token. It is the wrong call for tasks requiring broad, open-ended reasoning that genuinely needs the teacher's full capability, since the student will not generalize past the examples it was distilled from.
What the breakeven number does not capture
The calculator models direct compute and API cost, not the engineering time to build the distillation pipeline, evaluate the student, and stand up self-hosted serving infrastructure. Budget one to three weeks of engineering time for a first distillation project: generating and filtering the dataset, running the student fine-tune, building an evaluation harness that confirms the student actually matches the teacher on your task, and deploying the serving stack. For a workload spending several thousand dollars a month on a teacher API, that engineering investment typically pays back within the same quarter, but it is real cost that belongs in the total project budget even though it does not appear in this calculator's breakeven line.
How Netray builds distillation pipelines
Netray distills large models down to small, self-hosted students for high-volume, narrow tasks inside manufacturing and defense workflows, part classification, ERP data extraction, routing tickets to the right work queue, where a frontier model's broad reasoning is unnecessary overhead. We build the teacher-generation pipeline with quality filtering, fine-tune the student, and validate it against the teacher's own outputs on a held-out set before recommending a cutover. For regulated customers this also solves a data residency problem: production traffic stops leaving the building entirely once the student model is deployed on-prem.
Frequently Asked Questions
How much smaller can the student model be than the teacher?
For narrow tasks, dramatically smaller: distilling a frontier model down to a 1-3B student for classification or extraction tasks is common and often preserves 90-98% of task-specific accuracy. The right size depends on task difficulty; complex multi-step reasoning tasks need larger students, often 7-14B, to retain quality. Always validate empirically rather than assuming a ratio, since the right student size varies significantly by task.
Does the student need to match the teacher's quality exactly?
No, it needs to clear your evaluation bar on your specific task, which is a much lower bar than matching the teacher across every possible query. Build an evaluation set from real production queries and compare student output directly against teacher output on that set. Many teams find a well-distilled small student matches the teacher within a few percentage points on the narrow task while completely losing the teacher's general knowledge, which is an acceptable and expected tradeoff for a production workload that only ever sees the narrow task.
Can I keep improving the student model over time?
Yes, and you should. Route a small sample of production traffic to the teacher periodically, compare its outputs against the student's, and add cases where they disagree to a growing distillation dataset for the next student fine-tune. This creates a continuous improvement loop where the student gets progressively closer to the teacher's quality on your real traffic distribution without ever needing to serve the teacher model at production volume.
Get a distillation feasibility analysis for your highest-volume AI workload, including breakeven timeline.
Related Tools
Synthetic Training Data Cost Calculator
Model generator token cost plus human filtering and review labor to produce a synthetic fine-tuning dataset, compared against pure human authorship.
On-Prem AILoRA Fine-Tuning Cost Calculator
Turn model size, dataset tokens, epochs, and rank into a GPU-hour and dollar estimate for a LoRA fine-tuning run on rented or owned hardware.
On-Prem AILLM API vs Self-Hosted Cost Calculator
Model your API bill from requests and token mix, compare it against an all-in self-hosted monthly cost, and see monthly and annual savings.
Go Deeper
Model Distillation for Production: Shrinking Models Right
Model distillation for production: teacher-student data generation, sequence vs logit distillation, and real serving cost savings by model size.
The Real Cost of LLM Fine-Tuning in 2026: A Full Breakdown
The real cost of LLM fine-tuning in 2026: GPU-hour pricing by method, data preparation labor, evaluation cost, and the hidden line items teams miss.
Small Language Models for Enterprise: When Smaller Wins
When small language models like Phi-4, Gemma 3, and Qwen3 small variants beat large models on cost, latency, and task-specific enterprise accuracy.