For a narrow, high-volume task with a stable spec, a 4–8B student model fine-tuned on 20–50k filtered outputs from a frontier teacher typically reaches 90–100% of the teacher's quality on that task, at 3–10% of the per-request cost. That's the honest version of the title's 80/5 claim: conservative for classification and extraction, roughly right for constrained generation, and wrong for open-ended reasoning, where distillation fails predictably. The break-even is volume: below roughly 100k requests a month on the task, the engineering time costs more than it saves.
What "distillation" means in practice
Two techniques share the name. Logit distillation trains the student against the teacher's full output distribution; it needs teacher logits, which means an open-weight teacher, and buys a few extra points when you can get it. Data distillation — supervised fine-tuning on teacher-generated completions — is what almost everyone actually does, because it works through any API. Everything below is data distillation: generate, filter, train, evaluate.
Check the paperwork before the pipeline
Using API outputs to train a model is restricted by most closed providers' terms — OpenAI, Anthropic, and Google all have language limiting training of competing models on their outputs, and enforcement aside, it's a real contractual exposure for a company. The clean paths: use an open-weight teacher (DeepSeek's MIT licensing and Qwen's Apache-2.0 releases exist in large sizes precisely because distillation is a first-class use), use a provider's official distillation offering where one exists, or get the usage cleared in your enterprise agreement. Note the Llama license's own quirk: models trained on Llama outputs must carry "Llama" in their name. Read the actual terms for whatever teacher you pick; this paragraph ages faster than the rest of the post.
The pipeline: generate, filter, train, evaluate
Generate. Collect 10–100k real task inputs from production logs if you have them; synthesise the gap if you don't, with explicit diversity controls — temperature alone does not create coverage, and the failure modes are catalogued in synthetic data that works. Run the teacher with your full production prompt, not a simplified one: the student learns the task as prompted, edge-case instructions included. Generate via the batch API at half price; nothing about this workload needs real-time latency, and 30k generations at full price is money donated.
Filter, hard. Teacher outputs are training labels, and teachers make mistakes. Score every example with a judge model against a task rubric and drop the bottom 20–40%; enforce format validity mechanically (schema checks, label-set membership); deduplicate near-identical rows so the student doesn't memorise your most common input; and decontaminate against your eval set with exact match plus embedding similarity. Filtering is where distillation quality is actually made — a 20k filtered set reliably beats 50k raw.
Train. QLoRA on an 8B base handles most tasks and runs on a 24GB card in hours — config and hyperparameters in LoRA fine-tuning on a consumer GPU. Start from the instruct variant of the base model, not the raw base, and format examples with its chat template exactly.
Evaluate against the teacher, not against hope. Same held-out set, three-way comparison: teacher, student, and current production setup. Require the student within your tolerance (2–5% on the task metric) of the teacher on 200+ examples before anyone says "shipped", plus a general- capability canary to catch a student that aced the task and forgot how to decline out-of-scope requests. Harness discipline per evals before vibes.
The math, worked
Illustrative numbers for a structured-extraction task at 1M requests per month, 2,000 input and 400 output tokens each. Frontier at ~$3/$15 per million tokens: ~$12,000/month. Self-hosted 8B student on one rented or owned 24GB GPU, or a hosted small model at ~$0.15/$0.60: ~$540/month. One-time costs: teacher generation for 30k examples ≈ 45M output tokens ≈ $340 on a batch endpoint (estimate), GPU time for training experiments $20–100, and one to three weeks of engineering, which dominates everything. Payback at this volume is under two weeks; at 50k requests a month the same project takes most of a year to pay back, which is why volume is the first gate.
| Frontier via API | Distilled 8B student | |
|---|---|---|
| Cost per 1M requests | ~$12,000 | ~$400–700 |
| One-time cost | — | ~$500 + 1–3 weeks engineering |
| Quality on narrow task | Baseline | 90–100% of baseline |
| Handles spec changes | Edit the prompt | Regenerate data, retrain |
Where it fails predictably
The table's last row is the trap. A distilled student is a snapshot of one prompt's behaviour: every meaningful spec change means regenerating data and retraining, so a task whose prompt changes weekly should stay on the teacher. Open-ended tasks fail differently — the student mimics surface style but loses the reasoning that made the teacher good, which is why "distill a general assistant" disappoints while "distill this classifier" works. Long-tail input distributions fail quietly: the student matches the teacher on the common 95% and falls apart on the rare 5%, so if your rare cases are your expensive cases, route them — a small-first cascade with the teacher as the escalation tier is the standard hybrid, and often the permanent end state.
What I'd do
Gate on three questions: single dominant task, over ~100k requests a month, spec stable for a quarter. Three yeses: run the pipeline — production inputs, batch-generated teacher outputs, filter hard, QLoRA an 8B, three-way eval — and keep the teacher behind a router for the inputs the student flags or fumbles. Any no: stay on prompting plus routing, and revisit when volume or stability changes. And whatever you pick, keep the teacher-generated dataset under version control with the prompt that made it; the second retrain is 10× cheaper than the first only if the first was reproducible.