Distillation

Training a small, cheap model to imitate a larger teacher, trading some general ability for lower cost and faster responses.

“make a smaller model copy the big one”“cheap version of the smart model”“teacher student training”“train on answers from a bigger AI”“shrink the model without losing too much”“the big model is too expensive to ship”“distilation”“replace the expensive model with a small one”

See it

Live demo coming soon

What it is

Distillation trains a smaller student model to imitate a larger teacher. In classic knowledge distillation, the student learns from the teacher's full probability distribution, softened with a temperature setting, not just the winning label. For language models, the teacher often generates a large set of instructions and high-quality answers that become the student's fine-tuning data. The idea goes back to the 2015 paper 'Distilling the Knowledge in a Neural Network' by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.

Reach for it when one narrow task works well on an expensive model but needs lower cost, lower latency, or hardware you control. The usual loop is teacher data generation, cleanup, student training, then a head-to-head eval on examples neither model trained on. DistilBERT is the familiar product anchor: a smaller BERT-family model made through distillation.

Gotcha: the student copies the teacher's blind spots along with its good answers, and it will not retain every general capability of the larger model. A weak or repetitive synthetic dataset can make the student narrower still. Keep human-written test cases separate, measure the specific behavior you need, and compare cost only after quality clears that bar.

Ask AI for it

Build a knowledge-distillation training script with PyTorch and Hugging Face Transformers for SST-2. Fine-tune bert-base-uncased on the SST-2 train split first, freeze that checkpoint as the teacher, and use distilbert-base-uncased as the student. Train the student with a weighted sum of hard-label cross-entropy and KL divergence between teacher and student logits at temperature 2, scaling the KL term by the square of the temperature. Keep the validation split out of training, report accuracy and latency for both models, save the best student checkpoint by validation accuracy, and print the student's parameter count and disk size beside the teacher's.

You might have meant

fine tuningevalsgolden datasetinferenceopen weights model