Model distillation: definition

Distillation trains a smaller, cheaper model to mimic a larger one on a specific task, so you keep most of the quality at a fraction of the cost and latency.

A large model is capable but expensive to run. Distillation uses the big model to teach a small one how to do a narrow task well, so in production you run the cheap, fast small model and reserve the big one for the hard cases.

It is a practical way to cut inference cost at scale without giving up much accuracy, and a common step once a task is well understood.

Related terms

Fine-tuning: Fine-tuning is further training a base model on your own examples so it learns a specific style, format, or task more reliably than prompting alone can achieve.
Inference: Inference is the act of running a trained model to get an output, and inference cost is what you pay each time the model answers, measured in tokens.
Open-source model: An open-source model is one whose weights are published so you can run it on your own infrastructure, giving you control over data and cost at the price of hosting it yourself.