Quantization: definition

Quantization shrinks a model by storing its numbers at lower precision, so it runs faster and cheaper on smaller hardware with a small, usually acceptable, drop in quality.

A model’s weights are numbers, and storing them at full precision is expensive. Quantization rounds them to a smaller format, which cuts memory and speeds up inference, often enough to run a capable model on modest hardware.

It is a key lever for running open-source models cost-effectively or on-device. The trade is a slight quality loss, which for many tasks is not noticeable.

Related terms

Open-source model: An open-source model is one whose weights are published so you can run it on your own infrastructure, giving you control over data and cost at the price of hosting it yourself.
Inference: Inference is the act of running a trained model to get an output, and inference cost is what you pay each time the model answers, measured in tokens.
Latency: Latency is how long a model takes to produce its answer, and in live uses like chat or voice it matters as much as the quality of the answer itself.