Latency: definition

Latency is how long a model takes to produce its answer, and in live uses like chat or voice it matters as much as the quality of the answer itself.

A brilliant answer that takes eight seconds feels broken in a phone call and sluggish in a chat. Latency is the wait between request and response, and it is shaped by the model size, the prompt length, and whether you stream the output.

Designing for it, by choosing the right-sized model, trimming context, and streaming tokens as they arrive, is a big part of making an AI feature feel good to use.

Related terms

Inference: Inference is the act of running a trained model to get an output, and inference cost is what you pay each time the model answers, measured in tokens.
Token: A token is the unit a language model reads and writes in, roughly a word-piece of a few characters, and model pricing and limits are counted in tokens rather than words.
Context window: The context window is the maximum amount of text, measured in tokens, that a model can consider at once, covering the instructions, the conversation, and any documents you provide.