Rate limit: definition

A rate limit is the cap a provider puts on how many requests or tokens you can use in a window, and designing around it is part of running any AI system reliably at scale.

Model APIs limit how fast you can call them, by requests per minute and tokens per minute. Hit the limit and calls start failing, which in a live system means errors in front of users if you have not planned for it.

Handling it well, queuing, retries with backoff, and spreading load, is unglamorous but essential engineering, and a common reason a demo that worked falls over under real traffic.

Related terms

Inference: Inference is the act of running a trained model to get an output, and inference cost is what you pay each time the model answers, measured in tokens.
Latency: Latency is how long a model takes to produce its answer, and in live uses like chat or voice it matters as much as the quality of the answer itself.
API: An API is a defined way for two pieces of software to talk to each other, and it is how an AI system reads from and writes to the tools a business already runs.