Evaluation (evals): definition

An evaluation, or eval, is a repeatable test that measures how well an AI system does its job on real examples, so you can tell whether a change made it better or worse.

You cannot improve what you do not measure, and "it looked good in the demo" is not measurement. An eval is a set of real inputs with known good outcomes that you run the system against after every change.

This is the unglamorous discipline that separates systems that quietly degrade from ones that keep getting better. We build an eval set alongside the feature, not after it breaks.

Related terms

Hallucination: A hallucination is when a model states something false or invented as if it were fact, because it predicts plausible-sounding text rather than looking anything up.
Prompt engineering: Prompt engineering is the craft of writing the instructions, examples, and context you give a model so it produces reliable, correctly formatted output for a specific task.
Fine-tuning: Fine-tuning is further training a base model on your own examples so it learns a specific style, format, or task more reliably than prompting alone can achieve.