Evals
Unit tests for prompts: a fixed set of inputs, scored the same way every run, so you can prove a change made things better and not worse.
See it
What it is
An eval is a fixed set of inputs, run through your prompt or agent, scored the same way every time. It exists because vibes do not scale: you change one line of the system prompt, spot-check three examples, ship it, and silently wreck the fourth case you never tried. With an eval you get a number, and you can tell whether the number moved.
Scoring comes in three flavors, cheapest first. Deterministic checks (did it return valid JSON, is the total correct, did it call the right tool) are free and should cover most of the suite. LLM-as-judge handles the fuzzy ones (is this summary faithful to the source) but needs its own calibration against human labels. Human review on a small sample keeps the other two honest. The inputs themselves are your golden dataset, and the best ones come from real production failures, not from imagination.
Gotchas: start at twenty examples, not two hundred, or you will never start. Never delete a case because it fails; that is the entire point. And an eval suite you only run before launch is not an eval suite, it is a memory. Wire it into CI so a prompt edit shows a diff in scores next to the diff in text.
Ask AI for it
Build an eval suite for this AI feature. Define a golden dataset file of input plus expected-behavior pairs drawn from real usage, including the known failure cases. Write a runner that executes every case against the current prompt and model, scores each with a mix of deterministic checks (schema validity, exact fields, correct tool chosen) and an LLM-as-judge rubric for open-ended answers, then reports pass rate per category plus a diff against the last run. Make results reproducible (pinned model snapshot, temperature 0 where possible), fail CI on a regression above the threshold, and keep the whole thing runnable with one command.