Evals: testing non-deterministic output

An eval set: inputs plus expected properties

An eval set looks like a test suite, but each case pairs an input with expected properties of the output, not an exact string. For a booking-reschedule assistant: input "move Saturday's walk to Sunday" plus context, and the expected properties might be mentions Sunday, does not mention Saturday as the new day, returns a valid booking_id. You run the whole set whenever you change the prompt, the model, or the surrounding code — the same way a test suite catches a regression before it reaches users.

Three assertion styles, from cheapest to most expensive: structured checks (parse the output — usually JSON — and assert on fields, exact and fast), semantic similarity (embed the output and the expected answer, assert their similarity score is above a threshold — catches rephrasing), LLM-as-judge (ask a second LLM call to grade the first one's output against a rubric — the most flexible, also the slowest and least deterministic, since now your test itself is non-deterministic).