What is Evals?
Repeatable tests that score model behavior on fixed cases before and after prompt or model changes. Without evals every tweak is a guess; with them, regressions in tone, accuracy, or format get caught early.
Related terms
LLM Judge
Using a model to score other model outputs on rubrics like correctness, tone, or format compliance. Judges scale evals beyond hand-labeling, but need calibration against human grades or they bless fluent wrong answers.
History Replay
Re-running a recorded conversation prefix to reproduce or debug model behavior deterministically. Saved histories turn flaky one-off sessions into repeatable cases for evals and judge scoring.
Structured Output
Forcing model replies into a machine-readable shape such as JSON with fixed fields. Required wherever AI output feeds code, databases, or UI components, since free text breaks parsers the moment formatting drifts.
Temperature
Sampling randomness for generation: low values make output focused and repeatable, high values make it varied and surprising. BYOB chat configs default near 0.5 while tool calls use 0.3 for steadier structured actions.
Thinking Budget
The token allowance reserved for internal model reasoning before answering. Larger budgets suit hard multi-step problems; BYOB prices thinking tokens separately so runaway reasoning shows up in metering.
JSON Schema Mode
A structured-output setting where the caller supplies a JSON Schema and the model must conform to it. Types, required fields, and enums are validated automatically, which removes most post-processing repair code.