What is LLM Judge?
Using a model to score other model outputs on rubrics like correctness, tone, or format compliance. Judges scale evals beyond hand-labeling, but need calibration against human grades or they bless fluent wrong answers.
Example
BYOB backend configuration carries a dedicated judge model config alongside chat and tool configs, so scoring passes run on a calibrated grader instead of the model being tested.
What people get wrong
Letting the same model grade its own homework uncalibrated. Self-grading inflates scores; separate judges plus human spot-checks keep grades honest.
Related terms
Evals
Repeatable tests that score model behavior on fixed cases before and after prompt or model changes. Without evals every tweak is a guess; with them, regressions in tone, accuracy, or format get caught early.
History Replay
Re-running a recorded conversation prefix to reproduce or debug model behavior deterministically. Saved histories turn flaky one-off sessions into repeatable cases for evals and judge scoring.
Hallucination
When a model states something false with confidence, usually from parametric memory without grounding. Grounded, cited answers hallucinate less, which is why engines prefer quotable sources.
Grounding
Tying a model’s answer to retrieved sources instead of pure parametric memory. Grounded answers cite pages, which turns citation readiness into visible referral and brand presence.
Temperature
Sampling randomness for generation: low values make output focused and repeatable, high values make it varied and surprising. BYOB chat configs default near 0.5 while tool calls use 0.3 for steadier structured actions.
Thinking Budget
The token allowance reserved for internal model reasoning before answering. Larger budgets suit hard multi-step problems; BYOB prices thinking tokens separately so runaway reasoning shows up in metering.