Deterministic Checks, Model Judges, and Humans

#ai #evaluation #llm-as-judge #verification

Deterministic Checks, Model Judges, and Humans

Key jargon

Term	Plain-language meaning
Deterministic check	A rule with reproducible output, such as schema validation or exact comparison.
Model judge	A model asked to score or compare outputs against criteria.
Human calibration	Comparing automated grades with qualified human judgments.
Inter-rater agreement	The extent to which independent graders reach compatible judgments.

Key concepts

Use deterministic checks for properties code can decide; use judges for nuanced criteria and humans for calibration or high-stakes ambiguity.
Model judges inherit bias and variance, so test position effects, rubric sensitivity, and disagreement.

Concept map

flowchart LR
    A["Run objective checks"] --> B["Apply rubric-based judge"]
    B --> C["Sample human review"]
    C --> D["Calibrate thresholds and graders"]

Evaluator	Best for	Limits
Deterministic code	Schema, exact values, tests, invariants	Cannot judge open-ended usefulness alone
Model judge	Scaled rubric application and semantic comparison	Bias, instability, shared blind spots, prompt sensitivity
Human reviewer	Context, consequence, novelty, disputed cases	Cost, inconsistency, fatigue

Use deterministic checks first, calibrated model review for scalable judgment, and humans for high-impact or ambiguous cases. Compare model judges against human-labeled samples.

Evidence-backed grading

Grade demonstrated behavior, not a claimed success string. Preserve source spans, tool traces, test output, state transitions, and effect receipts.

Exercise

Score ten summaries with a deterministic citation check, a model rubric, and two humans. Calculate disagreement categories and revise the rubric.