Deterministic Checks, Model Judges, and Humans

#ai #evaluation #llm-as-judge #verification

Deterministic Checks, Model Judges, and Humans

Fact-check status

Reviewed claim by claim on 2026-09-02 against the primary or official sources identified inline as S1 and S2. Recommendations are design guidance unless a cited standard makes them normative.

Key jargon

Term	Plain-language meaning
Deterministic check	A rule with reproducible output, such as schema validation or exact comparison. S1
Model judge	A model asked to score or compare outputs against criteria. S1
Human calibration	Comparing automated grades with qualified human judgments. S1
Inter-rater agreement	The extent to which independent graders reach compatible judgments. S1

Key concepts

Use deterministic checks for properties code can decide; use judges for nuanced criteria and humans for calibration or high-stakes ambiguity. S1
Model judges inherit bias and variance, so test position effects, rubric sensitivity, and disagreement. S1

Concept map

flowchart LR
    A["Run objective checks"] --> B["Apply rubric-based judge"]
    B --> C["Sample human review"]
    C --> D["Calibrate thresholds and graders"]

Diagram semantics verified against S1 and S2 on 2026-09-02.

Evaluator	Best for	Limits S1
Deterministic code	Schema, exact values, tests, invariants	Cannot judge open-ended usefulness alone S1
Model judge	Scaled rubric application and semantic comparison	Bias, instability, shared blind spots, prompt sensitivity S1
Human reviewer	Context, consequence, novelty, disputed cases	Cost, inconsistency, fatigue S1

Use deterministic checks first, calibrated model review for scalable judgment, and humans for high-impact or ambiguous cases. Compare model judges against human-labeled samples. S1

Evidence-backed grading

Grade demonstrated behavior, not a claimed success string. Preserve source spans, tool traces, test output, state transitions, and effect receipts. S1

Exercise

Score ten summaries with a deterministic citation check, a model rubric, and two humans. Calculate disagreement categories and revise the rubric.