Evaluation Datasets, Rubrics, and Edge Cases

Key jargon

Term	Plain-language meaning
Evaluation dataset	A set of cases used to measure system behavior.
Golden label	A trusted expected answer, decision, or property for a case.
Rubric	Explicit criteria and scoring levels for judgment.
Edge case	A rare, difficult, or boundary input likely to expose failure.

Key concepts

Datasets should represent normal use, important failures, adversarial inputs, and slices of affected users.
Keep evaluation cases separate from training and prompt examples when unbiased measurement matters.

Concept map

flowchart LR
    A["Collect representative and risky cases"] --> B["Define labels and rubrics"]
    B --> C["Stratify important slices"]
    C --> D["Version and protect test set"]

An evaluation item needs input, expected behavior, scoring method, provenance, task/subgroup labels, and review history.

Include normal cases, boundaries, adversarial-but-authorized cases, ambiguous requests, missing information, conflicts, unsupported tasks, long inputs, multilingual content, tool failures, permission denial, and safe abstention.

Rubric design

Define observable criteria and examples at each score.
Separate correctness, completeness, relevance, grounding, style, and safety.
Weight critical failures explicitly.
Measure reviewer agreement and adjudicate disagreements.
Keep a hidden or newly authored set to reduce benchmark gaming.

Exercise

Create 20 evaluation items for a document summarizer. Label source type, difficulty, expected facts, prohibited inventions, and scoring rubric.