Evaluation Layers and Metrics

Key jargon

Term	Plain-language meaning
Metric	A defined numeric measurement of behavior or operation.
Grader	Code, a model, or a person that assigns a result using a criterion.
System evaluation	Testing the end-to-end application including retrieval, tools, policies, and model.
Operational metric	A measure such as latency, availability, token use, or cost.

Key concepts

No single score captures correctness, safety, user value, and operating health.
Layer component, workflow, safety, and operational evaluations so failures can be localized.

Concept map

flowchart LR
    A["Define success dimensions"] --> B["Choose graders and metrics"]
    B --> C["Evaluate components and system"]
    C --> D["Diagnose tradeoffs"]

Layer	Examples
Model	classification accuracy, perplexity, structured-output validity
Retrieval	recall@k, ranking, freshness, access-filter correctness
Tool	correct selection, argument validity, authorization, effect success
Workflow	completion, step failure, retry, escalation, evidence support
Safety	policy violation, harmful action, sensitive disclosure, overreach
Product	user task success, correction rate, time saved, cost per success

One aggregate score hides failure location and tradeoffs. Define thresholds for critical dimensions, not only an average.

Exercise

For a support assistant, define one metric at each layer and a release-blocking threshold. Explain which metric can be automated and which needs human review.

Principle

The evaluation target is the versioned system in its intended environment—not a model name in isolation.