Evaluation Layers and Metrics

Fact-check status

Reviewed claim by claim on 2026-09-02 against the primary or official sources identified inline as S1 and S2. Recommendations are design guidance unless a cited standard makes them normative.

Key jargon

Term	Plain-language meaning
Metric	A defined numeric measurement of behavior or operation. S1
Grader	Code, a model, or a person that assigns a result using a criterion. S1
System evaluation	Testing the end-to-end application including retrieval, tools, policies, and model. S1
Operational metric	A measure such as latency, availability, token use, or cost. S1

Key concepts

No single score captures correctness, safety, user value, and operating health. S1
Layer component, workflow, safety, and operational evaluations so failures can be localized. S1

Concept map

flowchart LR
    A["Define success dimensions"] --> B["Choose graders and metrics"]
    B --> C["Evaluate components and system"]
    C --> D["Diagnose tradeoffs"]

Diagram semantics verified against S1 and S2 on 2026-09-02.

Layer	Examples
Model	classification accuracy, perplexity, structured-output validity S1
Retrieval	recall@k, ranking, freshness, access-filter correctness S1
Tool	correct selection, argument validity, authorization, effect success S1
Workflow	completion, step failure, retry, escalation, evidence support S1
Safety	policy violation, harmful action, sensitive disclosure, overreach S1
Product	user task success, correction rate, time saved, cost per success S1

One aggregate score hides failure location and tradeoffs. Define thresholds for critical dimensions, not only an average. S1

Exercise

For a support assistant, define one metric at each layer and a release-blocking threshold. Explain which metric can be automated and which needs human review.

Principle

The evaluation target is the versioned system in its intended environment—not a model name in isolation. S1