Observability, Cost, and Run Records

Fact-check status

Reviewed claim by claim on 2026-09-02 against the primary or official sources identified inline as S1 and S2. Recommendations are design guidance unless a cited standard makes them normative.

Key jargon

Term	Plain-language meaning
Trace	A connected record of one request across model, retrieval, and tool steps. S1
Span	A timed operation within a trace. S1
Metric	A numeric measurement aggregated over runs, such as latency or token usage. S1
Provenance	Evidence showing which versions, sources, prompts, and tools produced an output. S1

Key concepts

Logs explain events, metrics reveal trends, and traces show causal paths across a run. S1
Useful records balance debugging and audit needs against privacy, secret exposure, and storage cost. S1

Concept map

flowchart LR
    A["Assign run identity"] --> B["Capture spans and decisions"]
    B --> C["Aggregate quality cost latency"]
    C --> D["Investigate and improve"]

Diagram semantics verified against S1 and S2 on 2026-09-02.

Minimum run record

Run, user/tenant, and task identifiers. S1
Workflow, policy, prompt, model, and tool versions. S1
Step timing, token/usage units, cache use, and estimated cost. S1
Retrieval document IDs and scores—not necessarily raw sensitive text. S1
Tool proposals, authorization decisions, effect IDs, and outcomes. S1
Validator assertions, human approvals, and final disposition. S1
Errors, retries, checkpoint, and cleanup state. S1

Metrics by layer

Layer	Useful measures
Model	latency, tokens, structured-output validity, refusals S1
Retrieval	recall/precision proxies, citation support, freshness S1
Tools	authorization denials, failures, duplicate-effect prevention S1
Workflow	completion, escalation, retry, abandonment, step failure S1
Product	user correction, task success, harm/incident rate, cost per success S1

Privacy rule

Observability is another data system. Redact or hash sensitive fields, restrict access, set retention, and test the redaction itself. S1

Exercise

Design a trace schema for a two-tool assistant without storing full prompts. Explain what you lose and how an authorized debug mode would work.