Observability, Cost, and Run Records

Key jargon

Term	Plain-language meaning
Trace	A connected record of one request across model, retrieval, and tool steps.
Span	A timed operation within a trace.
Metric	A numeric measurement aggregated over runs, such as latency or token usage.
Provenance	Evidence showing which versions, sources, prompts, and tools produced an output.

Key concepts

Logs explain events, metrics reveal trends, and traces show causal paths across a run.
Useful records balance debugging and audit needs against privacy, secret exposure, and storage cost.

Concept map

flowchart LR
    A["Assign run identity"] --> B["Capture spans and decisions"]
    B --> C["Aggregate quality cost latency"]
    C --> D["Investigate and improve"]

Minimum run record

Run, user/tenant, and task identifiers.
Workflow, policy, prompt, model, and tool versions.
Step timing, token/usage units, cache use, and estimated cost.
Retrieval document IDs and scores—not necessarily raw sensitive text.
Tool proposals, authorization decisions, effect IDs, and outcomes.
Validator assertions, human approvals, and final disposition.
Errors, retries, checkpoint, and cleanup state.

Metrics by layer

Layer	Useful measures
Model	latency, tokens, structured-output validity, refusals
Retrieval	recall/precision proxies, citation support, freshness
Tools	authorization denials, failures, duplicate-effect prevention
Workflow	completion, escalation, retry, abandonment, step failure
Product	user correction, task success, harm/incident rate, cost per success

Privacy rule

Observability is another data system. Redact or hash sensitive fields, restrict access, set retention, and test the redaction itself.

Exercise

Design a trace schema for a two-tool assistant without storing full prompts. Explain what you lose and how an authorized debug mode would work.