Online Monitoring and Drift

Key jargon

Term	Plain-language meaning
Drift	A meaningful change in inputs, behavior, or outcomes over time.
Telemetry	Structured operational data such as traces, metrics, logs, and feedback.
Service-level objective (SLO)	A target for a measured aspect of service reliability or quality.
Leading indicator	A signal that may reveal degradation before final outcomes are known.

Key concepts

Offline evaluation predicts behavior; online monitoring reveals behavior under real traffic and dependencies.
Monitor input, retrieval, model, tool, safety, user, and cost signals without storing unnecessary sensitive content.

Concept map

flowchart LR
    A["Collect privacy-aware telemetry"] --> B["Compare against baselines and SLOs"]
    B --> C["Detect drift or anomaly"]
    C --> D["Investigate and respond"]

Monitor input distribution, task mix, retrieval freshness, tool errors, model/provider changes, latency, cost, user corrections, abstention, escalations, policy events, and outcome quality.

Drift types

Data drift: input characteristics change.
Concept drift: relationship between input and correct output changes.
System drift: model, prompt, corpus, tool, or dependency changes.
Behavior drift: users adapt around the system.
Evaluation drift: test set no longer represents real use.

Exercise

Design a weekly scorecard with leading indicators, outcome metrics, alert threshold, owner, and response. Include one metric that could improve while user outcomes worsen.

Privacy

Sample and retain production traces according to data policy. Monitoring should not create a second uncontrolled corpus of sensitive prompts and outputs.