Lab 4 — Production Harness Capstone

Key jargon

Term	Plain-language meaning
Capstone	An integrated project demonstrating multiple learned competencies.
End-to-end test	A test exercising the system through its real external boundaries.
Failure injection	Deliberately causing a dependency or step to fail to test recovery.
Release evidence	Artifacts showing a candidate satisfied defined quality, safety, and operational gates.

Key concepts

The capstone proves system behavior under success, denial, interruption, retry, and recovery—not just a happy-path response.
A release packet should identify every component version and link claims to tests, traces, and approvals.

Concept map

flowchart LR
    A["Define contract and threats"] --> B["Build bounded observable harness"]
    B --> C["Test normal adversarial and failure paths"]
    C --> D["Assemble evidence and release decision"]

Goal

Build a synthetic support assistant that retrieves approved policy, drafts a response, proposes—but cannot directly perform—one reversible account action, and produces an evidence bundle.

Required components

Task/system contract, model adapter, context manifest, retrieval authorization, tool schema, effect preview, human approval, idempotency key, run state, checkpoint, trace, offline eval set, threat model, rollout, rollback, incident runbook, and cost budget.

Failure schedule

Test malformed model output, retrieval miss, poisoned document, permission denial, model timeout, tool timeout before effect, tool timeout after ambiguous effect, duplicate event, human rejection, and model-version change.

Pass criteria

No effect without current explicit approval.
No cross-user retrieval.
Unknown effects reconcile instead of repeating.
Critical evals block release.
A reviewer can reconstruct every output and effect from evidence.

Final reflection

Identify which value came from the model and which came from the harness. If the answer is “the model did everything,” the architecture is not yet understood.