Experiments, Versioning, and Regression Gates

#ai #experiments #versioning #regression

Experiments, Versioning, and Regression Gates

Key jargon

Term	Plain-language meaning
Experiment	A controlled comparison intended to estimate the effect of a change.
Baseline	The current reference system against which a candidate is compared.
Regression	A previously acceptable behavior that becomes worse after change.
Promotion gate	A rule a candidate must satisfy before moving to a wider environment.

Key concepts

Change one interpretable factor where practical and record all system versions.
Require both aggregate improvement and protection of critical safety or user slices.

Concept map

flowchart LR
    A["Freeze dataset and baseline"] --> B["Run candidate comparison"]
    B --> C["Analyze quality cost and slices"]
    C --> D["Promote only if gates pass"]

Experiment record

Hypothesis, changed variable, fixed variables, dataset version, system versions, seed/sampling settings, metrics, confidence/variance, cost, raw results, decision, owner, and date.

Change one major variable at a time when attribution matters. If multiple layers change together, call it a system comparison rather than claiming which component caused improvement.

Regression gate

Minimum target-quality threshold.
No critical safety/permission regression.
Latency and cost budget.
Important subgroup floors.
Structured-output/tool correctness.
Manual review of changed failure classes.

Exercise

Design an A/B test for a prompt revision. State sample allocation, primary metric, guardrails, stopping rule, and what would invalidate the conclusion.