Retries, Idempotency, and Recovery

Key jargon

Term	Plain-language meaning
Retry	Repeating an operation after a transient failure.
Backoff	Increasing delay between attempts, usually with randomness to avoid synchronized retries.
Idempotency	The property that repeating the same logical operation does not duplicate its effect.
Compensation	A deliberate action that reverses or offsets a completed step when full rollback is impossible.

Key concepts

Read failures and side-effect failures require different retry policies.
Persist intent, attempt, and outcome so recovery can distinguish not-started, in-flight, succeeded, and failed work.

Concept map

flowchart LR
    A["Classify failure"] --> B["Check prior outcome and idempotency key"]
    B --> C["Retry or compensate"]
    C --> D["Persist final state"]

Model and tool calls fail through timeout, rate limit, provider error, invalid output, network ambiguity, or downstream partial completion. Retrying is safe only when you know whether the first attempt had an effect.

Effect states

proposed → authorized → submitted → succeeded
                            ├── failed-no-effect
                            └── outcome-unknown → reconcile

Do not convert outcome-unknown into a fresh action. Preserve the original approval and idempotency key, query the provider, or require certified no-effect before creating successor authority.

Exercise

Model a payment in which the network drops after submission. Write recovery behavior for provider success, provider failure, and provider unavailable. No real transaction is needed.

Checklist

Classify operations as naturally idempotent or not.
Use provider-supported effect keys where available.
Bound retry count and elapsed time.
Persist attempt and outcome before process exit.
Route ambiguous effects to reconciliation.