Retry the model, never the payment

A pipeline reads an invoice with a model, verifies the result, posts it and sends a payment instruction. The model call times out after the provider has in fact finished. The orchestrator retries, gets a second reading — not necessarily identical to the first — and, if nothing guards the posting step, the invoice is booked twice.

Nothing in that sequence is exotic. Pipelines that mix models with ordinary systems retry far more often than classic integrations: model calls are slow, rate-limited and occasionally time out, and an agent loop can call the same tool twice because it never saw the first result. Retrying is unavoidable. Retrying safely is a design decision, and it is usually made late.

Two properties that make retries dangerous

Side effects. Something outside the pipeline changes: a ledger entry, an email, a payment instruction, a status in a supplier portal. Running the step twice means doing the thing twice.

Non-determinism. Calling the model again does not have to return the same answer. A retry of a model step is not a repeat; it is a new decision. If verification ran against the first reading and the posting used the second, nothing verified what was posted.

Classic integrations had the first problem and learned to solve it. Mixed pipelines have both, and the second one quietly undermines the usual solution to the first.

The rules that make it safe

Persist the proposal, then work from the record. The model’s output is stored once, with an identifier and the model and prompt version, before anything acts on it. Verification, posting and the audit trail all refer to that stored artifact. Retries of later steps reuse it and never ask the model again. Asking again is a deliberate new version, not a side effect of a timeout.

An idempotency key on every state change. Derived from business identity — supplier and invoice number, not the run identifier — so that a second attempt is recognised as the same operation. The target system, or a thin layer in front of it, refuses to apply the same key twice and returns the first result. Exactly-once delivery across systems you do not control is not something you can buy; at-least-once plus deduplication is what actually runs.

The workflow owns the state, not the agent. Where a case is, what has been done and what is pending lives in a durable workflow or state machine that survives restarts. It does not live in an agent’s context or in the memory of a loop. The agent is a step the workflow calls; it does not hold the process.

Retry by step type. Model calls: retry with backoff, a bounded number of times, then route to an exception. Deterministic verification: no retry, because it will fail the same way again. External writes: retry only under the idempotency key.

Separate failed from unknown. A timeout on a payment instruction means you do not know what happened. The next step is a status query, not a resend.

Infographic

Retries without duplicates, step by step

1. 01IntakeKey derived from business identity, not from the run.
2. 02Model readingOutput stored once, with model and prompt version.
3. 03VerificationDeterministic, on the stored proposal. Nothing to retry.
4. 04State changeIdempotency key; a repeat returns the first result.
5. 05External effectOn an unclear outcome, query the status before resending.
6. 06Audit recordAppended at every step, never overwritten.

The record a mixed pipeline needs

For each case, append-only: the input or a reference to it, the model and prompt version, the stored proposal, which checks ran and what they returned, what was committed and under which key, and every human decision with the name of whoever made it.

That record answers the question every auditor and every annoyed supplier eventually asks — why was this paid, and on what basis — three weeks after the fact. It also has an engineering benefit that teams tend to discover late: when a validation rule changes, you can run the new verification against stored proposals without calling the model again, and see exactly which past cases it would have stopped.

Test it by breaking it

Kill the process in the middle of a step. Make the model time out. Return an error after the write has succeeded. Deliver the same input twice, an hour apart. If any of these produces a duplicate effect or a case nobody can find, the pipeline is not ready, however good the model is.

These tests are cheap, mechanical and almost never in the plan. They belong in the same suite as the evaluation cases, and together with the decision record that makes any single case explainable, they are the part of AI agents engineering that nobody demonstrates and everybody relies on.