A release is the code, the prompt, the model and the index

Ask a team how they would roll back yesterday’s release and they will usually describe redeploying the previous build. In conventional software that is the whole answer.

In an AI system it covers one of several things that determine behaviour, and often not the one that changed. The answers got worse on Tuesday, nobody has touched the code since last week, and the build history has nothing to say about it.

What actually changes behaviour

Code. Built, tested and deployed through the pipeline. The part that release practice already handles well.

Prompts and instructions. Frequently kept in a configuration store, a database or an admin screen, which means they change outside the pipeline and outside version history.

The model version. Pinned by you, or retired by the provider on the provider’s timetable. An alias that always points at the current version is a release that nobody scheduled.

The retrieval index. Built from your documents with a particular chunking, embedding model and set of filters. Rebuilt in place, it has no previous version to return to.

Configuration. Thresholds, routing rules between models, tool definitions. Small values with large effects, usually changed by whoever happens to have access.

Each of these can change independently, on its own schedule and at the hands of a different person. A release practice that versions only the first is versioning the part that is easiest to see, not necessarily the part that changed.

Infographic

What a release of an AI system contains

1. 01CodeBuilt and deployed through the pipeline, as it always was.
2. 02Prompts and instructionsVersioned in the repository, never edited only in an admin screen.
3. 03Model versionPinned explicitly; a moving alias is an unscheduled release.
4. 04Retrieval indexBuilt next to the previous one, never rebuilt in place.
5. 05ConfigurationThresholds, routing and tool definitions, versioned with the rest.

One manifest, evaluated as a whole

The fix is unexciting. A single release manifest names the version of each of those five, and only manifests that have been evaluated together go to production. The evaluation score travels with the manifest, so a release is not just a set of versions but a set of versions with a known quality.

Rolling back then means returning to the previous manifest, all five parts at once, to a state that was measured, rather than guessing which piece moved.

It also answers the first question of every incident review: what exactly was running at the time? With a manifest that is a lookup. Without one it is an afternoon of reconstruction from logs, admin-screen history and the provider’s changelog.

Prompts belong in the repository for the same reason code does. If a prompt can be changed in an admin screen without passing through evaluation, the release process covers less than the team thinks it does.

Rolling out a change that throws no errors

A conventional rollout watches for errors. A change to an AI system rarely produces any, so the rollout has to watch something else.

Shadow first where you can. The new manifest runs on real traffic next to the old one; its answers are scored but not shown. You are looking for differences, not failures.

Canary on quality, not only on errors. A slice of users gets the new manifest, and widening it depends on the evaluation score and on behavioural signals such as refusals, corrections and abandoned answers, as well as on errors and latency.

Keep the previous index alive. Build the new index next to the old one, switch over, and keep the old one until the new manifest has proved itself. Rebuilding an index in the middle of an incident is the slowest rollback there is.

Treat provider deadlines as release dates. A model retirement announcement is a release you did not schedule. Put it in the plan on the day it is announced and send it down the same path as any other change.

Rollback should be boring

The usual first response to a quality incident is to fall back to the previous prompt or model version. That advice assumes the previous state exists, is recorded and was evaluated. Release practice is what makes the assumption true.

Rehearse it once in a quiet week. A rollback path that has never been exercised is a hypothesis, and an incident is a bad moment to test it. Decide in advance who may trigger a rollback and on what signal, too. A rollback that needs a meeting to approve it will happen late.

This is the deployment side of AI-native apps engineering: CI, environments and runbooks on your own infrastructure, documented well enough that your team runs the releases, and the rollbacks, without us.