The vendor retired your model. What happens to your system

The email arrives on a Tuesday, from the model provider, addressed to whoever registered the account. A model version is being retired. After a date a few months away, requests to it will be refused. A newer version is recommended, and the migration is described as straightforward.

The system it concerns has been in production for over a year. It classifies incoming claims, drafts the first response, and routes anything uncertain to a person. The prompts were tuned over months against that model’s habits, and the threshold for “uncertain” was set by watching that model’s confidence on real cases.

Nothing is broken. Nothing will be broken on the retirement date either, if somebody changes the model name in the configuration. The system will keep answering. It will simply answer slightly differently, and nobody will have decided whether the difference matters.

The cause is a mismatch of timescales. The provider manages model versions on its own calendar, measured in months. Your system was built as if the model were a fixed part, like a database engine. It is not. It is a dependency with a lifecycle, and that lifecycle is the one part of the system you do not control.

What goes wrong when the swap is improvised

Nobody knows what depends on the model. The classifier, the drafting step and the routing threshold may each call a model, and they may not all call the same one. Without an inventory, the retirement notice is read as being about “the system” rather than about three specific components, and one of them gets missed.

The prompts were tuned to a model’s habits, not to a specification. A prompt that says “be concise” produced the right length on the old model because of how that model read it. The new one reads it differently. The prompt has not changed, and the behaviour has.

The threshold is now wrong in a direction nobody can see. If the new model reports confidence differently, the share of cases routed to a person moves. Up, and your team is suddenly busier. Down, and more cases go through untouched, with no alarm, because nothing went wrong in a way a system can notice.

The evaluation set was never designed to detect a change of model. Many evaluation sets check that the system answers the easy cases correctly. A model swap breaks things at the edges: the awkward document, the ambiguous claim, the case that used to be refused. If those cases are not in the set, the score barely moves and the problem is real.

The swap happens once, in production, on the deadline. Nobody rehearsed it, so the change is made the week before the retirement date, under pressure, with no previous version to fall back to, since the previous version is the one being removed.

Infographic

Model retirement as a planned event

1. 01InventoryWhich components call which model version, with which prompts and thresholds.
2. 02Evaluation set that detects changeEdge cases and known refusals included, scored on the old model as the baseline.
3. 03Rehearsed swapThe new model run on the same cases, and on live traffic in shadow, before anything is switched.
4. 04FallbackA second model or a narrower mode the system can drop to if the swap misbehaves.
5. 05Switch, then watchQuality, routing share, cost and latency compared against the baseline for weeks, not days.

What a managed service does instead

Keeps a dependency inventory. For every system under management: which component calls which model version, through which prompt, with which threshold. Half of the work of a migration is knowing its scope.

Treats the retirement date as a release date. The deadline goes on the calendar the day the notice arrives, and the migration is planned backwards from it, with time for a rehearsal and time for the swap to go wrong once. A release is the code, the prompt, the model and the index together, even when somebody else set the date.

Scores the new model before it carries any traffic. The evaluation set run on the old model gives the baseline; the same set on the new one shows what moved. If the set does not contain the awkward cases, those are added first, from production logs, before the comparison means anything.

Re-tunes, then re-scores. Prompts and thresholds are adjusted for the new model with the evaluation set as the judge, one change at a time. The goal is not identical behaviour but behaviour whose differences are known and accepted by the business owner.

Keeps a fallback that is not the retired model. A second provider, a different model family, or a narrower mode that routes more to people. Chosen and tested before the swap, so that “roll back” means something on the day the old version no longer exists.

Writes down what changed. A short note to the business owner: what the new model does differently, where the threshold now sits, what the cost per task is expected to be. An SLA for an AI system should already require that note.

What cannot be promised

Not that the new model will behave the same. It will not, and a supplier who says otherwise has not done a migration. What can be promised is that the differences are measured before users meet them, and accepted by somebody on your side.

Not that every edge case will be caught. The evaluation set is only as good as the cases in it, and some differences surface in production. What can be promised is that the watching continues for weeks after the swap rather than ending on the day of it.

And not that a managed service is the right answer for every system. For a small internal tool with a forgiving process and its builder still in the building, a rehearsed swap is an afternoon’s work that needs no outside party. The service earns its place where the system touches customers or money, where the build team has moved on, or where there are several systems whose model dependencies nobody has written down.

A test to run today

Find the model name in your system’s configuration. Then ask two questions: when does the provider plan to retire that version, and what would you run instead if it were refused tomorrow.

If both answers come quickly, with a date and a tested alternative, the system is being operated. If the first is a search of the provider’s website and the second a guess, the notice is already on its way, and the time it gives you will go on the inventory and the evaluation set, not on the swap itself. It is the same gap that opens in year two of a transformation, when the build team has gone and the provider’s calendar keeps moving.

Our part

In AI managed services the model lifecycle is part of the service rather than an incident: inventory, evaluation set, rehearsed swap and fallback, tested against your cases before a change reaches you. predictes resells no model and no platform, so the recommendation of what to migrate to carries no margin.