AI systems do not crash. They drift

Every operations team knows what to do when a service stops. There is an alert, a runbook, an on-call rota and a post-incident review. The whole discipline is built around things that fail loudly.

AI systems mostly do not fail that way. They stay up, respond quickly, return well-formatted answers, and become gradually less right. There is no moment to alert on, which is why the average time to detection in systems we are asked to take over is measured in weeks.

The four ways quality leaves without an alert

The corpus moved. Documents were added, updated or withdrawn, and the system now answers confidently from material that no longer reflects how the business works.

The provider changed something. A model version was deprecated, a default was updated, safety behaviour shifted. The API contract is unchanged; the behaviour is not.

The questions changed. Users learned what the system is good at, or a new team started using it for something adjacent. The system did not get worse — the distribution did, and the effect is identical.

Someone made a small change. A prompt tweak to fix one complaint, shipped on a Friday, that made forty other cases slightly worse. Without an evaluation set this is undetectable until somebody notices a pattern.

What monitoring has to look like instead

A held-out evaluation set, scored continuously. The same cases, every day, with the score recorded. This is the only reliable early signal, and it costs very little to run.

Retrieval scored separately from answers. Most degradation happens in retrieval, and an end-to-end score tells you something is wrong without telling you which half.

Behavioural signals from real usage. Rising refusals, rising user corrections, changing output length, falling click-through on cited sources. Each is noisy alone; together they move before complaints do.

Alerts on movement, not on thresholds. A five-point drop over a week matters more than an absolute number, because the absolute number depends on the evaluation set and the movement does not.

Cost and latency alongside quality. They move together more often than teams expect, and a sudden cost drop is frequently a quality incident wearing a pleasant disguise.

The incident response that differs

When a conventional service breaks you restore it. When an AI system is wrong, the first action is not to fix it — it is to reduce what it does unattended while you investigate.

Narrow the scope, route more cases to a person, fall back to the previous prompt or model version. Cheap, reversible, and it stops the bleeding while the diagnosis happens. Systems designed without that reduction path leave an operations team with two options, both bad: leave it running while being wrong, or turn it off entirely.

The question worth asking today

If your AI system started being wrong this afternoon, how would you find out, and how long would it take?

If the honest answer is a user complaint, the detection time is however long it takes for enough people to be affected that one of them escalates. In most organisations that is weeks, and the answers were wrong for all of them.

That gap is what an AI managed services engagement is built to close — and it is why the first thing we do when taking over a system is build the evaluation set, even when the system is somebody else’s and already live.