Who runs your AI system the day after go-live?

AI projects are budgeted up to go-live. The plan, the team, the milestones and the acceptance all end at the moment the system gets real users. And that is exactly when the system starts to change: documents are added, users’ questions drift in new directions, and the model provider retires the version everything was tested on.

The question of who runs it is usually asked after the fact, when the project team has already dispersed. That is when the answer is most expensive.

What actually needs running

Not just servers. A cloud provider will keep the infrastructure up better and more cheaply than anybody else, but it will not tell you that the answers have been getting worse for a fortnight.

Answer quality. A continuously scored evaluation set, and somebody who looks at the result and acts on movement before users report it.

Model lifecycle. Providers retire versions and change how models behave. Every such change has to be checked against your cases before it reaches people.

Cost. The model bill grows with the system’s success and can jump after one innocent change. Somebody has to be able to attribute it to specific features.

The data underneath. Re-indexing, refreshing embeddings and enforcing retention as your documents change.

Incidents without an outage. The system does not go down, it starts being wrong. Somebody has to have the right to narrow what it does while the cause is found, without convening a committee.

Three options, one of which is the default

Your own team. The best answer in the long run, if you have people who understand the system and have time for it written into their schedule — not “when they get a moment” between projects.

An external managed service. Sensible when the team that built it was external or has moved on, and the system matters too much to wait for an internal team to mature. Provided the contract is built around handing over rather than around permanent dependence.

Nobody. The option nobody chooses deliberately, and the one that remains when no other is chosen. The system runs because nobody switched it off, and its quality is known precisely up to the first serious complaint.

Infographic

Who should run an AI system

High criticalityLow criticality

Managed externallyWith a handover schedule, so the dependency shrinks.

Your own teamWith a quality owner and time written into the schedule.

Light oversightBasic checks and a decision on whether the system stays.

Team routineRun alongside other systems, with the same checks.

Low capabilityHigh capability

How much attention it needs from you

Even with an external service, part of the responsibility stays on your side and cannot be handed over.

A quality owner on the business side. Somebody who defines what counts as an incident. A wrong answer about a price or a deadline is a different severity from an answer that is too long, and an engineer should not be the one settling that difference.

A decision on narrowing. A named person who can agree to restrict what the system does within hours, not at the next scheduled meeting.

One page a month, read. What changed, what broke, what it cost and what is recommended. If nobody reads that page, the service runs in a vacuum and its findings change no decisions.

It is not much time. But it needs a name against it.

What the handover must contain

Whoever takes the system over, the handover should contain a few things that are easiest to prepare before go-live and hardest to reconstruct after it.

The evaluation set. Real cases with known good answers. Without it nobody can measure whether the system is getting worse, and quality monitoring becomes a declaration.

A baseline. Current quality, cost and latency recorded on the day of handover, so that any later change is a fact rather than an impression.

Runbooks and access. What to do about the failures that will certainly happen, and who has access to what — written down, not passed on verbally at the last meeting.

A list of known weaknesses. The cases the build team knows the system handles badly. It is rarely written without an explicit request, and it saves the successors weeks of rediscovering the same things.

When to decide

Before go-live, not after. The cheapest moment is while the build team is still there: a handover with the evaluation set, runbooks and access costs days at that point. A few months later, somebody has to reconstruct it from the code and from the memory of people who now work elsewhere.

A good test is one question asked before acceptance: who gets the first alert about a drop in quality on the day after go-live, and what are they allowed to do? If there is no name, the decision about running the system has not been made. It has been postponed to the point where it costs more.

If the system is already live and nobody is running it, it is not too late. AI managed services often start with a system we did not build, and with building the evaluation set that was never there — with a support tier that is meant to shrink over time rather than grow.