An SLA for an AI system: what can be promised and what cannot

Procurement has a proven template for a service contract: availability, response times, resolution times, penalties for missing them. For ordinary software it works, because a failure is visible — the service responds or it does not.

An AI system can be fully available and have been answering worse for three weeks. A contract written to the old template will then be honoured to the letter, and your problem will not appear in it at all.

Why the conventional template falls short

It measures the thing that rarely breaks. Availability is the easy part. The hard part — answer quality — has neither a definition nor a measurement in a typical contract.

It assumes the fault is visible. Resolution time runs from the ticket. When a system is plausibly wrong, the ticket arrives weeks later or not at all.

Severity is counted technically. Users affected, components down. Yet one wrong answer about a price or a deadline can weigh more than an hour of downtime.

Infographic

Software SLA versus AI system SLA

Conventional SLA

• –Measures service uptime
• –Severity by technical impact
• –First response: restore service
• –Report counts tickets

SLA for an AI system

• →Measures answer quality too
• →Severity by business consequence
• →First response: narrow its scope
• →Note on what changed and why

What belongs in the contract

A defined scope of quality monitoring. Which cases are scored, how often, and what triggers an alert. Without that, “we monitor quality” is a sentence with nothing in it that can be enforced.

Severity described in business terms. Written by your side, not the supplier’s: what is a critical incident and what is a minor inconvenience.

Response times by severity, and a named engineer. Somebody who knows your deployment rather than a ticket queue, plus an escalation path for when that person is unavailable.

A path for narrowing. Who may restrict what the system does while an issue is investigated, and on what basis — agreed before the first incident, not during it.

Model changes tested before release. Deprecations and new versions from the model provider checked against your evaluation set before they reach users.

Data access. As narrow as possible, time-boxed, logged and written into the contract. Usually metrics, logs and a sampled evaluation set are enough, rather than the whole document corpus.

Maintenance windows, reporting and exit terms. What is handed over when the engagement ends, in what format and within what window — settled while you still have negotiating leverage.

How to write the severities

This is the one part of the contract the supplier should not write for you, and it usually takes the least time. A few examples per level are enough, phrased in terms of consequences rather than components.

Critical. The system gives wrong information on which somebody makes a financial, legal or safety decision — a wrong price, deadline or contract condition, for instance.

High. The system is wrong in a way users notice and correct, but which costs them time or undermines their trust in every other answer.

Low. The answers are correct in substance but worse in form: too long, inconsistent, less readable than before.

The examples matter more than the definitions. At the first incident nobody will interpret an abstract description of a level — everybody will compare the situation with the example.

What nobody can honestly promise

That the model will never be wrong. A supplier who promises that is describing something other than an AI system.

A fix within a fixed time for any defect. Diagnosing a quality problem can take longer than narrowing the system’s scope. What can honestly be promised is a response and containment, not a deadline for removing the cause.

Constant quality in a changing environment. When your documents or your users’ questions change, a contract can promise detection and response, not that nothing moves.

When any of these promises appears in a proposal, it is worth asking exactly how it will be measured. The answer usually shows that it will not be.

What to look at in the price

Pricing per user or per request means the fee rises because the system succeeded. Pricing per system and scope does not, and it aligns the two sides’ interests better.

The second thing is the model bill. Is usage cost inside the contract or beside it, and who is responsible for explaining its movement every month? A system whose cost nobody can attribute will eventually be judged on the invoice rather than on what it does.

That is how the SLA is built in AI managed services: response times by severity, a named engineer, a defined scope of quality monitoring — and a plain written statement of what cannot be promised.