AI SLA: what a service level agreement for AI should cover

An AI SLA is a service level agreement for a system with an AI model inside it: a chatbot, a search assistant, an agent that handles cases. Besides availability and response times, it covers the things that actually go wrong in such a system: answer quality, model changes on the provider’s side, cost per case, latency, security incidents, handing a case to a person, and where data is processed. Each item has a definition, a way to measure it and an agreed response when the result leaves the agreed range.

This guide is for leadership, procurement and IT leads at a company that runs an AI system in production, or is negotiating a contract to build or operate one. The narrower question of what a vendor should never promise is covered in an SLA for an AI system: what can be promised. This piece is the full map: definition, measurement, negotiation and a first step.

Why a classic IT SLA is not enough for AI

A classic service contract describes a service that either works or does not. It measures uptime, time to respond to a ticket and time to fix. For an HR system or an online store that is enough, because failure is visible: the page does not load, the invoice is not issued.

A system built on a language model rarely stops working. More often it starts working worse, and quietly. Seven differences that belong in the contract:

Quality drift. Answers degrade gradually because documents, user questions or the search index changed. The service is fully up, and for three weeks it has been quoting an outdated price list. The mechanism is described in AI systems do not crash, they drift.

Model changes by the provider. The model provider ships a new version, changes default behaviour or retires an old one. Your code has not changed, yet the system answers differently. A contract that does not anticipate this does not say who tests the change, or when. What to do once a retirement notice lands is in the vendor retired your model: what happens to your system.

Cost per call. In an ordinary application cost grows with users. In an AI system you pay for tokens, meaning the length of the question, the context and the answer. An agent stuck in a loop, or one that attaches the whole product catalogue to every case, raises the bill with no rise in traffic.

Latency. Response time depends on the model, context length, the number of agent steps and load at the provider. A monthly average hides the cases where a customer waited a full minute.

Security incidents. Malicious content in an email or a document can change an agent’s behaviour, and an agent with tools can perform an action nobody anticipated. The server log will show only “request served”.

Escalation to a person. Some cases the system should hand over to an employee. If the contract is silent on this, nobody knows who picks the case up, how fast, and with what context.

Where data is processed. Prompts and answers are company data, often customer data. Processing region, retention at the model provider and access by its operators are terms that belong in the contract, not in terms of service the provider can change on its own.

Infographic

Classic SLA vs AI SLA

Classic SLA

  • Service uptime
  • Time to respond to a ticket
  • Time to fix an outage
  • Priority by technical impact

AI SLA

  • Answer quality on a test set
  • Model changes tested before release
  • Cost and latency per case
  • Priority by business consequence

What an AI SLA should cover

Nine components follow. Not every system needs all of them, but for each one it is worth deciding deliberately whether it sits inside the contract or outside it. Each comes with a way to measure it. Threshold values in the examples are illustrations for the parties to agree on, not a standard and not an offer.

ComponentWhat it describesHow to measure it
Availabilitywhether the system accepts and handles casesshare of successful requests in the measurement window, excluding agreed maintenance windows
Answer qualitywhether answers are correct and grounded in the right sourcesscore on a fixed test set plus review of a production sample
Retrieval qualitywhether the system finds the right documentsretrieval accuracy on questions with a labelled correct source
Model changeshow a new model, prompt or index version is introducedtest-set score before and after the change, entry in the release log
Costwhat one case costs to handletoken and infrastructure cost per case, with an alert threshold
Latencyhow long the user waitsend-to-end time per case as a percentile, for example the 95th, not an average
Escalationwhich cases go to a person and how fastshare of cases handed over, time to pick up, completeness of handed-over context
Incidentswhat counts as an incident and what the first response istime from detection to narrowing the system’s scope, post-incident record
Datawhere it is processed and how long it is keptregion, retention, list of parties with access, confirmed in the report

A few notes on the table.

Quality needs a test set that belongs to you. Dozens to a few hundred real cases with expected outcomes, including hard ones and ones that have gone wrong before. The process owner builds it, not the vendor. Without it, “we monitor quality” has no content anyone can enforce. Where to begin is covered in build the evaluation harness first.

Quality is measured as a trend and a threshold, not a promise. An example to agree on: “the test-set score does not fall below the acceptance baseline by more than an agreed margin; a drop triggers an alert and an investigation.” That can be checked. “The model will not make mistakes” cannot.

Cost and latency are product decisions. A cheaper model answers faster and worse, a more expensive one the other way round. The contract should say who makes that call and on what data. More on this in cost per task and response time are product decisions.

Escalation has two sides. The vendor is responsible for the system handing a case over with full context. The company is responsible for somebody receiving it. How to design that handover is in where the automation stops, somebody has to start.

The first response to an incident is narrowing, not fixing. Diagnosing a quality problem can take days. Narrowing what the system is allowed to do, for instance switching to human approval, can take minutes. The contract should say who has the right to do it. The first steps are walked through in the first hour of an agent incident.

SLA, SLO and SLI: three acronyms, one structure

These terms come from the practice of running internet services, and they carry over to AI well.

SLI (service level indicator) is what you measure. For example: share of test-set answers rated correct, the 95th percentile of time per case, cost per case.

SLO (service level objective) is the value the team aims for. For example: “95th percentile handling time below an agreed number of seconds.” An SLO is an internal tool. Missing it means the team pauses new changes and works on stability.

SLA (service level agreement) is a commitment between parties, with consequences. It usually covers fewer indicators than the SLOs and has looser thresholds, because breaching it has contractual effects.

In AI systems the distinction matters in practice. Many quality indicators are better kept as SLOs: measured, reported and discussed monthly, without a penalty for every movement. What belongs in the SLA is what is unambiguous and genuinely in the vendor’s control: availability, response times, testing model changes before release, the scope of quality monitoring, data terms. Attaching penalties to every dip in a test-set score usually ends with the vendor resisting measurement instead of improving the system.

How to negotiate an AI SLA: questions to ask a vendor

Answers to simple questions tell you most about an offer. Ask them before you sign, while you still have leverage.

On quality. How do you define a good answer in our process? Which set of cases do you measure quality on, and who prepares it? How often do you run it, and what triggers an alert?

On model changes. Which models does the system use, in which versions? What happens when the model provider announces a version retirement? Does every change to the model, prompt or index go through our test set before release? Where are those changes recorded?

On cost. Is model usage cost inside the contract or next to it? Who explains each month why it moved? What happens when cost per case crosses the agreed threshold?

On incidents. What do you treat as a critical incident in our process? Who can narrow the system’s scope, and on what basis? How long from detection to that decision?

On people. Who on your side knows our deployment? What happens when that person is away?

On data. In which region are prompts and answers processed? Does the model provider retain them, and for how long? Who can access traces that contain conversation content?

On exit. What do we get when the engagement ends: the test set, instructions, configuration, change history? In what format, and how quickly?

“It depends” is not a bad answer if something concrete follows it. A bad answer is one that pushes the question to later. It comes back at the first incident, when the contract no longer settles anything.

Look at the pricing model too. A fee per user or per query grows because the system succeeded. Pricing by scope and system aligns interests better. A good operations contract should also shrink over time as your team takes over the routine, which we argue in support contracts grow by default: write one that shrinks.

How to start: an AI SLA in five steps

Do not start with a contract template. Start with one process where AI already runs or a pilot is about to begin.

Step 1. Name the process owner. The person who knows what a good answer looks like and who bears the consequences of a bad one. Without them, nobody writes the priorities or the test set. Why this role so often stays empty is in who runs your AI system the day after go-live.

Step 2. Build the test set. Real cases with expected outcomes. It becomes the quality measure in the contract and the gate for every model change.

Step 3. Measure the baseline. Before writing thresholds, measure how the system performs today: test-set score, handling time, cost per case, escalation rate. Thresholds written without a baseline are either unreachable or meaningless.

Step 4. Describe priorities in terms of consequences. A few examples per level: what is critical (a wrong price, deadline or contract term), what is high, what is low. At the first incident everyone will compare the situation to the examples, not to the definition.

Step 5. Separate SLOs from the SLA. Measure and report every indicator. Move into the contract, with consequences, only the ones that are unambiguous and in the vendor’s control.

Common mistakes

Copying the software contract template. The contract will be honoured to the letter, and the quality problem will not appear in it at all.

Quality without a definition. “The vendor ensures high answer quality” has no measurement, so it can never be breached.

A test set owned by the vendor. If the vendor picks the cases, scores them and reports on them, the contract measures what is convenient for the vendor to measure.

Averages instead of percentiles. Average response time can look fine while one customer in twenty waits a minute.

Nothing on model changes. A version change at the model provider is certain to happen. A contract that ignores it does not say who is responsible.

Penalties for every movement. Thresholds that are too tight make the vendor defend against measurement. Measure broadly, penalise narrowly.

Skipping where data is processed. EU rules on AI and on data protection place obligations on your company that a contract clause alone does not shift to the vendor. Leave the compliance assessment to your lawyer and data protection officer, but ask about region and retention before you sign. How the choice of where agents run shapes those terms is covered in where should your agents run.

How we approach it at predictes

In AI managed services we design the agreement to measure what actually breaks in an AI system: quality on a test set built with your process owner, model changes tested before release, cost and latency per case, priorities written in your language, and a path to narrow the system’s scope during an incident. We aim for the level of support to shrink as your team takes over the routine. It is one possible approach, and the questions in this guide are worth asking every vendor, us included.

More about predictes on the homepage.