Cost per task and response time are product decisions

In conventional software the cost of serving one more use of a feature is close to zero, and how fast it responds is an engineering concern with a target everyone already agrees on. Nobody in a product review asks what it costs to render a page.

AI features break both assumptions. Every request has a visible price that grows with use, and the response time is set by choices — which model, how much context, how many checking steps — that also set the quality. These are three dials on one machine, and somebody is turning them. Usually it is whoever wrote the code, on the day they wrote it.

Three dials on one machine

Quality costs money and seconds. A larger model, more retrieved context, a second pass that checks the first: each tends to improve answers, and each adds cost and waiting time. The improvement is real and measurable. Whether it is worth paying for depends on the task, which makes it a product judgement rather than an engineering one.

Cost grows with success. A feature that is cheap at pilot volume can become one of the larger lines in the budget once people like it. The unit economics have to be known before adoption takes off, not read off the first large invoice.

Latency changes what the product is. An answer in a second is part of the conversation. An answer in twenty seconds is a job somebody waits for, and while they wait they switch to another tab. The same output at a different speed is a different product, with a different design.

The questions product has to answer

What is one task worth? Not one request — one completed unit of work: a resolved ticket, a drafted reply, a checked invoice. Cost per task set against value per task is the only comparison that means anything, and writing it down is often the first time anyone has put a value on the task at all.

How long can the person wait here? Inline suggestions, chat, background drafting and overnight batch each have their own tolerance, and design can move a task from one to another. A slow answer that arrives as a ready draft is often better than a fast one that interrupts.

Where is quality worth paying for? Rarely everywhere. Usually a minority of hard cases benefit from the expensive path, and the rest would not notice the difference. Routing between the two is an engineering job; deciding what counts as hard is not.

What is the ceiling? A cost per task above which the feature is redesigned rather than tuned, written into the specification like any other requirement.

Infographic

Matching the path to the task

High task valueLow task value

Route and prepareFast default, costly path for hard cases, work done ahead of the request.

Pay for qualityLarger model and extra checks, delivered as a ready draft.

Small and fastThe cheapest model that clears the bar. Anything more is waste.

Batch itRun in bulk, off-peak, at the lowest cost per task.

Needs it nowCan wait

Who owns which numbers

Engineering should make cost and latency visible per feature and per task, and know the levers: smaller models for easy work, caching, batching, shorter prompts. Product should own the targets — the ceiling, the acceptable wait, the quality bar — and the call when they collide.

When one person owns both, or nobody owns the targets, the trade is still made. It is made by default, in the code, and rediscovered every time an invoice or a complaint arrives.

The prices move too. Providers change their pricing and release smaller, cheaper models, so a feature whose economics were marginal last quarter may be comfortable now, and the reverse. With the ceiling and the measured cost per task side by side, checking that becomes a re-run of the evaluation rather than a new project.

A workable arrangement is to keep the quality bar, the cost ceiling and the latency target in the same line of the specification, so that a change which moves one of them is reviewed against the other two. “Moving to the larger model raises the score on hard cases and raises the cost per task” is a decision with a name next to it. It should not arrive as a pull request.

How we work on it

In AI-native apps engineering cost and latency are measured per request and per task from the first week, with the numbers visible per feature so the product decision can take them into account, and with the levers documented rather than found at the first large invoice. Large bills usually turn out to come from one unexamined code path rather than from the product being expensive.