The demo is the cheap part. How to budget for the rest

The demonstration went well. Somebody built it in a fortnight, it answered the questions the room asked, and the obvious next question was how long it would take to roll out. The honest answer is usually a good deal longer than the demo suggests, and not because the demo was a trick.

AI prototypes are unusually easy to build because the model does the visible part. What it does not do is everything that lets people depend on the result: knowing whether it is right, finding the right document, staying inside a cost, respecting who may see what, and surviving a change. That work is most of the project, and it is almost invisible in a demo.

What sits underneath the demo

Not as a technical checklist, but in terms a budget holder can test.

A way to know whether it is right. A set of real cases with known answers, scored on every change. Without it nobody can say whether next month’s version is better than this one, and the project slows down until somebody stops touching it altogether.

Retrieval that finds the right thing. The demo was given a curated handful of documents. Production has every version of every document, including the out-of-date ones. Most bad answers start here.

Cost and speed inside a budget. Measured per request and per task, because a feature that costs pennies in a demo can cost real money at production volume, and the bill arrives after the commitment has been made.

Permissions and failure behaviour. Who may see which answer, and what the system does when it cannot answer: refuse, hand the case to a person, or produce something plausible that nobody catches.

The ability to change it. Monitoring, documentation and deployment that your team can run. A system that only its builder can modify is a dependency, not an asset.

Infographic

What sits underneath a demo

1. 01The demoThe model answering well-chosen questions. The visible part.
2. 02EvaluationReal cases with known answers, scored on every change.
3. 03RetrievalFinding the right document among every version of it.
4. 04Cost and latencyMeasured per request and per task, inside a budget.
5. 05Permissions and failuresWho may see what, and what happens when it cannot answer.
6. 06OwnershipDeployment, monitoring and documentation your team can run.

Fund it in stages, not as one number

The common mistake is to ask for a single estimate from demo to full rollout. Nobody can give an honest one, so you get a padded one or an optimistic one, and both are worse than staging the decision.

A structure that works: a short first stage that builds the evaluation set and puts one complete path into production for a small group. It is small enough to price honestly, and it surfaces the unknowns — the messy data, the permission model nobody wrote down, the cost per request at real volume — while they are still cheap. The decision to widen is then taken on what that stage measured, not on what the demo suggested.

What it costs you in time and attention

More than a demo, less than people fear. The work needs a product owner who can decide what counts as a right answer, a few domain experts to help build the evaluation cases, access to real data under real permissions, and your own engineers involved from the start if you intend to run the system yourselves.

The experts are usually the hard part. Their time is what turns “the answers look fine” into a set of cases with known answers, and they have day jobs. Book their hours before the project starts, not when the team is already waiting for them.

How to judge whoever builds it

Ask what they build first. If the answer is features, the project will be fast for three months and slow after that. If it is a way to measure quality, they have done this before.

Ask how they will know a change made things worse. A specific answer involves cases and scores. A vague one involves testing carefully.

Ask what the first stage delivers. Something running in production for a small group, not a report. A report at the end of a stage that was meant to run something means the risk was moved, not reduced.

Ask for cost per task, not per token. A provider who cannot give you that figure after the first stage has not measured it.

What a good outcome looks like

Not a bigger demo. A system in production on real data, with a quality number that moves when something changes, a cost per task that someone in finance can read, and a team on your side that can modify it without calling anybody.

That is the scope of AI-native apps engineering: the part after the demo, which is most of the work and decides whether the demo was worth building.