Write down what would make you stop before the pilot starts

Most pilots of agent-written code have a start date, a scope and a demonstration at the end. Few have a written answer to the question that matters most to whoever funds them: what result would make us stop?

Without that answer a pilot cannot fail. It produces mixed results, as pilots do, and mixed results get read as a reason to extend. Six months on, the method is half-adopted across a few teams, nobody can say whether it works, and the budget conversation is about sunk cost rather than evidence.

What to agree before the first specification

The pilot itself should be short and bounded: a well-understood, well-tested area of the codebase, and two or three real features specified to the precision the method needs. What makes it decisive is what is agreed before it starts.

The baseline. What the same kind of work costs today, in elapsed time and in engineering effort, from decision to working change. It does not need to be precise; it needs to be agreed. Without a baseline, any result looks like progress.

What will be measured. How long specifications take to write and review. How many generated changes pass review, and how much rework the rest need. How many defects escape to later stages. Measured openly, with failures counted rather than smoothed.

The widening criterion. The result at which the method is extended to more of the codebase, stated in terms of those measures.

The stopping criterion. The result at which it is not. Stated just as precisely, and treated as a legitimate outcome rather than an embarrassment.

Who decides. One named person with the authority to stop, who has agreed to the criteria in advance. Not a committee, which will usually find a reason to extend.

Two choices that decide the result early

The area. The least understood part of the codebase guarantees a stop for the wrong reason. A trivially easy one guarantees a decision to widen that will not hold anywhere else. The right area is ordinary, well tested and representative of work you do a lot of.

The people. A pilot run only by the most enthusiastic engineer measures that engineer. Include somebody sceptical. Their reading of the result is the one the rest of the organisation will trust.

Reading the result

Results rarely point cleanly in one direction, and the useful reading lies in why. Two questions do most of the work: was the specification hard to write, and did the generated changes need much rework?

Infographic

Reading the result of a spec-driven pilot

Little reworkMuch rework

Widen to similar workIt works, but specification effort decides where it pays.

WidenThe evidence supports extending to more of the codebase.

Stop hereThis area does not suit the method, which is worth knowing.

Tighten the criteriaSpecifications look easy but are vague, or the tests are weak.

Hard to specifyEasy to specify

Easy to specify with little rework is the case for widening. Hard to specify with heavy rework is the case for stopping in this area, which says something about the area and not necessarily about the method.

The two mixed cases are the interesting ones. Specifications that were easy to write but produced changes needing heavy rework usually mean the acceptance criteria were vaguer than they looked, or the tests did not test what mattered. Specifications that were hard to write but produced clean changes mean the method works where you can afford the specification effort, and the decision becomes one of where.

None of the four quadrants is a verdict on the team. Each is a statement about a kind of work in a particular codebase, which is exactly the level at which the budget decision should be made.

Why stopping is a good result

A clear stop after four weeks is cheap. It tells you that, in this kind of work and this codebase, implementation is not your constraint, or that test coverage has to come first, which pays off regardless. A pilot that drifts into partial adoption costs far more, because the cost is spread across teams and quarters and never shows up in one place.

A clear decision to widen, with criteria behind it, is equally valuable. It is much easier to defend at budget time than enthusiasm, and it tells the next team exactly what they are signing up for.

What it asks of the sponsor

Very little time and one uncomfortable commitment: to accept the stopping criterion if it is met. Most of the value of the pilot depends on that commitment being real, and the team running it will know whether it is. In practice that means saying out loud, at the start, that a stop is an acceptable result, and saying it again when the result arrives.

How we approach it

Spec-driven development is structured as exactly that: a bounded area, specifications written properly, the loop run with the failure rate measured openly, and a decision to widen or stop. It is better to know after four weeks than after four quarters.