Delivery metrics that lie once agents write the code

A quarter after a team adopts coding agents, the engineering dashboard looks like a success. More pull requests, more tickets closed, more code merged, velocity up. Leadership asks whether it is working and the dashboard says yes.

The dashboard is measuring activity, and activity is precisely the thing agents made cheap.

This is not a new problem, since any number that is easy to move eventually gets moved. What is new is that agents move these particular numbers a long way without anybody intending to game anything.

Why the familiar numbers inflate

Output metrics count the part that got cheap. Lines of code, pull requests and closed tickets were always proxies for human effort, and effort was a tolerable proxy for progress while it was the constraint. It no longer is. These numbers now count generation.

Cycle time starts at the wrong moment. Measured from the first commit or from opening a pull request, it skips the part that got harder, which is deciding precisely what to build, and gives too little weight to the part that got slower, which is review.

Velocity is self-reported. Story points are estimated by the people whose output they measure. When the work gets faster the estimates move, and the trend says more about calibration than about delivery.

None of this makes the numbers useless for a team planning its own week. It makes them useless as evidence that the method works.

What still means something

Most of what follows is the familiar DORA set (deployment frequency, lead time for changes, change failure rate and recovery time) with two adjustments that matter once agents are involved: the clock starts at the decision, and every number is split by who wrote the change.

Lead time from an approved specification to production. The clock starts when the decision is made, not when generation starts, and stops when users have the change. This is the number agents are supposed to move. If it has not moved, the speed went somewhere else.

Change failure rate, split by origin. The share of changes that cause an incident or a rollback, reported separately for agent-generated and hand-written changes. Without the split, a deterioration in one is hidden by the other.

Rework. How much merged code is changed again within a few weeks, for reasons other than new requirements. Plausible, subtly wrong code shows up here long before it shows up in incident reports.

Escaped defects by where they should have been caught. Specification, tests, pipeline gates or review. A raw defect count tells you something is wrong; the classification tells you which part of the process to fix.

Alongside these sit the review measures (changes reviewed per day and the size of the unreviewed backlog) that keep the real constraint visible.

Infographic

What to measure when agents write the code

Activity

• –Pull requests opened and merged
• –Lines of code
• –Story points per sprint
• –Time from pull request to merge

Delivery and quality

• →Lead time from approved specification to production
• →Change failure rate, generated versus hand-written
• →Rework within a few weeks of merge
• →Escaped defects by the stage that missed them

Split by kind of work

Agents help unevenly. On well-specified, repetitive work such as integrations, adapters and migrations, the gains are real. On novel or poorly understood work, writing a precise specification can take longer than writing the code used to.

An average across both hides the only question worth answering: where agents pay for themselves and where they cost. Tag the kind of work when it is planned, not afterwards, and report every number by tag.

Measure before you switch

The most common gap is the baseline. If nobody measured lead time and failure rate before agents arrived, the only comparison available is the team’s memory, and memory makes a poor baseline.

A month of measurement before the rollout costs very little. Afterwards it cannot be bought at any price.

Then measure the same way afterwards. Redefining lead time in the same quarter as changing the tooling makes any comparison meaningless, however favourable it looks.

A test for any metric

A number that rises when you simply generate more is not a delivery metric. Ask that of everything on the dashboard.

In spec-driven development the error rate is measured openly from the first loop, and it is these numbers, not enthusiasm, that decide whether the method expands to more of the codebase or stops. Stopping is a legitimate outcome, and better learned after four weeks than after four quarters.