A pilot with one crew is a specification, not a demonstration

Field pilots tend to be designed to succeed. The best-run site is chosen, the most enthusiastic crew volunteers, the provider’s engineers are on hand every day, and after a month the report says the system works. Then it goes to the other sites, where none of those conditions hold, and it starts failing in ways the pilot never saw.

The pilot did what it was set up to do. It was set up to answer the wrong question.

What a pilot is for

A pilot is the cheapest place to find out how a field system fails: where connectivity drops, which step people skip, what happens at shift change, which screen nobody can read in sunlight, where the paper workaround comes back. Every one of those is a requirement for the rollout, and none of them can be discovered from an office.

So the measure of a good pilot is not how few problems it had. It is how many it surfaced while there was still only one crew to apologise to.

Which is why a pilot that found nothing should prompt suspicion rather than relief. It usually means it ran in conditions the rest of the organisation does not have, or that nobody was collecting complaints.

Choosing the crew and the site

Typical, not best. Pick a crew and a site that look like the rest of the estate: ordinary connectivity, an ordinary shift pattern, an ordinary level of enthusiasm. A pilot at the head-office site with the best network proves that the system works at the head-office site. If your sites differ a great deal, choose the kind that occurs most often and leave the harder ones for a second wave.

One crew or one line. Small enough to watch closely and support properly, large enough to cover a full cycle of the work, including nights and handovers if those exist.

Include a sceptic. Somebody who has seen a previous system withdrawn will find the failure modes faster than anybody who wants it to work. Their complaints are worth more than their approval.

What to agree before it starts

A route back. The old process stays available, and everybody knows how to fall back to it. A pilot with no way back turns every problem into an incident, and people stop reporting problems.

How complaints are collected. Not a feedback form at the end. A short daily conversation, somebody from the build team on site for part of the time, and a written log of every workaround observed.

The exit criteria. Decided in advance, in operational terms the crew would recognise.

Infographic

How a pilot turns complaints into requirements

1. 01Work a shiftThe crew uses the system in the real conditions.
2. 02Collect complaintsA daily conversation and a log of every workaround observed.
3. 03Fix the causeChange the system or the process, not the crew.
4. 04Release to the crewBack to the same people, with what changed explained.

Exit criteria that mean something

A full shift without thinking about the system. The crew completes the work and nobody has to stop to work out what the device is doing.

Nothing lost silently. Every record made offline arrives when the connection returns, and conflicts are settled by the agreed rules, not by whoever happened to sync last.

The paper has gone without being banned. If people still keep a notebook next to the device, the notebook is doing something the system does not, and that is the next requirement.

Support without a site visit. The faults the pilot produced could be diagnosed and fixed remotely. If every fault needed somebody to drive out, the rollout will need a fleet of people driving out.

What it costs you in time and attention

An operations lead who can settle process questions within the week. A crew whose output targets allow for the slowdown that any new tool causes in its first weeks. Somebody from IT for devices and networks. One person on the operations side who reads the workaround log with the build team every week and decides which problems belong to the system and which to the process. And a willingness to extend the pilot when it surfaces something real, rather than treating the original end date as a promise made to the board.

A pilot with one crew is typically six to ten weeks, including the site work. Extending it by a few weeks because it found a genuine problem is cheap. Rolling a known problem out to every site is not.

What we do

In field deployment AI engineering the pilot is one crew or one line, watched closely, with a route back — and the complaints from that group are treated as the specification for the rollout, not as feedback to be managed.