A pilot should buy a decision, not a demonstration

Almost every hardware pilot succeeds. Devices get installed, data appears, somebody demonstrates a screen in a meeting, and the phase is written up as a success. Then the rollout does not happen, or happens two years later after a second pilot, and nobody can quite say what changed in between. The pilot was not a failure. It answered a question that was not the one blocking the decision.

The pattern is consistent enough to state as a rule. A pilot proves feasibility in the conditions of the pilot. If the pilot ran in the accessible corner of the site, on the machine that is easiest to instrument, with an engineer available throughout, then what it demonstrated is that the thing works under attention. Nobody doubted that. What the organization needs to know is whether it works at three hundred units with nobody watching, and the pilot as usually scoped is silent on exactly that.

Write the decision first¶

The discipline that fixes this is small and unpopular, because it happens before anything visible gets built. Write one sentence naming the decision the phase exists to enable, in the form: at the end of this we will commit to X, or we will not, and the thing that decides it is Y. Then scope backwards to whatever is minimally sufficient to produce Y.

The sentence is unpopular for a reason worth naming. It makes the possibility of a no explicit, and pilots are frequently commissioned by somebody who has already decided and needs evidence to carry a budget request. That is a legitimate political use of a phase and it is not the same activity, and knowing which one is being bought changes what should be built. A firm that will not say what a no looks like is not buying a pilot.

The four unknowns worth spending a phase on¶

Support load is the one that gets left out and the one that most often kills a rollout after it starts. During a pilot, small interventions are invisible: somebody power-cycles a unit, re-seats a connector, restarts a service, and none of it is recorded because it took four minutes and the person was already there. At fleet scale each of those becomes a dispatch. A phase that does not count them produces an operating estimate that is wrong by an order of magnitude, in the optimistic direction.

Run it where it is hard¶

A pilot attracts constant, reasonable pressure to run where it is easy: the accessible machine, the new building, the shift with the engaged supervisor. It makes the phase cheaper and faster and it makes the result worth much less, because the rollout is not going to be conducted in the accessible corner. The rollout is going to reach the far end of the yard, the machine from 1994 with no spare terminals in the panel, and the shift that was not consulted.

The economical compromise is to run a small number of units and place them adversarially: one in the easy location as a control, one in the worst physical environment in scope, one at the edge of the connectivity envelope, one on the oldest equipment. Four units placed that way tell you more than twenty placed conveniently, and they cost less.

Criteria are numbers, agreed before¶

After the fact, every pilot can be narrated as promising. The readings mostly tracked, the outages were mostly explicable, the operators mostly liked it. Nothing in that sentence is dishonest and nothing in it supports a decision, which is why the criteria have to be fixed in advance and have to be numbers.

• Agreement with the reference stated as a bound, not as a correlation: within this much, this often, including during the events that matter.
• Availability as a fraction of the period, with the definition of unavailable written down, because an instrument reporting a stale value is not available and will otherwise be counted as up.
• Interventions per device per month, with no exemption for the ones performed by whoever was passing.
• Time to recover a failed unit without a site visit, measured on a unit that was actually failed on purpose.
• One human criterion, measured by watching rather than by asking: did the work that was supposed to move to the terminal move there, or did it stay on paper alongside it.

That last one is regularly the decisive result and it never appears in a technical specification. A terminal deployment where the paper record survives beside the screen has failed, whatever the uptime figure says, and the only way to learn it is to stand on the floor at shift change and watch what people write on.

Budget the removal¶

A phase should be scoped so that a no is cheap and honest, which means somebody has to own taking the hardware out, restoring what it displaced, and saying what happens to the data collected. Where nobody owns it, the pilot hardware quietly stays, becomes load-bearing, and the operation ends up running production on units chosen for a trial and on a build nobody planned to support. That is the most common way a successful pilot becomes an expensive estate.

Where this argument stops¶

Insisting that a pilot buy a decision can be over-applied. A phase that tries to settle every unknown at once is a rollout with a smaller budget and a worse name, and there is a real cost to the deliberation: while the criteria are being negotiated, the plant is still running with the problem the project was meant to address. Two unknowns per phase is usually the honest ceiling, and picking which two is the work.

There are also projects where the answer is genuinely known and a pilot is ceremony. If comparable hardware is already running elsewhere in the same estate, in the same conditions, the useful phase is not a trial but a rollout with the first ten units watched closely. Insisting on a pilot there is a way of spending a quarter to learn what the building next door could have told you.