Why Do AI Pilots Stall Before Production?

The demo went well. The model answered every question, the numbers looked good, and everyone agreed it was promising. Three months later nothing is in use. It is a familiar ending for an AI project, and the model may be the one part that worked.

A pilot tests the model on the task; production tests the task inside the organisation. The pilot proves the model can read the claim, find the clause, or match the product, on data someone chose. Production asks whether it can do that in the real inbox, against the real systems, under the real rules, for people who have to change how they work. Pilots stall in the distance between those two tests.

How common it is

Common enough to have a forecast. Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs and unclear business value. None of those four reasons is about the model. Each is about something the pilot did not have to face.

What the pilot never met

The author's short film on the forward deployed consultant opens with the usual story, told as an illustration. In the author's film, a claims assistant scores 94% on 40 test files, then meets fourteen systems that nobody knows all of, and the rule that matters is on a sticky note: over $10k, ask Priya. The demo was honest. It was tested on the wrong thing.

The right column is where pilots stall:

• The real data is scattered, inconsistent and partly in spreadsheets that someone treats as the system of record.
• The real systems have permissions the pilot never asked for, versions nobody wrote down, and no API where the pilot assumed one.
• The unwritten rules decide most of the outcomes: the threshold on a sticky note, the person everyone asks, the exception that is handled by phone.
• The people were not in the demo. They have a way of working that the pilot quietly assumed would change.

The questions before the build

Most of this can be found before anything is built, by asking. The author asks these questions before anything is built: what kind of workflow it is, which parts are deterministic, whether retrieval helps, where a model adds value, whether anything needs an agent, how much autonomy is acceptable, where human review sits, how the system will be evaluated, what happens when it fails, and whether anyone will use it.

Several of those questions lead to a smaller answer than the pilot proposed. A part with clear rules becomes ordinary automation. A part that looks things up becomes retrieval. Only the part that needs judgement gets a model, and an agent only gets as much room as its bounds allow. The working rule is drawn on what is a forward deployed consultant.

Measure on the real work

A demo score answers whether the model can do the task. The question production needs answered is whether a person can use the result in the real workflow, and only a test on real work answers it.

A published retail case shows the difference. In a retailer's photo search for items with missing barcodes, the correct product came first in 50 of 89 test searches and appeared in the top five in 78, so the team made the result a shortlist that staff check before using it. The same numbers could have been reported as a 56% failure or an 88% success. Measured against the job, they said something more useful: the system is good at narrowing the choice, and a person should make it. The case study names its next step before wider use: testing photos taken in the aisle and measuring how long identification takes. That is a pilot moving toward production on evidence, one question at a time.

How to decide what counts as good before measuring is on evaluation is a human problem.

Ship small, put people where the stakes are

The film closes the illustration the same way. In the film, the spec routes claims over $10k to human review, approves the rest automatically and logs every step, and time per claim falls from 4.0 days to 2.2 over six weekly releases. The rule that lived on a sticky note became a route in the spec. The person who knew the override kept the decisions that need her. Each release was small enough that a Tuesday breakage was fixed by Friday.

Where people should approve, and what they need in front of them to do it, is human-in-the-loop design.

Who does this work

Somebody has to sit with the real work long enough to see the sticky note. That is the job the forward deployed roles were created for: people from the product company placed inside the customer until the product works there. What does forward deployed mean covers the roles; the forward deployed consultant is the one who owns the path from the problem to adoption.

Getting a pilot through is a different decision from declaring it ready. Whether the evidence justifies letting real users depend on it is when is an AI system ready for real use.

What this changes

If a pilot has stalled, look for what it was never tested on: the real data, the real systems, the unwritten rules, the people. If you are planning one, test it there from the start, measure on the organisation's own work, and decide where a person stays in charge before the demo, not after.