The Most Underrated AI Practice: Deciding in Advance When to Walk Away

Two surveys landed in the last few weeks that, read together, say something uncomfortable about how most of us are running AI programs.

Gartner surveyed more than 2,000 CIOs and found that only 13% say their enterprise has realized significant or transformational value from AI, and that roughly one in five use cases succeeds (Gartner). Ten days later, Deloitte's survey of 3,500 leaders reported that 81% of companies see faster decisions and execution, but only 25% are using AI to deeply transform the business (Deloitte).

Speed is showing up. Value, mostly, is not. And the detail in the Gartner release that stuck with me was not the 13%. It was this: the organizations getting results are far more likely to have written rules for when to stop.

As someone who leads an AI-agents initiative in banking and just finished an MBA, I think that is the most practical lesson in either report.

Volume is easy. Value is a discipline.

Gartner's analyst put it in three words: "Volume is not value." I have watched that play out. Once a team gets a capable model and a framework like Spring AI wired up, building a new use case takes days, not months. That is a good thing, until the portfolio fills up with pilots that each look promising and none of them has a clear finish line.

According to Gartner, organizations with guidelines for stopping a pilot before production were 2.3 times more likely to be in its high-performing group, and those with rules for retiring low-value AI already in production were 2.6 times more likely. Yet only 31% and 17% of CIOs reported having those practices. Most of us are better at starting AI work than ending it.

At Priceline, the culture around experiments was blunt: every test had a metric, a duration, and an expected decision. If the numbers did not move, the feature came out, no matter how clever it was. AI work deserves the same treatment, and in my experience it rarely gets it, because AI demos are persuasive in a way that A/B tests are not.

Why AI makes stopping harder

A few things make AI projects especially sticky.

• The demo is always impressive. A model that handles the happy path well creates momentum that the long tail of edge cases does not undo on its own.
• Costs arrive later and vary. Gartner notes that AI costs are probabilistic and set at run time, unlike a fixed license. Deloitte found 71% of companies track total inference and API spend, but only 44% track per-unit inference cost. If you cannot see cost per outcome, you cannot tell when a use case is underwater.
• Value is spread thin. Saving thirty seconds per task across a team is real, but it is hard to point to on a P&L. Deloitte's finding that AI mostly speeds up existing workflows without changing the work matches what I see.
• Nobody wants to own the "no." Killing a pilot feels like admitting failure, so it lingers in a half-supported state that costs more than either finishing or stopping.

What stopping rules look like in practice

When we scoped AI agents for a high-volume service ticket portal, the questions that mattered most were the ones we answered before writing code. Here is the shape I now push for on every use case:

1. Name the outcome and the baseline. Not "use AI for triage," but a measurable change against today's numbers, such as routing accuracy, time to first response, or reopen rate.
2. Set the pilot exit criteria up front. Write down the threshold for moving to production and the threshold for walking away. Put a date on it.
3. Track cost per outcome, not just total spend. Tokens per ticket, cost per resolved case, human review minutes per action. If the unit economics do not work at pilot scale, they rarely improve at production scale.
4. Count the human-in-the-loop cost honestly. We keep human approval on higher-risk actions, and I would not change that. But reviewer time is part of the cost, and it belongs in the business case.
5. Schedule a production review. For use cases already live, revisit value and cost on a fixed cadence, and be willing to retire one when a simpler approach does the job.

None of this slows a team down. It actually speeds up the good work, because people stop splitting their attention across projects that should have ended.

The people side is the real bottleneck

Both reports point at the same deeper issue. Deloitte found that 60% of companies have redesigned jobs around AI to some degree, but only 15% report a significant shift in who makes decisions. Gartner found only 4% of CIOs have explicit plans to redesign roles for a human-agent workforce.

That matches my experience as a manager. Tools like GitLab Duo make engineers faster, and agents can take routine work off a support team's plate. But if roles, approvals and decision rights stay exactly the same, the time saved gets absorbed rather than redirected. The value comes when you decide, deliberately, what people will do with the capacity AI gives back.

Takeaways for engineering leaders

• Make "when do we stop?" a standard question in every AI kickoff, right next to "what problem are we solving?"
• Build cost observability into the platform, so every use case reports unit cost and volume without a special project.
• Partner with the business on the decision to walk away. Stopping is a joint call, not an IT veto.
• Treat a retired pilot as a win when it frees people and budget for something better. Say so publicly so teams are not afraid to recommend it.
• Plan the role changes alongside the technology, not after it.

Closing thought

The surveys make it tempting to conclude that AI is overhyped. I do not read them that way. The tools are working; our management habits have not caught up. The teams pulling ahead are not the ones running the most pilots. They are the ones that decided in advance what success looks like, measured it honestly, and had the discipline to walk away when it was not there.

These are my personal views, not those of my employer.