Decision Doc: The Memo That Stopped a CTO-Sponsored AI Initiative

This series publishes sanitized versions of real decision documents from my work. Names, internal system references, and the sponsor's identity are removed. The numbers, the test design, and the outcome are real.

Context for readers. The initiative below had executive sponsorship, allocated engineers, and a demo that looked convincing. My job was to decide whether it should survive contact with a measured requirement. Killing funded work is the least fun and most valuable version of prioritization, so this is the document I am most often asked about in interviews.

The memo, as written

Recommendation. Hold. Stop feature work now, keep the learnings, redirect the three engineers to the workflow tool with the strongest adoption signal.

What we set out to prove. The concept requires 95 percent extraction accuracy on the target document class before its output can enter a case without full human re-review. Below that bar it adds a review step instead of removing one, which makes it net negative for throughput.

What the test showed. Over three weeks we ran the strongest available configuration against a representative sample. Accuracy plateaued at 71 percent. The error analysis matters more than the number: the residual failures concentrate in exactly the document subtypes with the highest downstream cost, and they stem from source-quality variance that model choice does not fix. This is a ceiling, not a tuning problem.

Why not iterate toward the bar. Every credible improvement path attacks the input variance, which is a data-acquisition problem owned upstream, not a modeling problem this team can solve. Spending another quarter here buys single-digit gains against a 24-point gap.

What we keep. The evaluation harness, the labeled sample, and the error taxonomy transfer directly to two adjacent document classes where the same test shows the bar is reachable.

Cost of being wrong. If a future model release moves the ceiling, we re-run the same three-week test at trivial cost. The harness makes this decision reversible. Continuing to build makes it expensive.

The redirect. The three engineers move to the transcript-interpretation tool, where shadowing work showed specialists spending 38 percent of their day on a problem our current stack handles well. That tool became the highest-adoption internal launch of the year.

What happened

The sponsor accepted the hold in one meeting. Not because the memo was persuasive prose, but because the requirement had been agreed in writing before the test started. The 95 percent bar was the sponsor's own number. The memo only reported which side of it we landed on.

That is the reusable mechanism: get the kill threshold agreed while everyone still believes the project will clear it. A feasibility test without a pre-agreed bar is a demo, and demos always pass.

I have now written versions of this memo twice in my career, once for this initiative and once for a 5G edge product where only 2 of 31 prospective buyers showed a genuine need for the latency we were selling. Both times the redirected team shipped something that generated real usage or revenue within a year. Both memos were one page.

Related: the release gate that produced the accuracy evidence is documented in the shadow-mode eval brief.