Decision Doc: A PRD Where the Core Feature Is Saying No

This series publishes sanitized versions of real decision documents from my work. Names and internal system references are removed. The numbers, requirements, and outcome are real.

Context for readers. The product was an internal research assistant answering tax-code questions for case specialists. Its answers informed filings, which means a confident wrong answer was strictly worse than no answer. The interesting product problem was not making it smarter. It was making it stop.

The PRD, as written

Problem. 14 percent of assistant answers contain claims unsupported by the retrieved sources. Specialists cannot reliably tell which 14 percent. Every unsupported claim that reaches a filing is a compliance exposure, and the review burden of catching them erases most of the assistant's time savings.

Non-goals. Raising answer coverage. Answering questions outside the retrieved-source boundary. Sounding confident.

Requirement 1: refusal is a first-class answer. When source support falls below threshold, the assistant returns a structured refusal: what it looked for, what it found, why that is insufficient, and the escalation path to a human researcher. A refusal that strands the user is a dead end. A refusal that routes them is a feature.

Requirement 2: the threshold is a product decision, not a model setting. Support confidence is scored against retrieved passages. The refusal threshold is set where the unsupported-claim rate reaches the compliance target, then reviewed quarterly against the false-refusal rate. Engineering owns the scorer. Product owns where the line sits.

Requirement 3: refusals are measured as rigorously as answers. Every refusal is logged with its confidence score and sampled weekly for review. A refusal on an answerable question is a defect with the same severity as an unsupported claim, because it teaches specialists to stop asking.

Requirement 4: the refusal rate has an owner-approved budget. Modeling put the projected refusal rate at 9 percent of queries at the target threshold. That number was taken to compliance and operations leadership for explicit written sign-off before launch, because the alternative was those 9 percent of queries silently receiving answers we could not defend.

Acceptance test. Unsupported-claim rate at or below 2 percent on the weekly audit sample, false-refusal rate below 3 percent, and zero unsupported claims in any filed document traced back to the assistant.

What happened

Unsupported claims fell from 14 percent to 1.8 percent. The refusal rate landed at the projected 9 percent, and compliance approved it in writing. Specialist trust moved in the direction that mattered: usage grew after launch, because an assistant that visibly knows its limits is one you can build a workflow on.

The design lesson I reuse everywhere: the refusal budget conversation is the actual product decision. Any team can lower a threshold. Getting a leadership signature on the sentence "we accept that 9 percent of questions will be declined, on purpose" is what turns a model behavior into an accountable product policy.

If you are specifying an AI feature in a domain where wrong answers have owners, write the refusal requirements before the capability requirements. The capability list will survive contact with reality. The refusal list is the reality.

Related: the release gate this assistant had to pass is documented in the shadow-mode eval brief.