Designing AI That Knows When to Stop

Imagine a career-support product that reads a project note and suggests a skill: "stakeholder management." The suggestion looks plausible. Beside it sits a green badge marked 91 percent confident.

The user still has four unanswered questions. What evidence produced that label? What did the system find ambiguous? Can it decline when the note does not support a skill? What happens when the user says the suggestion is wrong?

This is a synthetic scenario, not a description of a live portfolio feature. It exposes a common product gap: teams spend months improving model output, then reduce uncertainty to a badge. A confidence display may communicate that the system is probabilistic. It does not provide a path forward.

My product position is testable: for consequential suggestions, an experience built around evidence, uncertainty, abstention, and correction should produce fewer unreviewed errors than one built around a confidence score alone. The measures are straightforward: unsupported acceptance rate, successful recovery rate, time to correction, and task completion.

Confidence needs a recovery path

Generative systems can produce fluent claims that are false or unsupported. The NIST Generative AI Profile calls this risk confabulation and recommends actions across testing, monitoring, feedback, and incident handling. The important product implication is that a failure cannot be treated as a rare edge case hidden behind a disclaimer.

Human-AI interaction research points in the same direction. The Microsoft guidelines validated at CHI 2019 include making clear what a system can do, showing how well it can do it, supporting correction, and allowing dismissal. Google's People + AI Guidebook also frames trust as something to calibrate, not maximize.

These sources do not prove that one interface works in every domain. Microsoft evaluated general guidelines with 49 design practitioners across 20 products. Google's guide combines published research with proprietary studies whose details are not fully available. NIST is a risk-management profile, not a UX experiment. Together, they support a design constraint worth testing: the product should help a person act when the AI reaches its limit.

Four states make the limit usable

I would design the interaction as four distinct states. They can appear in a single response, but each has a different job.

Evidence shows what the answer rests on

The product should connect a suggestion to the input that supports it. In the synthetic career example, "stakeholder management" might point to the sentence describing coordination across design, engineering, and sales.

Evidence is not an explanation of the model's internal reasoning. It is the inspectable material the user can verify. If the product cannot show supporting material, that absence should affect what the system is allowed to claim.

Uncertainty asks for the missing piece

Uncertainty should change the interaction, not decorate it. If the note says only "worked with multiple teams," the system could ask what the user coordinated, which decision they owned, or what outcome followed.

A raw probability is often too abstract to answer those questions. The interface should communicate the source of uncertainty in task language: missing outcome, conflicting dates, unclear ownership, or insufficient evidence.

Abstention protects the task

When the product lacks enough support, it should stop. That may mean declining to infer a skill, offering a manual path, or saving the item for later review.

Abstention is not a universal cure. A 2024 research preprint on conformal abstention reported that a specific procedure bounded error rates on selected question-answering datasets. It relied on repeated samples, similarity judgments, calibration data, and assumptions that may not transfer to a career product. The practical lesson is narrower: abstention can be designed and evaluated as a system behavior. It should not depend on an untested self-reported confidence value.

Correction returns control

The user needs to edit, remove, or replace the suggestion without restarting the task. The product should preserve the original input, show what changed, and make the corrected state easy to confirm.

Correction also creates product evidence. Teams can study repeated rejection patterns without treating every edit as model training consent. Feedback collection, retention, and secondary use need their own explicit rules.

The Recovery State Card makes the decision inspectable

For each consequential AI output, I would require a compact product artifact with these fields:

Applied to the synthetic example, the system could show the proposed skill and the supporting sentence. If ownership is unclear, it asks one question. If no evidence exists, it abstains and offers manual entry. If the user replaces the skill, the product confirms the final wording and returns them to the draft.

This card is deliberately more operational than a confidence badge. It tells design what to render, engineering what state to support, data teams what events to log, and product managers what to evaluate.

The test should measure safe completion

I would compare two prototypes using the same synthetic cases. The control shows an answer and a confidence badge. The treatment uses the four recovery states.

The primary measure is unsupported acceptance: how often a participant advances an output that the case evidence does not support. Secondary measures are correction success, time to recover, task completion, and the user's understanding of why the system stopped. The test should include clear cases, ambiguous cases, contradictory evidence, and cases with no support.

Recovery can also create friction. Extra questions and evidence panels can burden people when the task is low stakes or the answer is obvious. The product should therefore scale recovery effort with consequence. A music suggestion can fail lightly. A claim placed in a resume deserves a stricter stop condition.

The four-state design should be rejected if it increases abandonment without reducing unsupported acceptance, or if a simpler evidence-and-edit pattern achieves the same result. The aim is to prevent unsupported output from advancing, not to make every AI interaction feel cautious.

Design the stop before polishing the answer

Before shipping a consequential AI feature, ask four questions: Can the user inspect the support? Can the product name what is missing? Can it decline safely? Can the user recover without losing work?

If any answer is no, a working happy path is not enough. The feature still lacks a safe route for uncertain cases.