Your Agent Can Win the Wrong Deal

An agent gets you a wonderful deal on a book you do not want to read. It stayed within budget. It followed the trading rules. It even improved on the starting offer.

What should the team fix?

A stronger negotiator might get you the same unwanted book for less. The earlier mistake was deciding what you wanted. For products that choose on someone's behalf, that deserves its own evaluation before the team invests in better execution.

My proposal is narrow: when an unresolved preference could change the selected option, test one concrete comparison before delegating the choice. Keep it only if it reduces unwanted selections enough to justify the interruption.

The preference model can be the weak link

Anthropic's September 24 Project Swap study put agents to work trading books for 201 employees. The agents' predicted preferences agreed with participants' held-out pairwise rankings 61% of the time. That is agreement between book pairs, not a claim that 61% of recommendations were good. The researchers attributed most of the gap from an ideal allocation to preference representation rather than trading. Project Swap.

The study involved employee volunteers, a bounded book market, and noisy preference measurements. It does not establish the main cause of failure across consumer agents. It does make a useful distinction available to a product team: is the agent pursuing the wrong objective, or pursuing the right one badly?

Those problems need different investments. Better search might uncover a previously unavailable book. Better negotiation might secure it. Neither tells you whether this reader wants it for this particular week.

Ninety percent can hide the deciding mistake

Here is an original, deliberately small example. The reader, books, rankings, and responses are fictional. No model was evaluated to produce these numbers.

A reader wants something for short train journeys. Five eligible books remain after checking price, format, and titles already read. Book A is a compact collection of independent stories. Book B is a long, acclaimed novel. C, D, and E are less attractive options.

The reader privately ranks them A, B, C, D, E. The agent ranks them B, A, C, D, E. There are ten possible pairs. They agree on nine. The only disagreement is A versus B, which is exactly the decision the agent will make.

The arithmetic is easy to reproduce: five items make ten pairs; swapping the top two reverses only their pair. This does not make pairwise agreement useless. It makes it insufficient on its own for a product that delivers one selection.

Nor is every first-choice mismatch harmful. If A and B are equally welcome, either is a good outcome. Ask people what they would reject, and why, rather than inventing precision from a forced ranking. A cheaper second choice may also be preferable once the real offer is visible.

Ask about the decision, not the whole person

The question in the example is useful because either answer changes the recommendation. "Tell me more about yourself" has no such guarantee. A product should be able to explain what it will do differently with an answer.

There is a recognizable workflow here. In OpenAI's September 29 customer story, Literary Fancy's Karen Rodriguez uses reader surveys and ChatGPT Work to create reading plans, including alternatives for books already read. That is a vendor-published account of a working process, not a controlled measure of recommendation quality. Reading-plan case study.

Start with the information that can rule an option out. Then separate stated preferences from guesses. In the fictional case, "short train journeys" does not prove that the reader dislikes long novels. They may enjoy returning to the same story each morning. The comparison exposes that assumption without demanding a life history.

Keep the answer attached to this task. A preference for short stories during a busy week should not silently become a permanent personality trait. If the product retains it, the memory needs a visible purpose and a correction path.

A helpful question can still cost too much

Some people want a pleasant surprise. Others would rather reject a cheap recommendation than complete an intake interview. Asking more questions can destroy the convenience the agent was hired to provide.

AgentHabit, a September 26 preprint from Seoul National University researchers, profiles how agents work across everyday tasks, including when they ask for clarification. Its interaction traces use simulated users. It helps name behaviors to inspect; it does not tell a PM which questioning style real customers will prefer. AgentHabit.

For low-cost, reversible recommendations, showing two options with a clear distinction may be enough. For a purchase, asking before committing can be worth more. The decision depends on the cost of a wrong selection and the effort required to correct it, not on an abstract goal of understanding the user completely.

Give the extra question a fair test

Compare the existing intake with an intake that adds one targeted comparison. Hold the candidate pool, prices, and selecting agent constant. Randomize people between the two experiences so answering the comparison does not teach the baseline their preferences.

Measure unwanted selections and correction time per assigned person, including those who leave without choosing. Report completion and time spent answering separately. Otherwise a burdensome flow can appear accurate simply because the least patient people disappeared.

Use fresh choices to test whether the answer transfers. Repeating A versus B after someone explicitly chose A only tests whether the system remembered the answer. Include ties, changing circumstances, and cases where no extra question is needed. Decide the acceptable added effort before looking at results.

If the shorter intake performs as well after those costs are counted, keep it. If the targeted question helps, you have evidence for a specific interaction, not permission to collect more personal data everywhere.

The Agent Autonomy Budget addresses the next boundary: what the product may do with a choice. Start one step earlier this week. Take the option your agent would select, put its closest alternative beside it, and ask whether the difference matters to the person who has to live with the result.