The AI features people like best can be the ones that make them decide worst

Satisfaction and accuracy pull in opposite directions

I design enterprise software where a wrong answer is expensive, and most of that work now turns on one question: what happens to a person's judgement when a machine offers to do the judging for them. The metric teams reach for, how good the thing feels to use, turns out to be a poor proxy for whether anyone using it decides well.

There is a finding in human-computer interaction research that most AI product teams have never had to reckon with, and it undercuts the way nearly all of them measure success.

Zana Buçinca, Maja Barbara Malaya and Krzysztof Gajos tested several interface designs meant to stop people accepting wrong answers from an AI. The designs that worked, the ones that most reduced AI overreliance, were the ones participants rated least favourably. The versions people preferred left them accepting more incorrect output.

Ship whichever variant tests better on satisfaction, as almost every product organisation does, and you may be selecting the version that makes your users worse at the task.

They will thank you for it in the survey.

This is the part of the AI conversation that gets skipped. The interesting question was never whether a model can imitate a person. It is that the qualities making an AI pleasant to use, fluency, confidence, a bit of personality, are not the qualities that help someone judge what it just told them.

Automation bias did not arrive with chatbots

None of this is new, which is what makes it worth trusting.

Linda Skitka, Kathleen Mosier and Mark Burdick published the core result in 1999, in the International Journal of Human-Computer Studies. In a simulated flight task, participants given a highly but not perfectly reliable automated aid performed worse on monitoring than participants given no aid at all. The tool was right most of the time, and that was the problem: people stopped checking.

They named it automation bias, and split it in two. Commission errors are acting on a wrong recommendation. Omission errors are missing something because the system never flagged it. Their proposed mechanism was cognitive laziness, which reads harshly but describes something ordinary. People spend the least effort a task seems to require, and a reliable-looking aid makes the task look like it requires very little.

Twenty-seven years later we have shipped that dynamic to hundreds of millions of people and given it a conversational interface.

Confidence in the tool replaces thinking

The clearest modern evidence comes from Hao-Ping Lee and colleagues at Microsoft Research, published at CHI 2025. They surveyed 319 knowledge workers who supplied 936 first-hand examples of using generative AI at work.

Their central quantitative finding is worth quoting exactly: "higher confidence in GenAI is associated with less critical thinking, while higher self-confidence is associated with more critical thinking."

Two different confidences, pulling opposite ways. Confidence in the tool substitutes for your own thinking. Confidence in yourself protects it. The same study found workers reported less cognitive effort in each category it measured, from comprehension through to evaluation.

That is a design finding, not a human failing. How confident a system appears is not a property of the model. It is a decision somebody made about tone, hedging, formatting and whether uncertainty is shown at all.

Explanations alone do not fix AI overreliance

The standard advice is to explain the output. Show your sources. Show the reasoning. Users will then calibrate.

Buçinca and colleagues tested that. Adding explanations to AI decisions did not reduce overreliance, and they noted studies suggesting it can increase it. Showing someone why an AI produced an answer is not automatically the same as helping them recognise when the answer is wrong.

An explanation is more content from the same system that produced the answer. It raises how checked an output feels without anyone having verified it.

What did work in their study was cognitive forcing: interventions that interrupt, that withhold the recommendation until you commit to your own answer, that make you do a piece of the thinking before the machine offers to do it for you. Friction, placed deliberately.

And people disliked it. That is the whole tension in one result. The intervention works because it costs something, and the cost is exactly what shows up in your satisfaction metric.

A personality is not a trust signal

The other common instinct is to make the system feel human. Give it a name, a warmth, a few conversational tics, and trust follows.

Research on the Human-Automation Trust Expectation Model tested this and found anthropomorphism had negligible impact on trust calibration unless its execution actually communicated information that helped people form accurate expectations of how the system performs. Warmth on its own did nothing measurable. Warmth carrying information did the work, and the information was doing it.

So a friendly persona is not a trust mechanism. It is a tone decision that people mistake for one. What calibrates trust is telling someone what the system is reliably good at, where it degrades, and how confident this particular output is relative to its usual standard.

The calibration gap

Here is the frame I use on AI work, and the one thing I would want a team to take from this.

The calibration gap is the distance between how much a user trusts an AI output and how much that output deserves to be trusted. Every AI product has one. The design question is not how to raise trust, it is whether your decisions widen the gap or close it.

Overreliance is only one side of it. An interface can also teach people to distrust correct output so consistently that a useful system goes unused, which is under-reliance and is the failure mode nobody puts in a deck because it looks like caution. The target is not scepticism. It is calibration: more scrutiny where the system is weak, more delegation where it has earned it.

Three levers do most of the moving, and each maps onto one of the studies above.

1. Expressed confidence. How certain the system sounds, independent of how certain it is. Tone, hedging, formatting, visual weight, whether uncertainty appears at all. Sounding equally sure across outputs of very different reliability is one of the easiest ways an interface can widen the gap.
2. Placed friction. Where the interface makes someone stop, compare or commit before it answers. This closes the gap and costs satisfaction, which is why it needs defending in advance rather than after the first survey. Applied everywhere it stops working, because people learn to click through it.
3. Communicated limits. What the product says about its own failure modes, in the moment of use rather than in a settings page nobody opens. The cheapest of the three to move, and the one that tends to end up on the marketing site instead.

Notice that explanations are not a lever. They move perceived diligence without moving the gap, which is why they are so popular and so ineffective.

What this looks like in a real product

I first ran into this properly on the CloudSOC security analytics redesign, where analysts triage alerts under time pressure and were losing about 40% of their day to false positives.

The instinct there is to make the system more assertive so the queue moves faster. The correct move was closer to the opposite. Analysts did not need every alert to look decisive, they needed to know which signals deserved immediate action and which deserved a second look. Confidence became part of the information hierarchy rather than metadata behind another click, and the speed came from no longer having to work out which alerts warranted hesitation.

The same shift fits almost any AI recommendation. Here is one alert, with nothing about the model changed between the two.

The first panel states a verdict. The second gives someone enough to disagree with it. That is the whole distinction, and it is an interface decision rather than a modelling one.

A calibration checklist for AI products

Six questions I would want answered before an AI recommendation ships.

1. Can someone tell when the system is unsure? If every output carries the same weight, the interface is claiming a confidence the model does not have.
2. Can they check it without leaving the task? Evidence that costs a context switch is evidence nobody looks at.
3. Are expensive decisions slower than cheap ones? Commit-first patterns work where a wrong answer costs something. Applied uniformly they stop meaning anything.
4. Do the limits show up at the moment they matter? A caveat in the documentation is a caveat nobody reads at the point of the decision.
5. Are you measuring wrong answers accepted, not only satisfaction? Seed a known error rate into a usability test and count the catches.
6. Have you tested both failure modes? Over-trust is the loud one. A system people override when it is right is failing just as expensively, and much more quietly.

The fifth is the one that gets skipped, and it decides the rest. Agree in advance which metric wins when satisfaction and accuracy disagree, because at some point they will.

The point

Most writing about artificial intelligence and psychology gets stuck on whether machines can think or feel, which is philosophy and is not a product decision. The useful version is narrower and much more uncomfortable.

Prediction and understanding are different capabilities, and a system can be excellent at the first while a person using it gets worse at the second. The only thing standing between a capable model and a worse-informed user is the interface, which makes it a design problem, and ours.

Teams that get this right should expect to look slightly worse on satisfaction dashboards and to produce better decisions. Deciding that trade in advance, and writing it into how the product is evaluated, is most of the job. It is the same argument I make about deciding constraints before you open the tool, and it is the same reason a default is never a neutral choice. It is where I start a product design engagement.