A study involving 3,132 participants across five experiments found that having access to AI advice virtually eliminated participants’ willingness to abstain from answering questions, even when the underlying models hallucinated. When AI was available, the rate at which participants withheld judgment dropped from up to 44 percent down to as low as 3 percent, while confidence scores rose significantly alongside a steep decline in overall accuracy.
The tendency to defer to AI answers persisted even when researchers introduced financial incentives for correct responses and penalized wrong ones. Furthermore, displaying unsolicited AI answers automatically yielded similar results, underscoring risks associated with embedded AI search summaries and writing assistants as users accept convincing output without verification.
Why it matters
Users significantly overestimate AI accuracy, leading to a severe decline in critical judgment and answer verification.
Financial incentives alone are insufficient to counter user overreliance on AI-generated answers in critical workflows.
Source: the-decoder.com



