Every risk covered so far in this series has an attacker somewhere in it. Prompt injection needs someone to write the injection, training-data poisoning needs someone to seed the corpus, denial of service needs someone to send the traffic. The ninth entry on the OWASP Top 10 for Large Language Model Applications needs nobody at all. Overreliance describes what happens when a system behaves exactly as built, produces something false, and no one checks.
OWASP’s 2023 entry defines the risk as excessive trust placed in model output without oversight or confirmation, and lists the results: misinformation, miscommunication, legal exposure, reputational damage, and security breaches. The 2025 revision renamed the category to Misinformation and pointed it more directly at the false output itself rather than the user’s trust in it. Both framings describe the same two-part failure, a model that generates something wrong and a human or downstream system that accepts it.
WHAT A HALLUCINATION IS
A hallucination is model output that is fluent, internally consistent, and false. The term is misleading in one important way, because nothing unusual is happening inside the model when it hallucinates. The same next-token prediction that produces a correct answer produces an incorrect one, and the model has no separate mechanism for checking either against the world.
Fluency and accuracy come out of one process, which is why the first is no evidence of the second. A weather forecast read in a confident voice is not more accurate than the same forecast read hesitantly, and a fluent model answer works the same way.
RESEARCH ON THE CAUSE. Work published by OpenAI researchers in September 2025 argues that the behavior follows predictably from how training and grading work, rather than from some mysterious defect. Most benchmarks award a point for a correct answer and zero for both a wrong answer and an admission of uncertainty.
Under that scoring rule a confident guess carries positive expected value and abstention carries none, so optimization pressure pushes models toward guessing. The incentive works just as it does for a student sitting a multiple-choice exam with no penalty for wrong answers, who therefore answers every question on the paper.
The paper’s conclusion is that hallucination is less a bug to be patched than an incentive to be changed.
THE RATES ARE NOT SMALL
Measured hallucination rates in specialized domains are high enough that occasional spot-checking is not adequate coverage. A 2024 study from Stanford’s RegLab, published in the Journal of Legal Analysis, asked general-purpose models specific, verifiable questions about randomly selected federal court cases. GPT-4 hallucinated on 58 percent of those questions. Llama 2 hallucinated on 88 percent.
The authors documented two patterns more useful than the raw rates. Models frequently failed to correct false premises embedded in a question, so a user asking about a case that does not exist tends to receive a confident description of it. Models were also poor at predicting when they were hallucinating, which removes the obvious defense of asking the model how sure it is.
WHEN NO ONE CHECKS
The clearest documented case of overreliance is Mata v. Avianca. In 2023 a lawyer preparing a filing in the Southern District of New York used ChatGPT for research and submitted a brief citing six federal decisions.
Opposing counsel could not locate any of them, and neither could the court, because the model had invented all six along with their citations, procedural histories, and quoted passages. On June 22, 2023, Judge Kevin Castel sanctioned the two attorneys and their firm $5,000.
The fabricated opinions read persuasively because they had the shape of real ones. A counterfeit note passes inspection not because it holds value but because it is well printed, and a fabricated citation passes the same way.
Moffatt v. Air Canada shows the same failure from the operator’s side. Air Canada’s website chatbot told a passenger that a bereavement fare could be claimed retroactively, contradicting the airline’s actual policy published elsewhere on the same site. Air Canada argued before the British Columbia Civil Resolution Tribunal that the chatbot was a separate entity responsible for its own statements.
On February 14, 2024, the tribunal rejected that argument, found negligent misrepresentation, and awarded $650.88 in damages. The sum was small and the precedent was not, because it established that an organization stays liable for what its model tells a customer.
THE SECURITY VERSION
Overreliance sits on a security list rather than a general AI-ethics list because its consequences include vulnerabilities. A Stanford study presented at ACM CCS 2023 by Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh gave participants security-relevant programming tasks, with and without an AI coding assistant. Participants who had the assistant wrote significantly less secure code than those who did not. They were also more likely to believe their code was secure.
That inversion is the risk in concentrated form: the tool lowered the quality of the work and raised confidence in it at the same time. The same study found that participants who trusted the assistant less, and who invested more effort in their prompts, produced code with fewer vulnerabilities.
A second failure mode converts hallucination directly into a supply-chain attack. Research presented at USENIX Security 2025 tested 16 code-generating models across 576,000 generated code samples, checking whether each recommended package actually existed.
Recommended packages that did not exist averaged at least 5.2 percent for commercial models and 21.7 percent for open-source ones, across 205,474 unique fabricated names.
Because those invented names recur across runs, an attacker can register a frequently hallucinated one and wait for installs to arrive. The technique works like a typosquatting attack aimed at the machine rather than the human, and it picked up a label in 2025, slopsquatting.
How often does a developer confirm that a suggested dependency exists before running the install command, rather than after?
WHAT MITIGATION LOOKS LIKE
OWASP’s mitigations for overreliance are procedural and mostly unglamorous. They begin with regular monitoring and review of model output, cross-checking against trusted external sources, and automatic validation wherever a checkable ground truth exists. Grounding techniques such as retrieval-augmented generation and fine-tuning on domain data lower the rate without removing it.
Breaking a complex request into smaller tasks makes individual claims easier to verify, and separating those subtasks reduces the chance that one error travels the whole chain unchallenged. For code specifically, ordinary secure development practice applies to model output as it would to any untrusted contribution, including validating dependencies before they reach a build. None of this is new engineering, it is existing verification discipline applied to a source that was mistakenly treated as authoritative.
The remaining mitigations target the interface rather than the model. Communicating limitations plainly, marking uncertain output, and designing interactions that invite checking all move some of the burden back to the reader. Remember the exam scoring rule from earlier, because the same logic applies to people: an interface that never signals uncertainty teaches its users to stop looking for it, exactly as a benchmark that never rewards abstention teaches a model to stop expressing it.
THE PART THAT IS NOT TECHNICAL
The last mitigation is the hardest, because the failure is a human one. A survey of 319 knowledge workers by researchers at Microsoft Research and Carnegie Mellon University, presented at CHI 2025, gathered 936 real-world uses of generative AI at work. Higher confidence in the AI tracked with less critical thinking applied to its output, and higher confidence in one’s own ability tracked with more.
Participants reported that the tools reduced effort across comprehension, analysis, and synthesis, while shifting the critical thinking that remained toward verifying and integrating what the tool produced. That is the trade at the center of overreliance, because the work does not disappear, it relocates. Whether it actually gets done in its new location is a question of habit rather than architecture.
Cognitive science calls this cognitive offloading, the practice of handing a mental task to an external resource. The learning series on this site covers the mechanism in detail and the conditions under which offloading helps or harms. The security-relevant version is simpler: a verification step that nobody performs offers no protection at all.
The next article in this series turns to a risk pointing the other way, not a model producing something worthless, but a model valuable enough to be worth stealing.
T.
References
- OWASP LLM09:2023 Overreliance - The original OWASP entry defining overreliance, its common vulnerability examples, and prevention strategies.
- OWASP LLM09:2025 Misinformation - The 2025 revision that renamed the category and refocused it on model-produced misinformation.
- Why Language Models Hallucinate (Kalai, Nachum, Vempala, Zhang, 2025) - OpenAI research arguing that training and evaluation procedures reward guessing over acknowledging uncertainty.
- Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models - Dahl, Magesh, Suzgun, and Ho in the Journal of Legal Analysis, reporting the 58 percent and 88 percent legal hallucination rates.
- Mata v. Avianca, Inc., Opinion and Order on Sanctions (S.D.N.Y. 2023) - Judge Castel’s sanctions ruling over the six fabricated judicial decisions submitted to the court.
- BC Tribunal Confirms Companies Remain Liable for Information Provided by AI Chatbot (American Bar Association, 2024) - Analysis of Moffatt v. Air Canada and the negligent misrepresentation finding.
- Do Users Write More Insecure Code with AI Assistants? (Perry, Srivastava, Kumar, Boneh, ACM CCS 2023) - The Stanford user study finding less secure code paired with greater confidence in its security.
- We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs (USENIX Security 2025) - Spracklen et al. measuring hallucinated package recommendations across 576,000 code samples.
- The Impact of Generative AI on Critical Thinking (Microsoft Research and Carnegie Mellon, CHI 2025) - Survey of 319 knowledge workers linking confidence in AI to reduced critical thinking effort.
