Why AI Makes Things Up, and What Actually Reduces It

A New York attorney submitted a legal brief in 2023 built partly on ChatGPT's research. The brief cited six court cases, complete with quotes and procedural histories. None of the six cases existed. The judge who caught it sanctioned the attorney, and the case became one of the most widely reported examples of what AI researchers call hallucination: a model generating specific, confident, plausible-sounding content that is simply not true.

This is not a bug that a future model update quietly fixes. It is a property of how these systems generate text, and understanding the actual mechanism is what separates a business that gets burned by it from one that uses AI knowing exactly where the risk sits.

Why does AI make things up instead of just saying it doesn't know?

A language model does not retrieve facts from a database and check them against a source. It predicts the next most statistically likely word, one at a time, based on patterns learned from its training data. When it "knows" an answer, that prediction lines up with something true. When it doesn't, the prediction process runs exactly the same way; it just lines up with something false.

There is no internal flag that says "I'm guessing now." The model has no separate mechanism for verified recall versus plausible invention; both come from the same generative process, which is why a fabricated case citation reads with the exact same fluency and confidence as a real one.

Why AI Models Sound So Confident When They're Wrong

Part of the answer is how these models get fine-tuned after initial training. Human reviewers rating model outputs during that process tend to prefer answers that sound complete and confident over ones that hedge or admit uncertainty. Over enough training rounds, that preference shapes the model toward always producing an answer, rather than flagging when it genuinely doesn't have one.

The result is a system that will answer a question about a court case, a citation, or a statistic with the same fluent tone whether it actually knows the answer or is filling the gap with something plausible. Confidence, in other words, is not a signal of accuracy. It is a stylistic property the model was trained to produce by default.

How common is AI hallucination, really?

Rates vary enormously depending on what's being asked and how it's measured, which is itself part of the honest answer: there's no single fixed hallucination rate for a model, because the risk depends heavily on the specific task. A model answering a well-documented factual question performs very differently from the same model asked to generate a legal citation or predict a niche, fast-changing detail.

This is why blanket claims like "AI is right 95% of the time" don't hold up to scrutiny; the honest framing is task-dependent, not a single number. What stays constant across every task is the shape of the risk: confident, fluent output that looks identical whether it's accurate or invented.

Is AI hallucination a bug the next model version will fix?

Every major model release genuinely reduces hallucination on the tasks it's been specifically tuned and evaluated against, and that real progress is worth acknowledging. But the underlying mechanism, generating the next statistically likely word rather than checking a verified fact, doesn't disappear because a newer model is better calibrated on average.

A model that hallucinates less often is not the same as a model that has stopped hallucinating. It still has no internal distinction between recalled fact and plausible invention; it has simply gotten better, on average, at producing outputs where the two overlap. That's real progress, and it's also exactly why treating any single model version as "solved" for anything consequential is a mistake.

This matters for how a business should read vendor claims too. A vendor advertising "hallucination-free AI" or "100% accurate AI" is describing something that doesn't match how these systems work at a mechanical level, regardless of how good their specific implementation is. The honest vendor claim is a reduction, not an elimination.

Why a Hallucination Is Different From an Ordinary Software Bug

A traditional software bug is deterministic: the same input triggers the same wrong output every time, which makes it reproducible, and therefore fixable through a normal QA cycle: reproduce it, patch it, ship it. A hallucination doesn't work that way. The same question asked twice can produce a correct answer once and a fabricated one the next time, because the underlying process is generating a probable response, not executing a fixed rule.

That difference is why the usual playbook for catching software defects doesn't fully apply. There's no single bug to find and fix. There's a probability distribution that sometimes lands on something false, which is why the actual fix has to be structural (grounding, verification, review) rather than a one-time patch.

Does the risk differ across domains, like finance versus general knowledge?

Yes, meaningfully. A model asked about a widely documented, stable topic has a large volume of consistent training data to draw from, which narrows the gap between plausible and accurate. A model asked about a fast-changing detail (a specific financial figure that updates monthly, a case that hasn't been widely written about, a niche regulatory requirement) has far less reliable signal to work from, and the plausible-sounding gap-filler shows up more often exactly where it matters most.

This is part of why hallucination in legal and financial contexts gets disproportionate attention: it's not that the model is uniquely worse in those domains, it's that those domains combine high real-world stakes with the kind of specific, fast-changing, or narrow detail that's hardest for a next-word predictor to get reliably right.

This same logic explains why hallucination shows up differently across business sizes too. A large enterprise with a dedicated data team can build custom grounding and review pipelines; a small business usually can't justify that investment for one narrow use case, which is part of why a general-purpose grounding and approval layer, built into the tools a small team already uses, matters more for them than for a company with the resources to build a bespoke fact-checking pipeline.

What Role Model Size and Training Data Play in Hallucination

Larger models trained on more data generally hallucinate less on average, because there's more real signal to draw from and less need to fill gaps with pure invention. But bigger isn't the same as solved: a larger model asked about something genuinely underrepresented in its training data will still generate a fluent, plausible answer rather than an honest gap, because the incentive toward always producing something doesn't change with scale.

This is also why a general-purpose model, however large, tends to hallucinate more on a specific business's internal details, its policies, its pricing, its client history, than on broadly documented public knowledge. None of that internal information was in the training data at all, which means every answer about it is necessarily either grounded in something the model was given directly, or invented.

Does hallucination happen the same way in a multi-step AI process?

It compounds. A single answer might have, say, a 90% chance of being fully accurate on a well-documented question. Chain five separate steps together (look something up, summarize it, draw a conclusion, draft a response, act on it) without a check in between, and the odds of the entire chain being error-free drop well below what any single step's accuracy would suggest, because each step's small chance of drift carries forward into the next.

This is the specific reason a multi-step, unsupervised AI process carries more hallucination risk than a single question asked and immediately checked. It's not that longer processes are inherently worse; it's that nobody was watching for the moment a plausible-but-wrong detail entered the chain, and by the final output, it's indistinguishable from everything that was actually correct.

What does this look like in an ordinary small business, not just a courtroom?

A 15-person accounting firm asked ChatGPT to summarize a client's quarterly tax obligations across three jurisdictions the firm doesn't handle often. The summary read cleanly and cited specific filing deadlines and thresholds. Two of the four cited deadlines were wrong, close enough to the real dates to look plausible, wrong enough to trigger a late filing if nobody had checked them against the actual jurisdiction's published calendar.

Nobody at the firm caught it in the chat window, because the answer read exactly like every other confident, well-formatted answer the tool had given them that week. It was caught during a routine second review, the same discipline that would have caught a junior associate's mistake. The tool wasn't behaving unusually. It was doing what it does by default, on a topic where the training data was thinner than the firm assumed.

Should a business ban AI use because of hallucination risk?

No, and treating hallucination risk as a reason to avoid AI entirely throws away the real, proven value of faster research and drafting over a risk that's manageable with the right process. The businesses that ban AI outright typically end up with employees using it anyway, informally, without any of the verification habits a sanctioned, supported process would have built in.

The more useful policy is scoped: use AI freely for drafting, brainstorming, and first-pass research, and build in a mandatory verification step specifically for anything with a name, date, number, or citation attached before it reaches a client, a filing, or a financial record. That's a narrower, more honest rule than either "never use it" or "trust it by default," and it's the one that actually matches how the risk behaves.

What actually happened in the Mata v. Avianca case?

In Mata v. Avianca, an attorney used ChatGPT to help research a personal injury case and submitted a brief citing prior court decisions to support his argument. Opposing counsel could not find the cases. The court could not find them either, because they did not exist; ChatGPT had generated case names, docket numbers, and judicial reasoning that read like real precedent but described rulings that never happened.

The attorney was sanctioned, and the case became a widely cited warning across the legal industry. It was not an isolated incident: similar sanctions have followed in other cases since, where attorneys submitted AI-generated citations without independently verifying that the underlying cases were real. The pattern is the same each time: fluent, specific, and false.

Does grounding a model in real documents actually reduce hallucination?

Yes, meaningfully. Retrieval-augmented generation gives a model real source documents to pull from and quote, rather than generating an answer purely from what it learned during training. When a model can point to an actual passage in an actual document, it has something to check its output against instead of only predicting the next plausible word from memory.

This is why grounding reduces the hallucination rate without eliminating it. A model can still misread a document, quote it out of context, or blend two real details into one wrong one. Grounding narrows the gap; it does not close it, which is why verification still matters even when a system cites its sources.

This is also why the same underlying AI model can perform very differently depending on what it's connected to: a general chat window with no access to a business's actual records is working from training data alone, while the same model given real access to that business's documents has a genuinely different, more checkable foundation to answer from.

For the specific mechanism behind ChatGPT's version of this, see why does ChatGPT hallucinate. For a step-by-step reduction checklist, see how to reduce AI hallucinations.

Practical Ways to Catch AI Hallucination Before It Costs You

A few checks catch most of what actually goes wrong in practice:

1. Ask for the source, every time, even when the answer sounds obviously right. If the model can't point to a specific, checkable document or citation, or the source it names doesn't actually exist when you look, treat the whole claim as unverified rather than assuming the rest of the answer is safe.

2. Spot-check anything with a number attached, dates, dollar amounts, statistics, case citations, before it goes anywhere consequential. These are exactly where fabrication shows up most, because a specific-sounding number reads as more credible than a vague claim, even when the number itself was invented to fill a gap.

3. Never let AI output go out the door unread on anything with real consequences. A draft reviewed by someone with real domain knowledge catches a hallucination before it becomes a sanctioned legal brief, a missed tax deadline, or a wrong number in a client-facing report.

4. Narrow the question. Break a broad research question into several specific, checkable sub-questions instead of asking one sweeping question and trusting the summary; each narrow question gives the model more real pattern to match against and less room to fill gaps with plausible invention.

Can I trust AI with financial or legal decisions?

Not unsupervised, and not because the technology is unusually bad at these domains specifically. It is because financial and legal answers carry the highest cost when something is wrong, and those are exactly the domains where a fabricated number or citation looks the most convincing. The mechanism that produces a hallucination does not know or care that the topic is high-stakes.

A bookkeeper at a 20-person retail business asked an AI tool to reconcile a vendor statement against the company's own ledger and flag discrepancies. The tool found three real discrepancies correctly, then reported a fourth, a duplicate payment, that didn't actually exist in either record; it had inferred a pattern from the other three and generated a plausible-sounding but fabricated fourth line. The bookkeeper caught it only because reconciliation work already requires cross-checking every flagged line against the source document.

The honest framing is the same one that applies everywhere else: AI can genuinely speed up research and drafting, but the final check on anything with real money or a real legal consequence attached still needs a person who verifies the source, not just trusts the tone of the answer.

Where This Leaves a Small Business Using AI for Real Work

None of this means AI is unreliable for everything; it means unsupervised output on anything consequential is the actual risk, not an occasional glitch. The fix a growing business actually needs is not a smarter model. It is a system that grounds its answers in the business's own real documents and defaults to asking before it acts on anything that matters, rather than guessing fluently and moving on.

That's the design a Kuvai teammate is built around: grounded in your own Company Context and connected documents rather than generic training data alone, and set to draft rather than send by default. It doesn't make hallucination impossible. It means a teammate's output is checked against your actual policies and documents before it reaches a customer, the same discipline this whole piece has been arguing for.

Want output you can actually verify, grounded in your own Company Context? Sign Up for Free — no credit card required, free to start, cancel anytime.