AI hallucination: why does AI make things up?
An invented price, a by-law that does not exist, a wrong address: how to spot these errors and limit them.
An AI hallucination is incorrect, inappropriate or fictitious content generated by an artificial intelligence model and presented as fact, as if it were genuine or matched the request. It stems from how generative models work: they predict plausible text without checking whether it is true, and they often sound confident either way.
In short
- NIST calls it confabulation: false content stated with confidence.
- The confident tone is the real danger, because people stop checking.
- It can also affect generated images and audio.
- Supplying sources and demanding quotes reduces it.
Why do AI hallucinations happen?
NIST's Generative AI Profile uses the term confabulation: the production of confidently stated but erroneous or false content, known colloquially as hallucinations. It calls this a natural result of how generative models are designed, since they generate outputs that approximate the statistical distribution of their training data. NIST also notes that some people object to the word hallucination because it anthropomorphizes AI [1].
Quebec's OQLF terminology entry points to two common triggers: training data for the large language model that is insufficient or has major gaps, and ambiguous wording in the original request. It cites a well-known case in which a popular AI claimed the James Webb telescope took the first pictures of an exoplanet, when those images date from 2004 and the telescope launched in 2021 [2].
The same entry notes that hallucinations are not limited to text; they can also show up in generated images or sound [2].
What does a hallucination look like in a small business?
| Type | Example in a service business | Consequence |
|---|---|---|
| Invented fact | A price or lead time that appears nowhere in your rates | A promise you cannot keep |
| Fake source | A municipal by-law or legal section that does not exist | Wrong advice given to a customer |
| Contradiction | Two different opening hours in the same reply | Lost trust |
| Off topic | A flat-roof answer to a question about eavestroughs | Wasted time, annoyed customer |
How many errors would slip through without review?
How can you reduce hallucinations?
Anthropic's documentation for its Claude models suggests a few simple techniques: explicitly allow the model to say it does not know; for long documents, ask it to extract word-for-word quotes first and work from those; and have it back each claim with a citation you can check [3].
- Supply the source: paste your rates or cancellation policy into the request instead of letting the model guess.
- Narrow the scope: a model that answers only from your documents invents less.
- Ask for references: treat any answer without a checkable source as a draft.
- Keep a person in the loop: human approval before anything is sent is the safest net.
- Publish clear facts on your site, including as structured data, so assistants and Google's AI Overviews find the right information.
Which habits make the problem worse?
- Asking find me a statistic about something without requiring a link: ideal ground for an invented source.
- Connecting an AI agent to customers with no check on outgoing messages.
- Mistaking fluent writing for accuracy.
- Fixing one error by hand without changing the instructions or the source material.
How does Pilotiko limit the risk?
Pilotiko's AI agents work from the business's real data, such as calendars, bookings and workload, rather than from memory. Above all, any action touching a customer, money or a send waits for a person's approval, so a possible hallucination stays a draft to fix rather than a message that went out.
Frequently asked questions
Why does AI invent an answer instead of admitting it does not know?
The model predicts the most plausible continuation of the text, and an invented answer can be very plausible. Without an instruction that allows it to express uncertainty, it tends to fill the gap. Anthropic's documentation recommends giving it explicit permission to say it does not know.
Will newer models stop hallucinating?
Models keep improving, but NIST describes confabulation as a natural result of how generative models are designed. Plan for checks even with the best tools.
Is my business liable for an AI hallucination?
If a wrong message goes to a customer in your company's name, your company sent it. For a specific legal question, talk to a lawyer. That is exactly why every send should be validated before it leaves.
How can I spot a hallucination quickly?
Check numbers, proper names, dates and references to laws or web pages first. Those are the details most often invented. A reference you cannot find anywhere is a red flag.
Is hallucination the right word?
It is the most common one, and the OQLF's French equivalent is hallucination d'IA. NIST prefers confabulation and notes that some find hallucination too human. Both describe the same problem.
Sources
- NIST, NIST AI 600-1 : Generative Artificial Intelligence Profile (accessed on October 15, 2026).
- Office québécois de la langue française (Vitrine linguistique), hallucination d'IA (GDT) (accessed on October 15, 2026).
- Anthropic (documentation Claude), Reduce hallucinations (accessed on October 15, 2026).
Read next
Large language model (LLM): how does it work?
What a large language model is, how it writes by predicting tokens, where it helps a service SME, a worked example and the limits to know before adopting one.
What is an AI agent?
An AI agent pursues a goal by chaining steps and tools on its own. Definition, the agent loop, a worked example for a Quebec SME and the guardrails it needs.
What does human-in-the-loop mean?
Human-in-the-loop puts a person before an AI's sensitive actions. Forms of oversight, Quebec and Canadian rules, a worked SME example and the mistakes to avoid.
Structured data: what are schema.org and JSON-LD?
Structured data, schema.org and JSON-LD explained: formats Google accepts, a LocalBusiness markup example for a Quebec small business and the usual mistakes.