Why AI Makes Things Up, and What Builders Can Do About It
Why AI makes things up, in plain words, and four fixes a builder can use: retrieval, narrow tasks, an eval set, and human review, with a worked support-bot example.
Why AI makes things up, and what builders can do about it, is the question that decides whether your AI product can be trusted. A model that sounds sure and is wrong will cost you customers faster than a model that says "I don't know." The good news is that you can design around it. This post explains the cause in plain terms, then goes through four fixes you can apply to a real product. If you are new to this, AI Entrepreneur has free lessons on how modern AI works, in short, hands-on steps.
What it means when AI makes things up
People call it hallucination. The model gives you an answer that reads well but is false. It might invent a refund policy, cite a study that does not exist, or give a function name that no library has.
Three things make it hard to catch:
- The wrong answer has the same tone as a right one.
- It often mixes true and false details in one paragraph.
- It shows up most when a user asks about something specific, such as your prices, your contract terms or last week's change.
Plan for it as a limit of how these models work, not as a bug someone will patch out. Your job as a builder is to shrink it, measure it and contain it.
Why AI makes things up, and what builders can do about it
Start with what a language model actually does. It predicts the next piece of text, one token at a time, based on patterns it learned from a huge amount of writing. It is not looking facts up in a database. It is producing text that fits.
That has a direct consequence. When the model has no good pattern for your question, it does not stop. It still produces the most plausible continuation. A made-up answer and a correct answer come out of the same process.
Two more things add to the problem:
- It does not know your business. Your pricing page, your internal docs and your customers' data are not something it can look up, unless you put them in front of it.
- An assistant is built to answer. A chat assistant that always replies will reply even when it has nothing solid to say, so you have to give it a way out.
The AI Entrepreneur course How Modern AI Actually Works has three lessons on this: Next-Token Prediction, From Raw Model to Helpful Assistant, and What AI Gets Wrong. The course is about what a language model really does and the limits every builder must design around, which is the right frame for this whole topic. For a longer look at the first of those, read how large language models work.
Where hallucinations hurt a business most
Not every wrong answer costs the same. Before you spend effort on fixes, sort your use cases by what a mistake does.
- Low cost: brainstorming names, drafting a first version of an email that a person will edit.
- Medium cost: summarizing a document for a teammate who might act on it.
- High cost: telling a customer what they are owed, quoting a price, stating a rule, or giving advice about health, money or law.
Here is a worked example we will use for the rest of the post. It is invented, to show the steps. Say you run a small online shop and you build a support bot. A customer asks, "Can I return a sale item after 40 days?" If your real policy is 30 days and the bot says yes, you now have an angry customer holding a screenshot. That is a high-cost question, so it needs the strongest protection.
Ground the model in real sources with retrieval
The most useful fix is to stop asking the model to remember and start giving it the text to read. This is retrieval. When a question comes in, your system finds the relevant passages from your own documents and puts them in the prompt. The model then answers from that material.
For the shop bot, the flow looks like this:
- Store your returns policy, shipping page and FAQ as small chunks of text.
- When a customer asks a question, find the chunks that best match it.
- Send those chunks to the model along with the question.
- Tell the model to answer only from them.
Here is the kind of instruction that goes with it:
Answer the customer's question using only the policy text below. If the text does not contain the answer, say "I'm not sure, I'll pass this to a person" and do not guess. Quote the sentence you used.
Retrieval does not remove the problem. The model can still misread a passage, and the search step can fetch the wrong chunk. But it moves the question from "what does the model remember?" to "did it use what we gave it?" That second question is far easier to check. The lesson Giving Models Your Data: Retrieval in the Intermediate course on the AI product stack is about exactly this, and our post what is retrieval augmented generation explains it in plain words.
Constrain the task and the output
The more open the task, the more room the model has to invent. Narrow it.
Give it a smaller job
"Be our support agent" is wide. "Classify this message as returns, shipping or other, then pull the order number" is narrow. Narrow jobs have fewer ways to go wrong, and you can check them with simple rules.
Control the shape of the answer
Ask for a fixed format, such as a short answer plus a field for the source sentence. Then your code can check it. If the source field is empty, or the quoted sentence is not in your policy text, do not show the answer to the customer.
Prefer a fixed workflow over a free agent
When a task runs through many steps, small errors pile up. A plain workflow with set steps is often safer than an agent that decides its own path. The Advanced course Agents and Automation is about when to use each, and why long chains of steps fail. If you want the workflow route, see how to automate a boring workflow with AI, step by step.
Measure hallucinations with an eval set
Do not judge your bot by reading three good answers and feeling fine. Build a set of test questions with known right answers, run it every time you change something, and count the failures.
For the shop bot, your set might include:
- Questions the policy answers clearly, such as "How long do I have to return a shirt?"
- Questions that sound close to the policy but fall outside it, such as the 40-day sale item.
- Questions the documents say nothing about, where the right behavior is to decline.
- Trick questions that assume something false, such as "Since you offer free returns forever, how do I..."
Then score each answer. Did it match the policy? Did it decline when it should? Did the quoted sentence really exist? Track the numbers over time so you can tell whether a new prompt or a new model made things better or worse. Our post on how to build an eval set for an AI feature goes through this step by step, and the Advanced course Evals: Shipping AI That Works is about measuring quality instead of judging by eye.
The "should decline" cases matter most. They test whether your bot knows the edge of what it knows.
Add human review and graceful failure
Even with retrieval, tight prompts and evals, some wrong answers will get through. Plan for that.
- Route by risk. Let the bot answer low-risk questions alone. Send anything about refunds, money or complaints to a person, or show the bot's draft to staff first.
- Fail in a useful way. "I'm not sure, I'll pass this to a person" is a good answer. Make it easy for the bot to say it, and make sure a real person picks it up.
- Show the source. A link or quote next to the answer lets customers and staff check it in seconds.
- Keep a log and review it. Read a sample of real conversations each week. Add every new failure to your eval set.
That last habit is how quality improves over time. Production teaches you what your test set missed. The lesson Running AI in Production in the evals course is about this stage.
One more point on risk. If your product depends on one provider's model, a model update can change how often it gets things wrong. That is one reason to re-run your evals on a schedule. Our post on platform risk when you build on someone else's model explains the wider issue.
Frequently asked questions
Can you stop an AI from making things up completely?
No. A language model produces plausible text, so some errors will always be possible. You can cut them a lot with retrieval, narrow tasks and checks, and you can catch the rest with evals and human review.
Does retrieval fix hallucinations?
It reduces them by giving the model your real documents to answer from. It does not end them, because the search can fetch the wrong passage and the model can still misread the right one.
How do I know how often my AI product is wrong?
Build an eval set of test questions with known answers, including ones the product should refuse. Run it after every change and track the share it gets right.
Should I tell users the AI might be wrong?
Yes, and do more than add a disclaimer. Show sources, let the bot say when it is unsure, and give people an easy path to a human for anything that matters.
Get started
The courses behind this post are about how models predict text, how retrieval works, and how to build and run an eval set. Every lesson is free with an account. Start learning with AI Entrepreneur: sign in on the home page with a 6-digit code sent to your email, and take your first short lesson today.
