Learn about AI · Hallucinations
Why does AI sometimes make things up, and what do you do about it?
Why does it happen?
A language model is trained to do one thing: continue a text in the way that most resembles everything it has read. It does not have a list of what is true. It has a very precise picture of what true sentences tend to look like. Most of the time those are the same thing. A sentence saying Denmark has five regions resembles the sentences it has seen, and it is also correct.
The trouble starts when the question is about something the model has never seen: your prices, your procedure, a customer, an article number. There is no correct answer in what it has read, and it writes the thing that most resembles a correct answer anyway. It does so in the same tone and with the same confidence. That is what makes it dangerous: you cannot hear the difference.
Three things make it worse: when the question presupposes an answer ("how much discount do we normally give?" assumes there is a normal discount), when the model is not allowed to say no, and when the answer has to be long. The longer the text, the more places it can slip.
An example: the same question, two set-ups
A pharmacy technician asks: "May we substitute product X with Y when X is on back order?" Put to a general chat service, she gets a well-written answer about substitution rules in general, perhaps citing a rule that sounds right. Whether it applies to those two products, and whether the pharmacy's own procedure says something different, the service does not know. It writes anyway.
Put the same question to a tool built for the pharmacy and something else happens. The tool first looks in the pharmacy's procedures and the substitution list. If it finds a passage that answers, it writes the answer from that passage and shows where it is. If it finds nothing, it says: "The procedures do not mention substitution between X and Y. Ask the pharmacist." The second answer is shorter and less impressive. It is also the only one you can act on.
How to build a solution that doesn't make things up
- Narrow it
The tool may only answer in one area: procedures, orders, appointments.
- Look up first
The answer must be built from your documents or data, not from the model's memory.
- Show the source
Every answer points to the document or record it came from.
- Allow "don't know"
If the source is missing, the tool says so instead of filling the gap.
- Test
A fixed set of questions with known answers is run again every time something changes.
The first step is underrated. A tool that may answer anything will hallucinate, because it gets asked about anything. A tool that may only answer about delivery times, and gets its delivery times from the order system, has very little to invent with. The narrow scope is not a limitation you put up with. It is the most important safety feature you build in.
The test set is the other underrated part. Thirty real questions from everyday work, each with an answer a professional has approved. It runs before the tool goes live, and again every time the model, the documents or the rules change. Without it you do not know whether the last change made the tool better or worse. With it, finding out takes fifteen minutes.
What helps, and what doesn't
Helps
- Answers built from your own documents and data (look up before answering)
- A source reference on every answer, so the employee can check
- A narrow area and a clear list of what the tool does not answer
- Short answers; long ones have more places to slip
- A person who approves before anything is sent or posted
Doesn't help, or not enough
- Writing "do not make anything up" in the model's instructions; it helps a little, and it is not a safeguard
- Waiting for a newer model; they invent less, not never
- Trusting an answer because it sounds confident; the wrong ones do too
- Letting the model check itself without a source to check against
What it takes on your side
- Documents or data the answers can be built from. If they do not exist, the tool can only guess, and then it should not be built yet.
- A professional who can approve the answers in the test set and judge when an answer is wrong.
- A decision on what the tool must not answer, taken before it is built.
- Willingness to accept "I don't know" as a good answer. It is what makes the other answers trustworthy.
Common misunderstandings
| What we hear | How it actually works |
|---|---|
| The newest models don't hallucinate any more | They do it less often. Ask them about something they have not seen and they still do. The fix is in how the tool is built, not in the model's vintage. |
| You can solve it with a good prompt | A good instruction makes it better. Looking up in your documents, source references and a test set make it safe. That is the difference between good advice and a seat belt. |
| If it sounds confident, it is right | The tone is the same whether or not the answer is correct. The source is the only thing you can judge by. |
| It is too risky to use AI for anything that has to be right | It is too risky to use a general chat service for that. A tool that looks up, shows its source and dares to say no can be used for things that have to be right, and that is what we build. |
What we would advise against
We would advise against using a general chat service for questions about your own rules, prices or customers, however good the answers look. It has not seen your data, so it guesses. And we would advise against putting a tool into use without a test set. It is the only way to know whether it works, and the only way to notice if it stops.
Frequently asked questions
Can hallucinations be removed completely?
Not in the model. In the solution they can be brought down to a level where they get caught, because every answer has a source and the important actions need approval.
What is a test set, and who makes it?
20–30 real questions from your everyday work, each with an answer a professional on your side has approved. We help collect them; the answers are yours.
How does an employee know whether an answer can be used?
By looking at the source. An answer without a source is not shown; an answer with a source can be checked in one click.
Is this the same as RAG?
RAG is the technique that makes the tool look things up in your documents before it answers. It is the single most important remedy against hallucinations, but not the only one.
Read next
Want help getting from explanation to practice?
Have a no-obligation chat with us about where it makes sense to start. We reply within 24 hours.