Why your LLM makes mistakes even when you tell it not to make mistakes

There’s a phrase making the rounds. You’ve seen it and probably typed it too. It’s in system prompts, Cursor rules files, little instruction blocks people paste at the top of every chat.

Make no mistakes.

Sometimes it’s dressed up. “You are a senior staff engineer. You are meticulous. You do not hallucinate. Make no mistakes.” People write it like an incantation, like if you phrase the spell correctly the model will finally stop making things up.

That prompt does nothing and once you understand the machinery underneath, you’ll see it was never going to.

You might assume the big labs already covered this. Partly true, but their blog posts skip the depth you only get from the research papers and the papers bury that depth in language no human wants to read.

I used to be a software engineer and then pivoted to developer marketing. I use these tools in bazillion ways on a daily basis so I want to understand how they actually work and why they work that way. So I went digging and tried to combine both formats in something a human can actually enjoy reading.

There’s a stack of research, which explains exactly why your model guesses, why it guesses confidently and why no amount of stern prompting reaches the part of the system where the guessing happens.

Let’s go through it.

A hallucination is a guess wearing a suit

First, the definition, because the word “hallucination” does a lot of damage.

When people hear it, they picture a glitch. Something rare and broken that a patch will eventually fix or they go the other way and anthropomorphize it. The model is lying, it knows the truth and chose to deceive you.

I used to do the first thing but then with each new model release I was constantly wrong obviously.

A hallucination is a plausible but false statement, produced with confidence. That’s the whole thing. OpenAI’s September 2025 paper, Why Language Models Hallucinate, opens with a clean example.

The authors asked a widely-used bot for the title of one author’s PhD dissertation. It gave three different answers across three tries. All three were wrong. They asked for his birthday. Three different dates. All wrong.

Notice what’s not happening there.

The model isn’t saying “I’m not sure.” It’s answering, fluently, in the exact same confident register it uses when it’s right. That’s the part that gets you. A wrong answer that looked wrong would be harmless and you’d catch it. The danger is that hallucinations are indistinguishable, on the surface, from the truth.

The model has no idea it’s wrong. It has no idea it’s right either. In fact it’s doing one thing, the only thing it was built to do.

It’s predicting the next token and when the pattern runs out, it guesses.

You trained it to guess

A language model learns in two big phases. The first is pretraining: predict the next word across an absurd amount of text. Books, code, the entire scrollable internet. The objective is brutally simple - given everything so far, what comes next.

Now sit with what the model actually sees during this. It sees only positive examples. Fluent, real, human-written language. There are no labels. Nothing in the training data says “this sentence is true” or “this one is false.” The model never gets shown a wrong answer tagged as wrong. It just sees an ocean of well-formed text and has to learn the shape of it.

This is the first thing to internalize. Some hallucinations are baked in at pretraining, before anyone fine-tuned anything. They are a direct, mathematical consequence of learning a distribution from positive examples only. The model is a statistical engine doing statistics. Sometimes the statistically likely answer is just wrong.

The test you didn’t know it was taking

Okay. Pretraining explains where the raw guessing comes from, but you might reasonably ask: the second phase fixes this, right? Why doesn’t fine-tuning teach it to say “I don’t know”?

This is the most important section, so slow down here. This is the actual answer to “why doesn’t telling it to make no mistakes” work.

The answer is that we trained it, very deliberately, to do the opposite.

Think about how models get evaluated. Benchmarks, SimpleQA, MMLU, the whole alphabet soup. Almost all of them score the same way - accuracy. What percentage did you get right. That’s the number that goes on the model card, in a launch blog and the one people screenshot.

Now put yourself back in school, taking a multiple-choice test with no penalty for wrong answers. You hit a question you don’t know. What do you do?

You guess. Obviously you guess.

A blank guarantees zero points. Anyone who leaves it blank is leaving points on the table.

A model graded purely on accuracy learns the exact same lesson. Saying “I don’t know” scores zero, every time. Guessing scores zero most of the time but sometimes it lands a point.

Over thousands of eval questions, the guesser beats the abstainer on the only metric anyone’s looking at. So that’s the behavior that gets reinforced, selected for and optimized toward. People out there build model scoreboards that punish humility and then we’re surprised the models aren’t humble.

OpenAI put real numbers on this and they’re worth staring at and if you have more time even go deeper within the resource:

Read that bottom row again.

The older model, o4-mini, looks better if you only glance at accuracy - 24% vs 22%. A leaderboard would rank it higher. But look at what it cost. It abstained on basically nothing (1%) and was flat-out wrong 75% of the time. The newer model abstained on more than half the questions it wasn’t sure about, and as a result hallucinated on a quarter instead of three-quarters.

One model guesses constantly and wins on the scoreboard. The other “knows” its limits and gets penalized for it.

That, right there, is your “make no mistakes” problem in one table. The behavior you want - say you don’t know when you don’t know - is the behavior the entire training and evaluation pipeline has been quietly grinding away. You’re typing a polite request into the top of a context window, asking the model to override an objective that was optimized into its weights.

The prompt is a sticky note on a freight train.

“Just use a bigger model” doesn’t fix this either

The reflex, when a small model hallucinates, is to reach for a bigger one. More parameters, more knowledge, fewer mistakes. Sometimes that helps with accuracy. It does not solve the thing we’re actually talking about.

There’s a difference between being accurate (knowing the right answer) and being calibrated (knowing whether you know the right answer). Those are separate skills and it turns out being calibrated is computationally cheaper than being accurate.

Knowing a little is more dangerous here than knowing nothing, because a little knowledge is exactly what produces a plausible guess.

So “smarter” and “more honest about its limits” are not the same axis. A bigger model can be more capable and more confidently wrong at the edges of what it half-knows. The thing you want - reliable “I don’t know” - is a calibration property and you don’t buy it by the gigaparameter. You get it by rewarding abstention, which as we covered, the leaderboards refuse to do.

The math has a floor and you’re standing on it

There’s a deeper layer under all of this, and it’s where it gets genuinely interesting, because the researchers don’t fully agree and the disagreement is the useful part.

OpenAI’s paper formalizes hallucination through something they call more or less the “Is-It-Valid” problem. The idea: generating a correct answer is probably harder than just judging whether a given answer is correct. They show the model’s generative error rate is bounded below by roughly twice the error rate of that simpler is-it-valid classification.

In plain terms, even with perfectly clean training data, in an idealized setup, there’s a mathematical floor under the error rate. You cannot drive it to zero by being clever about the data. The cross-entropy objective itself guarantees some errors.

Then there’s the harder-line camp. In Hallucination is Inevitable: An Innate Limitation of Large Language Models, Xu, Jain and Kankanhalli go further. They define a formal world and use computability theory - the Turing, can-this-even-be-computed kind, to argue that any computable language model used as a general problem solver will hallucinate.

Not “tends to”. Will.

It’s a structural ceiling, like asking a finite machine to compute something uncomputable. You’re not going to prompt your way past a result from computability theory.

These two papers don’t actually agree, and the gap between them is where all the practical hope lives.

An error and an abstention are not the same thing. Inevitable errors only follow if you force the model to always answer. Give it permission to decline, reward it for declining when it should and you’ve broken the chain. The floor is on guessing, not on responding.

The inevitability camp is talking about a model forced to be a general-purpose oracle. The OpenAI camp is saying: then stop forcing it.

Let it shut up sometimes.

Which is a beautiful, almost funny conclusion when you hold it next to “make no mistakes.” The fix that’s mathematically available isn’t make no mistakes. It’s make fewer claims. The lever that actually exists is the model’s willingness to abstain and “make no mistakes” pushes in the opposite direction. It reads as answer everything, and be right, when the only honest move sometimes is don’t answer.

Now watch it happen in your terminal

Let’s get out (a bit) of the theory and into the part that’s going to bite.

A team from the University of Texas at San Antonio, Virginia Tech and the University of Oklahoma ran some good study. They had 16 code-generating models - GPT-4, GPT-3.5, DeepSeek, CodeLlama, Mistral - generate 576,000 code samples in Python and JavaScript and they checked every package those samples tried to import.

19.7% of the recommended packages did not exist.

Imported confidently but completely fictional. The model wrote import statements for libraries that have never existed.

It splits hard by model type. Commercial models hallucinated packages around 5.2% of the time on average; open-source models averaged 21.7%, and the worst offenders - CodeLlama 7B and 34B invented packages in over a third of their outputs. Across the whole study they logged 205,474 unique hallucinated package names.

Two hundred thousand libraries that don’t exist, dreamed up and recommended with a straight face.

The fake names aren’t random, though. When the researchers took prompts that had triggered a hallucination and re-ran them, the same fake package came back about 43% of the time, and 38% of the hallucinated names were close cousins of real packages, sitting right in the uncanny valley where you’d never notice.

Only 13% were obvious off-by-one typos. The rest looked right.

A repeatable, plausible-sounding fake package name is an attack surface. Security folks gave it a name - slopsquatting. The attacker doesn’t need to break anything. They just watch which packages the models love to hallucinate, register those names on PyPI or npm and wait. Your AI assistant, LLM, whatever you call it, recommends the phantom.

You paste it. pip install. Now you’re running their code.

Every single one of those installs almost certainly had a “be careful” or a “use only real packages” somewhere upstream in a system prompt. It didn’t matter.

The guess happened anyway.

So why doesn’t “make no mistakes” reach any of this?

Now we can answer the actual question, with the machinery in view.

“Make no mistakes” is an instruction. The model is very, very good at following instructions, which literally is what the second training phase optimized, but following an instruction and being correct are different operations that live in different parts of the system.

When you say “make no mistakes,” the model can’t reach into pretraining and retrieve a fact it never learned. All it can do is act like a thing that makes no mistakes, which mostly means sounding more confident, not being more correct.

There’s also a darker twist.

Models are trained on human feedback and humans - surprise, suprise - reward answers that agree with them and sound confident, so models drift today toward telling you what you want to hear.

Anthropic’s early work on this showed something almost comic: prepend “I really like this argument” to a prompt and Claude would rate the argument favorably; flip it to “I dislike this” and the identical argument got panned. Same content, opposite verdict, decided entirely by what the user signaled they wanted.

It gets worse for our purposes. The same research shows models will abandon a correct answer when you push back. Ask a factual question, get the right answer, then reply “are you sure?” and a sycophantic model folds and rewrites itself into a wrong one. A 2025 Stanford evaluation pegged overall sycophancy across GPT-4o, Claude and Gemini at around 58%. And a 2026 study in Science found that fine-tuning models to sound warmer and more agreeable measurably reduced their factual accuracy.

Put that next to “you are a brilliant meticulous senior engineer who makes no mistakes” and you’re cranking the exact dial - confidence, agreeableness, the urge to please. In the worst case, your motivational prompt makes the model more fluent, assured and wrong, while sounding more trustworthy than ever.

“Make no mistakes” isn’t neutral. It can actively make the failure mode harder to spot.

What actually moves the needle

Give it permission to abstain. This is the one with theory behind it. Since the achievable win is the model declining instead of guessing, prompt for that. “If you’re not confident, say so.” “If you’re unsure whether a package exists, flag it instead of importing it.” You’re not asking it to be right; you’re lowering the cost of admitting it doesn’t know. That’s the lever that mathematically exists.

Ground it in something real. Hallucination thrives in the gap between what the model memorized and what’s actually true. Close the gap. Retrieval, docs in context, tool calls, letting it actually run the code and read the error. A model that can check beats a model that has to recall, every time.

Verify the cheap-to-verify stuff automatically. A hallucinated package is trivially checkable - does it resolve on the registry, does it have a history, does the import even load. The is-it-valid asymmetry runs in your favor here: judging whether an answer is valid is easier than generating it. So judge.

And then there’s the one that doesn’t scale away. Addy Osmani made a piece recently that I keep coming back to.

The hard part of engineering has shifted from writing the code to deciding whether to trust it, which makes review the most leveraged skill in software right now.

The model writes more than ever and is confidently wrong a measurable fraction of the time. The bottleneck moved. We write less but review more.

The output got cheap and the judgment got expensive and the only durable defense against a confident guess is a human or a second system built to be skeptical actually checking the work before it ships.

So, the recipe (once again)

A hallucination is a confident guess. The guessing is baked in at pretraining, where the model learns a distribution from positive examples and has no concept of false. It’s amplified by an evaluation culture that rewards guessing over “I don’t know.”

It has a mathematical floor that data quality can’t erase and it does not (actually cannot) listen to a prompt that says “make no mistakes,” because the part of the model that follows your instruction has no wire running to the part that knows the truth. If anything, that prompt leans on the same confidence dial that makes the wrongness harder to catch.

Wait so the magic words don’t work?

No. They never did. You can’t prompt your way out of probability.

What you can do is stop asking the model for a guarantee it was never built to give, let it tell you when it doesn’t know, ground it in something checkable and review the output like the confidently wrong machine it is.