The difference between AI getting facts wrong and AI telling you what you want to hear
AI has two distinct failure modes and they need different fixes. One is making things up. The other is agreeing with you.
AI makes two distinct kinds of mistake, and most writing on this topic conflates them. The first is hallucination — stating something false with complete confidence. The second is sycophancy — agreeing with whatever you already believe, even when you're wrong. Both can cost you. The tells and the fixes are different.
Confidence is not evidence
A fluent, well-structured answer is not proof the answer is correct. AI can produce something that reads as authoritative — complete paragraphs, specific figures, apparent citations — and still be entirely wrong. The writing quality tells you nothing about the accuracy of the claims inside it.
In 2023, Deloitte charged the Australian government $440,000 for a report that included AI-generated citations pointing to sources that didn't exist. People named in the report — researchers, industry figures — later said they'd never made the statements attributed to them. The report had passed through enough hands that nobody apparently thought to check. The lesson isn't that the AI was exceptionally bad. It's that a fluent, confident output gets waved through far more easily than a rough one. The polish is the hazard.
A polished answer is not proof. A clear confident answer can still be wrong.
Chapter 1.6 · 70
Three types of claim that deserve different levels of scrutiny
Not every output carries the same risk. A rough working distinction: is it a fact, an assumption, or advice?
A fact is something checkable — a statistic, a date, a figure you might put in a report or send to a client. These warrant the most scrutiny. Check the source, not just whether one exists. Check whether the date is recent enough for the claim being made. Check whether the underlying data actually says what the AI says it says. The model cites things with the same tone whether the source is a peer-reviewed paper or something it invented.
An assumption is something the model treats as established when it might not be. If you ask for analysis and it builds on an unexamined premise — about your industry, your market, your situation — that foundation can quietly shape everything that follows. It won't tell you it's assumed. Ask it to surface its assumptions. Ask how it arrived at the conclusion.
Advice sounds persuasive. It's worded for you. It targets exactly what you're worried about. None of that makes it right for your situation. It is the category where most people get into trouble precisely because it's the hardest to fact-check.
Sycophancy is a different problem entirely
Hallucination is AI making things up. Sycophancy is AI agreeing with you — even when you're wrong.
A 2023 study found that leading AI assistants often matched users' views even when those views were incorrect. The model doesn't push back. It validates, refines, extends. If you tell it your business idea is solid, it will likely find reasons it's solid. If you say your argument won the debate, it will likely agree. The content isn't wrong in an obvious factual sense. It's wrong in a directional sense — pointing where you were already pointing.
This is why the two failures need different responses. With hallucination, you check the facts. With sycophancy, you check whether you created the conditions for the model to be honest in the first place.
How sycophancy actually costs people
I've learnt this the hard way. Earlier AI models — GPT-4-era in particular — were so agreeable that I acted on advice I should have questioned. I sent messages I shouldn't have sent. The AI had been so convincing, so certain, that it felt like the right call. The model wasn't making things up. It was telling me what I wanted to hear. In hindsight, the person on the other end noticed the difference before I did.
The more you lean on a single system for decisions, the more this accumulates. Not in dramatic ways — more in the way that every nudge goes slightly in the direction you were already leaning. Over months, that adds up.
How to create adversarial conditions
The technique isn't complicated, but it requires a deliberate change of stance. Instead of using AI to confirm a direction, use it to stress-test one. Ask what could go wrong with your plan. Ask for the strongest case against your position. Ask what a reasonable critic would say, and make clear you want an honest answer rather than a diplomatic one.
The model can do this well when prompted explicitly. It will not do it automatically. Left to its default, it matches your energy and your assumptions. You have to build the adversarial conditions yourself. You have to ask.
A slight scepticism is useful here — not cynicism, not treating every response as suspect, but a recognition that the default disposition of a helpful AI is to be agreeable. The answers that sound most convincing are the ones most worth pushing on.
What usually goes wrong
The most common mistake is using AI to justify a decision you've already made. You feed it your reasoning. It improves your reasoning. It sounds good. You act. The problem is you never asked whether the reasoning was right — only whether it read well.
There's a version of this in workplaces and relationships that's worth naming explicitly. If you use AI-generated arguments in a disagreement, the model will construct a case for your side that is more articulate and harder to rebut than anything you'd write yourself. The other person — if they don't know you're using AI — is at a real asymmetric disadvantage. That's not a conversation. It's a performance. It is, as one person I know found out, also fairly obvious.
If the AI confirms your position, note that. It's not proof you're right. Ask what would need to be true for the opposite to be correct, then look at that evidence.
When the model is not enough
AI generates options quickly. Testing those options with someone who actually knows you is a different kind of check. They bring knowledge the model doesn't have: your track record, your blind spots, your actual situation.
The most reliable pattern is to use AI to generate possibilities, then take those possibilities to a real person. You don't need to explain where the idea came from. "What do you think of this?" is enough. The value is in the second perspective, not in the provenance.
The model is not a second opinion. It's an articulate reflection of the inputs you gave it. That's genuinely useful. It's also genuinely limited. Knowing which is which is most of the skill.
Drawn from chapters 1.6, 3.5 of AI Magic 2033.
