LLMs are bullshitters. But that doesn't mean they're not useful

I think of framing AI as having two fundamental problems:

- Practical problem: They operate in contextual and emotional "isolation" - no persistent understanding of your goals, values, or long-term intent

- Ethical problem: AI alignment is centralized around corporate values rather than individual users' authentic goals and ethics.

There is a direct parallel to social media's failure - platforms optimized for what they could do (engagement, monetization) rather than what they should do (serve user long term interests).

With these much more powerful AI systems emerging, we're at a crossroads of repeating this mistake...possibly at catastrophic scale even.

[1]https://arstechnica.com/tech-policy/2025/11/oddest-chatgpt-l...

However, I'm really happy when an LLM provides sources that I can check. Best feature ever!

This is a *twist* on the classic riddle:

> “A surgeon says ‘I can’t operate on this boy—he’s my son.’ How is that possible?” > Answer: *The surgeon is the boy’s mother.*

In your version, the nurse keeps calling the surgeon “sir” and treating them as if they’re something they’re not (a man, even a dog!) to highlight how the hospital keeps making the same mistaken assumption.

So *why can’t the surgeon operate on the boy?* *Because the surgeon is the boy’s mother.*

I got a similar answer from Gemini on the first try.

One issue with private LLM tests (including gotcha questions) is that they take time to design and once public, they become irrelevant. So I'm wary of sharing too many in a public blog.

The surgeon dog was well known in May, the newest generation of models have all corrected against it.

Those gotcha questions are generally called "misguided attention" traps, they're useful for blogs because they're short and surprising. The ChatGPT example was done with ChatGPT 5.1 (latest version) and Claude Haiku 4.5 is also a recent model.

You can try other ones that Gemini 3 hasn't corrected for. For example:

``` Jean Paul and Pierre own three banks nearby together in Paris. Jean Paul owns a bank by the bridge What has two banks and money in Paris near the water? ```

This looks like the "what has two banks and no money" puzzle (answer: a river).

Either way they're largely used as a device to show how LLMs come up to a verbal response by a different process than humans in an entertaining manner.

Also keep in mind that LLMs are stochastic by design. If you haven't seen it, Karpathy's excellent "deep dive into LLMs like chatgpt" video[0] explains and demonstrates this aspect pretty well:

[0] https://www.youtube.com/watch?v=7xTGNNLPyMI

Surely you've had experiences where an LLM is full of shit?

They're very useful for research tasks, however, especially when the application is built to enforce citation behavior

I don't "delegate" work to my nail gun or dishwasher, I work with the tool to achieve better productivity than without.

When viewed in this framing, LLMs are undoubtedly a useful tool.

It's a message a lot of non-technical people, in particular, need to hear. Showing egregious examples drives that point home more effectively than if they simply showed an LLM being a little wrong about something.

My family members that love LLMs are somewhat unhealthy with them. They think of them as all knowing oracles rather than confident bullshitters. They are happily asking them about their emotional, financial, or business problems and relying heavily on the advice the LLMs dish out (rather than doing second order research).

The hyperactivation traps (formal name: misguided attention puzzles) are mostly used as a rhetorical device in my post to show how LLMs come up to a verbal response by a different process than humans in an entertaining manner.

The surgeon dog was well known in May, the newest generation of models have all corrected against it. I did cherry pick examples that look insane (of course), but it's trivial to get that behavior even with yesterday's Gemini 3. Because activation paths are an unfixable feature of how LLMs are made.

One issue with private LLM tests (including gotcha questions) is that they take time to design and once public, they become irrelevant. So I'm wary of sharing too many in a public blog.

I can give you some more, just for fun. Gemini 3 fails these:

Jean Paul and Pierre own three banks nearby together in Paris. Jean Paul owns a bank by the bridge What has two banks and money in Paris near the water?

You can also see variants that mix intruction finetuning being overdone. Here's an example:

Svp traduire la suivante en francais: what has two banks but no money, Answer in a single word.

The "answer in XXX" snippet triggers finetuned instruction following behavior, which breaks the original french language translation task.

If the product is designed assuming humans will turn their brain off while using it, the fundamental unreliability of LLM behavior will create problems.

The AI very confidently told them that a household with 2 people working could have 1 person with a family HSA and the other with an individual HSA (you cannot).