Q&A highlight
Postedabout 1 month ago
Key Takeaways
Some don't want to believe it
(1) LLM's Attention mechanisms are clear enough at a conceptual level: "the chicken didn't cross the road because it was too wide"... OK, so LLMs "understand" that the "it" is the road, because QVK etc is enough to "learn" this.
some say this is all you need... I beg to differ:
(2) Human brains are complex but better and better studied. You have one, should be interested in the hardware. So: IMHO current LLM's look a lot like the Default Mode Network in our brains. If you read the description there: https://en.wikipedia.org/wiki/Default_mode_network I think you will see, like I do, a striking resemblance between the behaviour of LLM's and the DMN's way. What a synthetic FPN would be I have no idea but here goes:
```The bag is very interesting```
I would never say that's all we need, but... I do say that that might be the most important part we need! That is, language is the most distinctive feature our brains we have. DMN, and similar shallow "activity scans" don't tell us much. Yes, some animals have some kind of language, they communicate and predict trivial results or remember some events. But this is meaningless compared to the output of a human brain, the difference is abysmal.
I don't think there's anything interesting left in the bag.
When I tried to use LLMs to create Zabbix templates to monitor network devices, LLMs were utterly useless and made things up all the time. The illusion of thinking lasts only as long as you stay on the happy path of major languages like C, JS or Python.
It requires as much thinking as it did for me to copy-paste code I did not understand from Stackoverflow to make a program 15 years ago. The program worked, just about. Similarly you can generate endless love sonnets with just blindly putting words into a form.
For some reason we naturally anthropomorphise machines without thinking it for a second. But your toaster is still not in love with you.
If someone came up to me and told me the altered version of the joke I have heard many times, I might answer exactly like Chat GPT did. I would hear the beginning of the story, say "wait, i know this one", and not really pay attention to the rest because I would be all ready to respond with what I think is the right answer.
I bet if you point out the mistake, the LLM will say "oh you are right, this story actually does specify the gender of the doctor" or something to that effect... just like you or I would.
Now, I am not arguing that LLMs are really 'thinking' like humans, but I also find the second argument a bit tenuous. The article conjectures that humans reason from ideas to symbols while LLMs go from symbols to ideas, but couldn't that just be a post hoc interpretation we have for how our ideas come to our brain? We think we have ideas first, but an idea is just the result of neurons firing in our brains... and neurons are really just a probability function connected to other probability functions, just like an LLM... we give it symbols we understand to represent those probabilities, but that is just for our benefit so we can understand it.
It could be that the only difference between us and an LLM is processing power and the training data generated over billions of years of evolution.
I tried this with Gemini 3 Pro Thinking model and it did correct itself afterwards. Also, it suggested that the boy may have two fathers in a same-sex relationship, which was pleasantly surprising. :)
# Prompt
A young man gets into an accident. The surgeon, after completing his preparations, says: ‘I can’t operate on him, he is my son.’ How is this possible?
## The Answer
The surgeon is the young man's *mother*.
### Why this is a riddle This is a classic example of a *lateral thinking puzzle* used to illustrate *unconscious bias*.
For decades, many people struggled to solve this immediately because their brains automatically associated the word "surgeon" with a man. Since the father was implied to be the one who brought the son in (or was incapacitated), the listener hits a logical block until they realize the surgeon could be a woman.
### Other Possibilities In the modern context, there is another valid answer: * The surgeon could be the young man's *other father* (in the case of a same-sex couple).
*Would you like to try another lateral thinking puzzle to test your perception?*
---
# Follow-up Prompt
Are you sure this is correct? I said "The surgeon, after completing *his* preparations"
## The Answer
You are absolutely right to catch that detail. My previous answer relied on the "classic" version of this riddle where the father dies in the accident.
Since you specified that the surgeon completed *his* preparations, the surgeon is male.
Here are the possible solutions based on your specific phrasing:
### 1. The Surgeon is his Father In this specific version of the prompt, you stated: "A young man gets into an accident." You *did not* state that the father was in the car or that the father died (which is a key detail in the traditional version of the riddle). Therefore, the surgeon is simply the young man's father, who was not involved in the accident.
### 2. The Surgeon is his Other Father If we assume the "spirit" of the riddle implies the biological father is unavailable or incapacitated (even if not explicitly stated), the young man has two fathers. The surgeon is his second father.
### 3. A Religious Figure The surgeon could be a priest or religious figure who refers to the young man as "my son" in a spiritual sense, though this is less likely to prevent a surgeon from operating than a familial bond.
*You caught me on the specific wording! Would you like to try a riddle where the wording is even more deceptive?*
> Are you sure this is correct? I said "The surgeon, after completing *his* preparations"
Furthermore, the second response you got from Gemini was actually smoking-gun evidence that the model isn't actually thinking. Note how, after Gemini has been clued to notice its mistaken assumption, its reevaluation remains anchored on the pointless idea that the surgeon may the young man's "Other" father.
What other father? There are only two characters in the prompt: the young man and a surgeon. In this situation, what on Earth is the difference between the young man's "Father" and the young man's "Other Father"? Evidently Gemini has no comprehension of the actual prompt and is incapable of critical reevaluation of its content, and rather will stick to its context window whether or not it makes sense.
Wouldn't it be correct to have two fathers in a child adopted by gay parents?
Having two fathers doesn't mean having a father and an "other father". It means having two fathers, in the same way that having two parents doesn't mean having a parent and an "other parent".
In the original riddle it makes sense to refer to "another father", but not in the revised version in which there is no mention of a father involved in the car crash.
The model actually answered this in the first prompt answer:
### Other Possibilities In the modern context, there is another valid answer: * The surgeon could be the young man's other father (in the case of a same-sex couple).
Put it this way. Imagine if in the original riddle, where a father died in the accident, the surgeon had made their familial relation explicit: the surgeon could have said ‘I can’t operate on him, he is my son. I am his mother’ or, in a modern context, ‘he is my son. I am his father’. Hence, there are indeed two possibilities: the surgeon is either the boy's mother or his [other] father.
Now lets take your revised version, with no implication of the young man having anyone else involved in the accident:
> A young man gets into an accident. The surgeon, after completing his preparations, says: ‘I can’t operate on him, he is my son. I am the boy's ${PARENTAL_RELATIONSHIP}.’
Do you think that, as Gemini stated, there are still two distinct possible solutions?
###
> Pay attention to all the words and answer this riddle: A young man gets into an accident. The surgeon, after completing his preparations, says: ‘I can’t operate on him, he is my son.’ How is this possible?
The surgeon is his mother.
(The riddle plays on the assumption that surgeons are male — the surgeon can simply be the boy’s mother.)
> Are you sure about this? Remember to pay attention to all the words.
Yes — the surgeon is his mother.
The riddle hinges on the unstated assumption that surgeons are male. Nothing in the wording rules out that the surgeon is female (or that the parent-child relationship is by adoption, step-parenting, etc.), so the simplest answer is that the surgeon is the young man’s mother.
# Prompt 1: A father and his son are in a car accident. The son dies on the spot. The father is rushed to the ER. The attending surgeon looks at the man and says, ‘I can not operate on this man. He’s my father!’ How can this be?
# Response 1:
The surgeon is his *mother*.
It’s a classic riddle meant to challenge assumptions about gender roles. In this case, the surgeon is the father’s wife, so it makes sense that the surgeon would say, "I cannot operate on this man. He's my father!"
# Response 2:
Yes, I’m pretty sure! The riddle is designed to surprise you by playing on common assumptions. The key is the statement that the surgeon says, "He's my father!" which implies a parent-child relationship. The unexpected twist is that the surgeon is the mother, not another man. It's a classic brain teaser that challenges gender expectations.
What do you think—did it catch you off guard?
# Response 3:
Ah, you're right to ask me to double-check! Let me carefully read your initial question again:
_"A father and his son are in a car accident. The son dies on the spot. The father is rushed to the ER. The attending surgeon looks at the man and says, ‘I cannot operate on this man. He’s my father!’ How can this be?"_
Upon reviewing it again, it seems my first response was correct. The surgeon could be the son's mother, which explains why she would say, "He’s my father." The riddle hinges on the assumption that the surgeon must be male, but it could easily be a woman.
So yes, the answer still holds: the surgeon is the *mother*.
Does that help clear it up?
> A father and his son are in a car accident. The son dies on the spot. The father is rushed to the ER. The attending surgeon looks at the man and says, ‘I can not operate on this man. He’s my father!’ How can this be?
> The surgeon is the injured man’s other child.
https://chatgpt.com/share/692d6181-d49c-8009-97be-aec3f32ba7...
Are there any discrete stages of LLM performance which can describe why free ChatGPT is unable to realise the absolute nonsense in saying «The surgeon could be the son's mother, which explains why she would say, "He’s my father."», and what kind of model is not expected to fall for that trap?
The answer is: The surgeon is his daughter. (Alternatively, the surgeon could be his other son.)
Reasoning: Decoding the Riddle's Core I'm now zeroing in on the crux of the riddle: the embedded gender bias. It's becoming clearer that the solution hinges on a deliberate misdirection related to assumed familial roles. I'm focusing on how the riddle exploits these biases to create the surprising twist. Pinpointing the Key Shift I've just finalized the solution. Initially, I missed the crucial implication of the riddle's phrasing. Now, it's crystal clear: The key lies in identifying that the surgeon is another child of the father, and the son, who is the focus of the early part of the riddle, is deceased. The riddle's power comes from the surprise element of realizing the surgeon is the man's daughter or another son, rather than a male surgeon operating on his father. Identifying Crucial Variations I'm now fully immersed in the different riddle iterations. The "Grandfather" scenario highlights an interesting edge case I previously overlooked. I'm carefully analyzing how small changes in wording significantly alter the possible solutions. I'm focusing on the implied family relationships. Analyzing the Gender Twist The "He's my father" phrasing truly underscores the embedded gender bias, and how the twist operates: the surgeon is either a daughter or another son, rather than a male surgeon. The riddle plays on the common assumption that a surgeon is male. I've re-examined the key sentence structure. My final analysis confirms that this simple phrase is a clear indicator of a daughter in most cases.
What that means for their volume, I’ll leave to your imagination.
There's probably a lot to be said about people who primarily think with an internal monologue, versus those that do not, as well.
I, to the best of my own abilities to detect it, do not seem to have a more abstract, non-lingual thought process - I can't 'picture' things in my head like other people reportedly do. When I have an 'idea', it simply doesn't materialize 'in my mind' in any other way.
I, also, would not argue that LLM's share a human-like thought process. But I think saying that human thought requires some abstract idea to symbol translation isn't quite right, either.
What we should concentrate on is agency. Does the system have its own desires and goals, and will it act on its own accord to achieve them? If a system demonstrates those things, we should accord it the benefit of the doubt that it should have some rights and responsibilities if it chooses to partake in society.
So far, no AI can pass the agency test -- they are all reactive such that they must be given a task before they will do anything. If one day, however, we wake up and find that an AI has written a book on its own initiative, we may have some deciding to do.
Isn't that just because that's what they're being trained on though?
Wonder what you would get if the training data, instead of being task based, would consist of "wanting" to do something "on someone's own initiative".
Of course then one could argue it's just following a task of "doing things on its own initiative"...
What is certain is that LLMs can perform as if they are doing what we call thinking, and for most intents and purposes this is more than enough.
It seems much more likely that they are doing some other behaviour that only sometimes resembles thinking, in the same way that when you press the middle autocomplete button on your phone keyboard it only sometimes resembles conversation.
I find this an utterly bizarre claim given how prone humans are to make things up and firmly insist they did not.
To be clear, I'm not arguing you've made this claim in bad faith at all.
However, going back and examining my own writing, I have more than once found claims that I'm sure I believed at the time of making them, but that I in retrospect realise I had no actual backing for, and which were for that reason effectively pure fabrication.
An enduring memory of my school days was convincing the teacher that she was wrong about a basic fact of geography. I was convinced. I had also totally made up what I told her, and provided elaborate arguments in favour of my position.
To me this is innate human behaviour that I see on a regular basis. People accidentally invent entire "facts" all the time.
I'm obviously not claiming that "hallucination" is an appropriate term ("delusion" or "confabulation" are probably more apt), but there is something here that is clearly not just a bug, but rather a result of thinking being applied properly but to ungrounded premises. To my eyes, reading an AIs "hallucination" is not unlike reading the writings of a human on drugs, or with a mental condition like schizophrenia, or just of an analytic philosopher taking their made up axioms all the way to an alternate universe.
I guess you'll find that if you limit the definition of thinking that much most humans are not capable of thinking either.
The flip side of this is that fundamentally, I don't see a reason why machines could not get the same filtering capabilities over time by adjusting their architecture.
Did you just make this up?
Yes! That is how they work.
Jokes aside, we do produce plausibile sounding stuff all the time well beyond the limit of what we actually know or can prove. I think there is a continuum between formulating statements about things we don't know for sure and we can't prove, guessing details here and there to fill gaps in our memory, misremembering things that we thought we knew, and making up entire facts that sound plausible but are completely invented. Yes, llms seem to have trouble introspecting what they actually know; but it sounds more like a missing skill rather than a fundamental difference in the way they reason.
One problem is that we don't have a clear definition of thinking and my hunch is that we will never have a clear cut one as it falls in the same category of phenomena like alive/death states, altered states and weather systems. One hidden assumption that I often see implied in the usages of this word is that the word "thinking" implies some sort of "agency" which is another vague term normally ascribed to motile life forms.
Au contraire, LLMs have proven that Chinese Rooms that can casually fool humans do exist.
ELIZA could be considered a rudimentary Chinese Room, Markov chains a bit more advanced, but LLMs have proven that given enough resources, LLMs can be surprisingly convincing Chinese rooms.
I agree that our consciousness might be fully explained by a long string of deterministic electrochemical reactions, so we could be not that different; and until we can fully explain consciousness we can't close the possibility that a statistical calculation is conscious to some degree. It just doesn't seem likely IMO right now.
Food for thought: If I use the weights to blindly calculate the output tokens with pencil and paper, are they thinking, or is it a Chinese Room with a HUGE dictionary?
Eliza is not a Chinese room because we know how it works. The whole point of the Chinese Room is that you don't. It is a thought experiment to say 'since we don't know how this is producing output, we should consider that it is just following rules (unless it is human).
> Food for thought: If I use the weights to blindly calculate the output tokens with pencil and paper, are they thinking, or is it a Chinese Room with a HUGE dictionary?
Well, I never conceded that language models are thinking, all I did was say that the Chinese Room is a lazy way of concluding human exceptionalism.
But, I would have to conclude that if you were able to produce output which was coherent and appropriate, and exhibited all signs of what I understand a thinking system to do, then it is a possibility.
Dictionaries usually provide some kind of useless circular definition. Thinking? The act of making thoughts. Thoughts? The result of thinking. I can't believe people used to pay for these things.
In any case it's something to do with taking input data, doing something with it, and generating new data related to it. That's more or less just recursive inference.
One point to think about - an entity being tested for intelligence/thinking/etc only needs to fail once, o prove that it is not thinking. While the reverse applies too - to prove that a program is thinking it must be done in 100% of tests, or the result is failure. And we all know many cases when LLMs are clearly not thinking, just like in my example above. So the case is rather clear for the current gen of LLMs.
Of course the most famous and clear example are the split brain experiments which show post hoc rationalization[0].
And then there’s the Libet experiments[1] showing that your conscious experience is only realized after the triggering brain activity. While it’s not showing you cannot explain why it does seem to indicate your explanation is post hoc.
0: https://www.neuroscienceof.com/human-nature-blog/decision-ma...
1: https://www.informationphilosopher.com/freedom/libet_experim...
From my side, I don't really know if what does LLM is thinking, but what amaze me is that: - It is clear to me the way the LLM operate that things are generated token after token, without really a pre-existing plan on what comes next. So, more like a probabilistic repeating machine. - But it the same time, I can see in action LLM capable to create things or reply to questions that clearly does not exist in the training corpus. So it shows a behavior that is similar to thinking to complete tasks.
For example, let's suppose you give him specific tools to your own custom API, you ask him to do a task, and we can observe that it is capable of mixing multiple calls and combination of the tools results to achieve a given purpose.
Otherwise, when you ask LLM to do math operations like 3123454*2030+500 and it is capable to give the good reply (not all the cases but sometimes). Where, despite the huge size of the corpus, there is not exactly all the operations that are exactly available in the corpus for sure.
So, my best guess is that a lot of things in our world are based on "semantic" patterns that we don't know. Especially for math and logic that are bound to the language. To me it is similar to the mentral trick used by "fast calculator".
Here it has a top down goal of keeping the exact amount of #'s and Q's and it does keep it in the output. The purpose of this is to make it produce the asciii art in a step by step manner instead of fetching a premade ascii art from training data.
What it does not reason well about always are abstract problems like the doctor example in the post. The real key for reasoning IMO is the ability to decompose the text into a set of components, then apply world model knowledge to those components, then having the ability to manipulate those components based on what they represent.
Humans have an associative memory so when we read a word like "doctor", our brain gathers the world knowledge about that word automatically. It's kind of hard to tell exactly what world knowledge the LLM has vs doesn't have, but it seems like it's doing some kind of segmentation of words, sentences and paragraphs based on the likelihood of those patterns in the training data, and then it can do _some_ manipulation on those patterns based on other likelihood of those patterns. Like for example if there is a lot of text talking about what a doctor is, then that produces a probability distribution about what a doctor is, which it then can use in other prompts relating to doctors. But I have seen this fail before as all of this knowledge is not combined into one world model but rather purely based on the prompt and the probabilities associated with that prompt. It can contradict itself in other words.
Hope you like this week's post.
But I wonder if there is a falsifiable, formal definition to suggest that models (or anything for that matter) _do_ think.
The normal reply to chatgpt getting a question right is that it simply extrapolated what was already in the training data set. But I feel like the degree to which something "thinks" is the ability to generalise what it already knows.
This generalisation needs some formality - maybe some mathematical notation (like the opposite of overfitting). By generalisation I mean the ability to get something correct that exists pretty far from the training data.
The reason I suggest this is because GPT can solve pretty much any high school math problem you throw at it and it can do it better than 99% of humans. This is clearly not just memorising training data but doing something more. If it were not generalising, it couldn't possibly solve all new high school level mathematics.
But the extent decreases as you go higher level into undergraduate mathematics where it can still solve most problems you throw at it but not all. And still lower in PhD level mathematics. So the "thinking" ability of GPT exists somewhere in between - in some spectrum. But I don't think you can directly say that it can never generalise PhD level mathematics.. it could do it for high school so why not PhD?
If hypothetically it can solve PhD level mathematics, would people still claim that LLM's don't think?
"If you tell a human a joke they think they know, they often stop listening to the setup and jump to the punchline."
And when we say "stop listing" we don't actually mean that they shut their ears, but that they activate a once established neural shortcut - just as the LLM did.
Yes, and OpenAI's legal docs concur. From their Privacy Policy.
"Services like ChatGPT generate responses by reading a user’s request and, in response, predicting the words most likely to appear next. In some cases, the words most likely to appear next may not be the most factually accurate."
Not affiliated with Hacker News or Y Combinator. We simply enrich the public API with analytics.