The Why Question: Why Language Models Do Not Think, and Where a Different Question Starts
Abstract
A language model can read a long detective story with the last chapter cut off and name the murderer on the strength of clues scattered through the text, and still be unable to tell you why it chose him. We argue that this gap sits in the architecture: a model cannot answer a why-question about its own answer, and reasoning models, which repeat the model's pass in chains and trees, move the error around through the bias-variance trade-off without closing the gap. Thinking is defined here in a narrow, falsifiable sense: either you can follow an answer back through the steps that produced it, or you cannot.
A function that does not change
To see why a single language model cannot answer why, we only need to look at what kind of object it is. A deep neural network sends signals through layers of artificial neurons, and transformer models added three ingredients: embeddings that turn words into lists of numbers based on shared context, positional encodings that distinguish "the cat jumps over the dog" from "the dog jumps over the cat," and attention layers that weight statistical correlations across the input.
It looks remarkably lifelike, yet written down mathematically the entire network is a single static function with billions of fixed parameters. Once training ends, the function does not change. That is enough to guess the culprit in a detective story, because statistical cues point to the gardener. If you tell the model it is wrong, it appears to adapt — but only because your correction was added to the input prompt. The function itself has not learned a thing.
The decisive test is the why-question. Ask why the model named the gardener, and the only causally honest answer is that the weights at each layer multiplied out to give that word the highest score. Compare an intelligence test for children in which a ship floats even though it is made of iron. A child may reason that iron sinks, so there must be wood inside the hull. Factually, the answer is wrong. Yet there is a real thought process behind it — a clash between "iron does not float" and "this ship floats" — and you can ask the child why and test the premise. When you ask a language model why, you get a fluent paragraph citing Chapter 3. It may sound convincing; it may even be right. Structurally, however, it is nothing more than a brand-new run of the same black box on a new prompt.
Repeating the pass is not thinking
Reasoning models — chains of thought, tree search, and test-time token scaling — are often said to make the model think. They certainly get more answers right on benchmarks. To see what they actually do, consider the two kinds of controllable error every statistical model makes.
Think of an archer. If the bow is bent so that every arrow lands a foot too high, that is bias. If the archer aims dead center but the arrows scatter all around the target, the average is right while individual shots miss: that is variance. Expected error splits into bias, variance, and the wind in the world that nobody can control — and whenever you squeeze one of the first two down, the other tends to rise.
Every standard technique around a language model trades bias against variance at the input, inside the box, or at the output:
- At the input, a system prompt ("you are an expert programmer") raises bias on purpose in the hope that variance falls. Adding retrieved texts via RAG raises input variance in the hope that factual bias drops.
- Inside the decoder, temperature and beam search sample across likely word continuations.
- At the output, reasoning models repeat the pass. A chain of thought feeds the model's output back into its input; a tree of thought runs several passes in parallel and lets a scoring rule pick a branch.
Connect that back to where we started. At the input, in the black box, and at the output, you still cannot ask why. Ask why beam search chose one continuation, and the only truthful reply is that out of 125 combinations, that one had the highest product of probabilities. Ask why the tree took the left branch, and the reply is that the reward weights came out that way. However often you repeat the pass, it simulates the surface trace of thinking without ever becoming an inspectable thought process.
A different question
This is where a different architecture starts. Instead of asking a single black box to produce an answer and then asking it to invent an explanation afterward, we divide the inquiry into steps that follow how reflection actually proceeds: a thesis, an antithesis, and a synthesis.
In the detective story, the thesis states that a murderer needs a motive and no alibi — which points to the gardener. The antithesis counters that a missing alibi does not prove guilt if alibis can be forged or if another suspect lacked one too — which points to the butler. The synthesis weighs the exact tension between those two claims and decides.
Each individual step can still be carried out by a language model, and each step on its own remains a statistical black box. We do not pretend otherwise. What changes is the structure above the model. Because the perspectives, the counter-arguments, and the rules for weighing them are explicit in the architecture, you can walk backward step by step and ask why at every joint. If the final conclusion is wrong, you can see the exact step where the reasoning went astray — and use that stalled trace to form a better category.