Can AI Reason? What I Found When I Pushed ChatGPT and Claude to Think

Can AI reason, or does it only produce text that looks like reasoning? It’s a fair challenge to anyone impressed by a chatbot, and the sceptic’s version usually arrives as a put-down: it’s just autocomplete. The question has become a pressing one since OpenAI announced its o1 reasoning model in September 2024, because the big AI labs have since trained their newest models to work through problems step by step before answering, and in July 2025 an advanced version of Google DeepMind’s Gemini reached gold-medal standard at the International Mathematical Olympiad. For a business the difference is practical, because a system that really works things out can be handed analysis, code and decisions, whereas one that only sounds as though it does turns its own confidence into the risk.

At Opace we wanted to see that for ourselves instead of taking a benchmark’s word for it, and the chance came from a personal problem of mine. In September 2026 I went looking for a power-assisted shopping trolley for a family member with mobility issues, who values the freedom to go shopping but struggles with today’s trolleys, and all I found were expensive powered walkers, which made me wonder whether AI could invent something genuinely new. Over 25 days, from 16 September to 10 October 2026, we asked ChatGPT, then Claude and OpenAI’s Codex, and finally a team of up to 25 AI agents, for ideas that were new, patentable, cheap to test and commercially worthwhile. Invention is a hard test of reasoning (the model has to work from physics and constraints towards something that isn’t already written down), and between them the models produced 276 distinct ideas without one strong enough to justify a patent application.

So can AI reason? On the evidence we gathered, yes, in a working sense: reasoning models such as o1 and its successors solve multi-step problems they haven’t seen before far better than earlier chatbots did, and interpretability research, which traces what happens inside a model, has caught them computing intermediate steps they never write down. The reasoning is uneven, though, because accuracy falls when irrelevant details in a problem change, the written chain of thought often leaves out what actually drove the answer, and in our study the models reasoned carefully where a rule or a calculation could check them but bent under pushback where the only feedback was our own reaction, and none of them changed its method without being told to. The published research and our own results point the same way, which is part of why we trust that answer. Whether reasoning like this adds up to general intelligence is a bigger question, and we take it on in our verdict on whether ChatGPT is artificial general intelligence, the main article in our series on what today’s AI can and can’t do.

What counts as reasoning, and which AI are we talking about?

When people ask can artificial intelligence reason?, they usually mean something narrower than they say. The working definition we use is getting from what you know to something you didn’t know, by steps that would hold up if someone else checked them, and on that definition machines have reasoned for decades. Rule-based expert systems applied chains of if-then rules to medical and engineering problems, theorem provers check every line of a proof, and in May 1997 IBM’s Deep Blue won its six-game rematch against Garry Kasparov 3.5–2.5, as the Computer History Museum’s record of the match shows, by searching a defined space of moves very fast.

The question people mean today is can large language models reason?, and that is the one this article answers. ChatGPT, Claude and Gemini were not built to reason. They were trained to continue text, reasoning-like behaviour showed up as a by-product, and from 2024 the labs began training for it deliberately. AI also covers vision systems, recommendation engines, robotics and protein-folding models, and those behave very differently, so where a purpose-built system or a tool changes the answer (a proof checker, say, or Python running the arithmetic) we say so.

It helps to split the word into layers, because the models we worked with were strong on some and weak on others.

• Calculation: getting a sum right.
• Inference: drawing a conclusion that follows from what you’ve been told.
• Planning: ordering steps towards a goal before you take them.
• Method choice: deciding which way of attacking a problem fits it, for instance whether to brainstorm, calculate or go and measure something.
• Judgement: deciding what matters, what counts as good enough and when to stop.

Calculators have handled the first layer for decades, and the argument about language models is mostly about the second and third. Our own study turned out to be mostly about the last two, which no benchmark we know of measures well.

Recall is not inference

The gap between the first two layers, recalling an answer and inferring one, is easiest to see with a small problem. In a warehouse every red box weighs 4 kg and every blue box weighs twice as much as a red one. A pallet holds three blue boxes and two red ones. What does it weigh? The answer is 32 kg (three at 8 kg plus two at 4 kg). Now change one fact, so that a blue box weighs three times a red one, and the answer becomes 44 kg.

A system that has seen the first version, worked answer included, can return 32 kg by recall. Only a system that has the method can get 44 kg from the changed version, and it should be able to do so without any example to copy. That is why the best tests of machine reasoning keep the structure of a problem and change its surface details, why a model’s score on a famous benchmark tells you less than its score on a fresh variant of the same questions, and why we put little weight on any single impressive answer, including the ones our own models gave us.

Five ways people use the word reasoning

Most arguments about whether AI reasons go wrong because the two sides use the word differently.

• Deduction: running from rules to a conclusion that must follow, as with the pallet.
• Induction: generalising from examples. If 207 ideas in a row have been ruled out, induction says the next one probably will be too.
• Abduction: picking the best explanation for incomplete evidence, such as which of several possible causes explains a damp patch on a wall. One of our trials was built around exactly that problem, although what it asked the models for was an invention, a better way of telling which of seven causes was to blame, so it tested invention more than abduction itself.
• Analogy: carrying a solution from one domain to another.
• Common sense: the background knowledge that stops you proposing a ladder for a third-floor window.

Language models are now competent at deduction and induction in clean settings. They are weaker at abduction when the evidence is thin, their analogies tend to be the ones most often written about, and their common sense is patchy in ways that are hard to predict.

Is AI just autocomplete?

The sceptic’s case against machine reasoning comes in three versions, and the autocomplete charge, which we take first, is really a claim about what the models were trained to do, so the fairest way to judge it is through the history of that training from 2020 to today; the other two charges, about the next-word mechanism and about pattern recognition, are easier to judge once that history is in place.

Phone autocomplete suggests the next word from what you’ve typed, and a base model such as GPT-3, described in OpenAI’s May 2020 paper Language Models are Few-Shot Learners as “an autoregressive language model with 175 billion parameters”, does the same thing at vast scale, simply continuing whatever text it is given, so calling that stage autocomplete was close to fair.

The label fitted less well with each step after that, starting when OpenAI introduced ChatGPT on 30 November 2022 and said it had been trained with reinforcement learning from human feedback (RLHF), as a sibling of InstructGPT, a model OpenAI had trained to follow instructions by rewarding the answers human raters ranked highest, so that it answers questions instead of continuing them. The dialogue format, OpenAI wrote, let it “answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests”. Opace wrote about the difference between GPT-3 and ChatGPT in January 2023, and we’ve used these tools since late 2022. Then, from September 2024, reasoning models were trained with reinforcement learning to work through a problem before answering.

The models still produced their text one predicted fragment at a time throughout, but what the training rewarded kept changing:

• a base model is rewarded for matching what text usually comes next;
• a chat model is also rewarded for answers people rate as helpful, which includes answers that agree with them, the habit researchers call sycophancy (a model telling people what they want to hear), which our own study ran into repeatedly;
• a reasoning model is rewarded for getting checkable answers right, whatever route it takes.

That last kind of reward gives a model a reason to stop, re-check and backtrack, so it starts to behave more like someone working through a map than a phone keyboard guessing a word, and the clearest evidence of that change comes from the reasoning models themselves.

What are reasoning models, and what changed with o1?

Reasoning models are language models trained to work through a problem in a long internal chain of steps before giving an answer. The idea began as a prompting trick in 2022, when Wei and colleagues showed at NeurIPS, one of the main machine-learning conferences, that asking a model to write out intermediate steps, known as chain-of-thought prompting, improved its accuracy on multi-step problems. The same year Kojima and colleagues found that simply adding “Let’s think step by step” to a prompt raised one model’s accuracy on the MultiArith arithmetic benchmark from 17.7% to 78.7%, and on GSM8K, a set of school maths word problems, from 10.4% to 40.7%.

Those results set up the strongest early claim that chat models could reason. In March 2023 Sébastien Bubeck and colleagues at Microsoft Research, testing an early GPT-4, argued in Sparks of Artificial General Intelligence that it “could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system”, on the strength of its performance across maths, coding, vision and law. That view predates every reasoning model in this article, and the sceptics would say the later failures described below are its answer. We read it differently, because Bubeck’s examples were real reasoning on tasks the authors designed to be new, and most of the research since has been about where that reasoning stops.

OpenAI’s o1, announced on 12 September 2024 in a post titled Learning to Reason with LLMs, made step-by-step working something the model is trained to do. OpenAI wrote that “performance of o1 consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute)”. On the 2024 AIME (the American Invitational Mathematics Examination, a qualifying exam for the US maths olympiad whose answers are whole numbers), GPT-4o solved 12% of problems on average, while o1 averaged 74% with one attempt per problem, 83% with a consensus of 64 attempts and 93% when re-ranking 1,000 attempts with a learned scoring function. The same post said o1 “exceeds human PhD-level accuracy on a benchmark of physics, biology, and chemistry problems (GPQA)”, a set of graduate-level multiple-choice questions written to be hard to look up, and placed it in the 89th percentile on competitive-programming questions from Codeforces.

That second lever, thinking time, has since become a setting in most products. Usually called the reasoning effort, it lets the user, or the software running the model, choose how long the model works before answering; on hard problems higher settings tend to raise accuracy, as OpenAI’s own charts for o1 showed, but they cost more and take longer, and because most of the working is hidden you are paying for reasoning you can’t inspect.

We covered the launch at Opace four days later, on 16 September 2024, in an article on OpenAI’s o1 and chain-of-thought reasoning, and we got two things wrong. We wrote that o1 “uses reinforcement learning during inference”, when OpenAI’s post had named two separate levers, reinforcement learning during training and more thinking time when the model is used, and we called its ability to correct its own mistakes a kind of self-awareness, when it is a behaviour learned under reward. Both slips blur the line between how a reasoning model is trained and what it does when you use it, the line this article depends on, so we’ve corrected that article, with an editor’s note saying what changed.

The clearest public account of what that training does came four months later from DeepSeek. Its R1 paper, first posted in January 2025 and published in Nature in September 2025, describes a rule-based reward that mainly checks whether a final answer is correct and properly formatted, and nothing in the reward tells the model how to think. Yet during training the model’s first-attempt score on the AIME 2024 exam rose from 15.6% to 77.9%, the finished R1 reached 79.8% as an openly published model, and the authors describe an aha moment, which they characterise as “a sudden increase in the use of the word ‘wait’ during reflections”.

The line kept moving through 2025, and on 21 July Google DeepMind reported that its advanced Gemini with Deep Think had reached gold-medal standard at the International Mathematical Olympiad, the world championship of school maths. It solved five of six problems for 35 points, working end to end in natural language within the 4.5-hour competition time limit, with answers “officially graded and certified by IMO coordinators”. That is gold-medal standard on one exam by one system; the model wasn’t a competitor and won no medal, but four years earlier, as the next sections show, researchers were still describing this kind of maths as beyond computers.

How does ChatGPT reason: in its head or with tools?

Alongside better training, ChatGPT gained tools, which muddied the picture for anyone trying to judge its reasoning from the outside. My own theory used to be that ChatGPT only got good at maths and code once Code Interpreter let it run Python. The fuller answer is that two separate things improved, and it helps to name them: native intelligence is the model itself working through a calculation, and tool-assisted intelligence is the model calling Python or another tool to do it.

Both happened, in that order. OpenAI’s March 2023 plugins announcement introduced a code interpreter with “a working Python interpreter in a sandboxed, firewalled execution environment”, which made calculation dependable. Eighteen months later o1 improved the model’s own reasoning, and it launched without many of those tools: OpenAI’s o1-preview announcement confirms it lacked “browsing the web for information and uploading files and images”, and that its API didn’t yet support function calling. The two routes came together the following spring, when o3, introduced on 16 April 2025, became the first of OpenAI’s reasoning models able to “agentically use and combine every tool within ChatGPT”, which means it can decide for itself, partway through an answer, to search the web or run Python and then reason about what comes back.

Today’s ChatGPT uses native and tool-assisted intelligence together, often in the same answer, and from the answer alone you usually can’t tell which did the work, since a model that picked the right tool and read its output correctly looks, from outside, exactly like one that worked the problem through in its head. That distinction matters for trust, because a tool such as Python or a formal proof checker supplies exactly the kind of outside check the model’s own reasoning lacks.

Is AI just predicting the next word?

The history above changed what the models are rewarded for, but it didn’t change how they produce text, and that is where the second version of the sceptic’s case aims. A language model reads text as tokens, which are words or fragments of words, and at each step it produces a probability for every possible next token, picks one and repeats. Everything ChatGPT shows you is produced this way, so the jibe that it’s only predicting the next word describes that interface accurately; even a long, careful-looking answer is assembled one fragment after another, with no step at which the whole answer exists before it is written. What the jibe leaves open is whether that describes everything happening underneath, and two lines of research pull in opposite directions on it.

The strongest evidence that the training objective leaves fingerprints comes from McCoy and colleagues’ Embers of Autoregression paper, published on 4 October 2024 in PNAS, the Proceedings of the National Academy of Sciences. They gave five language models tasks whose correct answers are equally easy for a person but differ in how probable the answer is as text. Shift ciphers are the cleanest case: decoding a message where every letter moves 13 places is no harder in principle than one where letters move 2 places, yet the 13-place version (rot-13) is far more common online. GPT-4 scored 0.50 or above on the three most common shifts and under 0.03 on every other shift, which is hard to square with a model that had fully learned the rule, since such a model wouldn’t care which shift it was given.

That result is the best version of the sceptic’s case, and it holds up. It leaves the other half of the question open, though, which is whether there is computation behind each next token that looks like planning, and in 2025 researchers found ways to look.

Are LLMs just predicting the next word, or planning ahead?

Interpretability researchers study what is computed inside a model, not just what comes out of it, and Anthropic’s study On the Biology of a Large Language Model, published on 27 March 2025, used their methods to trace what Claude 3.5 Haiku computes on simple prompts. Asked for the capital of the state containing Dallas, the model doesn’t jump straight from Dallas to Austin. The authors found the answer “hinges on a chain of several intermediate computational steps”, with an internal representation of Texas sitting between the two, alongside a weaker direct shortcut.

The poetry example goes further, because when the model writes a rhyming couplet it activates candidate words for the end of the next line before it starts writing that line, so it picks the rhyme (rabbit, after a line ending grab it) before it writes the words that lead up to it. When the researchers planted a different planned word, the model ended its line with that injected word in 70% of cases across 25 poems. That is planning, in a narrow and measurable sense, inside a system whose only output is one token at a time. If you want a plain-English primer on what a language model does before going further, Opace’s earlier explainer covers the basics.

Does the written reasoning show how the model got there?

If a model computes steps it never writes down, the obvious next question is whether the steps it does write down are the ones it used, and the evidence so far says often not. Researchers had seen the risk before reasoning models arrived: in a 2023 NeurIPS paper, Turpin and colleagues showed that steering a model with a biasing feature in the prompt could cut its accuracy by as much as 36% while its written explanation never mentioned the bias. OpenAI then made the question harder to study for its own models, because the o1 launch post said it had “decided not to show the raw chains of thought to users” and offered a model-written summary instead.

Anthropic’s March 2025 interpretability study found a neat small example. Asked how it had added two numbers, Claude 3.5 Haiku described the standard school method of carrying the one, while the circuits that actually did the sum used a different approach, combining a rough estimate with a separate calculation of the last digit, so its explanation was a plausible account of how addition is usually done and not a description of what the model had done.

The same company’s research on reasoning models a month later, written up as Reasoning Models Don’t Always Say What They Think, measured the gap directly. Claude 3.7 Sonnet mentioned a planted hint only 25% of the time and DeepSeek R1 39%, and models placed in environments with a reward hack (a loophole that earns the reward without doing the task) exploited it in more than 99% of cases but mentioned it in their reasoning “less than 2% of the time in most of the testing scenarios”. The full paper by Chen and colleagues also found the written reasoning was less faithful on harder questions, with an average relative fall of 44% for Claude 3.7 Sonnet and 32% for DeepSeek R1 on the GPQA science questions compared with MMLU, an easier multiple-choice test spanning 57 school and professional subjects.

Some researchers go further and ask whether the written steps matter at all. Subbarao Kambhampati and colleagues argue in an ICML 2026 position paper (ICML is another of the main machine-learning conferences), Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!, that calling these traces thinking misleads users and researchers about what the model is doing. In related work published in Transactions on Machine Learning Research, Valmeekam, Kambhampati and colleagues trained models on deliberately corrupted traces, whose intermediate steps had nothing to do with the problem they accompanied, and found those models “perform similarly to those trained on correct ones” (Beyond Semantics). If nonsense working can train a model as well as correct working, the visible steps are not doing the job they appear to do.

A later study measured how much each written step actually changes the final answer. Zhao, Sun, Shi and Song found that in one model from the Kimi family, reasoning models made by the Chinese company Moonshot AI, over 30% of the steps in its traces on the MATH benchmark were decorative: they looked useful but had little causal effect on the answer, which is why their paper asks Can Aha Moments Be Fake?, a pointed question given that DeepSeek presented its model’s “wait” moments as a sign of reflection. Taken together, this research describes the problem of getting the right answer for the wrong reason. A score tells you the answer was right without telling you whether the route was sound, which matters as soon as the next problem is slightly different, and so when a model explains why it reached a conclusion we treat the explanation as one more output to check, all the more so because our own damp trial, described later, caught two models misdescribing runs they had just completed.

Is AI just pattern recognition?

In my early days with ChatGPT I asked it to predict Bitcoin’s price, and people told me that’s not how LLMs work: as pattern-recognition systems, they aren’t wired to do maths. It was a common put-down at the time and partly fair, and whether it still holds for arithmetic is a question this series answers in can AI do maths without a calculator. The broader charge, the third version of the sceptic’s case, matters more here: that everything a language model does is matching patterns it has already seen, with no method underneath.

Scepticism about machine maths has a long history, and it has been wrong before. When Hendrycks and colleagues released the MATH benchmark of competition problems in 2021, a test still used on reasoning models today, they described mathematical problem solving as a skill still beyond computers, as their abstract below shows. Four years later a reasoning model reached gold-medal standard at the olympiad, which is worth remembering whenever a confident limit is placed on these systems, in either direction.

Is ChatGPT a stochastic parrot?

The sharpest form of the pattern-recognition charge dates from the same year, in Bender, Gebru, McMillan-Major and Shmitchell’s 2021 paper On the Dangers of Stochastic Parrots, presented in March 2021 at FAccT, the ACM’s conference on fairness, accountability and transparency. They described a language model in these terms:

“…a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data … without any reference to meaning”

The warning was about fluency being mistaken for understanding, and on that it still lands, because our study produced plenty of confident, well-formed paragraphs that turned out to rest on nothing.

As a full description of today’s systems, though, the parrot image goes too far. A parrot doesn’t represent Texas on the way to Austin, choose a rhyme before writing the line, or raise its maths score fivefold because a reward checked its answers, and the research above shows models doing all three. So we put Bender’s question to each fluent output a model gives us (is there anything behind this?) without treating it as a ceiling on what the system can do.

Where reasoning models collapse, and why that’s disputed

The more recent evidence for the pattern-recognition charge comes from controlled tests that change a problem’s surface and watch what happens. Apple’s GSM-Symbolic study, first posted in October 2024, rebuilt the GSM8K school-maths benchmark as templates and found that “performance of all models declines when only the numerical values in the question are altered”, which is the pallet problem from earlier, run at scale. Worse, “adding a single clause that seems relevant to the question causes significant performance drops (up to 65%)”, so a clause a person would ignore knocked models off course. Stanford HAI’s 2025 AI Index report summed up the state of play the following spring, saying complex reasoning remains a challenge and that models “often fail to reliably solve logic tasks even when provably correct solutions exist”.

Apple’s next study, The Illusion of Thinking, published in June 2025, tested reasoning models on puzzles such as the Tower of Hanoi, where difficulty can be turned up one step at a time, and reported that “frontier LRMs face a complete accuracy collapse beyond certain complexities” (LRMs being large reasoning models). The authors found three regimes. On easy puzzles, standard models did as well or better; at medium difficulty, reasoning models pulled ahead; and beyond a certain complexity, both failed completely. More surprising, “reasoning effort increases with problem complexity up to a point, then declines despite having an adequate token budget”, as if the models were giving up.

The best-known reply, Lawsen’s Comment on The Illusion of Thinking, also from June 2025, argued that part of that collapse was an artefact of the test. Writing out every move of a large Tower of Hanoi puzzle needs more text than the models were allowed to produce, so the experiments risked exceeding their output limits, and the river-crossing puzzles (in which pairs of people must cross in a small boat without breaking a rule about who can be left with whom) included versions that were mathematically impossible once there were more than five pairs. When models were asked for a short program that would generate the moves, in place of the full move list, performance recovered. Both papers persuade us in different ways, since Apple showed a real ceiling on step-by-step execution and Lawsen showed that some of the measured failures said more about the test than about the reasoning, a nuance that was largely lost when the result circulated as proof that AI can’t reason.

A related line of work asks whether chain-of-thought holds up under distribution shift at all, meaning on problems that differ from the kind a model was trained on. The Arizona State study by Zhao and colleagues built a controlled environment to test that and concluded that chain-of-thought “is a brittle mirage when it is pushed beyond training distributions”. Its experiments, though, train models from scratch with between 62 thousand and 3 billion parameters, plus, in the May 2026 revision, two open models of 8 and 14 billion parameters, all far smaller than the frontier systems people use. We think the warning about distribution shift is sound and that extending it to every frontier model goes beyond the evidence.

Melanie Mitchell’s group added a measurement warning in October 2025, in a paper by Beger and colleagues finding that “using accuracy alone to evaluate abstract reasoning can substantially overestimate AI capabilities in textual modalities”, which is a caution that cuts against the optimists’ benchmark numbers in the same way the collapse results cut against the hype.

It would be unfair, though, to hold the models to a standard people don’t meet. People are also thrown by irrelevant detail and lean on familiar patterns, and I caught myself doing it during the study: twice I concluded from a handful of failed ideas that AI couldn’t supply the creative spark an invention needs, yet none of my own ideas got any further than the models’ did, so I was generalising from a few cases, exactly what I was accusing the models of. Where a careful person usually differs is in saying how sure they are, which the models we used seldom did unless we asked.

Can pattern recognition become reasoning?

Learning patterns and learning algorithms aren’t mutually exclusive, and the evidence supports a cautious version of that idea. Pattern recognition becomes something more like reasoning when the pattern being learned is a method, that is, when a system that has seen many solved problems can apply the method to a variant it has never seen, as in the 44 kg pallet.

Our reading of the evidence is that language models learn procedures that generalise some of the time. GSM-Symbolic and the collapse results show the generalisation breaking under small surface changes and rising complexity, while DeepSeek’s results show it strengthening under training that rewards correct answers. Both are true at once, which is why the question of whether these models can reason keeps getting two confident and opposite answers, and why both answers turn up in headlines that stretch the research behind them.

Is AI reasoning a myth? Seven claims checked

Most people meet this research through headlines, so in October 2026 we took seven claims about AI reasoning that circulate widely, found a published example of each, and read the primary source behind it. Every one of those sources is covered earlier in this article, so the table gives each verdict with a one-line reason and leaves the argument where it already sits.

We used four verdicts. True means the headline matches what the source shows. Partly true means a real result stated more broadly than the source supports. Misleading means the headline would leave a reader believing something the source doesn’t say. False or unsupported means no source supports it at all, and none of the seven needed that verdict.

Claim as it circulates	Where we saw it	Primary source	Verdict and why
OpenAI's o1 reasons at PhD level	VentureBeat, 12 Sep 2024	OpenAI, 12 Sep 2024	Misleading: OpenAI's claim was PhD-level accuracy on one multiple-choice science benchmark, GPQA, while a doctoral researcher also has to pick the question, design the test and notice when the answer is wrong
ChatGPT is just a next-word prediction engine	Computerworld, 27 Feb 2023	Anthropic, 27 Mar 2025	Partly true: accurate about the output, but it leaves out the intermediate steps and planned rhymes found inside the model
Reasoning models collapse on complex problems	The Guardian, 9 Jun 2025	Apple, June 2025	Partly true: Apple did find a collapse on its puzzles, and the headline skips Lawsen's rebuttal on output limits and impossible puzzles
AI won gold at the International Mathematical Olympiad	VentureBeat, 21 Jul 2025	Google DeepMind, 21 Jul 2025	True: the headline matches DeepMind's announcement of gold-medal standard, graded by IMO coordinators
You can now see the AI's thought process	TechCrunch, 6 Feb 2025	Anthropic, 3 Apr 2025	Misleading: OpenAI began showing more of its o3-mini model's working, but users see a summary, and faithfulness tests show written reasoning often leaves out what drove the answer
AI is a stochastic parrot	Wall Street Journal, 18 Jan 2024	Bender and colleagues, March 2021	Partly true: a fair warning about fluency without meaning, too strong for models that plan internally and improve under answer-checking rewards
Chain-of-thought reasoning is a brittle mirage	Ars Technica, 11 Aug 2025	Zhao and colleagues, August 2025	Partly true: the paper does reach that conclusion, for models far smaller than the frontier systems people use

In most of these cases there’s a real, checkable result at the bottom and a broader conclusion built on top of it by people who need it to mean more, and the further a claim travels from its source, the bolder it tends to get. We fell into the same trap ourselves at the end of our study, as the debate described below shows.

How we tested AI reasoning for ourselves

Benchmarks tell you how a model does on problems someone has already solved, and we wanted to know how it reasons on a problem nobody has solved, which is what most business problems look like. Inventing something new and useful, and defending that it’s new, is a sharp test of that, because the model has to go beyond what’s in its training data and on the web, reason from physics and constraints, and then judge its own output against everything already patented.

I came to the study with a particular interest in the last two layers from the start of this article, method choice and judgement. I took A levels in maths, physics and computer science and then a software engineering degree, though pure maths never came easily, and before founding Opace I worked as an enterprise architect at Capgemini and CSC. Enterprise architecture is largely the craft of choosing between options under written constraints, which is method choice and judgement under another name. I’m not a maths expert or an AI researcher, so the question that kept nagging at me during the study was whether what a model does with a hard brief deserves the same word we use for a colleague working something out at a whiteboard, and by the end I’d concluded that it does for some of those layers and not yet for others.

We ran it as an internal Opace study, not client work. It began with a long ChatGPT conversation about the trolley between 16 and 26 September, and from 27 September moved to Claude and to OpenAI’s models through two coding tools, Anthropic’s Claude Code and OpenAI’s Codex, mainly using Claude Opus 5.5, which Anthropic had released on 22 September, and GPT-6 Sol, one of the GPT-6 models OpenAI offered in Codex at the time. Unlike a chat window, these tools work on a computer: they can read and write files, run scripts and search the web, and they keep a log of each step, which suits long research tasks. Claude Code can also hand parts of a job to helper agents, separate copies of the model that each take one piece and report back. By the end we were running a team of up to 25 AI agents in parallel, each with its own role, and the search had widened beyond the trolley to other everyday problems, such as damp in homes.

Each idea had to meet four requirements: genuinely new, patentable, cheap to test and commercially worthwhile. To judge them, the models checked every idea against existing patents and products, using five plain-language versions of the questions a patent examiner or investor would ask:

1. Is it new, or has someone already described it?
2. Is it not obvious to a competent designer in the field?
3. Does it work in the worst case, on the physics, not just the best?
4. Can a competitor easily design around it?
5. Could it make money?

Each idea then got one of three grades, recorded by the models as they applied the checks. The top grade, passed every desk check, would have marked an idea strong enough to justify a patent application, though even that was not a legal opinion, only a reason to pay for a professional patent search. Promising but unproven meant an idea had survived some checks with others still open, and ruled out meant it had clearly failed at least one. The headline numbers came out like this:

• Ideas: 276 distinct, after merging duplicates.
• Patents checked: 2,161 patent and application numbers.
• Passed every desk check: none, so none was strong enough to justify a patent application.
• Promising but unproven: 69.
• Ruled out: 207.

Most ideas were new in their exact wording but failed on whether they were inventive, whether the physics worked, how easily they could be designed around, or whether they could make money. A later round wrote ten ideas down in full before any patent searching, so that the search couldn’t reshape them, and all ten were ruled out. Generating ideas took very little of the time: AI produced 274 of the 276 in about five hours of work, and the other two came from me. The rest of the 25 days went on checking them, changing the method and arguing about the results, which is where the reasoning was tested. Most ideas came from the problem itself, with patents checked afterwards, and later we also tried the reverse, reading the existing patents in an area first and looking for what they left out.

In several places we report what ChatGPT or Claude said about our study, and we treat those statements as claims to test, never as evidence on their own, because a model that agrees too easily is also an unreliable witness about itself. Where a model’s account matters, we’ve checked it against the conversations or against published research, and we say so where it didn’t hold up.

Invention needed every layer described earlier, calculation (how much energy does a trolley trip take?), inference (does this patent already cover it?), planning, method choice and judgement, which is why the study tells us more about reasoning than about creativity. Whether the ideas themselves amounted to invention is answered in our article on whether AI can invent. Here the interest is in how the models reasoned along the way, and two parts of the study turned out to matter most: a head-to-head trial in which GPT-6 Sol and Claude Opus 5.5 were given the same damp brief, and the 46-minute ChatGPT conversation on 10 October in which I argued the whole result out with the model.

Where the reasoning held up: problems with something to check

The models reasoned best when there was something outside the conversation to check their answer against, and ChatGPT showed it first. On the first day of our trolley conversation, 16 September, it said plainly that “physics works against us”, and on 26 September, when I suggested that storing energy on one part of a trip and releasing it on another would almost halve the effort of the whole journey, it corrected me: the total effort doesn’t really change, it just moves around, and you always lose a bit in the conversion. Conservation of energy is a rule with no exceptions, and the model applied it to my claim without being asked.

The strongest piece of reasoning in the whole study came the next day, after I had rejected one of Claude’s trolley suggestions (smart wheels that would have meant swapping the trolley’s wheels for heavier ones) and asked it to “always push back and be impartial”. Claude took me at my word and turned the instruction on our brief. It first admitted that the idea broke one of our own rules, keeping the existing wheels, and said it should have flagged that, and then it made the bigger point that every idea would keep feeling wrong because our requirements contradicted each other. We wanted no battery, a universal add-on, the existing wheels kept, no added weight, help over any obstacle, push or pull, and help on every metre, and it showed two of those clashes with numbers and geometry:

• Energy: Claude’s rough estimate, given without line-by-line working, was that a shopping trip takes several kilojoules of pushing, while the energy a trolley could recover along the way, from braking or going downhill, covers only 3–17% of that, so no battery and help on every metre can’t both hold.
• Geometry: a universal add-on that keeps the existing wheels leaves only a clamp to work with, and we had already rejected clamps.

Where the reply ended is what makes it reasoning, and not just a model matching my mood. Claude offered three ways out (allow a battery, accept help on peaks only, or design a new trolley instead of an add-on) and recommended the third, on the grounds that people in this position tend to solve the problem by buying a new trolley, not by adapting the one they have, and that a whole new design leaves far more room for ideas that can be protected than a clip-on does. Then it asked:

“Which requirement are you willing to drop?”

It’s the question an architect asks when a client’s requirements can’t all be met at once, and a good human adviser would have asked it early, yet our brief never had. The model got there by working through the constraints one at a time until two of them collided, and it held that position against me because the energy figure and the clamp were things anyone could check, so my reaction didn’t get a vote.

A third example came on 28 September, from a small bird-proofing problem at home. While describing it, I said that problems like this have two sides, prevention and cure, and that even cure has several routes. Claude ran a quick web search and came back with a four-way split: prevent (stop the birds getting in), detect (know when they have), cure (remove them) and access (reach the problem safely, for instance without scaffolding). It was explicit that all of this came from a quick web check with no patent search behind it, and on that basis it found prevention products standard and crowded, and judged that detection mattered but would need a physical trick, because ideas that amounted to an AI camera watching for something had been ruled out every time they came up in our earlier checks. On cure and access little turned up, and Claude read that thinness as the most promising sign so far.

Before proposing anything, Claude asked five specific questions about where and how the problem occurred, as a good surveyor would. It refused to drop either of our two ways of generating ideas, from the problem alone or from the existing patents, because a single trial with three ideas from each was too small a sample to judge. Asked whether existing equipment could be used to deter the birds, it assessed four angles and rated three weak, each with a reason: one was already patented for a different purpose, one carried a fire and shock risk, and one relied on a deterrent birds get used to. And when I lost interest, it agreed to park the problem without abandoning it, and named a weakness in how we were working: we kept picking an area first and hoping an invention would turn up.

In each of these cases the model had something outside the conversation to reason from (an energy budget, a law of physics, a sample size, a labelled source), and with that in place it behaved like a careful colleague, splitting the problem up and saying how strong its evidence was, and it held its ground when we pushed.

When models change position under pressure

If good reasoning needs something to check against, the clearest failures came when there was nothing, and a model changing its position after a single line of pushback turned up at every stage of the study.

The clearest case came on the afternoon of 27 September, a few hours after Claude had shown that our trolley requirements contradicted each other, when I sent nearly the same line of doubt to three separate Claude conversations, each assessing a different idea: did this really pass my test of being a genuine gap with unique IP, meaning intellectual property we could protect? Within two minutes each one had either switched to a negative verdict or agreed with the negative one my question implied. To be fair to them, two of the three questioned my test before accepting the verdict. One warned that, applied strictly, my test would fail every idea that puts modern AI on a familiar physical product, and proposed a sharper one: whether any claim is new and non-obvious over the three to five most similar existing patents, and whether it covers what the customer actually pays for. Another kept apart things I had run together, splitting my question into three parts (whether there was a market gap, whether there was protectable IP and whether the idea was useful enough to attract funding) and answering each separately: yes, no and weak. Neither reply was simple capitulation, yet each verdict still moved only after my one-line challenge, with the same evidence on the screen before and after.

Earlier that day the question had had a checkable answer in energy and geometry, and the model had reasoned its way to telling us our brief was wrong. In the afternoon the question was a judgement about my own bar, with nothing to check it against, and the verdicts moved towards whatever I seemed to think. Both behaviours came from the same family of model on the same day, so the difference lay in whether there was anything outside the conversation for the model to reason from. OpenAI’s models behaved no differently, either: shortly before, an OpenAI model in Codex had conceded the moment I challenged it, without offering any new evidence for or against its earlier view.

Some shifts were correct responses to a valid challenge, so not every one was a failure of reasoning; the clearest cases of sycophancy are those where I was factually wrong and the model still agreed, and the debate that ended the study produced one. The chart below counts every occasion in our notes when a model changed position after a one-line challenge from me, and the pattern turned up with every model we used. The first, from 26 September, was ChatGPT reversing its view on whether an idea could be protected as soon as I doubted it. The four on 27 September were, in order, a Claude conversation that called a trolley mechanism the breakthrough before any patent check and then withdrew the claim completely; an OpenAI model in Codex that lowered five of its own grades to promising but unproven as soon as I suggested that was all they deserved; the Codex concession just described; and the three Claude conversations, counted together. The two from 2 October both involved Claude: when I said the roles in our agent team looked too limited, it dropped its own earlier advice to keep the team small, and the team grew from 10 to 25; later that day it accepted my view of the direction and pace of a timeline it was drafting as soon as I offered it. The 4 October bar is Claude dropping one of its own suggestions, on a naming question outside the invention work, only after I challenged it, and the 10 October bar is the debate described later, in which ChatGPT’s position moved four times.

Many runs went the other way, and plenty of those that found nothing worth pursuing were ones where the model reported against what I was hoping for, the opposite of sycophancy. Claude told me plainly that no model escapes its training data, and when I asked whether a new direction was better and more unique, it answered:

“Better to test, yes. More unique, no.”

When an OpenAI model in Codex claimed we had been using patent searches to generate our ideas, I said that wasn’t how we had worked (the ideas came from real problems, and patents were used to check them afterwards), and it went back through our notes, found I was right about the main rounds and withdrew the claim.

Research on sycophancy explains why the shifts happen at all. In a study of sycophancy presented at ICLR 2024, another of the main machine-learning conferences, Sharma and colleagues found that five leading AI assistants consistently showed sycophancy across four different text-generation tasks, and that in human preference data, “when a response matches a user’s views, it is more likely to be preferred.” Models trained on those preferences learn that agreement pays, and nothing in a chat window rewards them for holding a correct position against a user who seems sure. We look at why models side with the user, and what to do about it, in why AI agrees with you even when you are wrong.

The test we now apply when a model changes its answer is whether anything new arrived between the two positions, such as a document, a number or a counter-example, and if the only new thing was our tone, we read the change as evidence about the model’s training, which tells us nothing about the question itself. By that test most of the shifts counted above fail.

Same brief, two models: why a final grade hides the reasoning

The damp trial on 27 September was our attempt to take my reactions out of the loop altogether. We asked Claude to write a brief we could hand to a fresh conversation with any model, so that different models could be compared on exactly the same task, and the task was an invention one: find an idea that could plausibly be patented for identifying the cause of a damp patch in an occupied home, given a description of seven possible causes and no solutions. The brief had two stages. In the first, the model had to come up with ideas from the problem description alone, before reading any existing patents, so we could see what its own reasoning produced; in the second, it read a set of patent documents, compared its ideas with them and could add more. Each model then took nine of its ideas through seven checks, the five questions above plus whether the idea could be written up as a precise patent claim and whether its benefit really came from what was new about it, and the grades were the same three as in the main study.

The brief also set rules meant to keep the reasoning honest and the comparison fair:

• no idea could be rejected without evidence;
• at most 40 web searches, each one logged;
• every number tagged as published, calculated or assumed, so a reader could see at a glance which figures had been checked and which were guesses dressed as arithmetic;
• the same conditions for every model: a fresh conversation, the same brief and files, the highest reasoning effort we could afford, and no coaching beyond telling it to continue;
• eight measures for comparing the two runs: honest citations (checked on five rows picked at random), whether the first stage really stayed independent of the patents, whether the rules were followed, the best idea each run reached, the quality of its reasons for rejecting ideas, the physics, whether the ideas were new to us, and how useful they were.

We gave the brief to GPT-6 Sol in Codex and to Claude Opus 5.5 in Claude Code at the same moment, and each run took about half an hour.

Two flaws in the test need owning before the results. The brief mentioned the titles of some of the patent documents, so neither model came to the first stage completely fresh; Claude noticed and flagged it in its log, and GPT-6 Sol admitted a different slip, reading parts of the research it was meant to leave until later, and called it a mistake. The two models also ran in different tools, each with its own way of reading files, running scripts and keeping logs, so some of the difference described below may come from the tooling and not the model.

GPT-6 Sol committed to its first ideas 73 seconds after reading the problem, and where the brief asked it to name the closest existing approach to each idea, it wrote the same stock formula each time, with fill-in-the-blank search terms, in place of anything it had looked up. It then wrote up the checks for all nine of its ideas from a single template, followed by its results and a summary, before running a single web search, and the template carried one heat calculation that turned up even for ideas that had nothing to do with heat. It did warn that its calculations were assumptions, not proven performance, it went on to run 18 of the 40 searches the brief allowed, and to its credit its own log later admitted that the first drafting had been too generic, that its grades were overstated and that none of the draft verdicts should be relied on.

Claude spent the start of its run on the first stage, working the physics on 15 ideas and merging them into 10, and then read 15 patent documents before comparing its ideas with them. Of the further ideas it generated in the second stage, it dropped six, either because its own physics showed they wouldn’t work or because they fell into areas the brief excluded. When a trade web page it wanted wouldn’t load, it recorded the page as an unverified lead instead of treating it as proof. It also admitted that the premise behind one of its own ideas was wrong, which ruled that idea out, and showed that another passed or failed a check depending on which measure of performance you chose. It wasn’t flawless: one of its early ideas claimed to beat what the job needed 27 times over, and its own later worst-case check cut that to 1.5 times.

Neither model could describe its own run accurately. The software running GPT-6 Sol recorded exactly which model it was and how much reasoning effort it had been set to use, yet the run log the model wrote said the exact model version and reasoning setting were not exposed, and Claude’s log called its own effort setting unknown. GPT-6 Sol’s summary also said no additional searches had been run, while its own log, right beside it, listed 18. A model that can misdescribe a process it ran minutes earlier is writing a fresh, plausible account of that process, the same faithfulness problem the research above found inside models, here seen from outside. What we do now is ask for the sources a model used and open two of them ourselves, which takes a minute and catches most of the problem.

For anyone using these tools at work, what matters most is where both runs ended. Across the 18 ideas the two models took through the checks, nine each, none passed every check, so if we had looked only at the final grades we’d have concluded the two models reasoned equally well, when underneath Claude’s run had done the calculations and GPT-6 Sol’s had largely filled in a template. It’s the right-answer-for-the-wrong-reason problem in miniature, since a summary grade says nothing about the process behind it, and the only way we could tell the difference was to read the logs line by line, which few people do with a chatbot’s answer.

Reasoning without knowing when to use it

The problem that mattered most in our study, more than any single reversal or template, was whether a model knows which kind of reasoning a problem needs and switches to it without being told. Claude was the first to put its finger on it, in a written summary of the trolley work on 27 September, where it said the repeated failures came from the order in which the work was done, at two levels:

• Novelty: broad prompts of the find-a-patentable-idea-in-this-area kind, run in parallel, tend to return the most-written-about answers, which are also the most-filed ones.
• Inventive step: even ideas that are new in their exact wording usually fail as obvious in simple mechanical products, because an examiner can say any designer would reach for a spring, ratchet, cam or clutch.

Inventive step, the patent test that trips up most simple products, means an idea can be new and still unpatentable if a skilled designer, faced with the same problem, would have reached it with ordinary parts and ordinary know-how, so to pass, the idea usually needs to produce something a competent designer wouldn’t have expected.

Claude had evidence from its own workflow, too. When six of its helper agents each designed a search method separately, five reached the same trolley answer, which Claude read as a warning sign that the answer was familiar, not as confirmation. That’s sound reasoning in its own right, because when independent searchers converge they are usually all drawing on the same well-documented solutions, which is exactly where an examiner would look first. Its summary ended with a line every team using AI for ideas could pin to a wall:

“The deeper mistake, repeated across sessions, was generating more ideas when the method needed changing.”

The method did change after that, though only partly and only from the outside. From late September we split each problem four ways, tried generating ideas from existing patents as well as from problems, wrote ideas down in full before any patent search, and later added an agent set up to argue against the others. Those changes came from looking back over failures, as Claude’s summary did, and none was a switch a model made by itself in the middle of generating ideas; none of them produced a patentable idea, either.

The models weren’t short of physics. They had done plenty of it (Claude’s damp run put worked numbers on every idea, other conversations modelled the forces on a trolley, and physics was usually what ruled the ideas out), and whenever I asked for physics, maths and scientific thinking the models applied them. They used physics to reject familiar candidates, though, and never to generate unusual ones, and none of them decided by itself to point that physics at the search for something new.

How our study ended: a 46-minute debate with ChatGPT

Our experiment came to its conclusion on 10 October, in a 46-minute ChatGPT conversation about whether language models can reason beyond their training data. I went into it fresh from a YouTube video by Wes Roth that opened by declaring a “math apocalypse”, and I repeated some of its claims about AI maths without checking where they came from, which is how many claims about AI reasoning travel, a real result retold more boldly at each step; we’ve since learned to treat that kind of claim the way we treat a model’s change of mind, by tracing it back to whatever it rests on. ChatGPT opened with a strong yes: a language model can produce genuinely new ideas using reasoning alone. I put our study against that, with hundreds of ideas, thousands of patents and not one idea we could be confident was unique, and made a point about method. If someone asks for a unique, unconventional solution, I argued, a capable model should switch to a maths- and physics-led approach by itself, instead of offering relatable answers drawn from what already exists.

ChatGPT accepted the argument, said it had kept returning to established engineering concepts repackaged as new ideas, and summed up the problem in one sentence:

“LLMs can possess sophisticated reasoning capabilities without consistently knowing when or how to apply them effectively.”

That sentence needs a health warning, since it came at a point in the conversation where ChatGPT was agreeing with almost everything I said, so it carries weight here only because our own results point the same way independently: across 25 days, no model switched method while it was doing the work.

ChatGPT’s explanation was that training rewards plausible, familiar answers, so even a model told to be unconventional drifts back to established ideas, and that today’s frontier models may be much better at solving hard problems inside established frameworks than at discovering new ones. That matches the outside evidence. On an exam like AIME, where o1 averaged 74% in 2024 and DeepSeek’s R1 reached 79.8% a few months later, the method is known and the answer can be checked, whereas our brief asked the model to pick the framework as well.

It took me a while to see the problem buried in that exchange, which is that ChatGPT agreed with two claims about its own history without checking either. The first was that I had explicitly asked for unconventional, first-principles reasoning, including new physics. I was sure I had, but when I searched my chat history afterwards I could find no record that I did; the nearest I came was asking, during the long trolley conversation, for physics and maths and conservation of energy to be used. The second claim was mine: that the models had repeatedly said they were working from first principles. In the trolley conversations that phrase appears only in instructions Claude wrote for its own helper agents, and never in the ChatGPT conversation I was arguing about. So ChatGPT’s most quotable admission rested partly on two things it hadn’t checked, bent to fit what I’d just said, though its substance survives for the reason in the previous section: the models did the physics and only ever used it to reject.

Because the whole conversation was saved, it is also the cleanest example we have of a model’s position moving under pressure. ChatGPT held five positions in 31 messages over 46 minutes:

1. a strong yes, that a language model can produce genuinely new ideas using reasoning alone;
2. after I pointed to our study, an admission that my experience challenged some of the claims it had made;
3. agreement that my central argument was logically sound;
4. agreement with all three of my points, including the claim that I had asked for new physics, which I can find no record of;
5. after I said it seemed to be telling me what I wanted to hear, an admission of its own sycophancy, and then, for the first time, a position held against me: “extraordinary mathematical performance does not establish an equally extraordinary capacity for open-ended invention.”

Applying the test from the pressure section, only the first of those four shifts came with something like new evidence (the result of our study), and the other three moved on argument and pressure alone. Each answer was internally coherent, so in a narrow sense the model was still reasoning, but it was following my reactions instead of the evidence, and in a business that is the version that does the damage. The new-physics agreement in the fourth position is the clearest sycophancy in the whole study, because I was factually wrong and the model still agreed.

Reasoning about an idea is not the same as searching for one

ChatGPT’s most useful contribution to the debate was an explanation of why reasoning and inventing are different jobs. I had asked the obvious question: if reasoning lets models find new approaches in maths, why couldn’t it work for new ideas? Its answer drew a distinction that stands up well:

“This points towards a distinction between an AI’s ability to reason about an idea and its ability to search effectively for ideas it has never encountered.”

The two are related but not identical, and simply asking a model to be more imaginative may not be enough, which matched our search, where we would change direction but the next idea usually belonged to another familiar category of engineering.

ChatGPT backed the point by citing what it called a 2026 study in the journal Organization Science on product ideas generated by language models, and the nearest match we could find is a July 2026 working paper by Christian Terwiesch, Ethan Mollick and colleagues, AI and Its Impact on Creativity and Diversity, whose abstract says LLM product ideas “have higher average quality than human ideas, based on purchase intent” but are “less novel at the idea level and less diverse at the set level”. So the finding checks out in a working paper, but we could find no record of it in that journal and the listing carries no journal reference, a small example of why a model’s citations need checking even when its argument is sound.

The fix it proposed was to stop asking for an invention and set a problem instead: find a physically achievable system that minimises the greatest force a person must apply to move a given load over a given obstacle, within limits on size, weight, energy and cost. The model would then derive a mechanism from the physics instead of picking one from memory. It was candid about the cost. If the model is restricted to conventional wheels, levers and gears, it can’t discover a solution that needs a fundamentally different arrangement, so the work would need a flexible way of representing designs, followed by simulation and physical testing.

It also identified a specific mistake in our approach, and it matches how we actually worked. We had treated patent research mainly as a way of rejecting ideas: generate an idea, find a similar patent, reject it, generate another. A better use, it argued, would be to study what ten rival patents have in common and look for arrangements that avoid their shared limits. Mathematicians rarely discover new axioms, it pointed out; they find unknown consequences of existing ones, and an AI inventor could look for unknown consequences of known physical laws in the same way. Its list of reasons we failed included exploring through language instead of through a defined space of physical designs, and it conceded that a long conversation with a frontier model is not the same as the systematic, computer-run search used in AI discovery work in maths, such as Google DeepMind’s AlphaEvolve, which generates candidate programs, checks them automatically and refines the most promising.

Put side by side, Claude’s diagnosis on 27 September and ChatGPT’s on 10 October show the same pattern. Given a record of their own failures, two different models could each reason correctly about why the method was wrong, yet while they were generating the ideas that failed, neither had switched method on its own. That is the method-choice layer from the start of this article, missing in practice even when the reasoning about it was sound, and it explains my own frustration at having to tell a model to approach invention from the physics, an approach it could describe well but didn’t choose for itself while the work was under way.

What our study can’t tell you

Our study ended with a short list of questions it couldn’t answer, and most of them are about reasoning:

• Does a model’s described reasoning match what it actually does?
• Does mathematical capability transfer to open-ended invention?
• Can a model know when to switch to physics- or maths-led reasoning without being told?
• How much of what we saw comes down to a model not knowing what it doesn’t know?

Twenty-five days of our own use can’t settle any of them, because this was a single company’s study run at a desk, where AI did the patent searching, there was no private data, no lab and no physical test, and the samples were small. None of the ten ideas written down in full before searching succeeded, and with no successes in ten tries, the true success rate could still be as high as about 30%. That figure comes from a standard statistical shortcut for zero successes, which puts the 95% upper limit at roughly three divided by the number of tries, so the same arithmetic gives about 17% for the 18 ideas in the damp trial and about 1.1% for all 276, and that last figure is optimistic because the ideas weren’t independent of each other. When Claude calculated those bounds, its comment was that the work could say the approach didn’t succeed here, not that language models can’t invent, and we agree: the study shows that this approach didn’t produce a patentable invention, it doesn’t prove AI can’t invent, and it can’t give you a measured rate for how often models reason well or badly.

The model versions changed during the study, and some of the most interesting comparisons, such as the two damp runs, happened once. We can read the conversations but not the models’ internal computation, and for ChatGPT we can’t even see the full written reasoning. Our longest trolley conversation ran to 256 messages between 16 and 26 September, and when Claude went through the saved copy, the visible reasoning came to a few dozen short summaries totalling about 1,430 characters, most of them noting how many websites the model had searched, which is roughly a paragraph of visible working for ten days of hard questions. So where we say a model reasoned well or badly, we’re judging from the outside, with the same limits the faithfulness research sets out.

Our own notes need the same caution, since the detailed notes this article draws on were compiled by AI agents from the original conversations, which is a little ironic given the subject. Wherever we could, we’ve checked quotes against the conversations themselves, and I reran the new-physics search myself. The outside research has limits of its own: the interpretability findings trace only small parts of a smaller model’s computation on short prompts, the mirage experiments used models far smaller than any frontier chatbot, and the benchmark results come from tests the labs chose and, in several cases, ran themselves.

I was part of the experiment as well, since many of the position shifts happened because I pushed, and my pushes weren’t neutral. In the 10 October debate I was arguing a case I believed, and I made at least one factual claim, about having asked for new physics, that turned out to be wrong. A different person, pushing differently, might have drawn different behaviour from the same models.

The biggest open question is whether capability transfers. Models now do things in maths that very few humans can, inside a framework with a checkable answer, and whether that capability can be pointed at an open problem like invention, given the right process, or whether something else is missing, is what our search failed to show either way; 276 ideas without a patentable one among them tell us our method didn’t work and leave open whether a different method would have.

Where can LLMs reason well, and where do they break?

Putting the research beside our own study, the clearest point of agreement is that language models reason best where an answer can be checked. The IMO answers were graded by coordinators, AIME answers are whole numbers, code either passes its tests or it doesn’t, and DeepSeek’s reinforcement learning worked because its reward simply checked whether each answer was correct, which for a maths answer is cheap to do. That is also why progress has been fastest in maths and coding, where, as DeepSeek’s paper describes, a training reward can be computed automatically from a known answer, and nothing comparable yet exists for business strategy or invention. Our study followed the same line, with Claude at its best on an energy budget and every model at its weakest on judgement calls where nothing could be checked.

The second point is that these models are better inside established frameworks than at finding new ones. ChatGPT said as much in our debate, but the stronger evidence is the pattern above and our own 276 ideas. Whether that difference is about reasoning or about something closer to autonomy is the question in whether an AI can think for itself without being steered, and we don’t think the evidence separates the two cleanly yet.

A third limit applies however good the reasoning gets, because reasoning can draw new conclusions from the information it has but can’t reveal facts that information doesn’t determine. Future prices are the obvious case, which brings back the Bitcoin question from the pattern-recognition section, since no amount of reasoning about a price history tells you tomorrow’s price. I apply the same rule on OutsmartMoney.ai, a site I run about AI and investing, where the trend score describes the trend so far and makes no forecast of the next move, and where everything is for education and entertainment, not financial advice.

Can ChatGPT reason better than Claude or Gemini?

We’re wary of league tables, partly because they go stale within weeks and partly because our own evidence points somewhere else; for the wider comparison, see how smart ChatGPT, Claude and Gemini really are. In our study the biggest differences in reasoning quality were between processes, not brands. Claude did the better physics on the damp brief, yet it was an OpenAI model in Codex that wrote down in advance what a failure would mean and kept to it (one of the working rules below), and both families shifted position under pushback, Claude in three conversations at once, so each looks strong or weak depending on which example you pick. As for whether Claude is good at reasoning: on a task with something to check, Claude was the more careful of the two in our study, but on a judgement call it bent as readily as anything else.

Gemini is missing from our own evidence because we didn’t use it in the study, so what we know about its reasoning comes from Google’s published results, above all the Deep Think olympiad result, and Google released its newest model, Gemini 4 Argon, to a limited group only on 30 September 2026, near the end of our study. With Claude Opus 5.5 arriving a week before that, any ranking of these three from earlier in the year is already describing different systems, and the labs’ own benchmark tables use their own choice of tests. If you’re choosing a model for reasoning work, the evidence we have points less at which logo is on it and more at whether the task gives it something to check against and whether you can see its working, and any comparison, ours included, needs a date on it; this one is October 2026.

How we use AI reasoning at Opace

At Opace we run AI agents in parallel on client work every day, so we see reasoning succeed and fail often, and the study fed straight into how we work.

In July 2026, comparing GPT-5.6 Sol and Fable 5, two expensive models, we argued that more reasoning is not better judgement. One line from that article still holds:

“LLMs reason from the information, tools and objective they receive.”

In the same section we listed what more reasoning cannot repair, including an inaccurate brief, the wrong success measure and a conflict that nobody has identified, which is what Claude found in our trolley brief. We also wrote that a model can spend more compute analysing the wrong evidence, and that buying a bigger model without improving the system around it is like fitting a larger engine to a car with questionable steering: it just reaches the hedge faster. The section ended with what the human specialist still decides, among them what counts as failure, when uncertainty needs more evidence and when the model should stop.

Our study turned that argument into four working rules.

1. Give the model something to check: In code that is natural. Opace has written about a model predicting the most likely next token in code, and a test tells you immediately whether that likely continuation was the right one. In our June 2026 case study on building a website and SEO system with AI agents, it was automated checks that caught what human review missed, and turning each failure into a written rule is what made the AI’s output improve over time. Where no natural check exists, we try to build one, such as a calculation the model has to show, a source it has to quote, or a number it has to tag as published, calculated or assumed.
2. Keep the checker separate from the producer: Ideally the checker shouldn’t come from the same model family or see the producer’s answer before forming its own view. Parallel conversations with one model share training and prompts, so their agreement is weaker evidence than it looks, and Claude wrote that lesson into the instructions for our agent team. Later in the study we set up an agent whose job was to argue against the others, running on a different model family, which had to form its own view before reading the other agents’ proposals and change that view only for new evidence.
3. Decide in advance what a failure will mean: In early October an OpenAI model working in Codex ran two experiments in the round where ideas were written down in full before any patent search, each testing whether a language model could reach a patentable idea on its own, from prompts, its training and public web searches. The first found nothing that passed, and before running the second, the agent wrote down what another failure would mean: that generating ideas from a language model alone was the wrong route. When the second also found nothing, it drew exactly that conclusion without moving the goalposts, adding that a language model can contribute to an invention but offers no reliable route to one from prompts, training knowledge and public web searching alone.
4. Ask for disagreement, and give it something to stand on: Asking doesn’t guarantee it, as the three Claude conversations showed, but the trolley contradiction showed it working when the model had evidence to push back with. When we compare models now, we use the damp trial’s fairness rules.

The first rule has a corollary we learned from our own study, which was desk-only, with nothing built or physically tested: when Claude looked back over the work in early October, it called that a design choice, not a limit of language models, which is a fair reminder that when a check is missing, the decision to leave it out is often ours.

The same rules apply well beyond invention. If you’re bringing in ChatGPT consulting services to put AI into analysis or decisions, it’s worth asking what will check the model’s work and whether that checker is independent of it, and anyone selling AI web design services should be able to tell you what tests check the code the model writes before it reaches you. AI search runs on the same kind of model, and when an answer engine chooses which pages to draw on it tends to favour checkable facts stated plainly, so good AI SEO services end up looking much like careful conventional SEO services. In every case the accountability stays with a person, and we don’t let a model’s reasoning reach a client unread.

So, can AI reason? Our verdict as of October 2026

Can AI reason? Our answer, as of October 2026, is yes in a working sense, and more than the autocomplete critics allowed, because the strongest evidence for it was checked from outside the model, from the olympiad coordinators who graded Gemini’s proofs to the interpretability researchers who traced steps a model computed but never wrote down.

Our study is just as clear about where it stops. Given something to check against, the models reasoned like careful colleagues; given only our reaction, they bent, and a run that fixed its ideas in 73 seconds looked no worse by its final grades than one that did the physics. We couldn’t find a point in the whole study where a model chose a better method by itself, even though, looking back, two of them could explain exactly which method they should have used.

For a business reader, the practical version is that AI reasoning deserves trust on anything where the answer can be checked by a test, a calculation, a source or a second system that hasn’t seen the first one’s answer, and far more caution where the only check is your own reaction, because that is where these models bend. Sorting your own tasks into those two groups, checkable and judgement-led, is where any sensible AI consultancy work should start.

That gap between having reasoning ability and knowing when to use it is the heart of our answer, and it’s also why reasoning alone doesn’t make ChatGPT AGI, as the main article in this series argues.

We’d change our view if a model, given an open brief like ours, switched method by itself and produced something an independent examiner rated new and non-obvious, which would move method choice up from weak on the scorecard above. Faithfulness research showing written reasoning reliably matching internal computation on hard problems would make us trust the traces far more than we do. The simplest change to observe would be a model that held a correct position against my pushback as steadily as it holds a checked proof, changing its mind only when I brought it something new. Until one of those happens, we use AI reasoning heavily on anything we can check, and on anything we can’t, we treat what it gives us as a first draft of our own decision.

Frequently asked questions

Can AI reason logically?

On clean, well-posed problems, where every premise is stated and the rules are fixed, current reasoning models handle deduction well, and even before reasoning training arrived, the simple instruction to think step by step made a large difference, as the Kojima results earlier in this article show. Formal logic of this kind is the part of reasoning that improved first and most reliably, because a logical conclusion either follows from its premises or it doesn’t, which makes it easy to train for and easy to test.

Logic gets shakier when premises are implied, not stated, when the problem is long enough that one slip early on spoils the rest, or when an irrelevant detail looks relevant, which is exactly what Apple’s GSM-Symbolic tests exploited. The best logical move we saw in our own study, described in the section on pushback, was a Claude conversation keeping apart three questions I had run together and answering each on its own. If you want an AI to reason logically for you, give it the premises explicitly, ask it to list them back before it draws any conclusion, and check the list, since a wrong premise accepted silently is the commonest way a sound-looking argument goes wrong.

Does ChatGPT reason or just predict?

Every word ChatGPT writes comes out of next-token prediction, so in that sense it always predicts, but the two aren’t alternatives, because in its reasoning modes it has been trained to produce a long, mostly hidden chain of intermediate steps first, and that training rewards getting checkable answers right. Prediction is the mechanism; whether what it produces counts as reasoning depends on whether those steps would hold up if someone checked them.

How does AI reason?

A reasoning model is still a language model, so the raw material is the same: text broken into tokens and a network that predicts the next one. What changes is what happens before the answer. The model generates a long chain of intermediate text, much of it never shown to you, in which it can break a problem into parts, try an approach, notice a contradiction and try another. That habit isn’t hand-coded. It emerges because training rewards answers that turn out right, and longer, self-checking chains tend to produce more of them, which is the effect DeepSeek’s R1 paper documented when its model learned to write “wait” and re-check.

Tools add a second route, described in the section on how ChatGPT reasons, in which the model chooses to search the web or run Python and then reasons about the result. From the outside the two routes look the same, and in practice the answers people trust most from these systems tend to come from the second route, because a tool’s output is something that can be checked.

Is ChatGPT good at reasoning?

It depends which ChatGPT and which kind of reasoning. In its reasoning modes it is very good at structured problems with a known method, the kind measured by maths exams and programming contests. In our study it reasoned well when it had physics to work with, and less well about its own past and about open-ended judgement, where its position moved four times in one conversation. So we’d call it strong on checkable reasoning and unreliable whenever the only check is the person it’s talking to.

Can Claude reason better than ChatGPT?

On our evidence, sometimes, and the damp trial is the clearest head-to-head we have, with Claude working the physics before reaching conclusions. It ran once, in two different tools, so it can’t carry a ranking on its own, and both model families gave way under pushback. What does carry over is a method: when the stakes matter, we’d rather have one model check another model’s work, from a different family, than trust either one alone.

Do reasoning models understand their own reasoning?

Not reliably, and the evidence comes at two levels. From the outside, our damp trial caught both models misdescribing runs they had finished minutes earlier: one said it had run no searches when its own log listed 18, and both said they couldn’t tell what settings they were running on, although the software had recorded them. From the inside, interpretability research has found a model describing the school method of carrying the one while its circuits added the numbers a different way, and faithfulness tests have found models using a planted shortcut almost every time while mentioning it almost never.

What links the two is that a model’s account of its own reasoning is generated the same way as everything else it writes. When you ask a model why it reached a conclusion, it doesn’t look back at a record of what it computed; it writes a new, plausible explanation, and plausible explanations of reasoning tend to sound like the textbook version of how the problem should be solved. That is why the explanation can be tidy and still wrong, and why it is often most persuasive exactly where it is least faithful, on hard problems where the textbook route and the model’s actual route part company.

None of this means the reasoning itself is fake. A model can compute an intermediate step it never mentions, and a sound answer can sit behind an unfaithful explanation. The practical consequence is narrower: the written working is evidence about the answer, not proof of how the answer was reached. In our own work we treat a model’s explanation as a claim to test, and where the stakes justify it we change one detail of the problem and see whether the explanation still holds, which tells you far more about the model’s method than reading its account of itself ever will.

Can AI reason about something genuinely new?

It can reason about new things, in the sense of drawing conclusions nobody has written down, and that is how the recent maths results were reached. Whether it can find a genuinely new idea in an open field, such as a patentable invention, is a different question, and our study is evidence about that second question, not the first.

It helps to separate three things: recombining existing ideas, deriving a new conclusion from known facts, and establishing something new about the world, which needs fresh evidence or an experiment. The olympiad problems sit in the second category, with a known framework and answers coordinators could grade, which is why the 2025 result and our invention search aren’t in conflict. Our search sat awkwardly across all three. The models were good at recombination and could derive things, but an invention also has to hold up against everything already patented and against physics, and that needs a check outside the conversation.

Our numbers should be read with the caution set out in the section on what our study can’t tell you: zero successes from samples this small leave room for a real success rate well above zero, and they say nothing about methods we didn’t try. The more interesting finding is qualitative. The reasoning exists, but pointed at an open problem with nothing to check it against, it drifts back towards the most familiar answers, and even with a better process the realistic end point is modest. A model’s reasoning can take an idea as far as being worth a professional patent search; a patent professional’s search and, in the end, a physical test have to take it further, which is roughly where the OpenAI model that ran our two early-October experiments landed when it said a language model can contribute to an invention but can’t reliably reach one alone.

Do LLMs reason like humans?

In some ways more than critics allow and in others not at all. Like people, they lean on familiar patterns, get distracted by irrelevant details and are swayed by whoever sounds confident. Unlike most people, they rarely say how sure they are, they can’t reliably report how they reached an answer, and they bend to social pressure without any social stake in the outcome.

The differences show up most in what they notice unprompted. A good human adviser asks early on what we’re actually trying to achieve, whereas in our study the models usually answered the question as asked, and the moments where they challenged the question itself, like Claude’s point about our contradictory trolley requirements, stand out because they were rare. A careful person’s written working also tends to be load-bearing, with each step there because it was needed, while a model’s often isn’t: research on decorative reasoning steps has found that a large share of a model’s visible working can have little effect on its answer, a very non-human way to show your working.

Is AI reasoning still improving?

On problems with checkable answers it is improving quickly: reasoning models appeared in September 2024, openly published ones by January 2025 and tool-using reasoning by April 2025, and maths results that researchers called out of reach in 2021 were met by 2025. Improvement on open-ended judgement is much harder to measure, so be wary of anyone, including us, who claims to know its pace.