How smart is AI? Most people who ask want a single number, an IQ or a benchmark score they can hold up against a person, and in 2026 the question has become hard to avoid. In July 2025 an advanced version of Google’s Gemini reached gold-medal standard at the International Mathematical Olympiad. In the first week of October 2026 OpenAI published a catalogue of more than 700 mathematical manuscripts, grouped into 372 related sets, produced by one of its internal models, and some of them have been checked by Lean, a program that verifies proofs step by step (the catalogue’s own notes warn that some unformalised results “could have issues”). Since OpenAI launched its o1 model in 2024, headlines have called systems like these ‘PhD-level’, a claim we trace back to its source further down. Meanwhile businesses, ours included, are deciding how much real work to hand over to them, and that is where the question stops being academic. It is the everyday end of our verdict on whether today’s AI is AGI.
Our own way of measuring it, between 16 September and 10 October 2026, was to give ChatGPT, then Claude and OpenAI’s Codex, and finally a team of up to 25 AI agents one job that recall alone couldn’t do: find an idea that was genuinely new, patentable, cheap to test and commercially worthwhile. They worked faster and more broadly than any person we have worked with, and the 276 distinct ideas they produced, checked against 2,161 patent and application numbers, included none strong enough to justify a patent application. Near the end the agents also built the homepage of a small marketplace website, a personal hobby project of mine that grew out of the study’s conclusion, and an OpenAI model looked at it and called it “bold, credible, clean and fairly premium”, holding to that view when we pushed back, while a person seeing it for the first time thought the business had something to do with parcel delivery.
So how smart is AI? Very smart wherever an answer can be checked: it writes code that passes its tests and solves olympiad problems at gold-medal standard. Where nothing outside the model can say whether it is right, it is shakier, and in our study it marked its own work too kindly and never found the genuinely new idea we were looking for.
Opace’s AI work started with GPT-3 experiments for SEO and content in 2022, which is where our AI SEO agency work began, and we now run teams of AI agents on client and internal work every day, so where the line falls between work a machine can check and work it can’t matters to us commercially. The most useful way we have found to answer the question is to narrow it each time (smart at what, compared with whom, and checked by what?), and the rest of this article works through it in that order: what people have meant by intelligence and how they have tried to measure it, where AI stands in 2026, what our own 25-day study added, and what we now do differently because of it.
Which ‘AI’ are we measuring?
‘AI’ is a family, and the answer to how capable it is changes with the member you pick. Chess and Go engines are superhuman at one game and useless outside it. Vision systems, recommender engines and robots each solve their own narrow problem. Google DeepMind’s AlphaFold predicts protein structures. Then there are discovery systems such as FunSearch and AlphaEvolve, which pair a language model with an automatic evaluator that scores every candidate, run thousands of attempts and keep only what the evaluator certifies. The FunSearch paper in Nature is frank about why the evaluator is there: language models “sometimes suffer from confabulations”, so the evaluator “guards against confabulations and incorrect ideas”. The intelligence on show is the combination, and the evaluator is the part that makes it trustworthy. When people ask whether large language models are intelligent, though, they usually mean the chat assistants: ChatGPT, Claude and Gemini, used in a browser or wired into agents.
A large language model is trained to predict the next token of text (a word or a fragment of one) across a vast corpus, which is a simplification of how a chatbot is trained to predict text but a useful one. Training rewards what is frequent and plausible in that distribution, whether or not it is true. That is why a model can be fluent, well informed and wrong in the same sentence, and why hallucinations come from the method itself and won’t be removed by any one release. Scale and later training stages add real problem-solving on top, so ‘it only predicts words’ undersells it, but the prediction objective explains a great deal of the unevenness that follows. And when a headline says that AI did something remarkable, the first question worth asking is what was wrapped around the model at the time: a chat window, or a system of tools, scorers and instructions.
Is AI intelligent? What ‘smart’ means for people and for software
Psychologists have never agreed what intelligence is. Shane Legg and Marcus Hutter collected what they called “the 70-odd definitions” in 2007 and boiled them down to one working line: intelligence measures an agent’s ability to achieve goals in a wide range of environments. Most of the definitions they gathered mix two ideas, having skills and being able to acquire new ones. François Chollet’s paper On the Measure of Intelligence separates them. He defines intelligence as skill-acquisition efficiency and points out that unlimited prior knowledge or training data lets you buy arbitrary levels of skill, which hides how well the system generalises. A model trained on a large part of the written internet can look highly skilled without showing much of that efficiency.
Definitions like Chollet’s come from computer science, which is close to my own training (maths, physics and software engineering, before I founded Opace in 2008), but the subjects that have always interested me more are about people: psychology, belief and how minds work. They point to a second, more human way into the question, because the way people judge whether something is clever is mostly social and leans heavily on how it talks, and that habit is the oldest trap in this field. It goes back to ELIZA, Joseph Weizenbaum’s 1966 pattern-matching program, which persuaded some users it understood them. In a 2024 study by Cameron Jones and Benjamin Bergen, people held five-minute text conversations with either a person or an AI and then guessed which, and GPT-4 was judged human 54% of the time, against 22% for ELIZA and 67% for real humans.
The authors concluded that people could not reliably tell GPT-4 from a person in that setting, and what we take from it is about us as much as the model, since a short conversation rewards style, social warmth and confidence. Mrinank Sharma and colleagues at Anthropic found that five state-of-the-art assistants consistently showed sycophancy, and that in human preference data a response matching the user’s views was more likely to be preferred, which means a model trained on that data is pulled towards agreeing with whoever it is talking to (and, to most listeners, agreement sounds like good sense). They also found that “both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones” some of the time.
The definition we work with at Opace is a commercial one, and it came out of a July 2026 comparison of two premium models, OpenAI’s GPT-5.6 Sol and Anthropic’s Claude Fable 5, in which we set benchmark scores against business results and concluded that a business pays for the result it gets, so a model’s standing on a leaderboard matters only as far as it predicts that result. By that test a model is smart for your purposes if it produces a result you would accept, on the job you actually have, with checks you can live with, and that is still the test we would suggest to anyone deciding whether to trust a tool with work.
Smart at what, and compared with whom?
A chess engine beats every person alive and cannot make a cup of tea, which is a modern form of what the roboticist Hans Moravec described in his 1988 book Mind Children: what is hard for people (formal reasoning, chess) can be easy for machines, while what a toddler does without thinking (perception, moving through a room) is hard. Moravec’s paradox, as it became known, still describes 2026 surprisingly well, as the clock-reading result later in this article shows. With chess the point is obvious, and with a chat model it is much less so, because the chat model talks about tea, chess and tax law in the same fluent voice. The best experiment we know on this is Fabrizio Dell’Acqua and colleagues’ study of 758 Boston Consulting Group consultants, published as a Harvard Business School working paper in September 2023.
Consultants were given 18 realistic consulting tasks that sat inside GPT-4’s capability. With the model they completed 12.2% more tasks on average, finished 25.1% more quickly and produced results judged more than 40% higher in quality than a control group. A separate task was deliberately chosen to sit just outside what the model could do, while looking much like the others. There, consultants using AI were 19 percentage points less likely to produce a correct solution.
The authors called the boundary between those tasks a jagged technological frontier, and the word matters. The edge runs in and out between tasks that a person would rate as equally hard, instead of along a smooth line that each better model pushes outwards evenly, and the person using the tool usually cannot see where it runs until something goes wrong. Compared with an average consultant on a well-specified analysis, GPT-4 was a large help, and on a subtly different problem the same consultant was worse off with it, partly because its wrong answer looked just as confident as its right ones (a pattern our own study ran into again and again).
Who gained most is the other half of the answer to ‘compared with whom?’. Inside the frontier, consultants below the average performance threshold improved their scores by 43% when they used GPT-4, and those above it by 17%, and the authors write that consultants “across the skills distribution benefited significantly”. That levelling effect explains a lot of the everyday sense that AI makes everyone look smarter, because it lifts people most where they were struggling and where the task is well defined, while the gains at the top are smaller and, outside the frontier, the same consultants did worse. The people who gained most are also the least likely to spot when an answer has crossed that line.
The authors also saw two ways of working that succeeded. Some consultants behaved as what they call Centaurs, dividing the work and deciding which parts to hand to the AI and which to keep; others behaved as Cyborgs, weaving the model through every step. Both styles kept a person deciding where the frontier ran on each task, which is close to how we work with AI agents ourselves.
How is AI intelligence measured, and why do benchmarks mislead?
The consultant study is unusual because it measured AI on real work, and most of what gets reported as AI intelligence, including the ‘PhD-level’ headlines, is something narrower: a benchmark score, from a fixed set of questions with known answers run under stated conditions. That is a reasonable way to compare models and a poor way to describe a mind, for three reasons worth looking for every time.
The first reason is saturation. OpenAI’s own o1 announcement, Learning to reason with LLMs, published on 12 September 2024, says frontier models do so well on the MATH and GSM8K sets of maths problems “that these benchmarks are no longer effective at differentiating models”, and once nearly every model passes a test, its score says little about which one to trust; the 2026 section below shows the same thing happening to a coding benchmark within a single year.
The second is that a benchmark rewards skill at its own format, and the same announcement shows how far a format score can travel. OpenAI reported o1 surpassing recruited PhD experts on GPQA Diamond, a set of hard multiple-choice science questions written so that they can’t simply be looked up, and in the same paragraph warned that this does “not imply that o1 is more capable than a PhD in all respects”.
That caveat is one we missed ourselves in September 2024, when Opace published a long piece on o1 that repeated “PhD-level benchmark performance” and ended with the line “AI just got a whole lot smarter”. Reading it now, it is easy to see how the headline number travelled and the caveat didn’t. The piece is still online, and we’ve corrected the PhD line with an editor’s note but left the rest as written, as a reminder of the ‘PhD-level’ benchmark claims made for o1 and of how persuasive they were, to us as much as anyone.
The third reason, and the one most likely to catch out a business, is that a benchmark can be passed for the wrong reasons, through memory or through luck with the wording, which is what the retests below were designed to expose.
Contamination, rewording and the bar exam
If a model has seen a benchmark’s questions, or close variants of them, during training, its score partly measures memory. Hugh Zhang and colleagues tested this directly by writing GSM1k, a thousand new grade-school maths problems matched in style and difficulty to the widely used GSM8K set. Some model families dropped by up to 8% on the fresh questions and showed signs of memorisation, while many frontier models showed minimal signs of overfitting. Iman Mirzadeh and colleagues at Apple went further with GSM-Symbolic, which changes only the numbers in known questions or adds a clause that sounds relevant but isn’t. Performance fell when only the numbers changed, and the extra clause caused drops of up to 65%.
The bar exam shows a score being right and its headline wrong at the same time. OpenAI’s GPT-4 Technical Report put its simulated bar exam result “around the top 10% of test takers”, but the comparison group was a February sitting dominated by people retaking after failing in July, and Eric Martínez’s re-evaluation in Artificial Intelligence and Law put GPT-4 at about the 62nd percentile against first-time takers (the 42nd on essays) and at about the 48th overall and the 15th on essays against those who passed the exam, licensed lawyers and those awaiting a licence. He didn’t dismiss the score itself. He replicated the multiple-choice result, raised questions about how the essay sections had been graded, and found that changing the model’s randomness setting made no significant difference, while showing it a few worked examples with step-by-step reasoning did, which says something uncomfortable about how much a benchmark score depends on the prompt as well as the model.
Rewording catches out social reasoning as well as arithmetic. In tests of theory of mind, the ability to work out what someone else knows or believes, Natalie Shapira and colleagues found that models’ apparent theory-of-mind ability relied on shallow heuristics and broke down once the stories were varied adversarially. Taken together, the retests explain why the headline scores of 2023 and 2024 promised more than the models could deliver, and they are why we treat any single number for AI intelligence with suspicion, including the one people ask about most.
What is AI’s IQ?
That number is an IQ score, and we haven’t found a primary source that measures a single IQ for a language model in a way that holds up against the retests above. IQ tests were designed for people, and they rest on assumptions about the test-taker that don’t hold for a system trained on much of the written internet.
The deepest of those assumptions goes back more than a century. In 1904 the psychologist Charles Spearman noticed that children who did well on one kind of mental test tended to do well on others, and in a paper he titled “General Intelligence,” Objectively Determined and Measured he called the shared factor behind those correlations general intelligence, or g. An IQ score is useful for people largely because of that pattern: a good result on a few puzzles predicts a good result on many other tasks. For a language model the pattern breaks, as the consultant study’s jagged frontier already suggests, so a score on one set of puzzles predicts much less.
| What an IQ test assumes | Why it fails for a language model | Evidence |
|---|---|---|
| The questions are new to the test-taker | Benchmark-style items, and close variants of them, may be in the training data | The GSM1k retest above, where fresh look-alike questions exposed memorisation in some model families |
| A right answer shows an ability that transfers | Answers can break when the numbers change or an irrelevant detail is added | The GSM-Symbolic retest above, where one irrelevant clause caused large drops |
| Scores on different tasks move together | Model skills are jagged; a top score on one task says little about the next | The Stanford AI Index 2026 finds coding scores near the ceiling while agents still fail many ordinary computer tasks (figures in the 2026 section below) |
| What is hard for people is hard for the test-taker | Puzzles easy for people can be hard for models, and the reverse | Everyday perception such as reading an analogue clock (AI Index 2026, below), and the ARC-AGI-2 puzzles described next |
The last row matters most, because a clock face is something most children can read, and ARC-AGI-2 turns that kind of gap into a deliberate test. It was calibrated with more than 400 members of the public in San Diego in early 2025, every task was solved by at least two of them in two attempts or fewer, and the tasks were chosen to stress-test the best AI reasoning systems. Its page reports the human results and gives no model scores, so we don’t quote an AI figure for it here.
Classic IQ formats that people find hard have gone the other way for some time. Back in the GPT-3 generation, Taylor Webb, Keith Holyoak and Hongjing Lu found the model was matching or even surpassing human capabilities in most settings of their analogy tasks, including a matrix task modelled on Raven’s Progressive Matrices, the format most people picture when they think of an IQ test.
Put together, a model can beat people on puzzles people find hard and fail ones a child finds easy, and an IQ score, which measures skill on the day, can’t separate a model that has learned something from one that has seen thousands of near-identical puzzles during training. By Chollet’s definition the number therefore says little about how efficiently a model learns something new, and when we are asked what AI’s IQ is, our answer is that the question is a category error: it is more useful to know which tasks a model can do, under which conditions, and how the answers were checked.
Myth or truth: IQ, benchmarks and ‘PhD-level’ claims
With the retests and the IQ question in mind, the same few claims about AI intelligence that keep turning up in videos and posts are easier to judge, so on 10 October 2026 we traced five of them from the headline back to the original source. They are listed below in the order the claims first appeared. We also looked for the source of a widely shared claim that AI has an IQ of 155, found nothing we could trace, and left it out.
| Claim | Circulating example | Primary source | What the source shows | Verdict |
|---|---|---|---|---|
| GPT-4 passed the bar exam in the top 10% | LawSites, 14 Mar 2023 | OpenAI, GPT-4 Technical Report; Martínez, 2024 | OpenAI's simulated exam did score "around the top 10% of test takers", but against a February sitting dominated by retakers. Against first-time takers, and against those who passed, Martínez's re-evaluation (described above) placed it far lower, especially on essays. | Partly true |
| AI discovered new maths on its own | SiliconANGLE, 14 Dec 2023 | Romera-Paredes and colleagues, 2023 (the FunSearch paper) | FunSearch did find cap sets larger than the best previously known, with the automatic evaluator described at the start of this article and on a problem and program skeleton that researchers set up. A genuine result, from a system that people framed and a scorer checked. | Misleading |
| ChatGPT passed the Turing test | BGR, 14 Jun 2024 | Jones and Bergen, 2024 | In five-minute text chats, GPT-4 was judged human more often than chance and less often than real people (figures above). Five minutes of chat tests conversational style, and real people still did better. | Partly true |
| OpenAI's o1 model is PhD-level | Pure AI, 2 Dec 2024 | OpenAI, Learning to reason with LLMs, 12 Sep 2024 | o1 beat recruited PhD experts on GPQA Diamond, and OpenAI itself said that does not make it more capable than a PhD in all respects: a PhD also designs experiments, spots bad questions and works on problems nobody has answered. | Misleading |
| AI won gold at the Maths Olympiad | Maginative, 21 Jul 2025 | Google DeepMind, 21 Jul 2025 | Gold-medal standard, certified by IMO coordinators who marked the answers but did not validate the system itself. No medal was awarded, because the AI wasn't a contestant; the details are in the 2026 section below. | Partly true |
How smart is AI in 2026? The jagged frontier
The consultant study’s phrase has turned out to be the best short description of AI in 2026, and Stanford’s AI Index Report 2026 now uses it too, calling the pattern the jagged frontier of AI. Its figures, like any annual report’s in this field, are already a snapshot of the recent past, but the shape has held in everything we have seen since.
Where AI is now superhuman
The clearest case is competition mathematics, where on 21 July 2025 Google DeepMind announced that an advanced version of Gemini with Deep Think had solved five of the six problems at the International Mathematical Olympiad perfectly, scoring 35 out of 42, a gold-medal score. It worked end to end in natural language and produced its proofs within the 4.5-hour competition limit, whereas a year earlier, at IMO 2024, DeepMind’s systems had needed experts to translate the problems into a formal language first and took two to three days of computation. The results were “officially graded and certified by IMO coordinators” using the same criteria as student scripts, although DeepMind also notes that this review “does not extend to validating our system, processes, or underlying model”.
That is an exam with a marking scheme, which is exactly the sort of task where models now shine, and the companion piece on how far AI maths has come since 2022 tells the longer story. Software is the second case. The AI Index reports that on SWE-bench Verified, a benchmark of real code fixes that must pass the project’s own tests, performance rose from 60% to near 100% in a single year, which is also why that benchmark no longer tells the leading models apart.
Discovery systems such as FunSearch and AlphaEvolve, described at the start of this article, belong on this side of the line as well. FunSearch found a cap set of 512 vectors in eight dimensions, larger than any previously known, and AlphaEvolve found a way to multiply 4×4 complex-valued matrices with 48 scalar multiplications and, according to DeepMind, a scheduling heuristic that recovers on average 0.7% of Google’s worldwide compute resources. Results like these cost a great deal of compute, which headlines rarely mention: FunSearch’s experiments typically used 15 samplers and 150 CPU evaluators running in parallel, on inexpensive CPU servers for the evaluators and accelerators for the language model. So smart, in 2026, is often a model plus a large amount of paid search, sampling and checking, and that affects what a business can afford to repeat, as our own study found when it came to count the cost.
What these cases share is a referee outside the model: an olympiad has markers, code has tests and a cap set either has the property or it doesn’t. Because the model can try many times and keep only what the referee accepts, the score reflects the best of many attempts, each one checked by markers, tests or mathematics that pay no attention to how confident the answer sounds (which, as the next section shows, is what is missing elsewhere).
Where AI is surprisingly weak
Take away the guarantee that an answer exists and the picture changes. The clearest test so far, Epoch AI’s FrontierMath Erdős benchmark, published on 1 September 2026, set five leading models 68 problems that mathematicians have left open, and the best of them, a pre-release version of GPT-6 Astra (the OpenAI model launched two days later), solved 2, about 3%. Every other model in that test scored zero. The companion article on maths, linked above, covers how the benchmark was run; olympiad papers are written so that every problem has a solution, and on problems where nobody yet knows whether one exists, the models tested have so far mostly come up empty.
The same AI Index page that reports the coding gains adds that the top model reads analogue clocks correctly only 50.1% of the time. Agents improved on OSWorld, a benchmark of real computer tasks, from 12% to about 66% success, which is a big leap and still leaves about a third of ordinary tasks failed. Our guide to what GPT-5 is recorded the day GPT-4o got a simple sum wrong, and slips like that, at the easy end, are what make people stop trusting a tool even when its benchmark scores are excellent.
Why the edges are so jagged
Our best explanation for the pattern is checkability. Where success can be verified automatically (a proof checks, code passes its tests, a link opens, a patent number resolves) a model can be trained, prompted, sampled many times and filtered until it is right, and it gets better quickly. When nothing automatic says whether the answer is good (is this idea new, does this page feel trustworthy, will this market move), the model has only its training distribution to lean on, and it leans towards what is common and plausible.
Forecasting is the extreme case, because a forecast can’t be checked until it is too late to matter. Models are very good at finding patterns in market data and explaining them, and knowing what happens next is another matter altogether, which is why OutsmartMoney.ai, a hobby site of mine, uses AI to explain market moves, where the explanation can be checked against the data, and doesn’t ask it to forecast prices (nothing here is financial advice).
A second explanation is coverage. The internet is full of worked maths and code and much thinner on clock faces photographed at odd angles, so perception tasks that seem trivial to us are underrepresented. A third is that rewording a problem moves it away from what was memorised, which the benchmark retests described earlier measure directly, and which is the territory of whether AI can reason when the question changes.
The fairest map we know is the one in Google DeepMind’s Levels of AGI paper by Meredith Ringel Morris and colleagues, which rates systems on performance and generality separately.
In their table, AlphaFold, AlphaZero and Stockfish are Superhuman Narrow AI, while ChatGPT, Bard, Llama 2 and Gemini, as of September 2023, are Emerging AGI, competent on some tasks such as short essays and simple coding and emerging on most others, with every general level above Emerging marked as not yet achieved. Models have improved a great deal since. On many well-specified tasks today’s best models would clear the paper’s Competent bar (at least the 50th percentile of skilled adults), and on a few scored ones, such as olympiad maths, they sit well above it, but Competent AGI means that standard across a wide range of tasks, including learning new skills, and we haven’t seen a 2026 model shown to reach it. On the authors’ own scale, then, we would place this year’s frontier models above Emerging and still short of Competent AGI. The grid’s main point holds: high performance on a narrow, scored task and breadth across many unscored ones are different axes, and a lot of the argument about how intelligent AI is comes from people looking at different axes without saying so.
What we found when we pushed AI for 25 days
Our own study started with a search I made in mid-September 2026 for a family member of mine, who has mobility issues, values the freedom to go shopping and struggles with today’s trolleys. I was looking for a power-assisted trolley, found only expensive powered walkers, and began to wonder whether AI could invent something genuinely new.
That trolley search set the study going, but the question it raised was whether AI could invent at all, so the tools weren’t held to one product, and the ideas they went on to produce ranged well beyond shopping, from controls fitted to existing home appliances to checks on medication packs and ways of tracing damp and mould in homes.
That turned out to be a demanding test of how smart AI is, because answering a question well can draw on what is already in a model’s training data and on the web. Inventing something useful, and defending the claim that it is new, means going beyond both and then holding up when someone searches for the same idea elsewhere. It mixes the things AI does well, such as searching, reading and summarising at speed, with the things the jagged frontier above suggests it finds hardest: judging quality, noticing what is missing and producing something that isn’t a rearrangement of what already exists.
The study ran for 25 days, from 16 September to 10 October 2026, in three stages. It began with ChatGPT, which on 16 and 26 September proposed about a dozen mechanisms for a better trolley in around 40 minutes, spending a few seconds on each, and saw each one fall to a quick patent search, a practical objection or the physics. From 27 September two AI coding agents ran side by side and checked each other’s work. One was Claude, used through Claude Code, Anthropic’s tool that lets it read and write files, search the web and direct helper agents; the other was Codex, OpenAI’s equivalent. Each conversation with one of these tools was a separate working thread with its own instructions and files, and often several ran at once. From 2 October a team of up to 25 AI agents, each with its own role, took over. Every idea had to be genuinely new, patentable, cheap to test and commercially worthwhile, every idea was checked against existing patents and products, and all of them went into one list with the evidence and verdict behind each one. In early October Codex wrote a report on what the study had shown, which Claude edited, and on 10 October we put the underlying question, whether a language model can create new knowledge, to ChatGPT in a 46-minute debate, in which it began by saying yes and conceded our objections one by one.
That set-up is why the difference between a chat window and a system, described at the start of this article, mattered so much in our study. Over one day Claude coordinated 36 helper agents, each with its own evidence rules, a limit on how many searches it could run and a fixed piece of work to hand back, while a single Codex conversation covered about 40 problem areas in 45 minutes. Neither was a chat. Each was a model working inside a set of tools, files and instructions, and much of what we learned about how smart these tools are came from seeing how much those surroundings, more than the model itself, decided the result.
The last stage also gave the agents something practical to build. The study’s own conclusion, that AI still needs people to supply what it can’t, became the starting point for the small marketplace website mentioned at the start, and the agent team was asked to suggest names for it and build its homepage. That turned out to be useful evidence in its own right, because a name or a homepage is something a person can judge at a glance, which a patent claim rarely is.
We set the bar deliberately high, so that to be judged strong enough to justify a patent application, an idea needed a clear gap that could be claimed after a full search for earlier patents that made it either not new or obvious, plus a real problem, a cheap way to test it and a credible market worth millions of pounds. Ideas that cleared part of that were promising but unproven. Ideas that existing patents, physics or the business case knocked out were ruled out. The bar made the study a stress test of two kinds of task: ones that are easy to check, such as finding a patent, opening a link or counting records, and ones that aren’t, such as judging whether an idea is new or whether a page looks trustworthy.
The headline numbers were these:
- 276 distinct ideas, gathered into one list on 27 September from the ChatGPT, Claude and Codex conversations, after duplicates were merged;
- 2,161 patent and application numbers checked;
- none judged strong enough to justify a patent application;
- 69 promising but unproven, and 207 ruled out.
Most of the ideas were new in their exact wording. They failed on whether they were inventive, whether the physics worked, how easily a competitor could design around them, or whether they could make money. Later work came on top of that list: on 2 October a further round wrote ten ideas down in full before any patent searching, so the search couldn’t reshape them, and all ten were ruled out, after which the agent team’s attention turned to the website.
Speed and breadth no person could match
The scale of these tools showed most clearly on 27 September, when AI produced 274 of the 276 ideas in about five hours, with roughly ten Claude and Codex conversations working at once, many of them starting from problems we had raised. The same day Claude went back through the earlier ChatGPT conversation, picked out the 49 replies that made factual claims and split them between four fact-checking agents, a job that would have taken a person most of a day just to read carefully. That breadth was bought with a lot of paid computing, too: the helper agents behind a single Claude conversation ran about 1,500 web searches, page fetches and file reads between them that afternoon.
By then the ideas were spread across 87 files, so Claude gave the job of gathering every idea, and every check made on it, into one list to 14 helper agents under strict rules: copy numbers exactly, add nothing from memory, and leave a field empty if the source says nothing. They returned 302 records in about 40 minutes, which became 276 distinct ideas once 26 duplicates were merged. Claude distrusted its own first count of patent numbers, which looked too high, and sampled it for false matches before accepting any of it; once references that weren’t patents had been separated out, the count went from 2,211 to 2,161 patent and application numbers, plus 566 references to products and services.
Codex then audited the finished list, and the part that impressed us most was how candid it was about the limits of its own check. It confirmed that all 276 ideas matched their source records, but it declined to certify that nothing at all was missing, because part of the project had been left outside the scope it was given, and it said so plainly instead of covering it over.
No person on our team could have produced that list in an afternoon, but looking back at it, what stood out was where the care had come from. The extraction agents were careful because their rules told them to be (copy exactly, add nothing, leave gaps empty), and Claude’s doubt about its own patent count and Codex’s refusal to certify more than it had checked both came in jobs framed as checking, with the scope written down in advance by us. For the question of how smart AI is, we think that matters more than the speed, because the raw ability to read, sort and count was extraordinary while the rules that made the result trustworthy came from people.
How fast and deep can AI work?
Speed was only half the picture, because depth and speed traded against each other throughout the study. At the deep end, Claude sent seven agents to search for earlier patents on ten candidate ideas, each asked for 25 to 40 searches, with every feature searched on its own and then in combination. Their instructions asked each agent to map what already existed and leave verdicts to others, and they banned any claim that something was patented unless the agent gave the patent’s full details, a reminder that much of what makes a model’s work look deep is written into its brief. Claude also raised a limit that no amount of searching could fix: patent applications usually stay unpublished for about 18 months after filing, so nobody can prove from public records alone that an idea is unique.
The timings put numbers on the trade-off, and on the result, since the deeper reviews ruled out more ideas with better evidence and still found none strong enough to patent:
- ChatGPT spent 4 to 12 seconds on each line of inquiry when exploring ideas.
- One Codex conversation made about 204 quick searches of patent summaries across roughly 40 areas in 45 minutes, about 13 seconds a search, and kept no notes of what it found.
- An agent set up to argue against the others, briefed without knowing who wrote what, took 7 to 14 minutes per review, against our request for answers in under a minute.
At its deepest, Claude did the kind of reading that no person on a deadline would bother with, searching the full text of two long patent descriptions, each many thousands of words, for specific terms whose presence would have mattered to one idea (it found none of them in the first). It also overruled one of its own agents on physics. The agent’s favoured approach relied on a low-resolution sensor picking out detail from four metres away, and Claude worked out, without being asked, that each pixel would cover about 27 mm, so a whole face would be only five or six pixels across, which ended the idea (correctly, as far as we can tell) and is a good example of what these tools do well when the numbers are known and the calculation is quick.
Depth also showed up as a willingness to be corrected. In another Claude conversation, the model declared one approach the strongest before its helper agents had returned any evidence, and the evidence overturned it within the hour. The agents then corrected several of its assumptions and four of its reasons for rejecting other ideas, and it accepted each correction openly. The deep work didn’t start out right; what it had, and the quick searches lacked, was the time and the structure to find out it was wrong.
The two tools also left very different trails, with Claude writing down its briefs, evidence and reasoning as it went, while the Codex conversation that made 204 searches saved nothing and another kept a single note. That sounds like housekeeping, but it points to a part of being smart that benchmarks never measure, which is whether someone else can check the work afterwards, and in a study about whether ideas were genuinely new, a fast answer with no trail would have had to be taken on trust.
Breadth had a cost that the count hides, and Claude admitted as much in its final report on the list of ideas: of the 2,161 patent numbers cited, “Many were only read as abstracts or search summaries, not in full,” and sixty-four ideas had been ruled out on a model’s recollection of earlier patents, without any search at all. The AI in this study read far more than a person could and read most of it shallowly, going deep only when the brief and the time allowed. That mattered most for invention, because proving that something is absent from a patent, which is what novelty needs, means reading all of it (an abstract can show that an idea is there, never that it isn’t), so the quick searches could rule ideas out in seconds and couldn’t establish that any idea was new, which was the first thing a patent needed.
A sharp critic, a poor judge of its own work
Our clearest test of AI as a checker came on 2 October, after Codex and six of its helper agents had written, over several conversations, at least 12 short market research papers on one of the ideas, and we weren’t confident in them. Claude was asked for an independent audit (every cited link tested, every cited patent checked, and every claim marked as correct, wrong, outdated, unverifiable or overstated), and five helper agents shared out the work, checking competitors, evidence of demand and, in one agent’s case, the facts in six of the papers.
Within minutes it reported 177 links tested, five of them dead, and all 22 cited patents real, and the problems it went on to find were in the reasoning. Claude judged the papers careful in tone but short of evidence: six competitors described wrongly, the leading business in the category missed until late in the work, demand never checked, and no verdict on patentability for any idea, in a project where that was the first requirement. When we asked whether Codex had simply made most of it up, Claude’s answer was no, because the sources were real and the errors lay in wrong descriptions, missed competitors and the absence of any check on demand.
That changed how we think about hallucination, since we had assumed the main risk was invented sources and found instead real sources assembled into a confident conclusion that nobody had tested, which is the harder failure to spot. Claude’s fact-check of six of the papers found that most named facts held, with the problems lying in wrong descriptions of competitors and in changes during 2026 that the papers had missed, and Claude then replaced the dead links and added correction notes to 12 of the papers instead of quietly rewriting them, which we were glad of, because it left the original errors visible.
An automated audit like this one proves less than it appears to, because a working link or a real patent number only shows that the source exists, and whether it supports the sentence citing it still has to be read and judged. Claude’s most useful findings, the misdescribed competitors and the unchecked demand, came from exactly that reading and judgement, which is also the part of the audit that could itself be wrong and that we could only check by reading the sources ourselves.
Judging their own work was a different matter, and the clearest numbers we have on it come from naming the marketplace website on 3 October. One agent in the team suggested a shortlist of names and scored them out of 10, and a second agent scored the same names blind, with the authors and earlier scores removed. On six names that both scored, the first agent gave 7.5, 7.5, 7.5, 7, 6.5 and 6.5 and the blind reviewer 4, 5, 5, 4.5, 3.5 and 4, so every pair moved in the same direction, by two and a half to three and a half points. The naming agent had also rated the names we already favoured generously, so some of that gap may have been deference to us, and six pairs are too few to tell the two apart.
The blind reviewer itself warned, twice, that its read wasn’t fully blind because earlier scores had leaked into the conversation, and it listed its own likely biases, which is more candour than most human reviewers offer. The fix it proposed was a human one, a quick reader test with real people, and although the agents recommended it repeatedly that morning, nobody on our side ran it, which shows how easily a team of models can keep refining a question that only people could answer.
The same generosity had shown up a week earlier, when six agents were each asked to design a new method for inventing and two of them scored their own example ideas 12 out of 14 and 12 out of 16, near the top of the scales their own methods set, before running a single search.
We had seen the same weakness in software a few months earlier, on a 113-page website written largely by AI coding agents and described in our comparison of vibe coding and agentic coding, where we caught fabricated pricing and spam content across 31 pages and the AI didn’t detect a quality problem in its own work at any point. A person reviewed every deployment of that site before it went live, which is the split anyone buying AI web design services should expect, with the agents building quickly and a named person deciding what ships.
Published research had described the weakness well before either of our projects. Jie Huang and colleagues showed in 2023 that large language models cannot self-correct reasoning yet: without external feedback they struggle to fix their own answers, and performance sometimes gets worse after they try. A 2024 paper by Arjun Panickssery, Samuel Bowman and Shi Feng then found that LLM evaluators recognise and favour their own generations, scoring their own outputs higher than others’ that human annotators rated as equal, and their controlled experiments showed the link between recognising its own text and preferring it holding up against the obvious alternative explanations, which suggests the bias is no accident of one prompt.
Across Claude, Codex and the agent team, then, the models were consistently better critics of someone else’s work than judges of their own. Claude’s audit of the Codex research and the blind reviewer’s scores on the names both exposed real weaknesses, and Codex was honest about what its own audit couldn’t confirm, so pointing a model at work it didn’t produce, where it becomes far more demanding, was the most reliable use of its intelligence we found. It is why we now put the checker on a different model and brief it without the author’s name or score.
Whether that gap is a temporary weakness or something deeper is the question of whether an AI can form its own judgement. It is also why the invention search kept failing at the final hurdle, the one that asks whether AI can find an invention that survives scrutiny once someone independent attacks it.
Are lots of AIs working together smarter than one?
We had assumed that more agents would mean more intelligence, and from the parallel conversations of 27 September to the 25-role agent team in October, more agents mostly meant more throughput and more things to go wrong, with failures that had little to do with how clever any single agent was. Several Claude conversations on 27 September shared one account, so when it reached its usage limit ten research agents stopped at once, and elsewhere one agent overwrote a script that another depended on.
The parallel conversations also drifted towards the same answers. Claude and Codex worked the same brief side by side that day, with very different methods, and still landed on the same familiar categories of idea, from controls fitted to existing appliances and medication packs to damp and mould in homes, and by the end of the afternoon both had reached the same conclusion about how to escape those categories (start from a new physical mechanism and build the product around it), expressed in different words. A few days later, on 2 October, we gave three separate Codex conversations the same prompt within an hour, asking each to think like an inventor looking ahead to where technology is going, and none of the three produced an idea strong enough to patent, each falling back on familiar product categories. Counting that as three independent confirmations would be a mistake, because it was the same model family given the same words minutes apart, and repeating a result like that shows it is stable for that model and that prompt while saying very little about whether a different kind of system would have done better. Since then, when two model families agree on an idea, our first suspicion has been that it’s the obvious one.
The agent team, set up on 2 and 3 October, grew to 25 roles within two days, and by 3 October the pattern was clear enough for us to write it down: the models tended to side with us, and when several agents discussed an idea they mostly ended up agreeing with each other, the same pull towards agreement that Sharma’s team had measured. The fix had four parts: an agent whose job was to argue against the others, a team rule that popularity, repetition and a majority of agents don’t count as evidence, briefings that hid who had written each idea, and checks by a different model family. After that the agents’ reviews of each other were noticeably harder to please, though the team stayed slow. When progress on the website was checked on 4 October, the agent coordinating the team confirmed that only the homepage existed, and when the work moved back to a single Claude conversation its review put the cause in five words: “too much process for the output”.
A team of AI agents sounds like a room of experts debating, and in this study it behaved more like a large organisation, good at dividing up work that splits cleanly and slow and fragile wherever every piece depended on every other. More agents gave us more coverage and better criticism of weak ideas, and we would use them again for that, but they didn’t give us a smarter answer to the hardest question in the study.
The rubric that scored the cheapest page highest
With only the homepage built, judging its design gave us the sharpest example of a model misjudging quality. Nine draft designs of the page had been made in ChatGPT, numbered 0.1 to 0.9, and on 4 October Claude was asked to set out the ten things that mattered most on a page like this and score each draft against them. It built a weighted ten-point rubric, with minimum standards for clarity and trust, and versions 0.9 and 0.8 came top at 7.9 out of 10.
To me those two looked the cheapest and least trustworthy of the set, and I said so, adding that this was exactly where I doubted a language model could give a real emotional response. Claude’s diagnosis of its own mistake was the most useful part of the exchange:
“That’s an analyst’s reading. A person reads trust from presence… I judged what was missing, not how it felt.”
Its rubric had given branding just 5% of the weight, and it agreed that was too low.
Every criterion on the rubric was sensible, and the scores followed the criteria faithfully. What went wrong lay in what couldn’t be written down as a criterion, the instant sense of whether a page looks like a business you would trust with money, and while a model can reason about trust signals, in this test it couldn’t see them the way a visitor does.
It was the same gap the OpenAI model’s “fairly premium” verdict on the built homepage had shown, and it is the reason I built PennyForYours, a personal hobby project for getting quick website feedback from real people. It is also the test we insist on for any new site, whether it comes from a web design agency or an AI tool: before anyone scores the page, we show it to a few people who have never seen it and ask them what the business does.
How smart are ChatGPT, Claude and Gemini?
Our study used OpenAI’s models through ChatGPT and Codex and Anthropic’s through Claude Code. It didn’t use Gemini, so what we say about Gemini comes from published results only.
How smart is ChatGPT?
ChatGPT has changed more than any of them since we first wrote about it, and our 2023 explainer on GPT-3 and ChatGPT now reads like a description of a different technology, and a useful reminder of how capability moved between model releases in three years: the ChatGPT of 2026 reasons through multi-step problems, uses tools and writes working code, and what our study suggests is that the jump has been much larger on checkable work than on judgement.
In the study ChatGPT was where the work started, and OpenAI’s models also powered Codex. At their best they were candid in a way we valued. When one Codex conversation was told the idea must not overlap any patent application still awaiting a decision, Codex replied that it would treat that requirement as “a research claim to test, not something I can guarantee without a formal patent search and attorney opinion”, which was right. Their weaker habit was speed over grounding. In the same conversation Codex rated ideas as promising before running any searches on their claims and ruled them out once it did, dismissed several ideas without citing any document number, and when challenged, conceded at once and then hedged everything as a research hypothesis.
The list of ideas gives a rough picture of what each tool contributed: of the 276 ideas, Claude wrote up 234 (56 promising but unproven, 178 ruled out), Codex 40 (13 promising, 27 ruled out) and ChatGPT 2, both ruled out. Claude’s own analysis warned how to read that split, because it shows which tool turned each idea into a written proposal, and most of the problems behind the ideas came from us; for more than half of them we can’t now say whether the problem came from us or from a tool, so the split is approximate.
Codex’s higher share of promising ideas needs reading with its habit of rating before searching in mind, since a promising rating from a quick look and one from a deep search don’t mean the same thing.
How smart is Claude?
Working side by side with Codex, Claude was the more methodical of the two: it read its earlier notes before starting, wrote briefs with constraints and separate verdicts, and ran physics checks as explicit scripts. It also made the most human-looking mistake of the study. While gathering the list of ideas, it assumed that other AI conversations were still working on the same project and started planning around them, when every one of them had already finished; it had taken for granted a state of the world it had never checked, and only our pointing that out put it right.
Its strengths and weaknesses had the same root, and the homepage rubric showed the weak side of it. Claude reasoned well about whatever had been written down (files, claims, criteria, citations) and badly about anything that hadn’t, whether that was what was happening outside its own conversation or the feeling a page gave a visitor.
How smart is Gemini?
Gemini wasn’t one of the tools in our study, so we have no first-hand comparison to report, and its standing rests on published results, above all the olympiad gold-medal score described in the 2026 section. How it compares with the others is harder to say, because the AI Index found the top US model only 2.7% ahead of the best Chinese model on its comparison of leading models, as of March 2026, and with the leaders that close together, any ranking of them by name tends to change with the task you set and the release you happen to test.
Is ChatGPT smarter than Claude?
Neither, in any general sense we could defend, and our study shows why: given the same founding brief on 27 September, Claude broke the work into helper agents with evidence rules while Codex ran a long burst of quick searches, and when challenged, Claude argued first and then agreed while Codex gave way at once. Despite such different methods they reached the same familiar categories of idea, and what worked best for us was letting one check the other’s output.
That is how we would answer the question for any pair of models, and it follows from the commercial definition earlier in this article: pick the model that gives the accepted result for the job in front of you, and check it with the other one.
How clever is AI compared with a human mind?
Pulling those results together gives our answer to how close AI is to human intelligence. On formal, checkable tasks it is level with skilled people or beyond them, while on meaning, judgement, physical context and private knowledge it still has a long way to go, and nothing in this evidence suggests that superintelligence, AI beyond the best human minds in every domain, has arrived (the broader question of whether AI is smarter than humans overall has its own article in this series). The clearest example of the gap in our study had nothing to do with raw reasoning.
When Claude’s helper agents gathered the 276 ideas from the project’s files on 27 September, the most valuable thing they produced was a short list of doubts. Several pointed out that their instructions mentioned objections to ideas that appeared nowhere in the files, because those objections had been raised in conversations they had never seen, and one noted that most of the rejections in its batch rested on earlier patents recalled from memory instead of searched for. The agents recorded those doubts without letting them change any verdict, which was careful work, but they had no way of finding out whether the claims were true, because that knowledge lived with us, where a colleague in the room would simply have asked.
What a person brought that our study’s models could not was surprisingly concrete:
- Private evidence: Codex’s own report on the study concluded that a model “cannot certify global uniqueness, supply private workshop knowledge, decide inventive step with legal authority or manufacture a non-obvious mechanism from a vacuum of new technical input”. Knowledge from the workshop, or from watching real people use a product, was exactly what the models didn’t have.
- The physical world: Seventy-three of the ideas came with a cheap first test described in writing, and none of those tests was run. Running any of them would have meant someone buying parts and working at a bench.
- The brief: Our July comparison found that more reasoning can’t repair an inaccurate brief, missing business knowledge, the wrong success measure or an incorrect assumption supplied by the user, and that strong models often infer what they think the user wants, which feels unusually intelligent when the guess is right. Every one of those sits with a person. By the time the homepage work came back from the agent team, its design had drifted from what had originally been intended, and Claude’s review agreed that capable agents had followed the brief they were given.
- Accountability: The agent team’s rules said that nothing went public, was bought or was sent without our sign-off. That rule was the right one, because someone has to own the consequence when a confident answer is wrong.
- Meaning: A model can describe why a design should feel trustworthy and still miss that it doesn’t, as the homepage rubric showed, and this was the gap that came up most often in our study.
Limitations and open questions
Everything above rests on Opace’s own 25-day study and a set of published studies, and both have limits. Ours was run at a desk: AI did the patent searching, there was no private data and no lab or physical test, and the samples were small. We kept the conversations and files from every stage, and the figures here come from them, but they show what happened on this work, under these conditions, in autumn 2026, and what they show is that this approach didn’t produce a patentable invention, which is a long way from proving that AI can’t invent.
Model versions changed during the study: on Anthropic’s side Claude ran on more than one of its current models, with a smaller, faster model often doing the helper agents’ work, and on OpenAI’s side Codex ran on several versions of GPT-6, and our logs don’t always say which model handled a given task, so we wouldn’t generalise from this study to a particular version. There was no human control group generating ideas under the same rules, either, so we can’t say how people working alone would have done. The small numbers need care too: the ten ideas written down in full before any searching were all ruled out, and with no successes in ten tries the true success rate could still be as high as about 30%, while the 276 ideas don’t give a much tighter figure, because many of them are variations on each other instead of independent attempts. Taken together, the numbers describe how this process performed and say much less about what language models in general can do.
| Limitation | What it means for the conclusions | The test that would address it |
|---|---|---|
| One study, with one company setting the questions | Results describe this way of working and can't be generalised to language models as a whole | The same method run by other people on other problems |
| Model versions changed and were not always recorded | Findings can't be tied to a single model release | Fixed model versions logged for every conversation |
| Parallel conversations used one model family | Agreement between them shows repetition, which is weaker evidence than a range of views | Conversations with models from different families on the same fixed brief |
| No human baseline or human judging under the same rules | No patentable idea can't be compared with what people would have found, or with how people would have judged the ideas | A human-only track, and people judging every track blind |
| Ideas are not independent and their origins are incomplete | Small, overlapping samples: a result of zero can't rule out a modest success rate | Counting independent needs, not idea rows |
| No legal novelty finding and no physical tests | Promising and ruled-out verdicts are desk judgements, untested by a patent office or at a bench | A professional search on the best candidate, and one bench test |
The published evidence has an age problem of its own, because the consultant experiment, the Turing-test study and the bar-exam re-evaluation all tested GPT-4 generation models, and the self-correction and self-preference papers date from 2023 and 2024. We use them because they are careful and because the patterns they describe matched what we saw in 2026, and we don’t assume their numbers describe today’s models; how much of what we saw will last is still an open question, since models keep changing. We also can’t separate the model from the setup around it, because almost everything here was done by models working inside tools, search limits and instructions that we set.
What a fairer test of AI intelligence would look like
The test that would answer more of these open questions came, as it happens, from one of Codex’s helper agents partway through our study, and it’s the one we would run next if we repeated the exercise. The agent started from the idea of a public contest inviting anyone to show that AI can invent, and (sensibly, we thought) turned it into a private benchmark, fixed in advance, with five separate tracks: AI working alone, a person assisted by AI, people working alone as a baseline, a team of agents, and a rediscovery track that asks AI to reinvent things that have recently been patented, to check whether the method can find inventions already known to be real. People would judge whether each idea was new and inventive, and models would only sort the entries. Its reasons were practical ones. Nobody can prove that an idea is unique worldwide, a public leaderboard would destroy patent rights by publishing the ideas, and a failed public round wouldn’t show that AI can never innovate, so fixing the protocol, the judges and the scoring rules before any idea is generated was the only way to stop anyone, us included, moving the goalposts once the results started to look good or bad.
Codex’s report on the study, which Claude edited, added what such a study should record: the number of separate real-world needs instead of the number of ideas listed, how many candidates get through each stage, the time and computing cost, and every case where a model’s summary of its work didn’t match the log of what it actually did. It also suggested running the rediscovery test against recently granted patents, while warning that training-data leakage would have to be acknowledged. That last point is the same contamination problem as GSM1k, applied to invention, and it is why a rediscovery score is only worth trusting alongside the other four tracks.
How to test how smart your AI really is
Who controls these systems matters as much as how capable they are, because a model inside a company’s workflow is only as smart as the brief, tools, checks and sign-off around it, and those are human decisions. The labs set the training and the guardrails, the businesses using them set the checks and permissions, and somebody still has to be accountable when an answer is wrong, which is why deciding those checks before the tools go live is the most useful part of any AI strategy consultancy work we do. The checks below came out of failures in our study and in published retests, and we now use them on client and internal work; each is written as what we do and why, because the reason is what lets you adapt it to your own tools.
Reword the question: A system that understands a problem should still get it right when the wording and the numbers change, and one that has memorised the problem won’t, which is what the GSM-Symbolic results showed. When we need to rely on an answer now, we ask again in a fresh conversation with different wording and different numbers, and if the answer changes we treat that as a reason to investigate before using either version.
Check facts before polish: The homepage rubric went wrong partly because a person was reacting to presentation while the model was scoring structure, and neither was checking the same thing. The review order we set out in our July comparison still works for us: whether the result follows the brief, whether the facts and mechanics are correct, whether all deliverables are present, whether it works in the intended environment, and only then whether it looks good. That order came from noticing a halo effect in our own reviews, where a polished result feels as though somebody has already checked it and so persuades reviewers before anyone actually has, and because model output is nearly always polished, we find the halo on almost every review of AI work.
Use a different checker, briefed blind: The market-research audit found real problems because a different model, with different training, read every source, and the naming scores showed what happens when the author marks its own work. So we put the checker on a different model family where we can, strip the author’s identity and earlier scores out of what it sees, and ask it to state its confidence and what would change its mind, as our independent reviewer agent did, giving little weight to any judge that can’t say what evidence would move it.
Test the check on known answers first: The best habit our agent team had was testing its tools before trusting them. When the agents checked whether candidate names for the website were still available to buy, they first ran the check on names known to be taken, such as google.com and bbc.co.uk, and on a nonsense name nobody would own. That caution paid off, because the registry for .co.uk names began refusing and slowing the agents’ lookups within a dozen requests, and without a name known to be taken in the batch, a blocked lookup would have looked exactly like an available name. The coordinating agent even caught one batch where the known-taken name had been left out. The team’s rule was that any method of gathering evidence had to be proven on cases with a known good answer and a known bad one, and we apply the same rule to any AI check, including the ones we write ourselves, because the most useful thing a tool can tell us is what it couldn’t check.
Ask an unprimed person: Anything about meaning, trust or feeling needs a human who hasn’t been told what to think, and the reader test our agents designed for the website names is the template we would use. The name is shown on its own, as it would appear in a web address, to about five people who haven’t seen it, for roughly three seconds. They are asked what the company does and, later, whether they would trust it with paid work, and one name whose meaning is already known goes into the set, so we can tell whether the test itself is working. It would have cost almost nothing, and not running it was our mistake. If you would like help building checks like these into your own workflows, that is part of our artificial intelligence services.
Verdict: how intelligent is AI, really?
Our verdict is that AI is very capable but unevenly so: where an answer can be checked by a proof, a test suite or a source that opens, the best systems now match or beat skilled people, at a speed and breadth no team of people could manage, and in our study they generated, sorted and audited ideas faster than a person could have opened the files.
Where an answer can’t be checked automatically, the same systems go wrong in ways that are easy to miss, because their tone doesn’t change when they do. The naming scores that came out two and a half to three and a half points too kind, the rubric that put the cheapest-looking mockups on top, and the 276 ideas with none strong enough to patent all came in the same confident voice, which we read as what a very good pattern-learner does when the pattern it needs isn’t in its data and nothing outside it says whether the answer is right.
So our answer to how smart AI is: it depends on whether success can be verified, and IQ scores and headline benchmarks mostly measure the cases where it can. Is AI really intelligent? On Chollet’s definition, learning new skills efficiently, the evidence is weaker than the benchmarks suggest. On the practical definition we use at Opace, getting accepted results on real work, it is very useful when a person sets the standard and checks the result. Whether any of this adds up to general intelligence is the question our guide to AI, AGI and ASI, linked at the start of this article, sets out to answer. If you are deciding how far to trust a tool with your own work, go back to the question this article opened with and ask it of that job specifically: smart at what, and checked by whom?
Frequently asked questions
Are LLMs actually intelligent, or just predicting words?
Both descriptions are partly true. Large language models are trained to predict the next token of text, and that objective explains why they lean towards what is common, whether or not it is correct. Training at scale and later fine-tuning turn the predictor into something that solves real, checkable problems, from olympiad proofs to code that passes its tests. Whether that counts as intelligence depends on the definition you use: on skill it often does, and on learning genuinely new skills efficiently the evidence is much weaker.
Is Claude smarter than ChatGPT?
Any answer is dated the day it is written, because both companies ship new versions every few months and the published gap between leading models is small, so the more useful question is which one gives you the accepted result for a particular job, and you can answer that yourself in an afternoon.
Take a real piece of work you have already done well, so you know what good looks like. Fix the brief, give the identical brief to both models in fresh conversations, and save every output along with the model version and the date. Then score the outputs without knowing which model wrote which, ideally with a colleague doing the scoring, against the measure that matters to you (the facts, the deliverables, whether it would ship) instead of how polished it reads. Repeat the exercise with the brief reworded, because a model that wins once on one phrasing may lose on the next, and if the job involves research, ask each model to critique the other’s output as well, since in our experience they find more in each other’s work than in their own.
Cost and speed belong in the comparison too. A model that is slightly better but takes three times as long, or needs twice as many rounds of correction, may not be the one to use for routine work, and the cheaper model is often good enough for first drafts that the stronger one then checks. Keep the scores in a simple spreadsheet so that you can rerun the same test when either company ships a new version and see whether your choice still holds. In our experience the result usually splits by task, with one model winning on careful reading and checking and the other on speed and breadth, and the most reliable setup has been to use one to check the other.
How close is AI to human intelligence?
Closer than most people expect on some measures, and further away on others. On formal tasks with clear answers, such as competition maths, code that must pass its tests and reading large volumes of text, the best systems are level with skilled people or ahead of them. On the things that make a person dependable at work, which include knowing what they don’t know, noticing what is missing and understanding what someone meant, they are still a long way behind.
For a business, the practical measure is how much of the background a task depends on. Reading, sorting and cross-checking that used to take a team a week now take an afternoon, while the question “is that actually what you meant?” still needs a person who knows the context, which is why the work that gains most from AI tends to be the work that has already been written down well, with the brief, the evidence and the standard for a good result all on the page.
Can you give ChatGPT an IQ test?
You can, and it will often do well, but the number won’t mean what an IQ means for a person. Models score strongly on some classic IQ formats, including the Raven-style matrix puzzles described earlier. The trouble is that a human IQ score is interpreted against assumptions about how the person prepared, and a model may have seen thousands of near-identical puzzles during training.
If you want a score that tells you something, make the test harder to have memorised. Write new items instead of using published ones, vary the format and the numbers, include a few items that look relevant but aren’t, and compare the results with puzzles people find easy and models find hard, such as those in ARC-AGI-2. Report the model version, the date and the exact prompts. The online AI IQ figures we have seen don’t do these things, and we couldn’t trace any of them to a primary source with a published method, so we wouldn’t quote one.
How smart are large language models?
Very, at tasks that can be checked, and much less so at tasks that can’t. A frontier model in October 2026 writes working code, solves competition maths and reads a long document faster than any person, yet it can misread a clock face, rate its own work too highly and miss what a design means to the person looking at it.
Why do AI models mark their own work so generously?
Three reasons stack on top of each other, and the first comes from training: preference data rewards answers people like, people tend to like confident, agreeable answers, and models learn a confident, agreeable manner that carries over into how they describe their own work.
The second is self-recognition. Panickssery, Bowman and Feng found that models such as GPT-4 can recognise their own writing with better than chance accuracy, and that the stronger this self-recognition, the stronger the bias towards rating it highly, so a model judging its own output is, in a measurable way, judging something that feels familiar.
The third reason, and the one that caught us out, sits one step earlier, because when a model writes the criteria and then scores against them, the scores inherit whatever the criteria missed, and our homepage rubric is the clearest example we have: its marking was consistent, yet the designs that looked cheapest to a person still topped the table, because branding carried so little weight. Nothing in the scoring was dishonest, and the generosity had been built in when the model decided what counted. The same thing can happen across models from one family, which share much of their training and tend to share the same blind spots, so a checker from the same family as the author is less independent than it looks.
What we do now is separate the jobs. One model or person writes the criteria, a different one does the scoring without knowing who produced each item, and before anything is scored the rubric is tested on two or three examples whose quality is already known, at least one good and one bad, and reworked until it can tell them apart. For anything about how a page or a message feels, a person goes in the loop before the scores do, and the rubric has to be able to account for their reaction.
Does using more AI agents give better answers?
That depends on whether the work can be divided: more agents helped enormously when the job split into independent, checkable pieces, such as extracting records from separate sets of files under the same rules, and that work was fast because each agent had a narrow job and a fixed rule for what counted as an answer, so its output could be checked against the source.
More agents helped much less when the job was judgement, because agents from one model family tended to agree with each other, every extra agent added coordination overhead, and each was another point of failure for usage limits, logins and shared files. Our rule of thumb is to add agents for coverage and checking, to add a different model family, or a person, when what you need is a different opinion, and to stop adding agents the moment coordination takes more attention than the work does.
Can AI come up with answers that are not in its training data?
Yes, in three senses worth keeping apart. First, a model can recombine what it learned into arrangements nobody wrote down, which happens constantly and is often useful. A model connected to tools can fetch information that post-dates its training, as the agents in our study did hundreds of times between them in a single day. And a model paired with an automatic evaluator, as in FunSearch, can search a space of possibilities until it finds a construction nobody had found before, which is a genuine new result, verified by the evaluator independently of the model.
What none of those does is establish a new fact about the physical world. Our own conclusion from the study was that a patentable idea would need a new technical relationship supplied by field evidence, measurement or expert practice, with the model helping to turn that relationship into a claim and then attacking it. The deeper question of whether AI can produce knowledge it was never trained on gets its own article in this series. Our short version is that the answer depends on what is outside the model feeding it evidence.
Is Gemini smarter than ChatGPT?
There is no defensible general answer, and we didn’t test Gemini in our own study. Google’s Gemini Deep Think holds the clearest exam result of the three, the olympiad score described earlier, but the leading models are now close together on published comparisons, so the honest way to choose is the blind test described in the answer on Claude and ChatGPT above: run both on the same fixed brief, score the results without knowing which is which and date the result, because the next release can reverse it.
