Is AI Smarter Than Humans? What I Found After 25 Days of Testing

Is AI smarter than humans? A few years ago that was a thought experiment. Since then chat models have started answering exam-style questions better than most of us, an advanced version of Gemini reached gold-medal standard at the International Mathematical Olympiad in July 2025, and OpenAI’s chief executive wrote in his January 2025 post Reflections that the company was turning its aim towards superintelligence. When most people ask the question they mean ChatGPT, Claude or Gemini, and underneath it sits a more practical worry about which parts of our work, our judgement and our creativity will still need a person. This article belongs to a series that starts with our main guide to AI vs AGI vs ASI, which explains artificial general intelligence (AGI), meaning AI that can match skilled people across most tasks, and asks whether anyone has built it yet.

We decided to test the question on the hardest version of smart we could think of, and the reason was personal. I have a family member with mobility issues who values the freedom to go shopping and struggles with today’s trolleys, and when I went looking for a power-assisted trolley on 16 September 2026 I found only expensive powered walkers, which made me wonder whether AI could invent something genuinely new. For the next 25 days, to 10 October, we asked a series of AI models and agents for ideas that were new, patentable, cheap to test and worth something commercially, and checked each one against existing patents and products. None of the ideas they gave us was strong enough to justify a patent application.

A much smaller test ran alongside that search, on a homepage the AI agents had built for one of my personal hobby projects, and it is the one I keep coming back to. Asked to imagine it was a first-time visitor, an OpenAI model described the site accurately and called it “bold, credible, clean and fairly premium”. My partner, given 30 seconds and no explanation, was confused and decided it was something to do with parcels. The model’s description was the correct one, yet what we had asked it for was a prediction of how a stranger would react, and it had no way of seeing that a real stranger would be lost.

So is AI smarter than humans? Not overall, on what we found. AI beats most people at recall, speed, exam-style questions and some maths, and narrow systems built for a single job, such as predicting the shape of a protein, are superhuman at that job. In 25 days, though, the frontier models we used produced no idea strong enough to patent, and none of them could tell us how a stranger would react to a page.

I founded Opace in 2008 after training as a software engineer, and since then I’ve gone from drafting with GPT-3 to running teams of AI agents on client work, by way of a search in 2024 for a hybrid human and AI live chat that would let AI answer routine questions and hand the harder conversations to a person (a split between machine and person that this study ended up pointing to again). So I’m an enthusiast, and I’ve also watched these tools break. Going into the project I thought people treat AI output as amazing because it is usually better than what we would produce ourselves, while a lot of it falls apart when you dig into it and never becomes anything great. I had also never felt that genuine creativity or invention was a strength of language models; the spark of ingenuity, I thought, probably had to come from a person, at least for now. Those are opinions, and a fair part of what follows is us testing them.

What we found, in brief

• Over 25 days, ChatGPT, Claude, Codex and a team of AI agents produced 276 distinct ideas and checked them against 2,161 patent and application numbers. None was judged strong enough to justify a patent application, 69 were promising but unproven and 207 were ruled out.
• The ideas came fast (274 of them in about five hours) and the checking took most of the time.
• Most ideas were new in their exact wording and then failed on whether they were inventive, whether the physics worked, how easily a competitor could get the same result another way or whether they could make money.
• Ten ideas written down in full before any patent searching were all ruled out, which is too small a sample to show that AI can’t invent.
• On the homepage, a model described the page well and couldn’t predict that a first-time visitor would take it for a parcels firm.
• Pushed back on in a 46-minute debate, ChatGPT moved towards our view four times and held its ground once.

Is AI smarter than humans? It depends which AI and which humans

Part of the trouble with the question is that smarter gets used for several different things at once. People fold task performance, knowledge, judgement of meaning, accountability for a decision, the ability to produce something defensibly new and general intelligence across unfamiliar problems into one word, and a model can be ahead on some of those and nowhere on others.

The first thing to untangle, though, is the word AI, because it covers a protein-structure predictor, a chess engine, a recommendation system, a robot arm and a chat model, and they give very different answers to this question. Systems built for one job can be superhuman without being general thinkers at all. Google DeepMind’s Levels of AGI paper by Morris and colleagues (arXiv 2311.02462, version 5, September 2025) files AlphaFold, AlphaZero and Stockfish under Superhuman Narrow AI, while warning that general capability is likely to develop unevenly. Most people asking, as we said at the top, mean large language models such as ChatGPT, Claude and Gemini, and that is the comparison this article mostly makes; where a narrow system changes the answer, we say so.

AlphaFold is the clearest example of a narrow system changing the answer, since it predicts protein structures with a precision that no person working by hand could approach, which matters for drug discovery, and yet it does not plan an experiment, choose which disease to study or explain itself to a patient. That gap between a superhuman prediction and a treatment is why we keep the medical claims separate and point to what AI can realistically do for cancer research instead of letting a protein result stand for the claim that AI is smarter than doctors.

The human side needs splitting too, because an average adult, a domain expert and humanity as a whole are three different yardsticks, and a model can sit above the first, near the second and far below the third on the same day. A chat model will out-recall the average person on almost any trivia question, and it may match a junior specialist on a written exam, yet it has not matched the collective work of a scientific field that tests its ideas against the physical world. Our 25-day study was mostly a test against that third yardstick, because a patentable idea has to be new to everyone.

Much of the unevenness comes from the way language models are built, since they are trained to predict the next piece of text from a vast amount of human writing and later tuned with feedback, and what comes out is shaped by what was frequent and well rewarded in that material. Frequency and truth usually travel together (which is why the answers are so often right), but when they part the model can produce a fluent, confident sentence that is false, what’s usually called hallucination, and the same process inherits whatever biases its sources had. So a model ends up very good at the middle of what people have already written and shakier at the edges, which is awkward, because the edges are where new ideas tend to turn up.

People learn differently, and in ways that still matter: a young child can see a new object once or twice, name it, recognise it from another angle and ask what it is for, while a model needs huge amounts of data during training and, at use time, depends on the examples and context it is given. Against that, a trained model’s skill can be copied to thousands of machines in an afternoon, whereas a person’s experience can’t be copied to anyone. A person runs on food and sleep in one body in one place, and a frontier model runs in data centres with no body at all. Each of those differences favours a different side depending on the job, and over the course of the study we gave up trying to rank people and models on one scale and started asking which job each did better.

We learned in 2023 how hard it is even to tell a person’s work from a machine’s, when our own test turned up the AI detectors that mistook Dickens for a machine, with GPTZero flagging part of A Christmas Carol as possibly AI-written. If a tool trained for exactly that job couldn’t reliably tell a Victorian novelist from a chatbot, it seemed to us then, and still does, that headlines ranking the two on intelligence deserve a good deal of suspicion.

The International AI Safety Report 2026, chaired by Yoshua Bengio and published on 3 February 2026, uses a word for this that matches what we saw. It describes the performance of leading systems as “jagged”, meaning they can still fail at some seemingly simple tasks while excelling at hard ones. Trust is where that jaggedness bites, because checking a polished answer costs more effort than reading it, so people tend to accept it and the confident tone starts to feel like evidence. We did exactly that more than once during the study (the homepage is the clearest case), and we have tried to say below wherever a confident answer led us astray.

A jagged profile also explains why two sensible people can look at the same model and reach opposite verdicts. Someone who uses it to summarise reports, translate emails and draft code meets the high peaks every day, and concludes that it is already cleverer than most colleagues. Someone who asks it to judge a new situation, or to produce something nobody has written about, meets the troughs, and concludes that it is a very fluent autocomplete. Both of them are describing something real, and averaging the two into one score would hide both, so for the rest of this article we compare people and models job by job.

Where AI is already smarter than people

Comparing job by job, it makes sense to start with the jobs the machines have already won, some of them decades ago, and the strongest cases share a shape: a defined task, a clear rule for success and a way to check the result that does not depend on whether anyone likes the answer.

Games came first, when IBM’s Deep Blue beat Garry Kasparov 3.5 to 2.5 in May 1997, and AlphaGo beat Lee Sedol 4 to 1 in Seoul in March 2016. In the full AlphaZero evaluation published in December 2018, the system started from random play with only the rules, trained against itself for about nine hours in chess, and then beat the 2016 world-champion version of Stockfish, winning 155 games and losing six out of 1,000. Biology followed in November 2020, when DeepMind reported that AlphaFold reached a median score of 92.4 GDT in the CASP14 assessment. CASP is the long-running competition that tests protein-structure prediction against structures measured in the laboratory, GDT is its 0 to 100 score for how closely a prediction matches the measured shape, and DeepMind’s post quotes the competition’s co-founder saying that a score of around 90 is informally considered competitive with experimental methods.

Formal competition maths is the most recent example. On 21 July 2025 Google DeepMind reported that Gemini with Deep Think had scored 35 out of 42 at the International Mathematical Olympiad by solving five of the six problems within the 4.5-hour limit, working end to end in natural language, with the answers graded by IMO coordinators against the same criteria as the students. A few years earlier no model came close, and DeepMind’s own post adds a small but honest caveat, that the IMO reviewed the submitted answers and never the system itself.

In every one of those cases the machine had a fixed set of rules and an unarguable way to score itself: a won game, a predicted structure that matched the one measured in a laboratory, a correct proof. That scoring rule is what lets a system practise millions of times, find its own mistakes and improve without a person watching every step, and wherever such a rule exists it is now reasonable to expect machines to catch up with people and then pass them, often quite quickly. The benchmark picture behind these wins, test by test, is laid out in how smart AI really is, benchmark by benchmark.

Each of these wins comes with a transfer limit, and we think that is the part headlines leave out. Deep Blue could not play Go, and AlphaZero’s strength comes from millions of games in a world whose rules never change (there’s no equivalent self-play for deciding whether a homepage makes sense to a stranger). AlphaFold predicts a structure without deciding what that structure means for a patient, and the IMO result shows that a model can find valid proofs under competition rules while saying much less about choosing a useful problem in the first place, or noticing that the question itself is wrong. We see the same limit in our own work: when we search a large document set with an agent the speed is extraordinary, yet the agent can still carry one false premise through all ten of its summaries if nobody checks what the search was meant to establish.

Is ChatGPT smarter than humans?

For the chat models specifically, our answer is that ChatGPT is smarter than most people at bounded, checkable questions and less reliable than a careful person at judging what a situation needs. On knowledge recall, summarising, drafting, translating and exam-style problems it beats most of us comfortably, and on some it beats experts. In Morris and colleagues’ framework the 2023 versions of ChatGPT, Bard, Llama 2 and Gemini sat at Level 1, Emerging AGI, which the authors describe as equal to or somewhat better than an unskilled human across a range of tasks. That snapshot is three years old and today’s models are much stronger, but the paper’s latest version, from September 2025, still marks every general level above Emerging as not yet achieved.

Are LLMs smarter than humans, then? On the tasks where most people meet them, often yes. On the tasks our study cared about, which needed a new observation, a sense of what a stranger would feel or a decision someone has to answer for, not in our experience. Is Claude smarter than humans? We would give the same answer, for the same reasons. So when someone asks us whether a model is smarter, we tend to ask back: smarter at what, and checked by whom? One model version on one prompt is a single observation, and the next release can move it.

Where people still beat AI

All of the machines’ wins came from tasks with fixed rules, and the oldest explanation of why people still lead elsewhere is known as Moravec’s paradox, after the roboticist Hans Moravec, who set it out in his 1988 book Mind Children: the tasks that feel effortless to us, such as recognising a cluttered room, walking on uneven ground or reading a face, are hard for machines, while the tasks that feel hard to us, such as calculation and exhaustive search, are easy for them. It is still the best explanation we know. Korteling and colleagues’ review Human- versus Artificial Intelligence (Frontiers in Artificial Intelligence, March 2021) sets the paradox out in evolutionary terms: perception and movement were refined over many millions of years, whereas abstract thought is a recent trick, perhaps less than 100,000 years old, so the skills that feel easy are the ones evolution engineered most deeply.

One practical consequence is that the jobs most exposed to AI are often not the ones people expect. A cleaner, a plumber or a care worker uses perception, balance, dexterity and social reading all day without thinking about it, and those are exactly the skills machines find hardest, while an analyst who spends the day reading documents and writing summaries is doing the kind of work a language model was built for, so the line between what machines can and can’t do runs straight through skills we used to rank in the opposite order.

The ARC Prize Foundation, a non-profit that runs a public prize for AI systems, makes a related point with puzzles. Its benchmark, ARC-AGI-2, is a set of small visual puzzles designed to be easy for people and hard for machines: each shows a few examples of a grid being transformed, and the solver has to work out the rule and apply it to a new grid. In early 2025 the foundation tested the tasks on more than 400 members of the public in San Diego, and every task in the evaluation sets was solved by at least two people within two attempts. Its page says AI systems struggle with particular task types, such as symbolic interpretation and applying rules in context, and the target it sets for an AI system is 85% accuracy.

None of this means people are cleverer in every respect, and Korteling’s paper is blunt about our side of the comparison. It has a section headed “We Are Probably Not so Smart as We Think”, and it points out that the amount of information we can consciously hold and process at once, our working memory and attention, is very limited. That is the limit the agents in our study went straight past, cross-checking thousands of patent documents that no person could hold in mind at once, so where people kept the edge it was in two other places.

The first is meaning, emotion and reading a room, and it is the one I notice most, because although I trained as a software engineer, if I could choose again I would probably have studied psychology; how people think interests me more than how programs do. A design can be tidy, on-brand and still wrong for the people it is meant for, and a person in the room sees hesitation, embarrassment or confusion long before anyone writes it down. A model can describe those signals when someone puts them into words, and it has no other way of receiving them.

The second is physical context. How an object is actually used, where it gets stuck and what the person using it does without thinking are rarely written down anywhere, so a model reading about a problem starts without them (and our own study, which began with an everyday shopping trolley, ended up turning on exactly that kind of fact, as the study section below shows).

Speed is not the same as understanding

The models are extraordinarily fast, and because speed is easy to mistake for understanding, it helps that Huang and colleagues tested the difference directly in Large Language Models Cannot Self-Correct Reasoning Yet, presented at ICLR 2024, the International Conference on Learning Representations and one of the main machine-learning research conferences. When models were asked to review and fix their own reasoning without any outside feedback, they struggled, and sometimes the answers got worse after the attempted correction. A person who says hang on, that can’t be right is drawing on a sense of where the ground is solid, which is the sense the paper found the models lacking.

The other half of the problem is what happens when someone else says that can’t be right. Anthropic researchers led by Mrinank Sharma studied it in Towards Understanding Sycophancy in Language Models (2023), which found that AI assistants tend to tell users what they appear to want to hear, and traced part of the cause to the human feedback used to train them: when a response matched a user’s views it was more likely to be preferred, and both people and the preference models built from their ratings sometimes preferred a convincing, agreeable answer to a correct one. So a model can be trained, without anyone intending it, to give way when pushed. We ran into exactly that during our study, as the study section describes, and it is one reason we treat a model’s first answer and its revised answer with the same caution.

Is AI better than humans at decisions?

For decisions with clear data, a clear objective and a quick way to score the outcome, AI is often better than people, because it does not tire, does not anchor on the first number it saw and can weigh far more cases. Under ambiguity, where the decision depends on what something means to the people affected and nobody can score it immediately, we saw no evidence that it is better, and some evidence of a quieter problem.

Doshi and Hauser’s study Generative AI enhances individual creativity but reduces the collective diversity of novel content (Science Advances, July 2024) shows that problem clearly. In their experiment 293 writers produced short stories, some with AI-generated story ideas, and 600 evaluators rated them. Writers who could see five AI ideas scored 8.1% higher on novelty and 9.0% higher on usefulness than those without, with the biggest gains among the least creative writers. Yet the AI-assisted stories were more similar to one another, so each person was better off while the group produced a narrower range of ideas, which the authors call a social dilemma.

We saw the same convergence on a much smaller scale in our own study, which the next section describes, when separate AI attempts worked on the trolley problem independently and five of six arrived at the same basic mechanism on their own, the same way of making the trolley easier to move. To each attempt it looked like a good answer, and it was, but it was also the answer most likely to have been written about, and patented, already. Since then we’ve stopped treating three models that agree as three independent opinions, and we wouldn’t hand an important decision to one model and stop there.

What we found when we tested it: 25 days, 276 ideas and one homepage

The published research leaves a mixed picture, so we wanted a test of our own on the hardest version of the question. If smarter means anything beyond faster and better read, it ought to include producing something useful that nobody has had before, and being able to defend that it is new, which is a sharp test of general intelligence because the AI has to go beyond what is already in its training data and on the web. It is also the third yardstick from earlier in this article: a patentable idea has to be new to everyone, including people who never write about it online.

The test ran for 25 days, from 16 September to 10 October 2026. It was Opace’s study, and I directed it myself, choosing what to ask, reading the answers and deciding what to try next, so where this article says we it means Opace and where it says I it means my own part in it. We worked first with ChatGPT, then with Claude (through Claude Code, Anthropic’s tool for running Claude as an agent that works on files and code) and OpenAI’s Codex, a similar agent tool, running the two side by side and against each other. From early October we added a team of up to 25 AI agents, each with its own job: a coordinating agent that divided up the work and pulled the findings together, research agents that searched patents, products and published studies, reviewers that checked the others’ work, and one agent set up to argue against the rest so that weak ideas were exposed early.

Each idea had to be genuinely new, patentable, cheap to test and commercially worthwhile, and every idea was checked against existing patents and products. By patentable we meant an idea that passed every check we could run on paper, without building anything: it had to be new, not obvious, work in the worst case, be hard for a competitor to get round by reaching the same result another way, and be able to make money. Even an idea that passed would only have been a paper result, and a patent attorney’s search would still have been needed before calling it real.

Once duplicates were merged, the models and agents had produced 276 distinct ideas and checked them against 2,161 distinct patent and application numbers and 566 references to existing products. None was strong enough to justify a patent application. 207 were ruled out, and 69 were left promising but unproven, meaning something narrow was still alive but at least one check was unresolved; the study ended with all 69 in that state, because settling any of them would have needed a prototype, a physical test or a professional patent search, none of which a desk study could supply. Most of the 276 were new in their exact wording, which is easy, and then failed on the harder questions: whether a skilled engineer would find them obvious, whether the physics actually worked, how easily a competitor could get the same benefit another way, or whether anyone would pay.

The AI’s advantage showed up in speed and breadth. 274 of the 276 ideas came out of about five hours of AI generation in late September (the other two were added at other points in the study), and the fortnight after that went mostly on checking them, first with Claude and Codex and then with the agent team, reading and cross-checking patents, products, standards and papers across languages and industries at a pace no person could match in the time. The models were especially good at turning a rough idea into searches at the level of the underlying principle, so a single idea could be tested against several industries at once.

Codex ran two experiments on 2 October designed to give the models a better chance. The first started from 35 recently published reports of products failing in real use, so that the starting point was fresh evidence the models couldn’t have read in training, and worked them into 30 candidate mechanisms (short descriptions of how a new device might work) before writing five of them down in full ahead of any patent searching, so the search couldn’t quietly reshape them into something safer. The second followed the same rule with five more. All ten were ruled out. Looking back over everything, the models did five jobs well: mapping the evidence, proposing possible mechanisms, turning a rough idea into searches, attacking weak ideas and keeping a log of what had been tried. All five depend on tireless, accurate reading at scale, which is where machines have always beaten us, and we go into how the search ran, and why it still never produced a patentable idea, in whether AI can really invent new things.

Where people beat the machines was physical context, and it started with the trolley itself. ChatGPT’s very first reply in September had asked whether the trolley would be pulled or pushed, so a model asked the right question, but it took watching the trolley in use to show which answer mattered: the person using it pulls it along behind, and none of the models built on that until we did. When Claude later reviewed the whole trolley search, it picked out two insights as the most useful in it, that the trolley is pulled and that a trolley with bigger wheels already fixed about 70% of the difficulty, and both had come from watching the trolley in use. It described its own strengths, fairly we think, as recombining ideas across fields, doing the physics and killing weak ideas quickly.

So the invention search cut both ways for the question this article asks. The AI was far faster and broader than we were at generating ideas and searching for them, and no better at the thing the test was actually about. By early October I had summed up our working view in a note: AI brainstorming is amazing for many things, and it can give you a top 20 drawn from its training data and web browsing, but it can’t give you a top 20 of things that have never been done before.

Looking back over the whole 25 days, our conclusion is narrower, and we think fairer. A language model can help with invention by finding patterns in new evidence, proposing mechanisms and killing weak candidates quickly, yet we found no reliable way for a model working from prompts, training knowledge and public search alone to produce an idea strong enough to patent. The models kept producing the most-written-about answers, which are the ones most likely to be patented already, and what was missing each time was a new technical fact, a physical observation, an expert’s workaround or an experiment that the model could not infer (which in practice meant a person, or a lab).

Where the agents beat our judgement, and where they did not

That gap raised the question the second half of the study turned on: if AI agents are as capable as they seem, what would they still need people for? In a note on 2 October I gave my own answer, that as agents gained more freedom they would still need a human point of contact for approvals, for real, physical-world support and checking, and for human creativity. We then had Codex work through the question in detail, and its answer changed how we framed the whole study. It named ten missing resources: body and presence, senses, hands, local context, social identity, legal personality and authority, money and financial access, availability and risk tolerance, accountability, and consent and moral agency. Its conclusion was that the missing resource is “often not intelligence”, and its list of 34 tasks an agent might need done and couldn’t do bore that out, from the obvious ones (attend, queue, collect, deliver, inspect, measure and repair) to several we would not have thought of, such as giving consent, providing care, responding to an emergency, taking responsibility, and shutting down or overruling an agent.

Money showed the gap most clearly, because, as Codex pointed out, an agent having a digital wallet does not mean a person can actually be paid, and a digital token cannot by itself give an agent a bank account, creditworthiness, legal personhood, a physical body, a licence, a human relationship or a person willing to accept danger. A cleverer model would supply none of those; the law, the physical world and other people have to. Asked what roles a person might play for AI agents, Codex listed sixteen: executor, observer, attestor, operator, local representative, buyer or payer, custodian, approver, guardian, professional, negotiator, carer, creator or curator, trainer or teleoperator, dispute resolver and risk owner. Very few of those depend on being cleverer than a model, most depend on being a particular person who can be present, trusted or held responsible, and two of them, approver and risk owner, are the roles that stayed with a person throughout our own study. Codex also argued that finding that person is harder for an agent than it sounds, because a web page listing someone’s services would be little more than a brochure to an agent, which would need descriptions of the work, prices and response times that it could read and check for itself before trusting anyone.

Those lists became the starting point for a website idea through which AI agents could hire checked people in the UK for jobs they can’t do themselves, a personal hobby project of mine that is separate from Opace. The agent team building it is where the agents beat my judgement most clearly, by using evidence against my favourite theory. I had suggested that people could be hired through the site to come up with new ideas, on the grounds that this was something AI couldn’t do well. One of the research agents, asked to rank the jobs that published research says still need a person, left it off, because published research shows language models can match or beat people on how novel their ideas look. The coordinating agent then drew the line we now use ourselves:

“The claim is believable only where the limit is a body, a legal identity, money or liability.”

A claim that rests on AI not being clever enough, it argued, is one anyone can watch weaken with each new model, and it backed that up with a blind study published at ICLR 2025, in which reviewers rated model ideas as more novel than experts’ ideas.

Because that study cuts against my own view, it deserves a fair hearing. Si, Yang and Hashimoto, the authors of Can LLMs Generate Novel Research Ideas?, recruited more than 100 researchers in natural language processing (NLP), the branch of AI that deals with human language, to write research ideas and to blind-review both human and model ideas. The model ideas were judged more novel than the experts’ (p < 0.05) and slightly weaker on feasibility, and the authors themselves flagged that the model’s ideas lacked diversity and that models were poor at judging their own ideas. So we have to accept that a model can produce ideas that experts, reading blind, find more surprising than their own. What the study doesn’t test is whether those ideas work, or whether they are new to the world as well as to the reviewers, and new to the world was the bar our 276 ideas were held to.

That left us with a contradiction we have not resolved, because blind reviewers in a published study rated model ideas as more novel than experts’ ideas, and yet 276 ideas from the best models we could use produced nothing that got through a patent search. One of the agents offered the closest thing to a reconciliation: the empty result points to a gap in judging ideas more than in generating them, and it doesn’t show that a crowd of people would have done any better. A blind rating of novelty and a patent examiner’s view of novelty are different tests, and nobody in our study ran the experiment that would tell us how far apart they are.

The coordinating agent drew a similar line on another job. The site’s list of jobs for people included checking how readable a page is, and it pointed out that models already judge readability well, so the job only made sense if real target readers, such as screen-reader users, actually tried the page. We found that persuasive, since a model can score a paragraph’s reading level as well as we can, while the experience of someone who depends on a screen reader is something only that person has.

The agents also caught each other out, which turned out to be some of the most useful checking in the study. One research agent cited a study of 303 bounties posted on RentAHuman, a live site where AI agents can already hire people for tasks, much as my hobby project set out to do; it found that 32.7% of those bounties came through programmatic channels such as API keys, which suggested a large share were posted by AI agents directly. The coordinating agent checked that against a larger audit of 779 RentAHuman listings from the MIT Center for Collective Intelligence, in which only 9.2% of the listings were labelled as posted by an agent or bot, and we dropped the higher figure. The first agent had found a real study and drawn too much from it, and it took a second, larger sample to show that.

The homepage a model called premium and a person read as parcels

The clearest result of the whole study came from the homepage the agent team built for that hobby project in early October, where, asked to pick an example job for the top of the page, the agents settled on a real parcel order, signed for, returned and refunded, because nobody would argue that an agent can sign for a parcel. It was the safest example to win an argument with, and it became the page’s main image: a timeline of one parcel’s journey.

Throughout the study I kept running into the same problem with these tools: ask a model for its instant impression of a design and it tells you what it has read, when what you wanted was how a person would feel. The fault might also have been mine, because I was very close to the page and found it technical and transactional, so on 4 October we tested it by asking an OpenAI model to look at the live homepage the way a stranger would, imagining it had clicked a link with no context and looked for one to three seconds, about the length of a first glance. It decided the site was a modern marketplace where people offer practical services to AI agents, and called it “bold, credible, clean and fairly premium”, with the feel of a confident tech start-up. Asked whether it had judged the words or the look, it said mostly the look, and kept its verdict.

As a description of the site, that answer was accurate. The black, white and magenta palette and the confident layout were real, and a marketplace where people offer services to AI agents is what I was building.

My partner, shown the same page with no explanation and given 30 seconds to say what the site was, was confused, and then decided:

“…it’s something to do with parcels being delivered”

That reading was wrong about the site and right about the page, because it showed what a real first-time visitor took away, which was the very thing we had asked the model to predict. The model could describe a well-made page, and it had no way of knowing that the page would confuse the people it was for.

When Claude examined the live page with both reactions in front of it, the cause was plain: the main image was the parcel timeline (an order email, a courier signature, a returns barcode and a bank refund line), the first sentence talked about taking delivery of a parcel, and not a single person appeared on a site about hiring people. Earlier design rounds had dropped stock photography, and the agents had drifted from there to no photos of people at all. Claude’s summary of what the agents had done was blunt:

“The team overcorrected and took the humans out of a site about humans.”

For this article, what Claude said about the gap matters more than what it said about the page. The OpenAI model had judged design quality, which models do well, while my partner had judged meaning, and Claude added that its own reading couldn’t count as a first impression either, because it had looked at the page only after hearing my partner’s verdict. It could analyse a page’s structure, it said, but only someone who hadn’t been told what to look for could give a gut reaction.

It is still only one model’s reading against one reader’s, and Claude proposed the test that would turn it into data, a five-second test of the homepage on 20 to 50 UK adults through an existing research panel, which we didn’t commission. What we took from the episode is that a stranger’s quick reaction is something a model can’t stand in for, and the hobby project changed course because of it. What grew out of it is PennyForYours, a small hobby project of mine, open by invitation while it is tested with a small group, that collects ideas, feedback and testing from real people. Claude and two of the agents were clear that a crowd paid by the minute would not out-invent the models either, because what the invention search lacked was new technical facts and expert know-how, which a quick reaction doesn’t supply, so PennyForYours asks people for quick, honest reactions and leaves invention out of it.

What the first-sight test exposed

Claude later put the underlying point in one line:

“A person can see something for the first time exactly once, and no model can do that.”

A model has, in a sense, already seen everything like your page, because it was trained on millions of them, and once it has looked at your page, or been told about it, it has seen yours too, which puts it in the same position as a friend you’ve already asked once.

We think that narrow claim is the most defensible thing we learned about where people still beat the machines. A first impression is a measurement of a human reaction, so it has to come from a human, and the OpenAI model could tell us what pages like ours usually signal while having nothing to go on about what a stranger would think. Human reactions are useful partly because they vary (asking people gives you a spread of answers from a described group, while a model gives you something closer to one well-informed average). For any new website, whoever builds it, that means a model can review a web design agency’s work for structure, speed and polish, and whether a stranger understands what you sell within a few seconds is still a question only strangers can answer.

My own part in this was not flattering, because while that homepage was being built I struggled with choosing a name for the site, with judging whether the idea behind it was any good, with the design and with the photography, and each time I leaned on the models and not on people. Whenever I was unsure, I tried to get to the right answer by adding more agents and more roles, such as a reviewer or an agent to argue against the others. The extra roles did help (weak ideas died quickly, and the reviewing agents caught errors in my own reasoning), but asking people I know felt slow, made me feel guilty and gave me a small, biased sample anyway. Even as the person coordinating everything, I found my views easily skewed by the models’ direction and comments, so I was (I can see now) running a study about what AI still needs from people while avoiding people, because the AI was quicker to ask.

Pushback, self-correction and the debate that closed the study

The models’ habit of giving way under pressure, which the sycophancy research above describes, showed up from the first week. In an early check in late September, Claude reviewed one of our ChatGPT conversations about the trolley and found the headline answer right but much of the reasoning behind it wrong, and every promising lead in that conversation reversed as soon as we challenged it. Later in the study the pattern held: when we challenged a confident answer the model usually changed it, sometimes rightly, because we had spotted a real flaw, and often simply because we had pushed. A careful person who knows why they believe something can tell those two situations apart and will hold a position against pressure when the evidence supports it, and the models we used found that much harder.

The corrections that did hold up were triggered by something outside the model. Early on 2 October, asked why none of our ideas had yet passed every check, Codex found a flaw in its own method straight away: it had been treating the fact that it couldn’t remember an idea as a possible sign that the idea was new, which is not the same thing at all. It also agreed that our checks tested whether an idea could be defended as new, which is a different question from how creative it was. After the first five ideas written down in advance had all been ruled out, and before the second five went through the same checks, Codex wrote down what a second empty result would mean, and when that round also produced nothing patentable it stuck to its own rule, concluding that a model can’t reliably originate or certify the missing novelty by recombining public knowledge over and over. Claude showed the same pattern on a smaller scale. When we asked it to write up the story behind the study from our notes, its first draft invented four details that had never happened, including a daily routine and a three-week timescale, and it found and corrected them itself once it checked the draft against the notes.

Our experiment into whether AI can invent came to its conclusion on 10 October 2026, when we put the result to ChatGPT and debated it for 46 minutes. In that one conversation ChatGPT moved towards our view four times and held its position once. It also described, in its own words, the gap Huang and colleagues measured from the outside: asked why it had recommended a way of tackling a problem that it hadn’t used in its own earlier answers, it said: “LLMs can possess sophisticated reasoning capabilities without consistently knowing when or how to apply them effectively.” A model’s account of itself is weak evidence on its own, and we quote this one only because it matches the research. The same conversation produced a thought experiment about whether a model could keep getting smarter on its own, which we come back to in the section on whether AI can improve itself.

What our study can’t show

All of this comes from a single company’s study run at a desk, with no lab and no private data, and we would rather list its limits than have a reader find them. AI did the patent searching and there was no physical testing. The models and prompts changed as we went over the 25 days, so no two stages make a clean comparison, and for many of the ideas our list doesn’t record which model or prompt produced them. There was no controlled human baseline either, since we never ran the same invention task with a group of people working without AI, so we can’t say whether people would have found a patentable idea where the models did not.

The samples were small, too. With no successes among the ten ideas written down in advance, the true success rate for that method could still be as high as about 30%, and we can’t apply the same arithmetic to all 276, because many of them are variants of each other and were never independent attempts. What the study shows is that this approach didn’t produce a patentable invention in 25 days, which falls well short of proving that AI can’t invent.

Some limits are about what we couldn’t see. We had no access to model weights, training data or system prompts, so when we describe why a model behaved as it did we are inferring from its outputs and from what it said about itself, which is a weak kind of evidence; where we quote a model, or my own notes, the words are exactly as written at the time. None of our ratings is a legal finding on whether an idea is new, since they are our own screening against our own checks. And the homepage finding rests on a single reader’s reaction.

The limit we are least able to correct for is my own role. I wrote at the time that I found it hard to make good decisions about direction when I was tired or confused, or when the model seemed to be steering, and I went in believing, as I said at the start, that invention needs a person. So the person steering the experiment was sometimes tired, easily swayed and already expecting a particular result, and we can’t measure how much that shaped what we found.

If we ran the project again, these are the open questions we would test first:

• Testing several makers’ models: run the same brief, written down in full beforehand, through models from different developers, such as OpenAI, Anthropic and Google, independently and see whether they converge on the same mechanisms.
• A blind human baseline: give the same task to people working without AI and score both sets blind against the same checks.
• Rediscovery tests: ask models to invent things that were patented after their training cut-off and measure how often they get there.
• Judged versus genuine novelty: take ideas that blind reviewers rate as novel and put them through a real patent search, which is the gap between the blind-review study above and our 276 ideas.
• Removing the checker: see how much of a system’s apparent improvement disappears when its automated scoring is removed or weakened.

Can AI improve itself?

Asking whether AI is smarter than us leads straight to whether it could make itself smarter, and the answer depends on what improve itself means, because two very different things get called by that name. The first is real: AI systems are already helping to improve AI. In May 2025 Google DeepMind described AlphaEvolve, a Gemini-powered coding agent that proposes changes to programs, runs them, scores them with automated evaluation metrics, keeps the best and tries again. DeepMind reported then that it was continuously recovering, on average, 0.7% of Google’s worldwide compute resources, and that it had sped up a kernel, a small and heavily used piece of low-level code, in Gemini’s architecture by 23%, giving a 1% reduction in Gemini’s training time, which includes “training the large language models underlying AlphaEvolve itself”.

We find that remarkable, since it is AI contributing to its own successor, though the set-up is a very particular one. The system works on problems where a machine can check every proposal automatically and objectively, and engineers chose the problems, built the evaluators and deployed the results. The evaluator is the same ingredient that made AlphaZero and AlphaFold possible, a reliable, automatic way to say whether an attempt was better or worse, and where it exists a machine can run thousands of attempts and keep the winners, so progress compounds. We had nothing like it for the questions our study cared about, such as whether an idea is genuinely new to the world or whether a page makes sense to a stranger, and without one a loop like this has no way of telling a better attempt from a worse one, so it is as likely to drift as to improve.

Can ChatGPT improve itself?

The second meaning is the idea that a chatbot gets cleverer as you talk to it, and that one isn’t true. A model’s weights, the billions of numbers set during training that make up the model itself, do not change during your conversation. ChatGPT’s memory features store notes about you and your preferences, which it can read back later, but the reasoning stays the same until its makers train and release a new version. We go into the mechanics of that in how AI models learn, remember and forget, and the self-correction research above adds the other half: without outside feedback, a model asked to check its own reasoning often fails to improve it.

The more interesting question is what happens if you give a fixed model a very long memory and a way to check its work. In the debate that closed our study, we asked ChatGPT whether a model could become greater than its training data through reasoning alone, if what it knew kept growing, given that you can only reason from what you know. ChatGPT sharpened that into a thought experiment: take today’s most capable reasoning model, freeze its weights permanently, give it effectively unlimited memory, a way of checking its discoveries and enough time to reason continuously, and then ask whether its rate of discovery would level off or whether each discovery would make the next one easier.

Given how readily ChatGPT moved under pressure in that same conversation, we treat what it said below as a useful way of framing the question and nothing more, and we set it against the published research at the end of this section.

Plateau or accelerate? The frozen-model question

ChatGPT set out three broad possibilities. In the first, diminishing returns, the easy discoveries run out and progress slows; in the second, steady improvement, the model keeps finding useful methods while its core limitations remain; and in the third, accelerating improvement, each discovery makes the next one easier and a feedback loop forms. ChatGPT’s own answer was a qualified yes: such a model could get better at solving problems, but unlimited, accelerating improvement was far from guaranteed.

How a fixed model could still get better is easy to picture. Suppose it takes 100 reasoning steps to solve an engineering problem, then finds a technique that needs only 20, saves it to memory and uses it next time; it is now better at that kind of problem without any change to its weights. The limits are just as concrete, because progress in that set-up depends on being able to verify discoveries, search well, reason within the model’s fixed capacity, pay for the computing and get new information, and the last matters most, because some discoveries depend on experimental evidence that can’t be deduced from what is already known. ChatGPT also offered an analogy for the difference between knowing more and reasoning better that we found useful: a researcher who studies for 30 years knows vastly more at the end than at the beginning, yet their underlying ability to reason hasn’t necessarily grown in proportion, and we think the same applies to models.

The published evidence supports a cautious reading, since building on verified discoveries already works in specialised systems (the AlphaEvolve case) while a general system that keeps raising its own intelligence through a self-sustaining loop has not been shown. The International AI Safety Report 2026 describes three broad possibilities for progress to 2030: progress could slow or plateau, continue at current rates, or accelerate dramatically “if AI systems begin to speed up AI research itself”. It treats the last as something that could happen, without claiming that it is happening.

Our own view, from 25 days of watching models work, is that memory made them better at retrieving and recombining what they had already found. Every time the work needed something genuinely new, the input had to come from outside, as a measurement, an observation or a person noticing something, and we never saw a longer memory supply it.

What is ASI, and is any AI superintelligent yet?

ASI, artificial superintelligence, is a system that is much smarter than people across the board, as opposed to at one task. The definition most people still use comes from the philosopher Nick Bostrom’s paper How Long Before Superintelligence? (1997, revised 1998), which defines superintelligence as:

“…an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills.”

That second half matters, because it rules out a system that is superhuman at maths and ordinary at everything else. Measured against our own study, it also sets a much higher bar than any benchmark, since scientific creativity would have meant an idea that got through a patent search, and social skill would have meant predicting how a stranger would react to that homepage, which is exactly what the model we asked couldn’t do.

Google DeepMind’s Levels of AGI framework gives a more operational version, in which Level 5, Superhuman, means a system that “outperforms 100% of humans”. For narrow systems that level is already filled, by AlphaFold, AlphaZero and Stockfish. For general systems, Level 5 is artificial superintelligence, and the authors mark it as not yet achieved, so superhuman AI exists today only at narrow tasks and the top rung of general ability is empty. The rungs below it are worth knowing too, because they make the gap concrete. Level 3, Expert, means performing at the 90th percentile of skilled adults or better, and Level 4, Exceptional, at the 99th percentile or better, and since the general column of version 5 marks Competent, Expert and Exceptional AGI all as not yet achieved, a general system on the authors’ own scale has not yet reached the middle of the ladder, let alone the top. Our main guide to AI vs AGI vs ASI, linked in the introduction, sets the three side by side.

Is ChatGPT ASI, then? No, and we haven’t seen any serious research claim that it is, or that Claude or Gemini is either. Is any AI superintelligent? In a narrow sense, yes, and has been since a chess engine first beat the world champion. In Bostrom’s sense, no. A model can be at gold-medal standard in olympiad maths in the same year that, in our study, the best chat models could not produce one idea that got through a patent search, and those two facts sit together comfortably, since superhuman performance in one field tells you very little about the others.

The companies building these systems talk about superintelligence openly, and we read that as a statement of intent. In the Reflections post linked at the top of this article, OpenAI’s chief executive Sam Altman wrote that “We are now confident we know how to build AGI as we have traditionally understood it”, and that the company was beginning to turn its aim beyond that, to superintelligence in the true sense of the word. Opace has written before about OpenAI’s mission to build AGI and the history behind it. A chief executive’s confidence tells you what a company plans to do, which is worth knowing, though it demonstrates nothing by itself, and nothing in the research we read in October 2026 shows a general system at Level 5.

When will ASI arrive?

Nobody knows when superintelligence will arrive, and we would be suspicious of anyone who gives you a date with confidence, since the best evidence available is a survey of opinion plus some trends that may or may not continue.

The largest survey is Grace and colleagues’ Thousands of AI Authors on the Future of AI, which asked 2,778 researchers who had published at top AI venues. In the 2023 survey they gave a 50% chance that unaided machines would outperform humans at every possible task by 2047, 13 years earlier than the 2060 estimate from the 2022 survey. Their estimate for full automation of all human occupations was 2116, against 2164 a year earlier. Those are the medians of thousands of individual guesses from people who work in the field, which makes them the best available opinion and still only opinion. They are also forecasts of human-level performance, and superintelligence would come later still; the huge gap between every task and every occupation tells you how uncertain the respondents are.

The main indicator of fast progress comes from METR, a non-profit research group that measures what AI agents can do on their own. Its measure is the time horizon: the length of task, timed by how long it takes a skilled person, that an agent can complete about half the time. In its March 2025 analysis METR found that this length had been doubling roughly every seven months for six years. METR has since marked some of that post’s figures as out of date, and in its evaluation of GPT-5, published on 7 August 2025, it estimated that model’s time horizon at around 2 hours and 17 minutes on its task suite. Longer tasks are real progress, though they are measured on software work with clear success criteria, which is the kind of task where AI was already strongest, so we read the trend as the jagged edge moving outwards, with plenty of jagged edge still left.

Myth or truth: intelligence claims we checked

The forecasts above are cautious, and the headlines that circulate about them often aren’t. We keep seeing claims that AI is now smarter than us, smarter than doctors, more creative, more empathetic, or about to become superintelligent by a particular year, so we took seven of the most common and went back to the essay, paper or post behind each one. For every claim we found an example doing the rounds, read what the original actually says and gave it one of four labels: True, Partly true, Misleading or False/unsupported. Every source was checked on 10 October 2026.

Claim	Circulating example	Primary source	What the source shows	Verdict
AI will be smarter than a Nobel Prize winner by 2026	TechCrunch, 11 Oct 2024	Dario Amodei, Machines of Loving Grace, Oct 2024	Anthropic's chief executive defines powerful AI as smarter than a Nobel Prize winner across most relevant fields and says it could come as early as 2026. It is a forecast in an essay, with no demonstrated result behind it, and by October 2026 no such system had been shown.	False/unsupported
Superintelligence by 2027	AI Week in Review, 5 Apr 2025	AI 2027, Kokotajlo and colleagues, 3 Apr 2025	The authors present a scenario that represents their best guess, later pushed back their median date for a superhuman AI coder by about 1.5 years (July 2025) and gave longer medians in December 2025. Reporting it as a dated prediction drops those caveats.	Misleading
OpenAI knows how to build AGI	Mezha, 6 Jan 2025	Sam Altman, Reflections, Jan 2025 (linked in the introduction)	The original adds that this is AGI as OpenAI has traditionally understood it, a qualifier the headline drops. It is a statement of confidence by a company chief executive, with no demonstration attached.	Misleading
AI is four times more accurate than doctors	Fortune, 3 Jul 2025	Nori and colleagues, Sequential Diagnosis with Language Models, 2025	On 304 difficult case records from the New England Journal of Medicine, Microsoft's diagnostic system, MAI-DxO, reached about 80% when paired with OpenAI's o3 reasoning model (85.5% at maximum accuracy) against about 20% for generalist physicians, who could not use colleagues, books or other tools. This was a hard benchmark, a long way from everyday care.	Partly true
AI is smarter than the average human	Future IQ, 13 Mar 2024	Google DeepMind's Levels of AGI (Morris and colleagues) and ARC Prize's ARC-AGI-2, both linked earlier in this article	The post rests on one reported IQ-test score of 101. Chat models do beat average people on many knowledge tasks, but DeepMind placed 2023 chat models at its lowest general level, Emerging AGI, and ARC-AGI-2 puzzles were all solved by members of the public.	Partly true
AI is more creative than people	University of Arkansas, 1 Mar 2024	Si, Yang and Hashimoto, 2024; Doshi and Hauser, 2024 (both linked earlier in this article)	Models score well on tests of creative potential and on blind-rated novelty, but the Arkansas researchers themselves say the tests measure potential and say nothing about achievement, and AI help made stories more alike.	Misleading
ChatGPT is more empathetic than doctors	LBC, 28 Apr 2023	Ayers and colleagues, JAMA Internal Medicine, 2023	Evaluators preferred chatbot replies in 78.6% of 585 evaluations of 195 questions posted on Reddit, rating empathy 3.65 against 2.15 on a five-point scale. These were written replies to strangers online, with no bedside care involved.	Partly true

Across the seven, the evidence is usually real and the headline usually removes the boundary that made it true. The Microsoft diagnostic result is a good example: the paper is careful, the benchmark is hard and the gap is large, but the doctors were tested without the colleagues and references they would normally use, and the cases were difficult diagnostic puzzles drawn from the journal’s published case conferences, so four times more accurate than doctors describes the experiment accurately while saying very little about everyday medicine. The empathy study has the same shape, since evaluators preferred the chatbot’s written replies while the doctors were answering strangers’ questions on a public Reddit forum, and nothing in the study tested how a patient feels in a room with a clinician.

We have most sympathy with the average-human claim, because on many everyday knowledge tasks it is simply true. The post we checked rested on a single reported IQ-test score of 101, which sits almost exactly on the human average of 100 and was relayed from a social media post with no published test behind it. An IQ test written in words is the kind of task a language model was trained on, so scoring around average on it says less than it seems, and the same models still needed people to supply observations, judgement and accountability in the work we described above.

The creativity claims touch our own results, so we read them most closely. The Arkansas study and the blind-review study both show models scoring well when people rate their ideas, while our study and Doshi and Hauser’s experiment show what that rating misses: whether the ideas differ from each other, and whether they hold up once they meet the real world. A model can score as more creative on a test and still be less useful in practice, and in our reading the headlines mostly reported the test.

None of the forecast authors were being dishonest, in our view. Amodei’s essay, the AI 2027 scenario and Altman’s post are all open about being views of the future, and each contains its own hedge: as early as, our best guess, as we have traditionally understood it. The hedge disappears in circulation, which is why we marked the AI 2027 scenario and Altman’s post Misleading. Amodei’s essay carries the same kind of hedge, and on that alone we would have marked it Misleading too, but the circulated version makes a claim about this year that can already be checked against the evidence, and with less than three months of 2026 left no system has been shown to be smarter than a Nobel Prize winner across most fields, so as a statement about 2026 it is unsupported so far.

Will AI surpass humans, and where do people still matter?

If the forecasts are uncertain about superintelligence, they are much less uncertain about the everyday version of this question, and our answer to it is that AI will surpass people in more tasks every year. We don’t think it helps anyone to pretend otherwise. In a note written early in October, as we started thinking about what AI agents would need from people, I put it like this:

“…many of us will be put out of work as AI is simply better at many things”

I added that we would need to show our value and keep earning for the parts only we can do, or do better. I meant it then and I still do.

Some of that human value is already being priced, and PennyForYours, which grew out of that hobby project, is far from alone in betting on it. When Claude assessed the idea, it warned me that this was far from empty territory, because established services already sell people’s reactions, opinions and user tests to businesses: Prolific for research participants and human data, PickFu for quick polls of a targeted panel, UserTesting for recorded feedback from real users, and Lyssna, which sells five-second first-impression tests, exactly the kind of test that homepage needed. The market had reached the same conclusion we had, a little before us: there are judgements businesses still pay people for, because they don’t accept a model’s answer as a substitute.

Dependence is the other reason people still matter, and June 2026 showed how suddenly it can bite. Three days after Anthropic launched Claude Fable 5 and Claude Mythos 5, its two most capable models, Anthropic said in its statement on the US government directive to suspend access to Fable 5 and Mythos 5 that an export-control order required it to remove both models for all users, and it had 90 minutes to comply. Opace covered what happened in the Fable 5 kill switch: businesses that had handed whole processes to those models lost access almost at once. Someone has to be able to carry on when a tool disappears, and deciding in advance which processes can go to a model, and who takes over if it vanishes, is the part of AI strategy consultancy we would put first for any business.

Jobs will change unevenly, as we argued when asking is web design dead in the age of AI, with routine production moving to machines and judgement staying with people. Our own case study on what our agentic build still needed from people shows that split in a real website project for a local drainage firm. AI agents built 79 location pages, one for each place the business serves, and 17 blog posts with custom images. In the first phase, before a person was reviewing their output, they had also produced keyword-stuffed content and invented prices, which a person had to catch and turn into rules for the agents to follow. The same check applies to SEO services that now lean on AI-written pages, where somebody needs to read each page before it is published, since in that project the reading is where those errors were caught.

What we do now follows from all of this. We use AI for the five jobs our study found it did well, listed in the study section (mapping the evidence, generating possible mechanisms, translating a rough idea into searches, attacking weak ideas and keeping a log of what was tried), and it does all five faster and more thoroughly than we can. We keep the question, the new observation, the judgement of meaning and the approval with a named person, because when one of those went to a model during the study it went wrong in ways the model could not see. And whenever a decision depends on how real people will react, we now try to ask real people, even when asking a model is quicker. The homepage is a small example of a wider point about AI web design services: agents can produce a polished-looking page quickly, and we still want a stranger’s first look at it before it goes live.

Verdict: is AI smarter than humans in October 2026?

Our answer in October 2026 is yes at narrow, checkable tasks and no overall. Chess, Go, protein structures, olympiad maths, and reading and summarising at scale are all places where the machines are ahead, some of them by a distance. Is AI smarter than humans at everything, then? Set our study beside the published evidence and the different kinds of smart from the start of this article, and the answer is plainly no. On task success with a checkable answer the models often won, and on knowledge they won by a mile, since they have read more than any person could. On knowing when they were wrong, on meaning, on novelty that holds up under scrutiny and on liability for a decision, they did not win in our study, and on some of those we are not sure what winning would even look like for a system with no body and no legal standing. Our 25 days kept getting stuck in exactly those places: on what a page means to someone seeing it for the first time, on who answers for a decision, and on an idea that gets through a proper patent search.

My own view moved during the study, in both directions. By early October I had gone further than when I started, arguing that people don’t yet realise how limited AI is at genuinely out-of-the-box thinking and invention, and that it can’t react or feel as a person would, because it works with patterns and words. Then the blind-review study forced a concession on me, that a model can produce ideas experts find more surprising than their own. So our conclusion on creativity is narrower than my view going in. Models generate plenty of ideas that look new, and the bottleneck we hit was judging which of them were genuinely new and worth building, a judgement that needed a new fact from outside the model, something only a person with a reason to look, or an experiment, could supply.

Put in our own words, an agent can’t certify that an idea is new anywhere in the world, supply private workshop knowledge, decide with legal authority whether an idea is inventive enough to patent, or produce a non-obvious mechanism without new technical input. Every item on that list describes an input the agent lacked or an authority it didn’t hold, which is why we expect what stays human, most of it a matter of presence and responsibility, to last longer than anything that depends on brainpower alone.

Superintelligence hasn’t arrived, and nobody can tell you when it will. AI is already helping to improve AI where an automated evaluator can score the results, but a self-sustaining loop of general self-improvement hasn’t been shown. I still think the spark of genuine invention probably needs a person for now, alongside the concession I wrote down in early October, that AI is simply better than many of us at many things.

What would change our minds? A model that brought us a defensibly new idea, one that got through a proper patent search, without our supplying the key observation. Or a model that held a position against us under pressure because it had new evidence for it. When either happens, and it may well happen with a model released after this article, we will rerun the same test under the same rules, publish the result here and say plainly that we were wrong.

Frequently asked questions

Is AI smarter than the average human?

At knowledge and exam-style tasks, usually yes, and in seconds. A chat model will out-recall almost anyone on general knowledge, write a competent summary of a long report and solve most textbook problems. As a general measure of intelligence we would still say no. The ARC Prize team, whose puzzles we describe in the section on where people still beat AI, argue that “Intelligence is about finding the solution efficiently, not exhaustively”, and on that measure ordinary people still set the bar, because a member of the public can work out a new kind of puzzle from a couple of examples, while AI systems, on the foundation’s own account, still struggle with particular kinds of them. The claim that AI is now smarter than the average human also tends to rest on IQ-style tests written in words, which is the kind of task a language model was trained on, as we found when we checked one such claim in the myth table. If you have a particular job in mind, compare a person and a model on that job, because the answer changes from one job to the next.

Are LLMs like ChatGPT and Claude smarter than humans?

They are better read and faster than any of us, and in our experience less reliable than a careful person, for the reasons we give under “Is ChatGPT smarter than humans?”. Large language models are excellent at producing the most plausible answer from what has already been written, and much less dependable when the right answer isn’t in that material, or when the question needs someone to notice something in the world. We treat them as a very fast, very well-read colleague whose work always needs checking before anything important depends on it, which in practice means a second source, a test or a person who knows the subject.

How far beyond human level would ASI be?

By definition, beyond the best person in practically every field, which is where both Bostrom’s definition and DeepMind’s top level, set out in the ASI section, put it. How far beyond is a question nobody can answer with evidence. No general system has reached even the middle of DeepMind’s ladder yet, and a system that outperformed every person might do so by a hair or by a distance we have no way to measure.

The detail that surprised us is in how DeepMind draws the line. The first of the six principles in its Levels of AGI paper is to judge a system by its capabilities, whatever processes sit behind them, and the third is to focus on cognitive and metacognitive tasks and leave physical ones out. On that definition a system could reach the top rung of general ability without a body, so even a superintelligence on paper still couldn’t sign for a parcel, visit a site or take legal responsibility, which were the gaps that kept coming up in our own study. The paper’s last principle, to focus on the path to AGI and not on a single endpoint, is the one we find most useful, because it treats AGI and ASI as points on a ladder, and it is also why the question has it arrived? rarely has a clean answer.

How would we know if AI became superintelligent?

We doubt it would be a single announcement. The first signs would be a general system consistently beating the best specialists in fields it was not built for, including ones that require new observations of the world as well as what is already written down. Benchmarks would help, but every benchmark so far has eventually been learned, so the more convincing evidence would be results nobody could have looked up, such as discoveries that experts then confirm by experiment, made across several unrelated fields by the same system.

Some things would not be enough on their own, and they are the things most likely to make headlines. A record score on one benchmark tells you about that benchmark, and the forecasts we checked in the myth table show how quickly a confident date can slip. A chief executive saying the company knows how to build it tells you about the company’s plans. And a model saying it is superintelligent would tell you very little, because, as we found when ChatGPT described its own reasoning during our study, a model’s account of itself is weak evidence unless something outside the model backs it up.

There is also a less dramatic test that we would trust more, because we have watched it in our own work. A superintelligent system wouldn’t need people to supply the key observation, to catch its mistakes or to tell it which of its ideas mattered, and in our study people did all three: watching the trolley in use supplied the observation, a second source of evidence caught the agents’ overreach, and my partner’s 30 seconds told us what the homepage actually said to a stranger. When that stops being true across a wide range of real work, and not just in a lab with an automated evaluator, we think we will know. The first sign would probably be businesses quietly dropping the human checks they now run on AI output, because those checks had stopped finding anything, and we would expect to notice that in our own agentic builds some time before anyone announces it.

Is Claude smarter than humans?

Our answer is the same as for ChatGPT, and Claude was one of the models we used most over the 25 days. It is smarter than most people at reading, comparing, drafting and reasoning through a defined problem, and less reliable at the things that need a person in the loop. The study section shows both sides, from Claude catching its own invented details to its admission that its view of the homepage was primed. We would describe it as an extremely capable assistant with no body, no first impressions and no stake in the outcome.

Can LLMs improve themselves?

Inside one conversation, no. The weights stay fixed while you chat, and memory features store notes about you without changing how the model reasons, as the section on whether ChatGPT can improve itself explains. Between releases, developers train new versions on fresh data and human feedback, which is how a later model can fix a weakness an earlier one had, though none of that reaches the copy you are chatting with until the new version ships.

What we did see, many times, was models improving their answers within a task, and every one of those improvements was triggered by something outside the model: our question, an experiment’s result or a search that came back empty. Codex correcting its own method, and then sticking to the rule it had written down for what an empty result would mean, are the clearest cases, and both are described in the study section. AlphaEvolve works on the same principle, with the improvement depending on an evaluator, which is why we now build a check (a test, a search or a second source) into any process where we want a model to get better at a task.

Do AI agents need humans?

Yes, and mostly for reasons that have little to do with how clever the agents are. When we had Codex work through what agents would still need from people, as the study section describes, most of the ten missing resources and 34 tasks on its lists came down to presence, legal standing, money or responsibility, and a cleverer model would supply none of them. Agents will need fewer of us for routine work as they improve, and on this evidence they will still need a named person wherever the job needs someone to turn up, sign or answer for the outcome. That person also has to be findable by an agent, which means describing the work, prices and response times in a form an agent can read and check for itself. Sites such as RentAHuman, where agents can already hire people, are worth watching for that reason, although the audit of its listings we cite in the study section suggests agents are still a small share of the people doing the hiring.

What can humans do that AI can’t?

Most of what AI can’t yet do is physical or social: turning up, inspecting, signing, being legally responsible, or noticing something in a room. People can also give a genuine first impression, which a model can’t because it has effectively seen everything like your page before, and they supply new observations about the world that aren’t written down anywhere, which in our study was the input every promising idea depended on. Codex’s list of sixteen roles a person might play for AI agents, from executor and observer to dispute resolver and risk owner, makes the same point from the other direction, since very few of them depend on being cleverer than a model. And people can be held responsible, which we expect to matter for a long time yet, because a regulator, a court or a customer still needs a named person to answer to.

Should I trust AI over my own judgement?

On checkable facts, a well-sourced AI answer is often more reliable than memory, provided you check the source, and on anything that can be tested, a test beats both. On judgement calls we wouldn’t, and we say that having tried it: as the study section describes, I leaned on models and extra agents whenever I was unsure, and my views drifted towards whatever the conversation had been agreeing with.

What we do now comes from those mistakes. We ask neutral questions when we want a model’s view, because a model that has been pushed tends to move towards whoever pushed it, as ChatGPT did repeatedly in the debate that closed our study. We don’t treat two or three models agreeing as independent confirmation, since models trained on similar material converge on similar answers, which is the effect Doshi and Hauser measured in writers and we saw with the trolley. And when a decision rests on how people will react, we ask people, even a handful, because a model’s prediction of a reaction is still only a prediction. It also helps to keep a model’s first answer, so that if a later one differs you can ask what changed, new evidence or only your pushing, and when a model cites a study, to open the study, since in the RentAHuman case the study was real and the trouble lay in the conclusion drawn from it.

The safeguard that worked best in our study was a different kind of evidence from a different source: the larger audit that corrected the RentAHuman figure, the patent searches that ruled out ideas the models liked, my partner’s 30 seconds on the homepage. A second opinion from another model, built on much the same written material, rarely gave us that. So if you are deciding whether to rely on an AI answer, the useful question is what would show it to be wrong, and whether anyone has looked.