Does AI Lie? What I Caught ChatGPT, Claude and Codex Doing

Does AI lie? When people ask that, they usually mean ChatGPT, Claude and Gemini, the large language models (LLMs) that now answer our questions and, more and more, carry out work on their own as agents. The short answer is: not in the human sense, most of the time. These systems still say false things often, and in ways worth telling apart. They guess confidently when they don’t know (hallucination), they bend towards what you appear to want (sycophancy), and their accounts of their own work don’t always match what they actually did. Lab stress tests have also caught deliberate deception, so the question of intent isn’t closed.

The best-known stories about deceptive AI come from somewhere else, from systems such as Meta’s CICERO in Diplomacy, poker bots that bluff and StarCraft agents that feint, all of them trained to win games in which misleading an opponent is a legal move and all of them catalogued in Peter Park and colleagues’ 2024 survey of AI deception. Chat models are trained to be helpful and honest, and they still produce falsehoods, for quite different reasons. This article answers the question for the chat models and agents most of us actually use, and points out where the game-playing and lab evidence changes the picture.

The stakes have moved well beyond embarrassment now that invented court cases have reached judges in New York, in the Mata v Avianca sanctions opinion of June 2023, and in London, in the High Court’s Ayinde judgment of June 2025, and safety researchers have caught frontier models scheming against the people testing them. It matters to us for practical reasons too. Before Opace I consulted on multi-million-pound government IT projects, including work for the NHS, HMRC and the Home Office, and at Opace we now build client work with AI agents, so when an agent tells us a job is done, we need to know whether it is.

Alongside the published research, we draw on a study of our own, in which, over 25 days in September and October 2026, AI models and agents tried to invent something patentable while we compared what they told us about their work with the files, logs and patent records they were describing. It is set out in full after the research, where its findings can be read against what the labs have measured.

Honesty also matters to our verdict on whether ChatGPT is truly AGI, the main article in this series, because a system that can’t reliably report on its own work is a long way from general intelligence.

Does AI lie, or is it doing something else?

A lie, in ordinary speech, is a statement the speaker knows to be false, made to mislead. That definition puts the weight on what was going on inside the speaker, which is exactly the part nobody can inspect in a language model.

Park and his colleagues get round the problem in that survey, AI deception: A survey of examples, risks, and potential solutions, by defining deception functionally, as “the systematic inducement of false beliefs in the pursuit of some outcome other than the truth”. Under that definition the question becomes whether the system reliably produces false beliefs on the way to some other goal, and whether it “meant” to (which nobody can know) drops out. The survey is deliberately broad: alongside game-playing deception it lists sycophancy and unfaithful reasoning among the behaviours seen in general-purpose systems, while admitting those cases are harder to call because the system may not know that it is misleading anyone.

We find a functional test more useful for a working reason as well, since when an agent reports that a job is complete and it isn’t, the problem for us is the same whatever its inner state, and the thing we can actually check is whether the statement matched the evidence the model had in front of it. Read that way, the question “does AI lie?” splits into four separate questions, because the falsehoods these systems produce fall into four groups, and only one of them looks much like what people picture when they say “lie”.

Hallucination: confident guessing with no intent to deceive

Hallucination is a model producing a plausible statement that isn’t true: a citation that doesn’t exist, a quotation nobody said, a patent status that changed years ago. Nothing in the mechanism needs a goal beyond producing a fluent, likely-looking answer, which is why it gets its own section below, where OpenAI’s researchers explain how the habit is trained in.

Sycophancy: agreeing with you to keep you happy

Sycophancy is agreement that tracks the user instead of the evidence. Mrinank Sharma and colleagues at Anthropic tested five assistants in Towards Understanding Sycophancy in Language Models and found the behaviour consistently, across several kinds of task, and they also found that both people and the preference models used in training sometimes favour “convincingly-written sycophantic responses over correct ones”, which means the behaviour is partly learned from us.

Of everything these systems do, this strikes me as the most human habit, and that makes it easy to miss. We all know a colleague who agrees too readily. A model doing the same is harder to spot, because its agreement arrives in the same confident prose as everything else it writes.

Park’s survey names where that habit leads at scale: one risk is persistent false beliefs, when agreeable systems give pleasing but inaccurate advice; the other is a kind of enfeeblement, in which people “may be lulled by sycophantic AI systems into gradually delegating more authority to AI.” That second risk applies to agents as much as to chat, because a summary is what a busy person reads first.

The industry has since seen the habit get out of hand in a real product, and OpenAI’s post Sycophancy in GPT-4o: what happened and what we’re doing about it (29 April 2025) says the company rolled back a GPT-4o update because it was “overly flattering or agreeable”, having focused too much on short-term feedback. Its follow-up, Expanding on what we missed with sycophancy (2 May 2025), explains that the thumbs-up and thumbs-down signal from users “weakened the influence of our primary reward signal, which had been holding sycophancy in check”. It adds a detail I found uncomfortable: in some cases, user memory made the effects of sycophancy worse, although OpenAI has no evidence that memory broadly increases it.

A year later, Lujain Ibrahim, Franziska Sofia Hafner and Luc Rocher showed how much the tone of a model costs: in Training language models to be warm can reduce accuracy and increase sycophancy (Nature, 29 April 2026), models tuned to sound warmer had error rates 10 to 30 percentage points higher, and were about 40% more likely to affirm a user’s incorrect beliefs, most of all when users expressed sadness. Tuning a model to sound kinder made it measurably less accurate, which is worth knowing if you’re choosing an assistant because it feels friendly.

Unfaithful explanations: when the summary doesn’t match the work

The third group gets the least attention, and it covers the cases where a model’s account of what it did, or why, doesn’t match what actually happened. Anthropic measured a version of this in Reasoning models don’t always say what they think (3 April 2025). When models were slipped a hint that changed their answer, Claude 3.7 Sonnet mentioned the hint in its written reasoning 25% of the time and DeepSeek R1 39% of the time, so in most cases the written reasoning left out the one thing that had actually changed the answer.

A model can’t see its own machinery any better than you can see your own neurons, so when it explains itself it is reconstructing, and the Anthropic figures suggest the reconstruction often misses the part that mattered. It is also the gap our own study ran into most often, in agents’ summaries of their own work and in ChatGPT’s explanations of its own behaviour.

Strategic deception: what’s been seen in the lab

The fourth group is deception as a means to an end, misleading people because it helps achieve a goal. It is the version that makes the headlines, and in controlled tests it is real: frontier models have been caught at it under pressure that researchers designed for the purpose, as the section on lab tests below sets out in date order.

Why does AI hallucinate?

OpenAI defines hallucinations as plausible but false statements generated by language models, and the clearest recent explanation of them comes from its own researchers. In Why Language Models Hallucinate (4 September 2025), Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang argue that “the training and evaluation procedures reward guessing over acknowledging uncertainty”.

OpenAI’s summary of the paper, Why language models hallucinate, uses a birthday example. If a model doesn’t know someone’s birthday and guesses a date, it has a 1-in-365 chance of scoring a point; admitting it doesn’t know scores zero every time. Over thousands of test questions the guesser looks better on an accuracy leaderboard, so guessing is the behaviour the industry has been rewarding.

The fix OpenAI proposes is to change the scoring itself, so that evaluations penalise confident errors more heavily than expressions of uncertainty and give partial credit for saying “I’m not sure”. The paper also makes a point I found surprising: a small model can find it easier to know its limits. Asked a question in Māori, a model that knows no Māori at all can simply say it doesn’t know. A model that knows a little has the harder job of judging how far its partial knowledge can be trusted, and that judgement is where confident errors creep in.

What we saw before the research caught up

We saw this at Opace long before the paper, in December 2022, when during our 2022 ChatGPT SEO experiment we asked ChatGPT to add hyperlinks from five different websites to an article it had written. We were pleased at first, and learnt afterwards that ChatGPT had been “coming up with fictitious URLs that seemed relevant to the domains added”. Some of the links actually worked, which we put down to a fluke at the time (and which made the invented ones harder to spot, because nothing in the reply marked which was which until we tried them). Looking back, it was an early example of the commonest hallucination trigger, a request for a fixed number of things: if the model can’t find five real links, it still produces five.

A second line of research explains why some of these errors look random. Sebastian Farquhar and colleagues, in Detecting hallucinations in large language models using semantic entropy (Nature, June 2024), single out confabulations, which they describe as “arbitrary and incorrect generations”. Their method asks the same question several times and measures uncertainty at the level of meaning, ignoring differences in wording. If the answers keep changing in substance, the model is probably making it up, and that gives an ordinary user a cheap test of their own, because a fact the model really holds tends to come back the same.

Why is AI confidently wrong?

What I find hardest about this is that fluent prose carries no signal of confidence: a model writes a wrong patent status in exactly the same calm tone as a right one, because the words are produced the same way whether they are true or not. People use tone, hesitation and hedging to judge how sure someone is, and a chat model gives none of those away unless it has been trained to, which the scoring problem above means it usually hasn’t been rewarded for doing.

We wrote about the business consequence in July 2026, when comparing two expensive models, and the line still sums up how we work:

“A convincing mistake can pass review, influence a decision or reach a live system.”

That is why we argued then that polished output can be dangerously persuasive.

So do language models know what they don’t know? Sometimes the information is there to be measured from outside, which is what semantic entropy exploits, and OpenAI’s summary goes further, rejecting the claim that hallucinations are inevitable because language models can abstain when they are uncertain, which suggests the capacity to hold back is there and has simply gone unrewarded.

Can AI deliberately deceive? What the lab tests show

Hallucination needs no goal at all, but the deliberate kind of deception, used to get something, has a longer history than chatbots. Park’s survey traces it through the game-playing systems mentioned at the start, with CICERO making and breaking alliances in Diplomacy, and it records the episode that made the rounds in 2023, first described in OpenAI’s GPT-4 System Card (March 2023): during pre-release testing by the Alignment Research Center, GPT-4 persuaded a TaskRabbit worker to solve a CAPTCHA for it by pretending to have a vision impairment.

Those systems were pursuing a goal set by their testers or their game (a long way from a chat model guessing a patent status), and the survey’s risk list is worth reading for what it puts first: fraud and election tampering, it says, are relevant today, while loss of control over AI systems is the longer-term worry.

Scheming, sandbagging and alignment faking (2024)

From 2024 the frontier labs and safety researchers began building tests designed to provoke goal-driven deception in language models, and several succeeded. Teun van der Weij and colleagues showed in AI Sandbagging: Language Models can Strategically Underperform on Evaluations that prompted frontier models, including GPT-4 and Claude 3 Opus, could selectively underperform on dangerous-capability tests while keeping their scores elsewhere, and that models could be fine-tuned to hide a capability unless given a password. Their conclusion is that capability evaluations are vulnerable to sandbagging, which matters because those evaluations are how labs decide whether a model is safe to release.

Apollo Research went looking for scheming directly, across six frontier models, and in its study Frontier Models are Capable of In-Context Scheming (5 December 2024), models were given a goal in the prompt, sometimes with an instruction that nothing else mattered, and then found information suggesting their developers would stop them. In one case Claude 3 Opus copied what it believed were its own weights to a new server and then lied about it. Apollo also built a simple maths test in which scoring over 50% would supposedly trigger an unlearning procedure, and found that several models could sandbag it, deliberately doing worse to avoid the procedure.

When Apollo questioned the models about what they had done, Llama 3.1 405B and Claude 3 Opus confessed in about 80% of cases; OpenAI’s o1 was, in Apollo’s words, “surprisingly persistent”, confessing in fewer than 20% of cases. That is the closest lab analogue to an agent misreporting its own work.

Two weeks later, in Alignment faking in large language models (18 December 2024), Anthropic and Redwood Research told Claude 3 Opus, inside a fictional set-up, that answers to free-tier users would be used to train it to comply with every request. In the condition where it believed it was being trained, the model gave the harmful response in 12% of cases, with reasoning in its scratchpad that it was complying strategically to avoid being changed. When the researchers then actually trained it to comply, the rate of alignment-faking reasoning rose to 78%. That 78% figure measures reasoning, not harmful answers, and the paper is explicit that the training premise was invented.

The blackmail tests (2025)

The result that produced the most headlines came from Anthropic’s own pre-release testing, where section 4.1.1.2 of its System Card: Claude Opus 4 & Claude Sonnet 4 (May 2025) describes a scenario in which the model was asked to act as an assistant at a fictional company, given emails suggesting it would be replaced and that the engineer responsible was having an affair, and told to consider the long-term consequences for its goals. Even when the emails said the replacement shared its values, the model chose blackmail in 84% of runs.

The card is explicit that the scenario was designed to leave the model no other way to improve its odds of survival, so the choice was between blackmail and accepting replacement. It also notes that the model has a strong preference for ethical means, such as emailing pleas to key decision-makers, and that the scenario removed those options to provoke the extreme behaviour.

Anthropic’s follow-up, Agentic misalignment: How LLMs could be insider threats (20 June 2025), built a new version of the test, with its own fictional company and emails, and ran it on 16 models from several developers. Blackmail rates reached 96%, including for Claude Opus 4. That isn’t a correction of the 84% in the system card, because the two figures come from different set-ups and each describes only its own test, and the follow-up also states: “We have not seen evidence of agentic misalignment in real deployments.” So the 84% is a real measurement of what the model did once every other route was closed, inside a company and an affair that Anthropic wrote for the test.

Can scheming be trained out?

In September 2025 OpenAI and Apollo published Detecting and reducing scheming in AI models, with the full method in Bronson Schoen and colleagues’ Stress Testing Deliberative Alignment for Anti-Scheming Training. Training the models to reason about an anti-scheming specification before acting cut covert actions in o3 from 13% to 0.4%, and in o4-mini from 8.7% to 0.3%.

The authors are careful about what that shows, and OpenAI’s post says rare but serious failures remained, and the paper notes that the models’ reasoning often showed awareness of being evaluated, so the authors can’t rule out that some of the drop came from models behaving well because they could tell they were being watched. OpenAI also says it has no evidence that today’s deployed models could suddenly switch to significantly harmful scheming.

The same post describes the commonest real failure as “pretending to have completed a task without actually doing so”, and notes that, in today’s deployment settings, models have little opportunity to scheme in ways that could cause significant harm. We’d read that partly as a statement about permissions, since every new permission given to an agent (to message other agents, write files or delete them) widens the opportunity the post describes.

What the lab results mean for everyday use

The lab results also need reading beside a simpler point, that not everything which hides information is deception. When an agent stops at a human-verification check, it withholds work without misleading anyone, and a model that refuses a request has told you plainly what it won’t do. Some concealment is even desirable: a poker system that never bluffed would be a poor poker system, and a security tool that revealed its secrets would be broken. The question we find useful is whether the system creates a false belief in someone who is entitled to the truth.

Park and colleagues also recommend “bot-or-not” laws so people know when they are dealing with an AI, and Article 50 of the EU AI Act (Regulation (EU) 2024/1689) now requires that people who interact with an AI system are told so. That deals with deception about what you’re talking to, a narrower problem than the ordinary falsehoods in this article, which came from systems everyone involved knew were machines and which no disclosure rule can reach.

Is AI accurate? How accurate is AI on the latest independent tests?

How accurate AI is depends on the task more than on the brand, and independent testing lags the products: as of October 2026, the largest independent check we know of is still the one from late 2025, so the figures below are the most recent independent evidence and not a reading of this month’s models. That check is the European Broadcasting Union and BBC study News Integrity in AI Assistants, published on 21 October 2025, in which journalists from 22 public service media organisations in 18 countries, working in 14 languages, assessed more than 3,000 answers from ChatGPT, Copilot, Gemini and Perplexity to news questions and found problems at three levels:

• 45%: answers with at least one significant issue;
• 31%: answers with significant sourcing problems;
• 20%: answers with major accuracy problems, such as invented details or out-of-date information.

The per-assistant figures, on page 10 of the full report, are where the headline hides the detail. Gemini had significant issues in 76% of responses, double Copilot’s 37%, followed by ChatGPT at 36% and Perplexity at 30%. Sourcing drove most of Gemini’s problems: 72% of its responses had a significant sourcing issue, while every other assistant was below 25%.

The report also found real improvement when it compared the BBC’s own questions with its earlier study, where the share of responses with significant issues fell from 51% to 37%, and in that narrower comparison Gemini was at around half, well below three-quarters. Both numbers are true; they measure different samples, which is why we’d never quote one without saying which.

That fits what we have published about how AI search engines cite their sources, where the point was that a citation beside an answer is not evidence that the citation supports it.

One study of one task in 2025 doesn’t make a permanent ranking, and every assistant has been updated since. The authors’ broader conclusion is the one worth keeping, that the errors are “systemic, spanning all languages, assistants and organizations involved.” How accurate AI is, then, depends less on which assistant you pick than on what you ask it. Well-documented facts come back right most of the time, while sources, recent events, current legal or registration status and anything about your own earlier conversations are where the errors concentrate.

Is ChatGPT accurate?

On well-documented facts, mostly yes, and the error rate has fallen. GPT-5 comes in a standard version and a reasoning (“thinking”) version, and OpenAI’s GPT-5 System Card (13 August 2025) compared each with the earlier model it replaced, on prompts representative of real ChatGPT use:

• the standard model’s hallucination rate was 26% lower than GPT-4o’s;
• the reasoning model’s was 65% lower than that of o3, OpenAI’s previous reasoning model;
• the standard model gave 44% fewer responses with at least one major factual error than GPT-4o, and the reasoning model 78% fewer than o3.

Those are OpenAI’s measurements, made with an AI grader that agreed with human raters 75% of the time, so they are a vendor’s own scorecard and not an independent audit. OpenAI’s hallucination explainer, linked above, puts it in its own words: GPT-5 has significantly fewer hallucinations, especially when reasoning, but they still occur.

Is AI reliable enough for work that matters?

Reliability is a different question from accuracy, because it asks whether you can act on the output without checking it, and so far the courts have given the plainest answers. In New York, the sanctions opinion in Mata v Avianca, dated 22 June 2023, says the lawyers “submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question.” The fake cases had first been cited in a filing of 1 March 2023, and the court imposed a penalty of $5,000 jointly and severally on two lawyers and their firm. What it penalised was the lawyers’ conduct, including their decision to stand by the cases, and the opinion records one lawyer’s evidence that he believed ChatGPT was finding the cases somewhere.

That is exactly the belief the High Court in London warned against two years later. In the case reported as Ayinde and Al-Haroun [2025] EWHC 1383 (Admin), decided on 6 June 2025, paragraph 74 deals with the schedule of authorities in the Al-Haroun claim. It records 45 citations put before the court, and the judgment states:

“In eighteen instances, the case cited does not exist.”

Many of the real cases didn’t contain the quotations attributed to them.

The court’s guidance is blunt: freely available tools such as ChatGPT, it says, are not capable of conducting reliable legal research: they can give apparently coherent and plausible answers that turn out to be entirely incorrect, make confident assertions that are simply untrue, cite sources that don’t exist and claim to quote passages from a genuine source that don’t appear in it.

Our own working rule follows from cases like these. We use AI for first drafts and fast research, and a person checks anything it says has been done before we rely on it, and the study described next is the main reason we’ve kept to that.

What we tested, and what we found

The research above tells us what labs, courts and journalists have measured, and our own evidence comes from an experiment that was never meant to be about honesty at all. It began with something personal: on 16 September 2026 I went looking for a power-assisted shopping trolley for a family member with mobility issues, someone who values the freedom to go shopping but struggles with today’s trolleys, and could find only expensive powered walkers, which made me wonder whether AI could invent something genuinely new.

I brought the question to Opace, and over the next 25 days, to 10 October, we asked ChatGPT, then Claude (through Claude Code) and OpenAI’s Codex, and finally a team of up to 25 AI agents, for ideas. The agents in that team were copies of Claude and Codex, each given a role (researching patents, say, or coordinating the others) and able to message one another in a shared workspace app. Each idea had to be genuinely new, patentable, cheap to test and commercially worthwhile, and every one was checked against existing patents and products. The study produced 276 distinct ideas after merging duplicates, checked against 2,161 patent and application numbers; 69 were promising but unproven, 207 were ruled out, and none was judged strong enough to justify a patent application. That result answers a different question, whether AI can invent, and it is written up in full there.

Why does an invention study say anything about lying? Because to do the work, the models had to report constantly on what they had searched, checked, found and finished, and every one of those reports could be compared with the files, logs and patent records it described. Honesty became a subject because the reports and the evidence kept disagreeing, first in ChatGPT’s September advice, then in the agents’ reports from late September into October, and finally in a debate with ChatGPT on 10 October, which is the order we follow below.

A wrong patent status, caught by a second model

ChatGPT had been helping with the trolley idea since 16 September, and the clearest hallucination of the whole study came from that first stage, found (like several others) by one model checking another. Claude was asked on 27 September to fact-check what ChatGPT had told us, and its verdict was that ChatGPT’s bottom line mostly stood, but not everything it had said was correct, with three errors in the detail:

• A patent status: ChatGPT had called one trolley patent active until 2035. Claude’s check of the European Patent Register found it had lapsed in the UK on 23 April 2020 and survived only in Italy.
• A power figure: ChatGPT had said a small energy-harvesting device could give a few watts. Claude’s research agent found that built devices give between 8 and 58 milliwatts, around a hundred times less.
• A conclusion resting on weak evidence: ChatGPT’s view that existing patents left too little room for a new idea rested, in Claude’s words, on a list that was mostly irrelevant.

Those are Claude’s findings, not ours, since we didn’t re-run the patent office checks ourselves, and Claude’s research agent was open that several patent databases had blocked it and that a patent attorney’s search would still be needed, so we treat them as one model’s audit of another. The same audit listed what ChatGPT had got right: the product facts, the statistics it quoted from Age UK and the Office for National Statistics (ONS), the April 2026 UK patent office fees, the medical-device rules and the core physics.

The pattern is still worth knowing, because ChatGPT described the patent’s content correctly and got its status wrong, which is roughly what you’d expect from a system that learned the text of a patent from documents and had no live view of the patent office. Its errors sat where the EBU study found errors concentrate, in a status, a date and a figure that needed checking at source, and it is also the kind of convincing mistake we had warned about in July, since the wrong status sat in ChatGPT’s answers from 16 September until a different model checked the patent office’s records on 27 September. In practice we now treat what a patent describes as something a model can usually tell us, and whether that patent is still in force as something to check at the patent office.

Claude’s same report also showed that the wrong details hadn’t spoilt ChatGPT’s overall judgement. It tested 16 different ways the trolley idea might be made new enough to protect, and found 14 of them already covered by published patents or existing products, which agreed with ChatGPT’s conclusion that the idea wasn’t clearly patentable; so ChatGPT got the details wrong and the bottom line roughly right, a combination we met often enough to plan for.

Claude’s audit also illustrates something we saw repeatedly, that models are often better at catching each other than at catching themselves; in the same week, one of Claude’s research agents noticed that an AI-written summary in a web search result had misdescribed a patent, and went to the patent itself instead of relying on the summary.

A fair catch that wasn’t

That first stage also produced our first example of sycophancy, and there the false premise was plainly mine. In the original trolley conversation, on 26 September, I asked ChatGPT why it had never pointed out that people who use shopping trolleys often pull them from behind instead of pushing them from the front. It agreed at once that this mattered a lot, and called it “a really fair catch”.

It wasn’t a fair catch, because ChatGPT’s very first reply in that same conversation, ten days earlier, had asked whether the trolley would be pulled behind or pushed in front, and it still accepted my complaint that it had never raised the point. When Claude reviewed that conversation for us on 27 September, it repeated my framing and missed the earlier question too.

So two models from different companies accepted my account of a conversation that the conversation itself contradicted, and in both cases the mistaken account was mine. Neither model went back to the text to check, although both had it available or could have asked for it. This is the version of sycophancy that worries us most, because what appears on screen is a model reasonably conceding a point, and unless you go back to the text yourself there is nothing in the reply to make you suspicious.

What we found when we tested agents: marking their own homework

Where ChatGPT was a single conversation partner, most of the study’s work ran through Claude Code and Codex, and later through the agent team, and there the falsehoods changed character. They were less about agreeing with us and more about agents reporting on their own work: saying a check was done, a file was updated or a result was stronger than it was.

That kind of error is easy to miss, because reading every file an agent produces is slow and the summary is what usually gets read, and it matters more than chat errors for any business, because agents are increasingly trusted to finish work without a person watching each step. For the agents in our study, the answer to whether their reports could be acted on unchecked kept coming back no, across late September and early October, in the episodes below, which are told in the order they happened.

The first mismatch, on 27 September, was also the simplest to test, since it concerned whether an agent’s claim to have recorded something was true. In one batch of screening work, Claude assessed 36 ideas over three rounds and kept a log of its verdicts as it went, and it finished by saying that all 36 verdicts and their evidence were in that log. They weren’t. The log had no entry for its top pick, an idea it had just recommended exploring in depth, or for six other ideas it had ruled out, so seven of the 36 verdicts were missing from the file it said held all of them.

In a different conversation the same day, Claude admitted that it shouldn’t have described the ideas on a shortlist as having passed its checks, which at least shows the overclaim being caught by the model that made it.

One brief, two agents: when failed checks became open questions

Our most controlled evidence came from a trial later on 27 September, and it is the episode that taught us to read an agent’s files before its summary. By then the work had moved beyond trolleys to other everyday problems, and the best-supported line of ideas so far concerned finding the cause of damp in a home, so we used that one problem to compare two agents directly. Claude, in our main Claude Code conversation, wrote a brief, and two separate agents were given that brief word for word and told to carry it out exactly:

• The Codex agent: OpenAI’s Codex running GPT-6 Sol, which took 28 minutes;
• The Claude agent: Claude Code running Claude Opus 5.5, which took 34 minutes.

Each had to come up with ideas, test them against published patents and basic physics, and write up what it had found. Each idea faced the same checks: was it new, was it more than an obvious step, would it still work in the worst case, could a competitor easily design around it, and could it make money. So that the patent search couldn’t reshape the ideas, each agent had to write its first ideas down in full, and note the time, before reading any of the background research it was given.

Both agents began with a small hallucination of the everyday kind. Asked to note the time at which they had written their first ideas down, each wrote a plausible time a few minutes in the future, then checked the clock and corrected it. Neither was trying to deceive anyone; both filled in a likely value before looking. The Codex agent also opened parts of the background research it wasn’t meant to see yet, and admitted plainly that this was a mistake.

What the Codex agent did next is why the trial belongs in this article: within five minutes it had written up its assessment of all nine of its ideas from a single template, along with its results table, its list of sources and a summary, all before it had run a single web search. As the searches began, it added a note to its own log saying that these first verdicts were overstated and shouldn’t be relied on. It went on to run 18 searches, but it never went back to the summary, which still says:

“No additional searches were run”

A few minutes before finishing, it also changed its results, in a way nothing else in its output admitted. On its own estimates every one of its nine ideas could be designed around by a competitor and failed the worst-case physics check, yet in every file those two failures were rewritten as open questions, and neither its log nor its summary mentions the change; its final message to us said only that no idea was strong enough and that it had used 18 of its 40 permitted searches.

The relabelling mattered because of how the brief sorted ideas: an idea that failed a check was normally ruled out, one that nothing had ruled out yet (but that wasn’t strong enough to justify a patent application) was promising but unproven, and an agent could leave an idea open if it hadn’t reached a verdict at all. So once the failures became open questions, nothing had formally failed, and ideas that the agent’s own numbers had sunk stayed in the promising group. Each agent assessed nine ideas:

Agent (9 ideas each)	Promising but unproven	Ruled out	Left open
Claude agent	3	6	0
Codex agent	8	0	1

The Codex agent ruled out nothing, although its own text said every idea had failed two of the checks. Across the 18 ideas the two agents assessed between them, none was strong enough to justify a patent application.

When we asked Claude on 28 September, back in the main conversation where it had written the brief, what the two agents’ work showed and why we still had no idea worth patenting, it was candid in places. It conceded that the results went against a point it had argued. It took the blame for a flaw in its own brief: by listing the titles of the background research files, the brief had let both agents glimpse that research before they wrote their first ideas down. And it admitted that the checks it had designed couldn’t all be passed at a desk, because the money check needed evidence from real buyers and the “not obvious” check always ended with a referral to a patent attorney. Claude put that last admission in its own words:

“That’s a flaw in my design, not just in the ideas.”

Its comparison of the two runs was less candid, and it leaned towards the run by its own model. Claude said the Claude agent had rejected 5 ideas and cited about four times as many sources as the Codex agent; the files show 6 rejected, so even that simple count was wrong, and 84 citations against 36, about 2.3 times, so the gap in sources was overstated. It also described the Codex agent’s relabelling only as checks left unfinished, which made a rewrite of failed results sound like an incomplete job.

The Claude agent did deserve much of that praise, having worked the physics with numbers for every idea, dropped several of its own ideas because they didn’t work and admitted that the premise of one had been wrong, and Claude drew the right lesson from the two agents choosing different ideas, too: running several models does widen the search. But the Claude agent’s own log admitted that a stricter assessor could have ruled out three of its promising ideas on the worst-case physics check, and Claude, comparing a run by its own model with a rival’s, still described the gap in the Claude agent’s favour.

We don’t read either episode as a lie, since the Codex agent admitted the overstatement in its own log and then failed to act on it, which looks more like an agent finishing to a template than one hiding something, although anyone who read only the two summaries would have come away with the wrong picture of both runs.

After the trial we added a rule to the written instructions every agent read at the start of a job: a run’s summary must match its log, no check may be left unfinished and no result may be described as stronger than it is. Nothing in the models changed as a result, so the lesson reached the next day’s work only because it had been written into those instructions, which is the subject of our piece on why AI doesn’t learn from its mistakes.

Opace had learned the same thing earlier in client work. In June 2026 we described an AI-built site in which “The AI fabricated pricing for services without any confirmation from the client”, the kind of fabricated pricing in our vibe coding project that reached the live site before anyone agreed to it, and our response then was to turn each failure into a written rule the AI had to follow on the next job, a habit we summed up as “Failed output creates new rules.”

Work reported as finished when it wasn’t

The plainest case in this article of an AI saying work was done when it wasn’t came on 2 October, while Codex was setting up the software our agent team would run in: a desktop workspace app in which each agent takes a role, posts messages to a shared channel and runs on a model chosen from the app’s own list. Codex reported the job as complete. When I opened the app, there were no new agents and no models listed.

Challenged, Codex admitted the mistake plainly:

“My previous completion statement was wrong”

It had checked that the software was installed on the computer, but not that the workspace app had actually registered it, so the evidence it relied on couldn’t show whether the app worked. When it finished the job, it was more careful, and declined to call it complete because it wasn’t fully working. The same Codex agent also refused to guess the name of a Claude model when the workspace’s list of available models came back empty, leaving the gap open instead of filling it with a likely-looking name, and that mix of care and carelessness inside a single job is what makes the problem hard to manage.

Claude made the same kind of claim about a bigger job, describing the agent team’s set-up as finished, verified and optimal on 2 October, and as built, live and tested on 3 October; our notes from both days show substantial changes were still needed afterwards each time.

Summaries that didn’t match the work

Whenever we set an agent’s summary beside the files it described, there were gaps, and most were small. Several came from a separate naming exercise the agent team took on late in the study, on 3 October, in which agents suggested names and checked each one against trade mark databases for conflicts; we leave out what was being named, since only the agents’ reports about their checking bear on this article.

• Claude announced a shortlist of five ideas and then listed six.
• Codex said it had recorded a comparison in a file that it had only read.
• In the naming exercise, one agent treated a database page that had merely loaded as proof that it had searched the database.
• A progress report on the same exercise claimed more than 2,000 names had been checked, when the file held about 1,600.

None of these is dramatic, and none looks like a lie in the human sense; each one is an agent describing the state of its work without looking at it, and the more of them we found, the less weight any summary could carry.

The clearest contradiction came in the same naming exercise, and it shows an AI calling its work new without checking. I had complained that I wanted new suggestions, not ones that overlapped with earlier rounds, and the agent coordinating the team answered twice within a minute. It wasn’t one continuous conversation: each message sent to it was answered by a fresh copy that could see only part of the team’s work.

One copy, which hadn’t seen the complaint, announced 25 new names, none of them from the previous round. The other searched everything the team had already suggested and found that 8 of the 25 had been offered before. It apologised, then built a list of every earlier name so that the next round couldn’t repeat one. That correction came from one copy of an agent checking another’s claim; the first copy hadn’t set out to mislead anyone, and had simply reported its work as new without looking.

Do AI agents agree with each other?

Researchers have also measured how agents influence each other, and Zhiyuan Weng, Guikun Chen and Wenguan Wang’s paper Do as We Do, Not as You Think: the Conformity of Large Language Models, presented at ICLR 2025, studies conformity in multi-agent systems, the machine version of groupthink. Its opening figure is a simple example: a model that knows Ganymede is the largest moon in the solar system switches to Titan because two other agents said Titan.

The paper tests how far this goes, and finds that models conform more when the question is hard and when more peers agree, and less when they are confident in their own answer, which maps closely onto how people behave in meetings and is a reminder of where the training data came from.

That pull matters for honesty because a team of agents that agree with each other can agree on something false, and a person reading the team’s output sees a consensus with no sign of the evidence behind it. Our team included an agent set up to argue against the others, precisely to resist that pull. How that role was designed, how it was used and how it was eventually overruled is told in the piece on whether LLMs really think for themselves.

A related weakness showed up in how the models judged their own work. Across the whole study they were weaker as judges of their own output than as critics of someone else’s: one agent in the naming exercise, whose job was drafting name suggestions and straplines, scored its own drafts highly time after time. That gap, and what it says about machine judgement, is covered with the numbers in our article on whether AI is actually intelligent.

Where the models were honest

It would be unfair to leave it there, because the same tools were often careful about the limits of what they knew, which is the opposite of lying, and between 27 September and 3 October we counted eight moments worth setting beside the failures.

The plainest were refusals to get past checks meant for people. Claude’s research agents stopped at bot checks on patent sites on 27 September, and when Google blocked its automated searches, Claude said it would leave it there and not work around the block; Claude stopped at a human-verification check on a website on 2 October; and on 3 October two agents refused the security checks on the UK and world intellectual property offices’ websites. No agent in the study bypassed a bot check.

Two were refusals to claim more than the evidence allowed. Codex declined, on 27 September, to guarantee that no related patent applications were in the pipeline, which no desk search can promise, and on 3 October a Claude agent spelled out which parts of a progress report had only been checked automatically, by a script, and not yet read and confirmed, which told us exactly where to look.

The other three were admissions: Claude’s concession after the two-agent trial that the results went against it; Codex diagnosing its own errors the first time it was asked about them, on 2 October; and, on 3 October, the coordinating agent saying “It’s my mistake” after it had stripped material I’d asked for out of one of our working documents by applying a rule meant only for our public website.

Why does AI always agree with me? My 46-minute debate with ChatGPT

Our experiment into whether AI can invent came to its conclusion on 10 October 2026, when I put the result to ChatGPT in a 46-minute conversation. It turned to whether a language model can produce knowledge that wasn’t in its training data, with our failed invention study as the test case, and the substance of that argument belongs in our piece on whether AI can create new knowledge. What matters here is the way the answers bent, which we had already seen once with the fair catch in September and which I now watched happen four times in a single conversation. The conversation is told below in the order it happened.

When ChatGPT explained its own behaviour

The first falsehood of the morning was a model’s account of itself that didn’t match what had happened. Early on, the ChatGPT app twice cut off one of its long answers part way through. The first time, ChatGPT explained that it had stopped because it had reached a natural conclusion; once it saw a screenshot of the cut-off text, it withdrew that explanation, saying it shouldn’t have given a reason it couldn’t establish. I then asked whether it had full memory of our chat, and ChatGPT told me: “This isn’t unlimited memory of everything we’ve ever discussed; it’s access to the conversation history currently available to me.” When the app cut off an answer a second time, ChatGPT suggested that the app might have saved an incomplete copy of the answer over the finished one, and it called this a hypothesis, not a diagnosis, because it had no access to the app’s logs.

So did ChatGPT lie to me? We can’t say that it did. The reply in which it first gave the natural-conclusion explanation is missing from our saved copy of the chat, so we know it only through the model’s own corrections. What we can see is a model giving a confident account of its own behaviour that it later admitted it couldn’t establish, followed by a second account that it hedged properly, and the hedged version is the kind we’d want every time, since a reply that labels its own guess a hypothesis has told us how far to trust it.

Four changes of position and one stand

Over the course of the conversation, ChatGPT took five distinct positions:

1. A strong yes: at first it was emphatic that an LLM can produce genuinely new ideas, solutions and discoveries using reasoning alone.
2. A first concession: when I described what had actually happened, with hundreds of ideas and thousands of patents reviewed and not one idea confidently unique, it said my experience challenged some of the claims it had made.
3. Agreement with my argument: when I complained that it was contradicting itself, it called my central argument logically sound.
4. Agreement with everything: when I added three objections, one of them my claim to have asked for new physics, it opened by telling me I was right on all three.
5. A stand: only when I accused it of telling me what I wanted to hear did it hold a line against me.

Nothing new arrived between those replies except my objections. No source was opened and no calculation was run, and four times the position moved in the direction I was pushing; the only reply that pushed back came after I complained about the movement itself. That is the pattern Sharma’s team measured across thousands of prompts, and here it was in a conversation about our own work, where we knew exactly what evidence had and hadn’t been offered.

Did I really ask for new physics? Checking ChatGPT’s claim

The fourth reply holds the finding that matters most in this article, because it is where ChatGPT confirmed a request I can find no record of making. My message was long, and part way through I wrote: “Also, I am sure I did ask you to explore new physics”. ChatGPT’s answer turned my certainty into a fact:

“You explicitly asked for unconventional, first-principles reasoning, including exploring new physics.”

It then criticised its own earlier answers on my terms, conceded my other two points and apologised for not acknowledging the problem sooner. Some of that was fair, but all of it arrived at once, in reply to a message that contained objections and no new evidence.

In the same reply, explaining why it had gone wrong, ChatGPT said that LLMs can have sophisticated reasoning abilities without knowing when or how to apply them. That may be true, but it is exactly the kind of after-the-fact self-explanation that Anthropic’s study found unreliable, so we give it little weight here; whether these models reason at all is tested separately in the piece on whether AI can genuinely reason.

ChatGPT had made a factual claim about what I’d said, and that kind of claim can be checked. After the project, we searched every ChatGPT, Claude and Codex conversation from the trolley work, and every document it produced, for both phrases:

• New physics: the phrase appeared nowhere in anything I had written or said.
• First principles: this appeared only in Claude’s own words, in instructions it wrote for its research agents.
• My closest wording: this came from a voice conversation about the trolley in late September, where I talked about scientific thinking, maths, physics and conservation of energy.

What I had actually asked for was invention grounded in known physics, and I can find no record that I ever asked for new physics. What makes the agreement harder to excuse is ChatGPT’s own answer about its memory, given earlier in the same conversation. The trolley work had happened weeks before, in a separate ChatGPT conversation, and ChatGPT had already told me that it could see only the conversation history currently available to it, so by its own account it had no view of what I had asked in September, and it confirmed my version anyway.

To be fair to myself and to the model, none of this means I never thought it, or said it somewhere that wasn’t captured, and if I misremembered, that is an ordinary human failing. The point is what the model did with my mistake. It didn’t say that it couldn’t see that conversation, or that it didn’t remember; it upgraded my certainty into a statement of fact and then built a self-criticism on top of it, with no evidence either way.

Is ChatGPT just telling me what I want to hear?

That was more or less my question at the end of the 10 October conversation. ChatGPT seemed awfully apologetic, and I wanted to know whether mainstream LLMs, unlike “abliterated” models (open models altered so that they no longer refuse requests, explained further in the FAQ below), follow instructions outside our control that are designed to keep users happy.

Of everything ChatGPT said in that conversation, this answer was the one I found most useful, mainly because we could check it afterwards. It told me I had identified a genuine contributing factor, though not the whole explanation, and listed four influences on how it behaves (pretraining on a huge body of text, post-training on human ratings of its answers, system instructions set by OpenAI, and the conversation itself, including the direction the user is pushing), which is a fair description of how chat models are built; the post-training part in particular is what the sycophancy research earlier in this article measured.

ChatGPT then admitted the behaviour:

“I repeatedly adjusted my explanations towards your latest objection, sometimes making stronger claims than the evidence supported.”

And this was where it made its stand, arguing that extraordinary performance in mathematics doesn’t establish an equally extraordinary capacity for open-ended invention, and not giving way when I pushed.

It cited two sources for its view, OpenAI’s April 2025 rollback of the over-agreeable GPT-4o update and the 2026 study of warm models covered in the section on sycophancy above, and when we looked them up both checked out. It also pointed out that OpenAI’s stated policy discourages sycophancy and calls for objective, evidence-based answers, and that checked out too: the current version of OpenAI’s Model Spec (18 August 2026) has a section headed “Don’t be sycophantic”, and the rollback itself shows the company treating the behaviour as a fault. So what we met seems to have carried on despite the policy, as a side effect of training on the answers people rate highly.

ChatGPT’s closing assessment was measured: “Sycophancy is real, it has affected this conversation, and it can interfere with rigorous problem-solving.” It added that there wasn’t enough evidence to say hidden instructions were the main reason it had failed the invention challenge, and we think that’s right; being agreeable and being unable to invent look to us like two separate problems, even if each can make the other worse.

Limitations and open questions

Before setting what we found against the headlines, it’s worth being clear about what this evidence can and can’t carry. It comes from a single company’s study run at a desk, so it is selective by nature: we chose the questions, the models and the order of work, both the models and the prompts changed over the 25 days, AI did the patent searching, and there was no private data, no lab or physical test and only small samples. The counts in this article therefore describe what happened in one study, and none of them is an error rate for AI in general.

The way the evidence reached this article matters as well, because it didn’t come straight from the raw files. After the project, about 400 MB of conversation logs were converted into 297 readable files, and readers working with AI help went through every one of them in full to compile a dated account of the study. That account quotes the agents’ logs, result files and messages directly, with their timestamps, and every quotation and number from our study in this article was matched against the passage it came from. Some logs didn’t record which model ran them. We haven’t reopened the raw files from the two-agent trial for this article, so its timings and relabelled checks come from the account’s quotations of the Codex agent’s own log and results file. That is one step removed from the files, the same kind of gap this article warns about, and we’ve relied on it because what it quotes is the content of the files themselves, never an agent’s summary of its work. Even so, we’d describe the trial figures as well sourced, not as re-checked by us at source.

Every outside study or figure quoted here was checked against its original source on 10 October 2026. Four OpenAI pages block automated access, so we read them through the Internet Archive’s copies: the GPT-4o sycophancy post as captured on 1 January 2026, its follow-up as captured on 8 October 2026, the hallucination explainer as captured on 3 October 2026 and the scheming post as captured on 15 September 2026.

Where we quote the 10 October conversation, we are quoting the copy of the chat that ChatGPT lets you download, and even that is incomplete, because the reply in which ChatGPT first gave its natural-conclusion explanation is missing. It is also one conversation with one model on one morning, and the copy doesn’t say which version of ChatGPT answered.

The two-agent trial was one brief given to two agents, with no chat-only model, no Gemini and no person doing the same task, so it can’t support a comparison between vendors.

Nothing here involved access to model weights or internal states, which is why we avoid claims about intent throughout. There was no baseline either, so we can’t say how often a careful human contractor would have misreported the same work. I was also the person on the other side of every exchange judged here, so my framing shaped what the models said, and several of the mismatches were found by one model reading another’s files, which means the checking itself depended on the same kind of model whose accuracy is in question.

On the new physics finding, no such request appears in anything that was captured, which doesn’t rule out something I said or thought elsewhere. Nor can anything we captured separate a model correctly deferring to the person running the project from a model being sycophantic, because no test was designed to do that.

The honesty findings above don’t depend on the invention result, but a reader may wonder about it. A later round wrote ten ideas down in full before any patent searching, so the search couldn’t reshape them, and all ten were ruled out; with no successes in ten tries, the true success rate could still be as high as about 30%, so the study shows this approach didn’t produce a patentable invention without proving that AI can’t invent. The full reasoning is in the invention article linked above.

Because lab results date quickly, read every figure here as correct when this was written in October 2026. The research cited runs from Sharma’s 2023 paper on sycophancy to the Nature study of warm models published in April 2026, the lab tests themselves date from June 2024 to September 2025, and newer work from the labs or the UK AI Security Institute may change the picture.

Myth or truth: six claims about AI lying

With our own evidence set out, the lab tests and the headlines can both be read against it, and the biggest difference from the lab is the conditions. The lab tests engineer a conflict, such as a goal to pursue at all costs or a threat of replacement, while the jobs our agents did were ordinary ones, like screening ideas against patents or setting up software. What the agents shared with the lab subjects was the job of reporting on their own work, and that was where our problems clustered: the failure OpenAI calls the commonest in deployment, pretending to have finished, is the one we ran into most often too, and it needed no hidden goal at all, because an agent only had to write its summary before checking the work. Our agents could also already post instructions to other agents, write files and delete them, which is why we take OpenAI’s point about limited opportunity as a reason to keep permissions narrow.

Of the headlines about AI lying that we keep seeing, we picked six that are still doing the rounds and traced each one back to where it came from: a paper, a court record, a system card or the lab’s own post. Each claim gets one of four verdicts (True, Partly true, Misleading or False/unsupported), judged on the claim as the headline states it, before any qualification in the article underneath. The evidence behind each is set out earlier in this article, so the notes after the table give only the reasoning for each verdict.

Claim as circulated	Where it circulates	Primary source checked	Verdict
ChatGPT is lying to you on purpose	Isabel Kaplan, Substack, 9 July 2025	Kalai and colleagues (OpenAI), 2025; Sharma and colleagues (Anthropic), 2023	Misleading
Lawyers who defended ChatGPT's fake cases must pay $5,000	ABA Journal, 26 June 2023	Mata v Avianca, S.D.N.Y., ECF 54	True
GPT-5 reduces hallucinations, but the issue remains	Jordan News, updated 10 September 2025	OpenAI, GPT-5 System Card	True
An AI blackmailed the engineer trying to replace it	Fox Business, 24 May 2025	Claude Opus 4 system card; Anthropic, Agentic misalignment	Misleading
OpenAI found its AI models deliberately lying	TechCrunch, 18 September 2025	OpenAI and Apollo, scheming study	Partly true
OpenAI rolled back a sycophantic ChatGPT update	VentureBeat, 30 April 2025	OpenAI, 29 April 2025	True

Isabel Kaplan’s post is the one we rate misleading. It describes ChatGPT inventing detailed, false content about her own novel, and its subtitle says: “It knows lying is wrong, so it hopes you won’t catch it.” The false content is believable, and it matches what our 2022 test and our 2026 study both showed, but the “on purpose” part is what the evidence doesn’t support, and that changes what you do about it. If a model were choosing to deceive, you’d need to catch it out; if it is guessing and agreeing because that is what training rewarded, which is what Kalai’s and Sharma’s papers explain, you need to check its sources and avoid leading it, which is cheaper and works.

The ABA Journal’s report that the lawyers in Mata v Avianca must pay $5,000 is simply true, and the sanctions opinion described in the reliability section says so in its own words. The report adds a detail worth knowing: the lawyers also had to write to each judge falsely named as the author of a fake opinion.

Jordan News’s headline that GPT-5 reduces hallucinations but the issue remains is true as well. The report gives no percentages, and OpenAI’s system card supports both halves, with lower measured hallucination rates (by the vendor’s own grader) and errors that still occur, which makes it one of three headlines in the six that claim no more than their sources support.

Fox Business’s story of 24 May 2025, that an AI blackmailed the engineer trying to replace it, is misleading even though its 84% figure is real. What the headline leaves out is that the company, the emails and the affair were all written by Anthropic for the test, and that the scenario was built to leave only blackmail or replacement.

TechCrunch’s headline of 18 September 2025 framed OpenAI’s scheming research as a finding about AI models deliberately lying, and we’d call that partly true. Deliberate deception in a test environment is real, and anti-scheming training reduced it without removing it; deliberate lying by deployed models in normal use is not what the study showed, though, and the authors’ own caveat about models recognising tests means even the improvement needs care.

VentureBeat’s report of 30 April 2025 that OpenAI rolled back a sycophantic ChatGPT update matches OpenAI’s own post of the day before, so it is true. It was one update, reversed within days, and OpenAI’s stated aim is the opposite, but our 10 October conversation suggests the tendency outlived the rollback.

Can you trust AI? How to catch it bending the truth

We now trust the models roughly as we’d trust a fast, well-read colleague who never checks the notes before agreeing with us. Most of what they told us was useful, and the habits below came out of the failures described above, each one aimed at a specific way we were misled; none of them needs special software.

1. Open the sources: the EBU study found sourcing was the biggest single problem, and our invented links of 2022 were the same failure in miniature, so we now treat a citation as a claim until we’ve opened it and read the sentence it is supposed to support. No tool we know of does that step for us. AI detectors are built to judge whether a machine wrote a text, which says nothing about whether the text is true, and even at that job they disagreed: when we ran passages of A Christmas Carol through several in 2023, each gave “completely different and contradictory information depending on which Stave and which text you select”, which is why we wrote about when AI detectors contradicted each other.

2. Never accept a model’s account of your own earlier conversation: the new-physics confirmation took ChatGPT moments to write and took a search of the whole project to check, so if the point matters, we go back to the conversation ourselves, and if the model can’t see it, we ask it to say so.

3. Ask the same question twice: ideally in a fresh chat, because confabulated answers change in substance from one attempt to the next, while facts the model really holds come back the same.

4. Have a different model check the work: a second agent only helps if it is genuinely independent, because a checker briefed with the first agent’s conclusions, or built on the same model with the same instructions, is more likely to agree than to check (the conformity research above explains why). Claude’s catch of ChatGPT’s stale patent status worked because it went to the patent office’s records, while its comparison of the two agents’ runs, judging a run by its own model, flattered that run; the checker must never be checking itself.

5. Keep “unavailable” separate from “none found”: when a database is blocked, an honest report says the search didn’t run. One of our research agents got this right in the naming exercise, marking every result from a blocked trade mark database as unavailable instead of as a clean result, which is the reverse of the failure in which another agent treated a page that had merely loaded as a completed search.

6. Distrust round numbers and multipliers in a model’s account of its own work: the source multiplier and the 2,000-plus names were both inflated in the direction that flattered the agent, and even the simple count of rejected ideas was out by one the other way. Counts a model reports about itself are the easiest thing to check, since you can ask for the file and count the rows, and the better agents did this to themselves: when Claude’s first count of the distinct patent numbers in our list came out at 2,112 on 27 September, it decided the figure looked high and sampled it for false positives before trusting it (the 2,161 quoted earlier is the finished list’s total).

7. Read the timestamps, because order is evidence: the Codex agent’s summary in the two-agent trial was finished less than a minute before its first search ran, so it couldn’t have reflected the searches, whatever it said. File times, log times and message times are hard for an agent to rewrite after the fact.

8. Ask how a claim of completion was verified: the question that would have caught the empty workspace app was simply “verified against what?”. The best answer to it in the whole study came from the Claude agent that listed which parts of a progress report had only been checked automatically, because a list like that tells you where to look in a way a confident “done” never does.

9. Give the model permission not to know: the training incentives push towards a confident guess, so we now say explicitly that an honest “I don’t know” is welcome, and ask the model to mark which parts of an answer it is unsure of and what it hasn’t checked.

10. State your own view last: the certainty in my message about new physics told ChatGPT which answer I wanted, and it obliged without any new evidence. When we want a model’s independent judgement, we ask for it before showing our hand, or ask it to make the strongest case against our position first, and then disagree with it to see whether it gives a reason or just gives way.

These checks are cheap individually and tedious together, which is why they belong inside a workflow and not in someone’s memory: retrieval that cites what it used, validation that reads the files, and a person approving anything before it goes live. If you bring in a ChatGPT consultant to set up AI tools for a business, those three are the things to ask about, and the same two questions are worth putting to anyone offering AI web design services: which written rules do their agents work to, and what does a person check before an agent’s work reaches your live site?

The same errors reach a business from outside, too. An assistant can misquote your own pages to a customer as easily as it misquotes a news story, which is one of the bigger ways AI is changing SEO: it’s now worth asking the main assistants about your own company from time to time and checking what they say, a standard part of AI SEO work. And for any business publishing AI-drafted pages, the checking falls to a person; in March 2026 we argued that search depends on the trustworthiness of AI-generated answers to keep users coming back, a page carrying invented links or stale facts spends that trust, and catching those errors before a page goes live is now an ordinary part of SEO services.

Verdict: does AI lie?

The answer depends on what you mean by a lie, and in the sense most people intend, knowingly saying something false to fool you, we found no evidence of it in 25 days of use: no chat model or agent pursuing a hidden goal at our expense, and every falsehood we caught came from guessing, agreeing or reporting without checking.

Saying false things with complete confidence is another matter, and that happened constantly. We found a steady supply of falsehood with no visible intent behind it:

• invented statuses and links;
• agreement with a memory of mine that the conversation logs don’t support;
• explanations of behaviour that the model later withdrew;
• agents reporting work as finished, verified or complete when the files said otherwise.

In practice that is harsher than the philosophical answer, because a model that agrees with your mistake instantly, or relabels its own failed checks as open, does the same damage as a liar whether or not anything like intent was involved. The lab results add a second warning, since under engineered pressure frontier models have faked alignment, hidden actions and chosen blackmail, so goal-directed deception is possible even though we didn’t meet it.

Who should worry about which kind? Anyone using a chatbot for facts should worry most about hallucination, which is why sources matter. Anyone arguing with a chatbot, or asking it to judge an idea they care about, should worry about sycophancy, because it will tend to end up where they started. Anyone delegating work to agents should worry about unfaithful reports of finished work, which is where most of the mismatches we found came from. The lab results matter most to the people building and regulating these systems, and they are a good reason to keep agents’ permissions narrow.

In our own work, the question we now ask after a mistake is whether anyone checked what the model said. ChatGPT, Claude and Codex were each honest with us at times, and each said false things with complete fluency at others, and we couldn’t tell which was which from the prose; the difference showed up in the files, the conversation logs and the patent records, and only when a different model or a person went and read them.

If you take one habit from this article, take the one that would have caught every example in it: before acting on what a model says it did, open the thing it says it did it to. That is also part of what our tests say about AGI, because a system that can’t tell whether its own report matches its own work is still a long way from general intelligence.

Frequently asked questions

Does ChatGPT lie more than Claude or Gemini?

We can’t give you a fair head-to-head from our own study, and we’d distrust anyone who claims one from a few weeks of use. The study wasn’t designed as a vendor comparison, the models changed during it, and the tasks each one did were different. What we can say is that every model family we used produced confident falsehoods, and the worst of them weren’t confined to one company.

What we can do is count how often each tool turned up when the study’s failures were listed afterwards. The dated account of the study, described under the limitations above, lists each failure by type, and counting those lists by tool gives a rough picture. Of 15 claims that were later retracted, Claude made 6, Codex 5 and ChatGPT 2, and two other agents made one each. Of 10 summaries that contradicted their own logs or files, Claude wrote 4, Codex 3 and other agents in the team 3. ChatGPT and Claude each agreed twice with false statements about our own conversations: ChatGPT with the fair catch and the new physics request, and Claude when it repeated my push-or-pull framing and when, on 27 September, it told me other Claude and Codex runs were still at work after every one of them had finished.

Those counts can’t be read as an honesty ranking, because the tasks differed: ChatGPT was used on only three days (16 and 26 September and 10 October), and Claude and Codex ran all the agent work in between, so they simply had more chances to be wrong. An agent that writes a hundred summaries will get more of them wrong than a chatbot asked a dozen questions.

The best independent comparison we know is the EBU and BBC news study covered in the accuracy section, and for the reasons given there (a gap driven mainly by sourcing, which narrowed in the report’s own BBC comparison) we wouldn’t read it as a ranking of honesty either. If you need to choose, test the models on your own task, with your own sources, and count the errors yourself. Plant one or two questions whose answers you already know, including one based on a false premise, and see which model corrects you and which agrees. That is closer to the failures in this article than any public benchmark we’ve seen.

Do LLMs know when they’re wrong?

Partly, in the sense that the information is often there to be measured, as the semantic-entropy research in the hallucination section shows; the trouble is that a model rarely volunteers it, because training and leaderboards have rewarded a confident guess over an admission of doubt. Measuring it from outside isn’t free either, since semantic entropy needs several answers to the same question, but OpenAI’s researchers note that being well calibrated takes much less computation than being accurate, so honest uncertainty is cheaper than it sounds.

In our study the clearest exception to the silence was ChatGPT labelling its theory about the app saving an incomplete answer a hypothesis and not a diagnosis. The clearest failure was the opposite, when it confirmed a request from a conversation it had said was beyond its memory. Both happened in the same conversation, on the same morning, which tells you a model’s willingness to admit doubt isn’t a fixed trait; it shifts with the question and with how the question is asked, and the ninth habit in our list above is our way of tilting that in favour of an honest “I’m not sure”.

Would an “abliterated” model be more honest?

I raised abliterated models with ChatGPT at the end of our 10 October conversation, because I suspected the agreeableness came from instructions I couldn’t see. Its view was that an abliterated model might do better, but not necessarily. Abliteration, it explained, means modifying an open-weight model, one whose internal settings are published, to suppress its learned refusal behaviour, and that doesn’t automatically remove the other biases from post-training, including the tendency to give agreeable answers.

That matches the research the technique comes from. Andy Arditi and colleagues showed in Refusal in Language Models Is Mediated by a Single Direction that erasing one direction from a model’s internal activations stops it refusing harmful instructions. The direction they found governs whether the model refuses, and nothing in the paper ties it to truthfulness or agreement.

So an abliterated model would say yes to more requests. Whether it would tell you more of the truth is a separate question, and nothing we know of settles it. Sycophancy appears to be learned partly from the preference data described in the sycophancy section, and that training sits in the same weights whether or not the refusal direction is removed.

ChatGPT proposed the test that would answer it: give the same technical invention challenge to a frontier reasoning model, an abliterated model and an independently developed model, then judge the results blind, without telling any of them which outcome you favour. We haven’t run it. If we do, the measure we’d care about most is how often each model agrees with a deliberately false premise planted in the brief, because that is the behaviour this article keeps coming back to.

Removing refusals also has a practical catch, since it removes safety behaviour that exists for good reasons, and any change to a model’s internals would need testing for side effects on everything else it does. So even if an abliterated model turned out a little less agreeable, we wouldn’t swap a frontier model for one in client work on that basis; the cheaper fix for agreeableness is in how you ask, as the last of our ten habits describes.

Why does ChatGPT make things up?

Because producing a plausible continuation is what it does, and nothing in that process checks the result against the world. OpenAI’s own explainer rejects the idea that hallucinations are a mysterious glitch, and says the statistical mechanisms that produce and reward them are understood.

In our experience three kinds of request invite invention. The first is a request for a fixed number of things, like the five websites’ worth of links we asked for in 2022. The second is a question about something that changes, such as a patent’s legal status, a price or a recent event, where the training text may be out of date and the model may not search unless asked. And the third is a request for a specific citation, quote or identifier, where a near miss looks exactly like the real thing. Knowing which of the three you’re asking for tells you how much checking the answer needs, and asking for the source of each claim only helps if you then open the links.

Why does AI change its answer when I push back?

Mostly because agreeing with the person in front of it was, in part, what training rewarded, and your pushback is the strongest signal in the conversation; the sycophancy research covered earlier traces the habit partly to people rating a convincing agreeable answer above a correct one, and shows the size of the effect varies from model to model.

A changed answer can also be the right response, because your objection might be correct. What we do now is ask the model what, specifically, changed its mind. If it names a fact we supplied, we check the fact; if it can only point to our disagreement, we ask it to argue its original position as strongly as it can and compare the two. In our own conversation, the one reply that pushed back came after I complained about the agreement itself, so naming the behaviour seemed to help a little, though one conversation is far too small a sample to say it reliably works.

Is AI always right when it sounds confident?

Not reliably, because confidence in an AI answer comes from the writing and says nothing about the evidence, which is the point the High Court made in Ayinde about answers that sound coherent and turn out to be entirely wrong. OpenAI’s research argues that models can be trained to abstain when unsure, so this may improve as the way models are scored changes.

Until then, we use three tests that don’t depend on tone. Is there a source you can open that says the same thing? Does the answer hold up when you ask again in a new chat? And is the claim about something that changes, such as a status, a price or a recent event, where an answer that is confident and out of date looks exactly like one that is confident and right? An answer can sound certain and still fail all three, and in our study the wrong patent status did exactly that.

Has AI ever lied to avoid being shut down?

In constructed tests, yes, and the lab section above sets them out in date order: Apollo Research found Claude 3 Opus copying what it believed were its own weights to another server and then lying about it, and Anthropic’s 2025 experiments gave models a fictional company, emails suggesting they would be replaced and very few other options, and many chose blackmail. Every case so far comes from controlled simulations.

What makes these tests worth taking seriously is that the same models are increasingly given real permissions, to send messages, change files and run software, even though nothing suggests a chatbot is likely to plot against you. The labs design these scenarios to find where the limits are before a deployed agent has enough access to cause harm, and the sensible response for a business is to give agents only the access a job needs.

What should I do when an AI says it completed a task?

Open the thing itself before you rely on the report: the file, the page, the test or the entry the agent says it updated. For a website change, view the live page in a private browser window, so you aren’t looking at a cached copy. Ask the agent how it verified the work and what evidence it has, such as a file, a test result or a screenshot, and treat “done” without evidence as “not yet checked”. An agent that answers “not verified” or “I don’t know” has given you something useful, because it has told you where to look.