On this page
01 / The finding
Given the markdown, the live model flags a quarter of structured human writing
114 of 418 never-trained human documents wrongly flagged, 95% interval 23.2–31.7. GOV.UK-class business and technical pages with headings and lists intact — exactly the register people paste. The largest false-positive figure this programme has ever measured on its own live model.
Stripping the markdown syntax un-flagged almost all of it: 5 of 3,529 structured human documents at cycle-2's live pair, against 794 raw. The checker normalised its model input this way (contract md-strip-v1, both routes and the server) from 31 August until 1 September 2026, when cycle 5 shipped on raw-v1 input and the strip retired with it.
1 of 418 — with long-form detection up, not traded away (97.5% v 95.7% at matched false positives). Deployed 1 September 2026; through the browser runtime the same 418 read 1 of 418 (0.24%).
That number had never been measurable before this corpus existed, and it belongs in the published weakness table whatever is decided about the new model.
02 / How it surfaced
A corpus built to test evasion turned around and measured us
The matched-pairs corpus was built for a different question: take real human documents, extract only their brief — topic, length, structure outline, tone, never the text — and have model families the detector never trained on write to it. The AI half answers "how detectable is structure-matched machine text with no leakage at all". The human half — the 418 partners, on held-out topics, kept out of every training split by a hash guard that aborts the build on contact — turned out to answer a question nobody had been able to ask: how does the live model read modern, structured, licence-verified human writing, markdown and all?
Measured at a rule that no longer ships
- Corpus
- 418 structured human documents (held-out topic slugs, never trained by any cycle) and 192 matched AI documents from held-out families and topics; scored beside the full eval view of 675 AI and 4,500 human long-form documents with every trained or overlapping row excluded from measurement.
- Retired flag point
- 0.9855 / 0.9763
- Detector
- tier3-cycle2-e5small-fp32.onnx, SHA-256 e313ab00de1fffd2…4d2788d (cycle 2, superseded 1 September 2026)
- Runtime
- fp32 server analogue of the shipped cycle-2 model at its live pair; the harness reproduced the published 883/922 and 45/4,636 exactly before any new figure was taken. Text scored as the model then received it — raw, markdown included.
- Measured
- 30 August 2026
- Also
- Intervals are 95% Wilson. Thin cells (google family n=16, family-plus-topic n=5) are printed with their denominators and must not be quoted without them.
The rule that ships today is margin 3.570935 / gap 0.34 (display 0.9679 / 0.9562). The figures under this stamp answer the same question at a flag point this tool no longer uses, and they are not a description of the tool as it runs now.
| Slice | n | Shipped | Cycle-5 model (now live) |
|---|---|---|---|
| Human partners wrongly flagged (structured, held-out topics) | 418 | 27.3% (114) | 0.2% (1) |
| Topic-bucket AI detected (fully independent) | 176 | 83.5% (147) | 86.9% (153) |
| Meta-family AI detected | 59 | 67.8% | 81.4% |
| Google-family AI detected — thin cell | 16 | 56.2% | 62.5% |
results-c5.json gate2_humans and gate2_by_family, verified at source. At the cycle-2 pair convention the replacement reads 0 of 418; the 0.2% quoted is its refitted matched-FP pair — the pair that now decides verdicts — the more conservative reading.
Why no earlier measurement caught it. Every published false-positive figure was measured on the 4,636-document human corpus — which is plain prose, because web scraping had stripped its structure. The people most exposed — docs teams, developers, technical marketers, anyone who writes in markdown natively — were invisible to every corpus this programme had, until a structure-preserved human corpus was banked and its held-out topics became this evaluation slice.
03 / The mechanism
The model reads the syntax, not just the writing
The same investigation isolated the mechanism with a controlled strip: take the identical 600 human documents and score them three ways. As raw markdown, 22.5% flag (135 of 600). With only the syntax stripped — every word kept, every paragraph break kept — 0.0% flag (0 of 600). As raw HTML, 0.2%. Same documents, same model, same thresholds; only the characters around the words changed. Of 60 flagged documents individually re-scored with their markdown removed, 59 un-flag.
So the exposure is the ## and - and ** characters themselves, not document structure: the model learned that markdown syntax accompanies machine writing, because in its training corpora it did — AI output arrives as markdown source while scraped human text has lost its markers. That is a truth about corpora, not about authors, and a detector must not charge people for it. The live demonstration is one strip away: an 18F handbook page scored 0.9873 — flagged — as raw markdown, and 0.3465 with the syntax removed.
04 / What was fixed, and what replaced the fix
The input fix went live 31 August 2026 — and retired itself 1 September 2026 when cycle 5 shipped
This section describes a bridging fix that no longer applies. From 31 August to 1 September 2026 the checker normalised the model's input to the plain-prose rendering the published cycle-2 figures were measured on: heading markers, list markers and bold, code and link syntax removed before the model read a word, on both routes and again on the server as defence in depth (input contract md-strip-v1, the measurement's own strip, ported byte for byte with golden fixtures). On the corpus behind this page, that one change took the shipped cycle-2 model from 794 of 3,529 structured human documents wrongly flagged (22.5%) to 5 of 3,529 (0.14%) at cycle-2's live pair. What it deliberately did not do was change the model: a small detection cost on markdown-carrying AI text was the price (93.6% raw falls to 88.4% stripped — the syntax was doing about five points of the work), and the model itself still carried the habit the strip routed around.
Since 1 September 2026 the checker no longer strips markdown before scoring. tier3-cycle5-v1 was trained on raw text (input contract raw-v1) specifically so it reads markdown without the crutch, and feeding it md-strip-v1 output would remove the structural signal its features exist to read. The strip retired with the model that needed it; the current model's input handling is raw-v1, not md-strip-v1.
05 / The path forward
The cycle-5 model — measured, then deployed
A new training cycle was built with the structural measurements of the last two days as model inputs, trained on the structured human corpus's licence-clear bucket, and evaluated on the never-trained slices. On the same 418 documents it reads 0.2% false positives (1 of 418); on the eval view it detects 97.5% (658 of 675) against the shipped 95.7% at matched false positives (42 v 41 of 4,500); fiction false positives fall from 8.8% (20 of 227) to 3.1% (7 of 227); and short-form detection at 100 words moves from 19.6% (11 of 56) to 76.8% (43 of 56, interval 64.2–85.9). The disclosed trade: heavy LLM rewrites of human originals flag more, 28.5% against 21.2% (137 documents) — a direct consequence of training heavy rewrites as machine-written, which they are.
It shipped on 1 September 2026 — after the missing pieces were built and measured. The eight structural input features now have a TypeScript port under a parity contract (features-v1): on 1,051 documents the port and the Python pipeline produced zero segmentation mismatches and 6 verdict disagreements (0.57%). The browser runtime was then measured directly — the int8 file through onnxruntime-web WASM, the engine build the site ships, over the full 5,558-document corpus: 900 of 922 AI documents detected (97.6%) at 73 of 4,636 human false positives (1.57%, against the fp32 server route's 0.99% — the browser route is the weaker of the two and each route prints its own figure). Verdicts disagree with the fp32 route on 37 of 5,558 documents (0.67%). One cell stays open: commercial humanisers, which escaped the cycle-2 build 96.4% of the time, have not been re-measured against this model, and no claim is made for them.
06 / Limits
What this page does not prove
The 27.3% describes the cycle-2 model given raw markdown — not the live product today. The input fix normalised markdown before scoring for the rest of that model's life, taking the same corpus to 0.14% through that path; the cycle-5 model that replaced it on 1 September 2026 was trained on raw text and declares raw input, so the strip retired with the model that needed it. The figure is published because it is true of the cycle-2 model, it was true of the product until 31 August 2026, and the register it lands on — structured professional writing — is precisely who was exposed.
One corpus, one snapshot, some thin cells. The 418 documents are professional, edited, licence-recorded web writing from a corpus whose registers and sources are coupled; the google-family and family-plus-topic AI cells are 16 and 5 documents; the 100-word short-form cell is 56. Every one of those carries its denominator wherever it is quoted, and none may travel without it.
The two runtimes are measured separately and do not share figures. The server-route numbers on this page are fp32; the browser figures were measured through onnxruntime-web WASM on 1 September 2026 and are printed beside their own denominators. The earlier int8 CPU proxy numbers were retired the day the real browser measurement replaced them.