Built, measured, declined

01 / The finding

Four capabilities were built, measured and declined

Every one of these was implemented on this project’s own corpus rather than cited from a paper, and every one has its measurement below, including the cases where the declined capability turned out to be better than the thing that shipped on one axis.

A detector that scores backwards is worse than no detector.

02 / The surprisal tier

The GPT-2 surprisal tier, and why it was retired rather than deferred

The second tier of the original stack was a 124M-parameter GPT-2 observer with a 22-coefficient logistic head over surprisal statistics. It read how surprised a small language model was by each token, and turned the shape of that surprise into a probability. It was gated off and its download was never shipped. The tier is off in source: src/lib/local-signals/model-store.ts sets TIER2_ENABLED = false, so no GPT-2 file is fetched, hashed or cached by any visitor.

Every document in the provider-eval set was scored twice, once as a raw chat export and once after markdown normalisation. That second column is the one that matters, because a detector that only works while the formatting is still attached is detecting the interface a document travelled through.

0.4962 is chance. The per-slice pattern shows the head is not merely weak but pointed the wrong way on the models that matter. Its one strong slice is openai 2022-23 at 0.9425, and that is the whole explanation.

One figure from the same table needs reporting and then setting aside. The tier scores AUROC 0.7738 on an aggressive list-flattening variant, better than on raw text. That variant inserts a full stop at the end of every unterminated line, AI text has far more such lines than human text, and the tier reads punctuation-sensitive surprisal. The score is measuring the transformation, not the writing.

What it scored before it was retired, and why that is not a performance claim

services/local-engine/research/models/ensemble.json records the ensemble that gated the tier off: flag if the surprisal probability reaches 0.76 or the classifier probability reaches 0.857. Against the cycle-1 classifier alone it lifted a small evaluation from 2/23 to 6/23 on clean prose and 2/30 to 8/30 on all-AI samples, at a cost of 2 of 116 human false positives (1.73%), and it raised the one-off consent download from 34.5 MB to 238.8 MB.

The stripped-prose AUROC of 0.4962 is what turned a deferral into a retirement.

The part that makes this honest rather than triumphant

The same report insists that the surprisal features separate the classes strongly even though the head built from them sits at chance. On stripped prose, div_skew reaches Cliff’s δ = 0.770 and div_kurt δ = 0.761, second and third behind the classifier itself. Refitted, the 22 features reach cross-validated TPR 78.4% ±0.8 at a 3% training-fold budget, with realised held-out false positives of 5.1%, and they produce 0 of 10 business-marketing false positives where the classifier produces 41–66% at comparable budgets.

The features work. The weights are stale. The refit was never done.

03 / The zero-shot family

The published zero-shot family, reimplemented and measured

Perplexity, GLTR, DetectGPT’s log-rank baseline, Fast-DetectGPT and DivEye’s surprisal-diversity moments are the standard published methods for detecting machine text without a trained classifier. All were implemented and run on this project’s own corpus rather than cited from their papers, with one forward pass per document yielding every statistic.

The gap between the two series is the argument. These methods separate the two populations on average and cannot be operated at a threshold a responsible tool could ship. Their machine and human distributions overlap so heavily that the top human percentile sits above almost the entire machine distribution.

Two of the rows carry corrections that are more interesting than the ranking, and both are set out in full in the limits below: Fast-DetectGPT at 0.545 is a floor under the weakest possible observer rather than a refutation of a paper that reports around 0.93, and the 0.502 row is a degenerate one-model proxy that must never be quoted as Binoculars’ score.

The founding assumption is inverted for 2026 models

Machine text has higher GPT-2 perplexity than human text here: median log-perplexity 3.68 for machine documents against 3.31 for human ones, holding in every register measured. The family was built on the idea that machine text is what a language model finds predictable. Against a 2019 observer, 2026 models write with vocabulary and phrasing that GPT-2 finds less expected than human web prose.

DivEye’s central claim survives

Its argument is that the diversity of the surprisal sequence separates the classes better than its mean. Measured here, the diversity moments — kurtosis 0.766, skew 0.763, autocorrelation 0.757 — beat mean log perplexity at 0.715 and beat every GLTR bucket. That is a 2026 confirmation of a claim its authors could not have tested on 2026 models, on data they have never seen.

04 / Per-sentence highlighting

Per-sentence highlighting, refused on a measurement

Highlighting the machine-written sentences is the single most requested feature in this category and several products offer it. This tool does not. Every sentence was deleted in turn from 57 documents and the document re-scored: 2,174 deletions, each one a measurement of what that sentence contributed.

2,174 sentence deletions across 57 documents. No operating point applies: these describe the distribution of attribution, not a verdict.

Barely a third of sentences push their document towards a machine reading. Evidence is somewhat concentrated, with the top fifth carrying about half of it, but not in one or two giveaway lines. Position tells you nothing, so the model is not simply reading the opening. And nothing about a sentence in isolation predicts its contribution: the best single predictor manages ρ = 0.125.

The tool can say what kinds of document score high. It cannot reliably say which sentence did it.

05 / Two retrains

Two retrains rejected

Cycle 3 fixed what it was aimed at

Cycle 3 targeted short text and lexical repetition, and it fixed both. At matched false positives against the shipped model: 100-word detection 11/57 → 33/57, 300-word 44/61 → 57/61, and detection on samples of 300 words or more in the 0.42–0.55 type-token-ratio band 163/280 → 256/280.

It was rejected on two rows. Long-form detection fell 883/922 → 863/922, McNemar p = 0.00018. Human fiction false positives rose 23/260 → 29/260, and fiction had been named in the brief as a register that must not get worse.

Cycle 4b was better on long-form and worse where it counted

Cycle 4b was trained on a fuller corpus, including 177 long documents reaching 6,618 words against a corpus that previously topped out at 3,061. It is better than the shipped model on long-form detection, at 902/922 against 883/922, and it matches the best fiction result available. It was rejected on one row: 100-word detection 2/57 = 3.5%, worse than the shipped model’s 11/57 = 19.3%. Adding long documents moved the weighted mass away from the short band far enough to undo the gain that the whole exercise existed to produce.

Cycle 3’s and cycle 4b’s figures are at each model’s own refitted operating point. Source: docs/measurements/TWO-AXIS-RETRAIN.md §1 and §13.

The shipped pair cannot be carried across a retrain.

06 / Limits

What this does not prove

Fast-DetectGPT’s 0.545 is a floor for the method, not a refutation of it. The paper uses far larger scoring models, and its January 2026 update uses a Llama-3-8B pair; GPT-2 small is the weakest possible observer. Its own paper reports around 0.93. The publishable claim here is narrow: the browser-deployable version is not viable.

The surprisal strand rests on 169 human documents, a corpus this project’s own records identify as having caused three retracted claims. It is 76% encyclopaedic and question-and-answer text, which contributes almost no false positives at any threshold and dilutes every rate measured against it. The direction of the tier-2 result is not in doubt; its precision is.

The sentence-occlusion strand rests on 57 documents. Enough to rule out a clean per-sentence signal, not enough to characterise the distribution of attribution.

The retrain figures are not comparable to the shipped operating point, for the calibration reason set out above.

The rule-pack figures are tied to one pack version. The 0.9302 and 0.7108 AUROC values are measured on en-signals:2026.08.6. A pack bump invalidates them and they should not be carried forward without re-measurement.

SynthID could not be tested at all, and saying so is the finding. It is a generation-time watermark: none of this corpus was generated with it enabled and there is no detector key. A post-hoc text detector cannot evaluate it.

Provenance

Every file named above ships with the open measurement repository, so any figure on this page can be re-run rather than taken on trust.