On this page
01 / The finding
A sentence score supports an ordering, and nothing else
The same detector, the same corpus: AUROC 0.9695 on whole documents, 0.7639 on single sentences (balanced 60,000 a side). Handing the model a sentence costs about twenty points.
39,592 of 68,916. The majority of a machine-written document reads human one sentence at a time — a number on every sentence would show a correctly flagged document apparently contradicting its own verdict.
877 of 922 (and 0 of 4,636 human documents have one). The document rule cannot be reused at sentence scale: on most AI documents it would find nothing at all.
The informative sentences are rare but strongly enriched — useless in the middle, sharp at the top.
02 / The unit
Nine sentences in ten are below the engine's own scoring floor
This project's central finding is that detection collapses with length — 16.9% at 100–199 words — and a sentence is roughly twenty words. The engine already refuses to score any fragment under 50 WordPiece tokens, because a fragment that short cannot carry a printable probability. Measured against that floor: 89.6% of sentences in AI documents and 83.8% in human ones sit below it (median sentence: 25 and 30 tokens). A per-sentence percentage would be printing exactly the number the engine was written to withhold.
Measured at a rule that no longer ships
- Corpus
- The 5,558-document long-form corpus of 28 August 2026. 269,732 scorable sentences; a further 13,447 below the five-word floor were never scored. The corpus's AI half is not fully independent of training (268 of 922 in a cycle-2 split) — which would tend to understate every weakness finding here, not create it.
- Retired flag point
- 0.9855 / 0.9763
- Detector
- tier3-cycle2-e5small-fp32.onnx, SHA-256 e313ab00de1fffd2…4d2788d (cycle 2, superseded 1 September 2026)
- Runtime
- fp32 server checkpoint, tier3-cycle2-e5small-fp32.onnx, SHA e313ab00de1fffd2…, temperature 0.8324 — the EU server's scoring path. Browser figures are separate and labelled.
- Measured
- 30 August 2026
- Also
- Each sentence is its own forward pass; scores do not depend on neighbours.
The rule that ships today is margin 3.570935 / gap 0.34 (display 0.9679 / 0.9562). The figures under this stamp answer the same question at a flag point this tool no longer uses, and they are not a description of the tool as it runs now.
03 / The contradiction
What a number beside every sentence would actually display
The two populations overlap heavily through the middle of the scale: the median sentence in an AI document scores 0.452, in a human document 0.294. A cut at 0.5 selects 42.6% of AI-document sentences — and 11.5% of human ones, 23,005 sentences of ordinary human writing marked as leaning machine. Only at the very top does the signal sharpen:
| Cut | Sentences from AI documents | Sentences from human documents |
|---|---|---|
| 0.50 | 29,324/68,916 (42.6%) | 23,005/200,816 (11.5%) |
| 0.80 | 7,821/68,916 (11.3%) | 1,399/200,816 (0.7%) |
| 0.90 | 3,126/68,916 (4.5%) | 199/200,816 (0.1%) |
| 0.95 | 1,305/68,916 (1.89%) | 24/200,816 (0.012%) |
| 0.98 | 231/68,916 (0.3%) | 0/200,816 (0.0%) |
A reader shown a number on every sentence of a document the tool correctly called AI would see most of those numbers read "human", and would reasonably conclude the tool was contradicting itself. The display would be honest arithmetic and a dishonest impression at once.
04 / What survives
The ordering points at the right passages — except where it doesn't, and it cannot tell
1,996 splice trials: a human document with a contiguous quarter of its sentences replaced by AI sentences of the same register, then ranked by score. The single highest-ranked sentence is one of the inserted ones in 78.0% of trials (1,557 of 1,996, against 25% by chance); precision at k is 0.575 against 0.250; within-document AUROC 0.804. The ranking works where the scale does not — and this is precisely what licenses a highlight and nothing more.
| Register | Trials below chance | Precision@k |
|---|---|---|
| White paper | 0.00% | 0.676 |
| Company update | 0.95% | 0.660 |
| Research summary | 1.38% | 0.624 |
| Academic literature review | 1.83% | 0.661 |
| Academic discussion | 1.84% | 0.574 |
| Long-form journalism | 9.21% | 0.424 |
| Fiction | 14.96% | 0.408 |
PER-SENTENCE-RELIABILITY.md §6. Sentence-level AUROC is not inverted in any register (fiction 0.666, CI 0.660–0.672); the failure is in per-draft ranking, which is what a highlight actually uses.
At section level — the unit the verdict actually uses — nothing is weak anywhere: 0.944 on fiction, 0.986 and above everywhere else. The section deep-dive stands on measured ground; the sentence layer does not, and the interface must never present the two as equally solid.
05 / Fragility
Changing one spelling moved a sentence by 0.475
596 documents were re-scored under five mechanical, meaning-preserving edits — expanding contractions, swapping British and American spelling, replacing "However," with "But", removing serial commas, straightening quotes. Removing a serial comma moves a sentence's probability by 0.024 at the median. Changing organise to organize moved one sentence by 0.475 — from one end of the scale to nearly the other, for a spelling.
The same experiment is the second reason a relative display is right: with all five edits applied at once, the top-ranked sentence stays the top-ranked sentence in 93.3% of documents (556 of 596) and the top fifth of the ranking keeps 94.5% of its members. A printed percentage would visibly change when a writer fixed a comma, and would deserve every complaint it got; a ranking barely notices.
06 / What ships
Marks behind a counted floor, ranked, never scored — and quiet where it is unreliable
What the checker's sentence layer actually does, each choice traceable to a section above. One disclosed gap travels with this section: tier3-cycle5-v1 replaced tier3-cycle2-v1 for the document verdict on 1 September 2026, but the sentence-evidence floors below remain fitted to tier3-cycle2's score distribution — they have not been re-fitted to cycle 5, and the product states that on every run rather than presenting the floor as newly validated.
No number beside any sentence, ever. The rendering code consumes the probability and returns a rank and a tier; the value is not in scope downstream, so a percentage cannot be rendered even by mistake.
An absolute evidence floor before any ranking. On the shipped browser route (WASM) the floor is 0.945, fitted to a counted human false-mark rate: 25 marked sentences in 200,890 human sentences — 0.012% — against 1,292 in 68,916 on the AI side. A passage is marked in 40.3% of AI documents and 0.52% of human ones. Without the floor, a purely relative layer would always paint something — including on the one fiction draft in seven where it would paint confidently and wrongly. With it, the failure mode becomes "no highlight" rather than "wrong highlight".
The floor is per-runtime, because the routes disagree on the decision. Of 300 sentences at or above 0.95 on the server, 58 (19.3%) fall below it in the browser — close agreement on values, frequent disagreement on decisions, because so much score mass sits just above the floor. The browser floor of 0.945 was fitted on the browser runtime to reproduce the counted rate; a server figure is never quoted for the browser.
A stated absence. 95.1% of AI documents contain no sentence above the document flag point, so a confident verdict above an unmarked draft is the common case — and the interface says so rather than leaving it to read as the tool retracting itself. The layer goes quiet by itself exactly where §4's table says it is unreliable: 9.6% of fiction and 16.1% of journalism AI documents get any mark at all.
07 / Limits
What this page does not prove
The corpus is not fully held out. 268 of the 922 AI documents touch a cycle-2 split. That inflates detection figures; every finding here is a statement about weakness, which contamination would tend to understate rather than create — but the caveat travels with the numbers.
Long-form only, English only, one build. The sentence distributions describe prose of roughly 600 words and up from thirteen models. Marking behaviour after light editing may legitimately change between runs — a median trivial-edit movement of about 0.02 sits against a floor where the marked count moves 27% per 0.01 — which is a property of the instrument the interface states rather than hides.
A ranking licence is not a detection licence. Everything above supports "this passage is stronger evidence than that one, in this draft". Nothing above supports a claim about what any single sentence is, and the tool never makes one.