The text the model never saw

01 / The finding

A passing test, and text the classifier was never shown

The tool cut every document into sections of 340 words, on the stated premise that 340 words sits comfortably inside the classifier’s 512-token window. On dense prose it does not. Words are not tokens, and the sections that ran over the window did not fail loudly. They were quietly cut off at 512 and scored on what was left.

1,348 of 23,318 sections ran over the 512-token window and had their ends silently thrown away, across 684 of 5,558 documents and 276,466 of 9,287,413 tokens.

Recovering the lost text then changed no verdict at all on this corpus. That result is published here because it is true, not because it flatters the fix, and the section it sits in explains why maximum aggregation absorbed the defect.

02 / How it was measured

What produced these numbers

Every section of the corpus was tokenised with the checkpoint’s own WordPiece tokeniser, run without truncation, so what is counted is the length a section would have needed rather than the length it was allowed. That distinction is the whole measurement: with truncation on, every section reads as 512 tokens and the defect is invisible.

The shipped segmentation contract is segments-v3. It produces the identical 21,093 sections as the segments-v2 rule described in the source record, so every coverage quantity on this page is true of the shipped product. Only the contract string moved on.

03 / The assumption

What the rule assumed, and where the assumption broke

WordPiece splits unfamiliar, technical and hyphenated words into several pieces, so a 340-word passage of biomedical or policy prose can need far more than 340 tokens. The proxy fails on exactly the dense writing the tool is weakest on anyway.

Measured across the corpus, tokens per word run at a median of 1.27, a 95th percentile of 1.62 and a maximum of 3.89. The section token distribution tells the same story from the other end.

A 95th percentile over the limit is not a tail risk. It is the ordinary behaviour of the rule on a twentieth of its work, and it had been the ordinary behaviour for months.

04 / Concentration

Where it concentrated

It fell where you would expect: on dense academic prose and on long human-written reports. On the AI side, academic literature reviews lost 80 of 597 sections and white papers 83 of 634. On the human side, fiction lost 122 of 1,031 and white papers 327 of 3,363. Student essays lost nothing at all, 0 of 1,011, which is a measured zero rather than an absent row.

The AI literature-review figure covers 28 of 107 documents, and the human fiction figure 65 of 260. A register absent from one side’s source table is printed as absent rather than as a zero, because those are different claims.

05 / Why it hid

A test that could not see a fault both routes shared

The tool has two implementations of segmentation, one in TypeScript for the browser and one in Python for the server, and a route-parity test exists precisely to catch them drifting apart. That test saw 0 of 284 token mismatches.

Both routes truncated identically. A test that asks “do these two agree” cannot see a fault they share.

There was a second signal, and it was misread. The server’s truncated flag carried a comment saying that a true value means the two routes have drifted apart. The flag was firing constantly, and never for that reason. A flag that fires all the time stops being read, which is how a loud warning becomes a silent one.

06 / The replacement

A token-exact split, with the exactness verified rather than assumed

Two facts make a word-granularity split exact. Whole-document tokenisation decomposes precisely across /\S+/ boundaries, checked token for token on all 5,558 corpus documents and on the eight golden texts, 5,558 of 5,558. And a WordPiece token never consumes fewer than one code unit, which bounds the one case word granularity cannot handle.

Each word is one atom, tokenised and measured. A document whose measured tokens plus the two special tokens fit in 512 is one section, verbatim. Otherwise it is cut into the fewest sections that all fit, as near equal in tokens as word boundaries allow, using integer arithmetic only so no rounding rule can drift between the two languages. Every resulting section is then re-tokenised and measured, and the split widens by one part if any section overshoots.

One word in 6,916,005 needed the oversized-word path: a 987-character run of punctuation that BERT basic tokenisation splits into 987 tokens before WordPiece runs.

07 / What the rewrite found

Three cross-language divergences that were sitting in shipped code

Running the two implementations over the whole corpus and diffing every section’s start offset, end offset, word count and measured token count found 5,558 of 5,558 documents identical across 21,093 sections. Getting there exposed three genuine divergences that the eight golden test texts could not reach.

All eight golden token-id sequences still match after those two tokeniser changes: 8 of 8.

08 / The result

The result nobody predicted

The obvious hypothesis is that recovering the dropped text raises scores on the documents that were being truncated. That is not what happened.

Retired flag point 0.980. Not one verdict on that subset changed, at either threshold measured. Of the 15 AI documents newly flagged at the retired 0.984 point, none had a truncated section under the old rule.

Maximum aggregation is why. A truncated section still contributed its first 512 tokens, and on a document where any section reads as machine-written the maximum was already being set by some other section. The lost text would have had to be the only machine-looking part of the document to change anything, and on these 5,558 documents it never was.

The detection that did improve came from somewhere else. The new rule produces fewer, longer, equal-sized sections where the old one produced a full 340-word run followed by short rebalanced remainders, and detection is length-sensitive. Every newly flagged document has a lower section count under the new rule. There is a symmetric cost, recorded rather than buried: 72 documents that the old rule split needlessly are kept whole by the new one, all 72 human, and at the retired 0.980 point two of them are now flagged that were not before.

The fix removed a real, measured, unbounded coverage defect. On this corpus that defect happened not to be costing verdicts. That is luck rather than design.

It also costs 9.4% fewer forward passes on the same documents, 21,093 against 23,318, so the inference bill went down.

09 / Limits

What this does not prove

The coverage figures are threshold-free; the detection figures are not. Every before-and-after detection number in the source record is at 0.980 or 0.984, both superseded by the shipped minimum-evidence pair 0.9855 / 0.9763. The detection deltas are quoted above only with their own threshold named, and they must not be reprinted under a shipped-pair heading. The coverage quantities carry no operating point and publish as they stand.

The browser check is a subset, and it is register-balanced rather than corpus-proportional. Its 2.33% human false-positive rate is not comparable to any corpus-wide figure: fiction, the worst register, is 10% of it against 5.6% of the corpus. Read the deltas, not the levels. At denominators of 296 and 300, one document is 0.34 percentage points.

Earlier figures for the same phenomenon are superseded. A 60-document route-parity sample reported 12.7% of sections and 21.7% of documents. The corpus here is 93 times larger and differently composed. Those figures should not appear beside these.

The Python halves of the measurement code are not reproducible from the repository. They live in a session scratchpad. The TypeScript verification scripts are committed.

Register labels are machine-assigned, so every per-register split in Figure 3 inherits that.

10 / Provenance

Where every figure on this page came from

Every file named above ships with the open measurement repository, so any figure on this page can be re-run rather than taken on trust. The Python halves of this particular measurement are the exception noted in the limits above.