On this page
01 / The finding
The quarantine was well designed and pointed at the wrong file list
Every accuracy figure this tool publishes is measured on 5,558 documents assembled for that purpose: 4,636 human documents drawn under an explicit licence, and 922 machine-written documents generated through OpenRouter after the shipped model had finished training. Each candidate was hashed against the material it had to be independent of, with an exact collision set to abort the build.
The index held 6,114 hashes from four held-out files, and the build returned zero exact collisions against all four.
cycle2-train/dataset.jsonlwas not one of the four.
Eight source pools, each row carrying its licence string. Where two authorities disagree on a licence, both are recorded rather than resolved.
13 current models through OpenRouter, 800–2,000 word targets, eight long-form registers, three prompt styles.
168 in the train split, 72 in test, 28 in calibration. The independent subset is 654 documents, and that is the population any unseen-data argument has to use.
That belongs at the top of this page rather than in a footnote at the bottom of it. The design was sound: normalise, hash, abort on a collision. The list it checked against was incomplete, which is a more useful lesson than a clean result would have been, and it is why the phrase “hash-quarantined against every training split” has been withdrawn from this project’s records and corrected at the head of the file it appeared in.
The size of the effect is the second half of the finding, and it cuts the other way. Measured directly, seeing the training data was worth roughly one percentage point. That is small. It is also not the correction: a figure published as a measurement on unseen documents was, for 29.1% of the machine-written half, a measurement on documents the model had fitted on.
02 / Composition
What is in it, and under what licence
Every row carries its licence string. The human sources named in the shipped measurement file are Europe PMC, GOV.UK, CRS, Global Voices, Mongabay, SEC EDGAR and PERSUADE 2.0, and the eighth pool is the Internet Archive’s Creative Commons text collection, taken per item under whatever the uploader declared. PERSUADE 2.0 is the one row with a conflict: the dataset is CC BY 4.0 upstream and the mirror used declares MIT. Both are recorded, because they differ.
| Source | Documents | Licence | Registers |
|---|---|---|---|
| Europe PMC open access | 1,425 | CC BY family, or CC0 | academic introduction, literature review, discussion, conclusion |
| GOV.UK research and policy | 851 | Open Government Licence v3.0 | white paper, research summary, company update |
| Congressional Research Service | 420 | US government work, 17 U.S.C. 105 | white paper |
| Global Voices | 420 | CC BY 3.0 | long-form journalism |
| Mongabay | 420 | CC BY-ND 4.0, stored verbatim | long-form journalism |
| SEC EDGAR 10-K Item 7 | 420 | mandatory public disclosure | company update |
| PERSUADE 2.0 student essays | 420 | CC BY 4.0 upstream, MIT on the mirror used | student essay |
| Internet Archive CC texts | 260 | per item, the uploader’s licenceurl | story |
| Total, human half | 4,636 | eight licences, one of them disputed | eight long-form registers |
longform-corpus/MANIFEST.md. Corpus built 28 August 2026. Counts are documents after de-duplication, capping and the prose test; no detector, no runtime and no operating point applies to any value here.
The exclusions show the judgement as clearly as the inclusions. Abstracts were dropped from Europe PMC because they are formulaic and are not what anyone pastes into a detector. PDF-only GOV.UK publications were skipped rather than OCR-guessed. Financial tables were stripped from 10-K filings before chunking, and any passage still mostly figures failed the prose test.
03 / What was refused
Four sources were examined and turned down
The Conversation would have been the single best fit for long-form journalism written by academics. Its republishing terms state that its Creative Commons licence prohibits using its content as training data for AI systems, so it is not here. Strange Horizons reserves all rights on behalf of its authors. BAWE is registration-gated with no redistribution right, and ICLE is commercially licensed.
Excluding the best-fitting source on the publisher’s own terms is the largest single loss in the build, and it is visible downstream: the journalism register rests on two outlets rather than three, and the student-essay register rests on one dataset.
04 / The machine half
922 documents from 13 models, generated after the model was trained
The machine-written half was generated through OpenRouter with 800–2,000 word targets, across eight long-form registers and three prompt styles, from a fixed bank of 68 topics. It cost $12.33 of the $13 authorised, and the per-call cost is stored on every machine-written row so the total is recomputable from the delivered file rather than taken on trust.
Show the per-model counts
| Model | Documents |
|---|---|
deepseek-v4-pro-0813 | 131 |
openai/gpt-5.6-luna | 121 |
google/gemini-3.7-flash | 121 |
x-ai/grok-4.6 | 121 |
qwen/qwen3.8-max | 110 |
meta-llama/llama-4-maverick | 101 |
z-ai/glm-5.3 | 67 |
mistralai/mistral-medium-3-5 | 41 |
anthropic/claude-sonnet-5 | 26 |
moonshotai/kimi-k3 | 26 |
anthropic/claude-opus-5 | 23 |
google/gemini-3.1-pro-preview | 21 |
openai/gpt-5.6-sol-pro | 13 |
| Total, machine half | 922 |
docs/measurements/DETECTION-BY-LENGTH-AND-MODEL.md Table 2, denominators column. Generated 28 August 2026, after the cycle-2 model finished training. Counts only: no detector, no runtime and no operating point applies to any value here.
Three models contribute under 30 documents each. Any per-model figure drawn from those rows carries an interval wide enough to make a ranking meaningless, which is why per-model detection is published with its denominators on the detection-rates page rather than summarised here.
05 / The quarantine
The quarantine, as designed
Every candidate document is hashed on normalised text: NFKC, lower-cased, punctuation stripped, whitespace collapsed. That hash is checked against an index of material the corpus must be independent of, and an exact collision aborts the build, because an exact collision is proof rather than evidence.
Near-duplicates are screened separately, at 12-word shingle overlap with a 25% threshold. Those rows are dropped and listed in the manifest rather than aborting the build. A shingle hit is a heuristic, and a build that halts on a heuristic will be disabled by whoever runs it next.
Splits are group-aware: 60/15/25 by SHA-256 of a group key, with every variant of a source sitting in the same split as its source. In the cycle-3 build the group key is the post slug, chosen so that one key covers both “same source URL” and “same topic”.
The guard is not decorative, and there is a measurement showing it fires. The cycle-4 corpus build checked every candidate against the normalised hashes of all 11,004 documents in five measurement sets and their 2,684 source references, and caught and excluded three rows: duplicate uploads of the same public-domain work. That build asserts if more than 25 rows are caught, on the reasoning that a large number would mean the exclusion-by-source step had failed rather than that coincidence had happened.
| Step | Stage | What happens | What it produced |
|---|---|---|---|
| 1 | Fetch | One candidate document is pulled from a licensed source, under the licence recorded on its row. | A candidate with its licence string attached |
| 2 | Normalise | NFKC normalisation, lower-casing, punctuation stripped, whitespace collapsed. | One canonical text per document |
| 3 | SHA-256 | The normalised text is hashed. | One norm_sha256 per document |
| 4 | Exact-collision check | The hash is compared against the index of material the corpus must be independent of. A collision aborts the whole build, because an exact collision is proof rather than evidence. | 0 exact collisions against the four indexed files |
| 5 | Shingle screen | 12-word shingle overlap at a 25% threshold. A hit drops the row and lists it in the manifest rather than halting the build, because a build that halts on a heuristic gets disabled by whoever runs it next. | 2 near-duplicates dropped, 505 internal duplicates dropped |
| 6 | Group-aware split | 60/15/25 by SHA-256 of a group key, with every variant of a source landing in the same split as its source. In the cycle-3 build the group key is the post slug. | Train, calibration and test splits with no source spanning two of them |
| — | Not in the index | cycle2-train/dataset.jsonl, the training file for the model this corpus was built to evaluate, built the same day. No candidate was ever hashed against it. | 268 of 922 AI documents were later found in it, 168 of them in the train split |
longform-corpus/MANIFEST.md for the pipeline and the index contents; corpus-reconciliation-2026-08-29/analysis.txt §2 for the counts in the dashed box. Schematic, not a measurement: the only figures in it are the four index counts, their total, and the overlap counts named in §2.
06 / The index
What it was pointed at, and what it missed
| Held-out source | Texts | In the index? |
|---|---|---|
eval-samples.json | 34 | yes |
provider-eval/eval-set.jsonl | 1,896 | yes |
tests/battery/human-corpus-v1.json | 40 | yes |
tests/battery/human-corpus-v2.json | 4,144 | yes |
| Total hashes checked against | 6,114 | — |
cycle2-train/dataset.jsonl | not counted | no |
Against what it was pointed at, the design worked exactly as specified: 0 exact collisions, 2 near-duplicates dropped, 505 internal duplicates dropped. Source: longform-corpus/MANIFEST.md.
cycle2-train/dataset.jsonl is the training file for the model this corpus was built to evaluate, and it was built the same day. The index covered the evaluation material and the human battery, and not the training set itself.
| Corpus file | Rows | Also present in the cycle-2 dataset |
|---|---|---|
longform-corpus/ai-longform.jsonl | 922 | 268 — 29.1%: 168 train, 72 test, 28 calibration |
longform-corpus/human-longform.jsonl | 4,636 | 11 — 0.24%: 5 train, 3 calibration, 3 test |
Measured afterwards on normalised SHA-256. Source: corpus-reconciliation-2026-08-29/analysis.txt §2, recorded in CORPUS-RECONCILIATION-2026-08-29.md §2.1.
PERSUADE 2.0 is not held-out material. It also appears in the cycle-2 training corpus, and the manifest says plainly that anyone combining the two corpora must deduplicate on norm_sha256. That was recorded on the day the corpus was built and it was correct. It is the most important single caveat on this page and it is not softened anywhere on this site.
07 / What it cost
What the leak was worth, in points
The effect was measured directly, by splitting the machine-written half into the documents that appear nowhere in the cycle-2 dataset and those that do. That comparison exists at one operating point and one only: the superseded 0.984 single-threshold rule, which is not the rule this tool ships.
Measured at a rule that no longer ships
- Corpus
- The 922-document machine-written half, split into 654 documents independent of every cycle-2 split and 268 that are not, of which 168 are in the train split.
- Retired flag point
- 0.984
- Detector
tier3-cycle2-e5small-fp32.onnx, SHA-256e313ab00de1fffd2…4d2788d- Runtime
- Python onnxruntime, CPU, fp32, maximum-over-sections aggregation
- Measured
- 29 August 2026
- Also
- No seen-against-unseen split has been measured at the shipped pair. These three rows are the only scoring of that split this project has.
The rule that ships today is 0.9855 / 0.9763. The figures under this stamp answer the same question at a flag point this tool no longer uses, and they are not a description of the tool as it runs now.
| Subset | What it is | n | Detected | Rate |
|---|---|---|---|---|
| Never in the cycle-2 dataset | the independent subset | 654 | 620 | 94.80% |
| In the cycle-2 dataset, any split | train, test or calibration | 268 | 257 | 95.90% |
| In the cycle-2 train split alone | the documents the model fitted on | 168 | 163 | 97.02% |
corpus-reconciliation-2026-08-29/analysis.txt §2. 654 independent, 268 seen in any cycle-2 split, 168 in the train split alone; the three subsets are drawn from the same 922 machine-written documents, so the second and third rows overlap by construction.
Both gaps should be quoted with the pair they describe. Against the seen subset the independent subset reads 1.1 points lower. Against the training split alone it reads 2.2 points lower. Weighted across the corpus, contamination is worth about 0.3 points on the headline detection figure.
Seeing the training data was worth roughly one percentage point. The correction is not that the effect is small.
The three rows above are at 0.984 and must not be relabelled or reprinted under a shipped-pair heading. A re-measurement at the shipped pair is outstanding.
08 / Limits
What this corpus does not cover
The 1.1-point gap in Figure 4 was measured at the retired 0.984 single-threshold rule, which is the only operating point that split has ever been scored at. No seen-against-unseen figure exists for the pair that ships, and the 1.1 points must not be quoted under a shipped-pair heading. Until that re-measurement is run, the honest statement is that the corpus is not fully held out and the size of the effect at the current rule is unknown.
- Humanities full text at scale. Europe PMC is a biomedical index first. Twenty-two subject queries widened it deliberately, into education, sociology, linguistics, history, ethics, policy and law, anthropology, business, economics and media, and the discipline label is carried per row. It remains health-adjacent humanities rather than literary criticism, theology or philosophy proper. A detector tuned on this material should not be claimed to be validated on humanities essays.
- Modern open-licensed short fiction barely exists, and the fiction register is the weakest source in the corpus for that reason. What remains is the Internet Archive’s Creative Commons text pool, which is uneven self-publishing and scanned material; some passages are creative non-fiction or essays about literature rather than fiction, and some are OCR of scanned pages. This register’s numbers deserve more suspicion than the others, and it is also the register with this project’s worst human false-positive rate.
- The white-paper register is two governments, the UK and the US. Think-tank, NGO and standards-body publications were sought and are largely PDF-first or non-commercially licensed.
- Global Voices is substantially translated into English, which is a different kind of English from an originating newsroom.
- The delivered GOV.UK era span is 2014–2022, not the 2018–2022 the filter asked for, because the search filter and the recorded year are different fields. This is stated in the manifest.
- Pre-2022 corporate blogs under an open licence were searched for and not found at useful scale. SEC management discussion and analysis stands in for corporate communications, and it reads differently: it is written under legal review.
- The machine-written side is the easy case. These are single-shot generations from a fixed bank of 68 topics across three prompt styles. Nothing here is edited, human-revised or adversarially rewritten, and nothing measures hybrid human-and-machine text.
- The corpus holds no mixed documents and no machine-written document above 3,061 words. The server accepts up to 4,000 words, so the unmeasured range starts inside what it will score.
- Register labels are machine-assigned. Every per-register figure anywhere in this project inherits that.
- The 25,723-document figure from the signal study is a different corpus — six pooled sources, de-duplicated — and must never be conflated with these 5,558.
09 / Provenance
Where every figure on this page comes from
| Figure | File | Section |
|---|---|---|
| Corpus 5,558 = 4,636 human + 922 AI; built 28 August 2026 | services/local-engine/research/longform-corpus/ | — |
| Per-source human counts, licences, registers, capping rules and exclusions | longform-corpus/MANIFEST.md | whole file |
| Per-model machine counts, 13 models totalling 922 | docs/measurements/DETECTION-BY-LENGTH-AND-MODEL.md | Table 2, denominators |
| Spend $12.33 of $13 authorised, per-call cost stored per row | longform-corpus/MANIFEST.md | generation run |
| Pipeline, index contents (34 / 1,896 / 40 / 4,144 = 6,114), 0 collisions, 2 near-duplicates, 505 internal duplicates | longform-corpus/MANIFEST.md | quarantine |
| Cycle-4 build guard: 11,004 documents, 2,684 references, 3 rows excluded, assert above 25 | longform-corpus/MANIFEST.md | build guard |
| 268/922 and 168 train-split overlap; human 11/4,636 | corpus-reconciliation-2026-08-29/analysis.txt | §2 |
| 620/654, 257/268, 163/168 at the retired 0.984 rule | corpus-reconciliation-2026-08-29/analysis.txt | §2 |
| The same overlap figures as published on this site | docs/measurements/CORPUS-RECONCILIATION-2026-08-29.md | §2.1 |
Every file named above ships with the open measurement repository, so any figure on this page can be re-run rather than taken on trust. The counts in the masthead and in the prose are read at build time from the same thresholds.json the browser fetches, so a correction to the measurement lands here without anyone editing this page.