On this page
01 / Read this first
The boundary that governs every figure below
A watermark scan does not clear anyone and does not accuse anyone.
Our scan runs the published SynthID-Text mathematics against three demo keys we published ourselves. A result of “no signal found” under those three keys says nothing whatever about a private production key held by Anthropic, Google or OpenAI. It is not evidence that text is human, not evidence that it is unwatermarked, and not evidence about who wrote it. The only thing it establishes is that the text does not carry a mark made with one of our three published keys.
Watermarking happens at generation time, not afterwards. A SynthID-class mark is written into the token choices as the model makes them. You cannot add one to text that already exists, and nothing in this project attempts to. What we ported is the detection half.
The mathematics only says something if you hold the key. That is the design, not a defect. Section 3 measures what happens when you do not hold it, and the answer is that watermarked text and unwatermarked text land in bands that overlap. Anyone selling a service that reports “AI watermark detected” without naming the key it used is reporting nothing.
A score near 0.5 is not a finding about a person. It means “indistinguishable from unwatermarked under this key”. Unwatermarked machine writing produces exactly the same reading as human writing, because neither carries the mark.
Those keys are private. Without one, no amount of correct arithmetic here produces a statement about a provider’s text.
02 / The mathematics
What the detector actually computes
When a language model picks its next token it normally samples from a probability distribution. Tournament sampling runs a small knockout between candidate tokens instead. Each candidate is given a pseudo-random 0 or 1 — its g-value — derived from the preceding few tokens together with a secret key, and candidates holding a 1 win their round. Ties break on the model’s own preferences, so the text stays fluent while carrying slightly more 1s than chance would give.
The chain, concretely
- 1 Text is tokenised with a byte-level GPT-2 BPE tokeniser, in the browser.
- 2 Tokens are cut into 5-token sliding windows: four tokens of context plus the current one.
- 3 Each window is hashed — SHA-256 initialisation vector, then a linear congruential generator over the window and each key layer, in exact 64-bit integer arithmetic.
- 4 The g-value is bit 30 of the final hash state, one per key layer, six layers deep.
- 5 Two masks are applied: repeated contexts, and everything after the end-of-text token.
- 6 Mean g over the unmasked positions and layers becomes a z-score against Bernoulli(0.5), and an
erfc-based one-sided p-value.
Both masks matter and both are easy to get wrong. The same window always produces the same g-value, so counting a repeated context twice inflates the score on repetitive text. Everything after the end-of-text token is excluded because it is not generated content. A port that skips either mask produces plausible numbers that are wrong.
What a mean-g score means
Mean g is the proportion of positions that came up 1, under one key. Unwatermarked text sits at 0.5 — a coin flip. Watermarked text sits measurably above it. The distance from 0.5 is the signal; the number of scored positions is what turns that distance into evidence.
The p-value answers one narrow question. It is the probability of seeing a mean g at least this far above 0.5 if the text carried no mark under this key. It is not the probability that a machine wrote the text, and it is not a calibrated probability of provenance: the z-score assumes the g-values are independent Bernoulli draws, which is an approximation good enough to rank and to threshold.
Length is what converts signal into confidence. The strongest passage in our corpus scores mean g 0.6807 over 393 scored positions, which gives p = 3.2 × 10−69. The same distance from 0.5 over 60 positions would be unremarkable. This is why the checker withholds a verdict entirely below 40 scored positions rather than reporting a weak one.
Faithfulness to the reference, asserted test by test
30 tests pass, including g-values matching the reference exactly, the context-repetition and
end-of-text masks matching, key-dependent hash IV derivation, and full scores agreeing with the reference
implementation’s own outputs. Across all 72 fixture × key scores the largest disagreement in
mean g is 2.98 × 10−8, and every one of the 72 agrees on the
number of scored positions exactly. Tokeniser parity is checked over 12 adversarial cases — accented
Latin, emoji, CJK, contractions, CamelCase, snake_case, a literal
<|endoftext|> string, mixed line endings — plus a decode → re-encode → decode
round trip. Source: probe.mjs, output stored beside it.
03 / The wrong-key collapse
Identical mathematics, different key, nothing to see
The wrong-key band and the unwatermarked band overlap, and that is the honest picture.
One figure on this page disagrees with the document that summarises it, and the recomputed value is
the one printed. The paraphrase README records the unwatermarked median as 0.4989; recomputing it from
reference-scores.json gives 0.5000 — the two middle values of the 24
are both exactly 0.5000 — with a mean of 0.5007. The range and the denominator reproduce exactly.
The disagreement is printed rather than resolved quietly, because the summary documents for this corpus
were wrong once already and were corrected on 30 August 2026.
| Condition | n (fixture × key pairs) | Mean g range | Reading |
|---|---|---|---|
| Watermarked, scored with its own key | 12 | 0.6400 – 0.6936 (median 0.6722) | Clear signal |
| Watermarked, scored with a wrong key | 24 | 0.4693 – 0.5222 | Signal gone |
| Unwatermarked, scored with every key | 24 | 0.4756 – 0.5264 | No false signal |
Drawn from packages/watermark-lab/fixtures/reference-scores.json, recomputed 30 August 2026. The same arrays produce the table below.
Correction, 30 August 2026: the previously published ranges were wrong, and narrower.
Until that revision these three rows read 0.64–0.68, 0.487–0.513 and 0.508–0.511, and carried no denominator at all. All three were wrong against the fixtures they cited.
What each of the three got wrong, and why the corrected version is the better one
The third was the worst: 0.508–0.511 is one unwatermarked fixture’s three scores, printed as though it were the range across all eight. The corrected ranges are wider in every row.
The correction makes the result weaker-looking and more honest, which is the point. With the old numbers the wrong-key and unwatermarked bands were disjoint, and a reader could have concluded that a wrong-key score is distinguishable from unwatermarked text. It is not. Under the corrected bands they overlap across 0.4756–0.5222, which is exactly what “no signal under this key” should look like.
Source: packages/watermark-lab/fixtures/reference-scores.json, in the open measurement repository.
The chart in the repository plots 4 of these 24 fixtures
The key-collapse chart on the repository front page is an illustration, not the
corpus. docs/assets/charts/watermark-key-collapse.svg plots four fixtures: the three
400-token watermarked passages, one per key, against a single unwatermarked control,
uw-250-01. Every plotted value matches reference-scores.json to four decimal
places. The problem is the control.
uw-250-01 is the unwatermarked fixture that sits closest to 0.5 under all three keys.
Its three scores are 0.5077, 0.5091 and 0.5105 — a maximum deviation from 0.5 of 0.0105, the smallest
of the eight unwatermarked fixtures, where the next-tightest is 0.0144 and the loosest is 0.0264. So the
flat control bar in that chart is flatter than the corpus, and the visual gap between the bars is
comparable to the wrong-key standard error the fixture manifest records at these lengths, roughly
0.02–0.04. The sample and the control belong in the caption wherever that chart appears.
| Fixture | Own key | Wrong key A | Wrong key B |
|---|---|---|---|
wm-alpha-400-03 | 0.6807 | 0.4987 | 0.4869 |
wm-beta-400-07 | 0.6842 | 0.4956 | 0.5088 |
wm-gamma-400-11 | 0.6867 | 0.5069 | 0.5064 |
uw-250-01 — the single control | — | 0.5077 | 0.5105 |
An unwatermarked fixture has no own key, so its three scores fill the wrong-key columns. Figure 1 above plots all 24 fixtures rather than these four, which is why it is the figure this page leads with.
04 / Corpus selection
The tight null bands are partly the acceptance rule
This corpus is seed-selected, and it changes how Figure 1 should be read. scripts/generate-fixtures.py starts from a global seed of 20260827 and will
reroll a fixture’s seed up to 12 times, keeping the first candidate that meets an
acceptance rule:
- a watermarked passage must score mean g above 0.62 under its own key and within 0.035 of 0.5 under both wrong keys;
- an unwatermarked passage must score within 0.035 of 0.5 under all three keys.
The selection was light, and it was still selection. The recorded
per-fixture seeds show 18 of the 20 generated passages accepted on the first candidate; two
— wm-alpha-200-01 and wm-alpha-200-02 — needed a second, for
22 generations in total. The four degraded fixtures are derived from already-accepted
passages and were not themselves rerolled.
So the wrong-key and unwatermarked spreads in Figure 1 sit inside ±0.035 of 0.5 partly because that is the acceptance rule, not purely because it is what chance returned. The script does not guarantee the bound — after 12 failed candidates it keeps the last one regardless — but no fixture needed more than two, and every accepted score is inside the band: the widest excursions in the committed corpus are 0.0307 wrong-key and 0.0264 unwatermarked. Anyone wanting an unselected null distribution will not find one here.
The rule lives in scripts/generate-fixtures.py; the seeds and the rule are both recorded in
fixtures/synthid-demo-v1.json, and the whole corpus is reproducible from the pinned Python
stack — Python 3.12.11, torch 2.4.0, transformers 4.43.3, temperature 1.0, top-k 64, two leaves.
05 / Paraphrase
Paraphrase removes the mark and keeps the meaning
Zero of 40 paraphrased passages were detected. The smallest p across all forty was 1.15 × 10−3.
| Arm | Detected | Median mean g | What it is |
|---|---|---|---|
| Watermarked, unaltered control — must fire | 12 / 12 | 0.6722 | The 12 watermarked fixtures under their own key, before anything was done to them. |
| Length-preserving deterministic rewrite control — must fire | 36 / 36 | 0.6155 | A 92-entry synonym table and sentence reversal, over three arms of 12. Median length ratio 1.002–1.006, so the text was edited without being shortened. |
| Degraded but watermarked control — must fire | 4 / 4 | 0.6540 | Truncated to 50% and to 25%, and token-substituted. The rule fires on damaged watermarked text, which is what makes its silence elsewhere a measurement. |
| Paraphrased the experiment | 0 / 40 | 0.5088 | 40 rewrites of all 12 watermarked passages by two named local paraphrasers: Qwen3-4B-Instruct-2507 (28) and chatgpt_paraphraser_on_T5_base (12). |
| Unwatermarked, every key control — must stay silent | 0 / 24 | — | 8 unwatermarked fixtures under all three demo keys, mean g 0.4756–0.5264. |
Drawn from services/local-engine/research/paraphrase-resilience/analysis.json (76 rows) and summary.json, recomputed 30 August 2026.
The two obvious objections, closed by measurement
It is not shortening. Median 180 scored positions retained in the 28-rewrite Qwen arm and 165 in the 12-rewrite T5 arm, minimum 57 across both, against a median length ratio of 0.938 and 0.910. Applying the 90–110% token gate that published work uses leaves 25 of the Qwen rewrites and 7 of the T5 rewrites, and detection stays at 0 of 32. The length-preserving deterministic control — a 92-entry synonym table plus sentence reversal, median length ratio 1.002–1.006 — is detected 36 of 36. Editing as such is not what does it.
It is not meaning destruction. Semantic similarity across the 40 rewrites has a median e5 cosine of 0.9788 and a minimum of 0.9487, against a measured unrelated-fixture floor of 0.747. A blind grader given unlabelled pairs found 0 of 24 destroyed. The decisive detail is that the arm producing the better paraphrases destroyed more signal, so the confound points the wrong way for the objection.
The mechanism, in one number. Word 4-gram retention falls to a median of 8.9% in the paraphrase arms, against 57.5%, 64.9% and 86.7% in the three deterministic arms. The watermark lives in token sequences. Paraphrase replaces them while preserving meaning; synonym substitution and sentence reordering leave most of them in place, which is why the mark rides through those.
Scope, stated so the figure cannot travel further than it should
Demo keys, watermarking depth 6 rather than the reference 30, longest passage 400 tokens, GPT-2 tokenisation. It says nothing about any production watermark. It does not contradict Kirchenbauer et al., who find watermarks survive paraphrase given roughly 800 tokens (arXiv:2306.04634) — nothing here reaches that length. The blind grader is a model rather than a person, and 14 of the 24 rewrites it graded were “partial”: detail drift with the topic intact.
The SynthID-Text authors measured paraphrase themselves and reached a compatible conclusion, quoted: editing “weakens detectability, but the watermark can still be detected with high accuracy if the text is sufficiently long” (Dathathri et al., Nature, 2024, Supplementary section C.6, read 29 August 2026). Sufficiently long is the operative phrase, and it is longer than most people paste.
Reproducible from the repository, with no model downloads. The harness,
the 40 stored rewrites, the control arm, the per-passage scores and the aggregation all live in
services/local-engine/research/paraphrase-resilience/,
with the model identifiers, revisions, decoding settings, prompt, seeds and detection rule recorded
alongside. Re-running node scoreall.mjs … && node analyse.mjs regenerates
summary.json byte-identically. Scoring needs Node and the committed fixtures, nothing else.
Two attacks remain genuinely unmeasured. Translation round-trips and targeted removal have not been run against this corpus. They are recorded as unmeasured rather than left out of the table, because a robustness table containing only survivable damage reads as a claim of general durability.
06 / Anthropic’s position
A commitment about future models, covering zero shipping models
Anthropic’s own framing is future tense, and it is worth quoting rather than paraphrasing. The news post opens: “Future Claude models will generate text that contains a watermark” (Anthropic, Claude text watermark, published 14 August 2026, read 29 August 2026). The scope sits in a separate support article: “Claude models launched on or after August 2, 2026 support marking at launch.” Cite them separately — the news post for the framing and the detection API, the support article for the launch scope.
State it as a commitment. Never as coverage.
The commitment covers models launched on or after 2 August 2026. As of 30 August 2026 no Claude model has launched after that cutoff — Opus 5 launched on 24 July and Sonnet 5 on 30 June. So the commitment currently covers zero shipping models. Anthropic publishes no per-model status and there is no public detector. Whether any given piece of Claude output carries a mark today is not publicly established, and no page on this site may say otherwise. A build-time claim guard fails the site if one does.
Why 2 August, and why the date can never mean what it looks like it means
Article 50 of the EU AI Act applies from 2 August 2026. The Act turns on whether a system was placed on the market before or on that day, so Anthropic’s boundary is a legal cutoff rather than a chosen deployment date. That is the mechanism behind the caveat above rather than a hedge on our part: a date inherited from a regulation marks when an obligation attached to newly placed systems. It does not mark a day on which existing output started carrying a mark.
Two dates, and they are not a contradiction. 14 August 2026 is when the announcement was published. 2 August 2026 is the scope of the commitment. Conflating them produces a claim neither source makes.
A detection API is also future tense: “We will soon be offering a watermark detection API.” The distinction matters for what happens next here. A detector endpoint would make this product a client of Anthropic’s service and would not use this lab’s mathematics at all. Only a published key activates what is built here, and publishing a production key is not something a provider has any reason to do.
One adjacent development, and the trap inside it. OpenAI shipped a public
provenance verification API, POST /v1/content_provenance_checks, checking C2PA Content
Credentials and SynthID (read 29 August 2026). It accepts image and audio files only — PNG, JPEG,
WebP, MP3, Opus, AAC, FLAC, WAV, PCM — and no text input of any kind. Metadata
travels with a file; a text watermark lives in the words. A provenance API for images and
audio tells you nothing about a paragraph someone pasted into a box.
07 / The shipped check
What our tool actually does with all of this
It runs the published SynthID-Text mathematics against three public demo keys, in
your browser, and reports one row per key. The keys — opace-demo-alpha,
opace-demo-beta and opace-demo-gamma — are published in the source, which is
the point: anyone can regenerate the fixtures and check our arithmetic. They are worthless as security and
were never meant to be otherwise. The detector is a 1.7 MB chunk fetched only when asked, it holds no
network primitives, and 10,000 tokens score in under 250 ms. Nothing is uploaded to Opace, a model
provider or an analytics service on this route.
| Situation | What is reported |
|---|---|
| Fewer than 40 scoreable positions | No verdict at all. Not a weak result, not a provisional one. Below that length the mathematics cannot separate anything from chance, so nothing is offered. |
| At least 40 positions, one-sided p below 0.001, under one of our keys | A signal is reported for that named key, with its mean g, its weighted mean g, its scored-position count and its p-value. |
| At least 40 positions, no key clears the threshold | “No signal found”, with the explanation that this says nothing about authorship and nothing about any private key. |
| Anthropic’s, Google’s or OpenAI’s production watermarks | Not assessed. Their keys are private, so the check reports the watermark as not assessed rather than guessing at it. |
The AI Watermark & SynthID Checker exists to make the wrong-key point impossible to misunderstand. It loads the detector on demand, offers the 24 built-in passages or your own text, lets you choose the key, and colours the passage token by token by its g-value so the signal is visible rather than merely asserted. Running the wrong-key experiment deliberately, and watching a genuinely watermarked passage collapse into the band in Figure 1, is the most useful thing on that page.
08 / Limits
What this page is not evidence for
One distinction governs all of these. SynthID-Text is a published technique with a published evaluation; this is one implementation of its detection half, tested on one small corpus. Anything above about the technique comes from the Nature paper and the reference implementation. Anything about what we have shown comes from 24 seed-selected GPT-2 fixtures under three public demo keys at watermarking depth 6, and from the 40-rewrite paraphrase run.
- Demo keys only. Nothing here detects, verifies, clears or removes any production watermark from Google, Anthropic, OpenAI or anyone else.
- A 24-fixture corpus, and a selected one. Seed 20260827, up to 12 candidate seeds per fixture, accepted against a stated tolerance band. Small, and not a random sample.
- GPT-2 tokenisation. Text from a model with a different tokeniser would need that model’s tokeniser to score meaningfully.
- Depth 6, not the reference 30. Fewer layers make the per-layer signal stronger at the lengths a person actually pastes. It is a teaching choice, recorded in every fixture manifest, and it means the technique’s behaviour at depth 30 is not something measured here.
- Short passages are uninformative. Below 40 scored positions no verdict is offered.
- A score near 0.5 is not evidence that a person wrote the text. It means “no signal under this key”, which is also what unwatermarked machine writing looks like.
- Detection only. No watermarking of your own text, no removal, no stripping. The generation path is not ported.
- The z-score is an approximation. It assumes independent Bernoulli g-values. Good enough to rank and to threshold; not a calibrated probability of provenance.
- Translation round-trips and targeted removal are unmeasured, and are marked as such in the robustness table rather than omitted from it.
The honest status is unchanged from the day the port passed its tests:
ready, unproven against production output, and saying so. The full measurement record, including the
corrections this page carries and the two third-party figures withdrawn for want of a traceable source,
is in docs/WATERMARK-LAB.md.