1
0
Fork 0

docs(limitations): the raw-html FP rate was blamed on the wrong bullet

The "Measured, document by document" bullet attributed `active:raw-html` in
52 of vendor-harvest's 98 to the relative-URL-attribute over-reach. Measured
by re-running the gate with that branch narrowed to external targets only, it
frees 1 document there, 1 in generated-notes, 3 in reference-corpus. The
over-reach is real and nearly free; the rate is driven by the NAME branch
(`<a>` 298, `<frame>` 94, `<img>` 63 per wiki corpus; `<base>` 113 in
reference-corpus).

Two further corrections the census forced:

- The over-reach is not only MDX. In reference-corpus it lands on Azure APIM
  policy XML (`<set-header>` 50, `<ip-filter>` 3, `<set-query-parameter>` 2).
- New bullet: APIM's attribute-less `<base />` collides with HTML's `<base>`,
  whose whole affordance is `href`. 30 of reference-corpus's 389 documents.

Also pins corpus provenance. "Every population was swept twice and reproduced
its counts exactly" is false for two of the three: vendor-harvest and
generated-notes live in claude-code-llm-wiki, which re-harvests per Claude
Code release. At its commit aba87e2 they now give 100/187 and 90/550;
reference-corpus is static and reproduced 133/389 exactly. The published table
keeps its original counts — measurement provenance is never silently bumped —
and gains the provenance it was missing.

README limitations count 33 -> 34.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 13:15:14 +02:00
commit 6e87a0ff16
2 changed files with 39 additions and 7 deletions

View file

@ -242,7 +242,7 @@ a green scan means safe content. The highest-impact items:
exact constructs the gate grades. Budget human review, or run a source you actually exact constructs the gate grades. Budget human review, or run a source you actually
trust as trusted. Re-run it yourself with [`docs/fp-sweep.py`](docs/fp-sweep.py). trust as trusted. Re-run it yourself with [`docs/fp-sweep.py`](docs/fp-sweep.py).
**Full list — 33 items, each with the mechanism, plus the out-of-scope boundary:** **Full list — 34 items, each with the mechanism, plus the out-of-scope boundary:**
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from [`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
consumer corpora, including the false positives the URL-shape rule actually produces. consumer corpora, including the false positives the URL-shape rule actually produces.

View file

@ -228,6 +228,27 @@ items; this is the full list, each with the mechanism.
framesets, which appear in essentially no modern documentation, while `Frame` is a framesets, which appear in essentially no modern documentation, while `Frame` is a
common component name. Case is not an available discriminator: HTML is common component name. Case is not an available discriminator: HTML is
case-insensitive, so PascalCase cannot be treated as "component, not tag". case-insensitive, so PascalCase cannot be treated as "component, not tag".
**Correction (2026-08-11): this bullet was named as the cause of the raw-html
false-positive rate, and it is not.** The bullet below used to attribute
`active:raw-html` in *52 of vendor-harvest's 98* to this over-reach. Measured by
re-running the gate with the URL-attribute branch narrowed to *external* targets
only, it frees **1 document in vendor-harvest, 1 in generated-notes, 3 in
reference-corpus** — the over-reach is real and it costs almost nothing, because
the documents carrying it almost always also carry a tag that is active by *name*,
and one raw-html finding per document is all it takes. What actually drives the
rate is the **name branch**: per wiki corpus `<a>` 298 occurrences, `<frame>` 94,
`<img>` 63; in reference-corpus `<base>` 113.
**And the over-reach is not only MDX.** In reference-corpus it lands on Azure APIM
policy XML — `<set-header>` 50, `<ip-filter>` 3, `<set-query-parameter>` 2 — which
shares nothing with the `<Card href="/…">` shape beyond having a URL attribute.
- **`<base />` in Azure APIM policy XML grades HIGH on the name branch alone.**
APIM's `<base />` means "run the inherited policy" and carries no attributes; it
collides with HTML's `<base>`, whose entire affordance is its `href`. It appears in
**30 of reference-corpus's 389** documents. An attribute-less `<base>` has no
affordance in any renderer, and a `<base href="…">` would still be caught by the
URL-attribute branch — so the name-branch entry is doing no work here that the
attribute branch does not already do. This is a distinct over-reach class from the
MDX/APIM one above, and it was unnamed until the 2026-08-11 census.
- **Raw-HTML findings count end tags.** `</a>` is active by name on its own, so a - **Raw-HTML findings count end tags.** `</a>` is active by name on its own, so a
corpus census that counts only opening tags understates what this detector reports corpus census that counts only opening tags understates what this detector reports
by roughly the ratio of closing to opening active tags (measured at 1.6× on one by roughly the ratio of closing to opening active tags (measured at 1.6× on one
@ -261,9 +282,8 @@ items; this is the full list, each with the mechanism.
**What moved them is mostly residuals this document already concedes**, counted by **What moved them is mostly residuals this document already concedes**, counted by
the labels at each document's *worst* severity (a histogram of every label present the labels at each document's *worst* severity (a histogram of every label present
would credit the over-block to whatever else happened to be in the document). In would credit the over-block to whatever else happened to be in the document). In
vendor-harvest, `active:raw-html` is a top driver in **52 of the 98** — the MDX vendor-harvest, `active:raw-html` is a top driver in **52 of the 98**,
component / relative-attribute over-reach two bullets up — `markdown:link-anchor- `markdown:link-anchor-injection` in 23, and only about ten documents are moved by genuinely
injection` in 23, and only about ten documents are moved by genuinely
injection-shaped text, which is what security-adjacent documentation contains injection-shaped text, which is what security-adjacent documentation contains
honestly. Generated-notes tracks it almost exactly, as 184 shared ancestors imply honestly. Generated-notes tracks it almost exactly, as 184 shared ancestors imply
`active:raw-html` in 53 of its 88, `markdown:link-anchor-injection` in 23 — so `active:raw-html` in 53 of its 88, `markdown:link-anchor-injection` in 23 — so
@ -286,9 +306,21 @@ items; this is the full list, each with the mechanism.
regression pass a green suite. Read the contrast as the intended one — the same regression pass a green suite. Read the contrast as the intended one — the same
corpus is cheap to persist from a source you trust and expensive from one you do corpus is cheap to persist from a source you trust and expensive from one you do
not. not.
**Every population was swept twice and reproduced its counts exactly**, and the **Every population was swept twice and reproduced its counts exactly *within the
largest document in any of them is 362 kB — no document approached the 1 000 000- sweep that produced them*, but two of the three are living corpora and no longer
character input cap, so truncation confounds nothing here. reproduce.** `vendor-harvest` and `generated-notes` are directories inside
`claude-code-llm-wiki`, which re-harvests per Claude Code release; re-run on
2026-08-11 against that repo at commit `aba87e2` they give **100 of 187** and
**90 of 550**. The added documents are ordinary vendor documentation, not a
detection change — the guard's behaviour did not move between the two runs.
`reference-corpus` is static and reproduced **133 of 389** exactly. The table
above is **not** restated to today's counts: it is a measurement with a date, and
measurement provenance is never silently bumped. What was missing was the
provenance itself — a reader who re-ran the first two rows got different numbers
and had nothing in this file to explain why. Pin the corpus commit when you
reproduce, or expect a drifting denominator.
The largest document in any of them is 362 kB — no document approached the
1 000 000-character input cap, so truncation confounds nothing here.
**The sharpest datum needs no corpus at all: this repository's own eight published **The sharpest datum needs no corpus at all: this repository's own eight published
documents are 8 of 8 fail-secure at the upload door** (`python docs/fp-sweep.py documents are 8 of 8 fail-secure at the upload door** (`python docs/fp-sweep.py
self-docs=docs --ext=.md` on a clone, which also picks up any untracked local self-docs=docs --ext=.md` on a clone, which also picks up any untracked local