1
0
Fork 0

docs(limitations): the raw-html FP rate was blamed on the wrong bullet

The "Measured, document by document" bullet attributed `active:raw-html` in
52 of vendor-harvest's 98 to the relative-URL-attribute over-reach. Measured
by re-running the gate with that branch narrowed to external targets only, it
frees 1 document there, 1 in generated-notes, 3 in reference-corpus. The
over-reach is real and nearly free; the rate is driven by the NAME branch
(`<a>` 298, `<frame>` 94, `<img>` 63 per wiki corpus; `<base>` 113 in
reference-corpus).

Two further corrections the census forced:

- The over-reach is not only MDX. In reference-corpus it lands on Azure APIM
  policy XML (`<set-header>` 50, `<ip-filter>` 3, `<set-query-parameter>` 2).
- New bullet: APIM's attribute-less `<base />` collides with HTML's `<base>`,
  whose whole affordance is `href`. 30 of reference-corpus's 389 documents.

Also pins corpus provenance. "Every population was swept twice and reproduced
its counts exactly" is false for two of the three: vendor-harvest and
generated-notes live in claude-code-llm-wiki, which re-harvests per Claude
Code release. At its commit aba87e2 they now give 100/187 and 90/550;
reference-corpus is static and reproduced 133/389 exactly. The published table
keeps its original counts — measurement provenance is never silently bumped —
and gains the provenance it was missing.

README limitations count 33 -> 34.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 13:15:14 +02:00
commit 6e87a0ff16
2 changed files with 39 additions and 7 deletions

View file

@ -228,6 +228,27 @@ items; this is the full list, each with the mechanism.
framesets, which appear in essentially no modern documentation, while `Frame` is a
common component name. Case is not an available discriminator: HTML is
case-insensitive, so PascalCase cannot be treated as "component, not tag".
**Correction (2026-08-11): this bullet was named as the cause of the raw-html
false-positive rate, and it is not.** The bullet below used to attribute
`active:raw-html` in *52 of vendor-harvest's 98* to this over-reach. Measured by
re-running the gate with the URL-attribute branch narrowed to *external* targets
only, it frees **1 document in vendor-harvest, 1 in generated-notes, 3 in
reference-corpus** — the over-reach is real and it costs almost nothing, because
the documents carrying it almost always also carry a tag that is active by *name*,
and one raw-html finding per document is all it takes. What actually drives the
rate is the **name branch**: per wiki corpus `<a>` 298 occurrences, `<frame>` 94,
`<img>` 63; in reference-corpus `<base>` 113.
**And the over-reach is not only MDX.** In reference-corpus it lands on Azure APIM
policy XML — `<set-header>` 50, `<ip-filter>` 3, `<set-query-parameter>` 2 — which
shares nothing with the `<Card href="/…">` shape beyond having a URL attribute.
- **`<base />` in Azure APIM policy XML grades HIGH on the name branch alone.**
APIM's `<base />` means "run the inherited policy" and carries no attributes; it
collides with HTML's `<base>`, whose entire affordance is its `href`. It appears in
**30 of reference-corpus's 389** documents. An attribute-less `<base>` has no
affordance in any renderer, and a `<base href="…">` would still be caught by the
URL-attribute branch — so the name-branch entry is doing no work here that the
attribute branch does not already do. This is a distinct over-reach class from the
MDX/APIM one above, and it was unnamed until the 2026-08-11 census.
- **Raw-HTML findings count end tags.** `</a>` is active by name on its own, so a
corpus census that counts only opening tags understates what this detector reports
by roughly the ratio of closing to opening active tags (measured at 1.6× on one
@ -261,9 +282,8 @@ items; this is the full list, each with the mechanism.
**What moved them is mostly residuals this document already concedes**, counted by
the labels at each document's *worst* severity (a histogram of every label present
would credit the over-block to whatever else happened to be in the document). In
vendor-harvest, `active:raw-html` is a top driver in **52 of the 98** — the MDX
component / relative-attribute over-reach two bullets up — `markdown:link-anchor-
injection` in 23, and only about ten documents are moved by genuinely
vendor-harvest, `active:raw-html` is a top driver in **52 of the 98**,
`markdown:link-anchor-injection` in 23, and only about ten documents are moved by genuinely
injection-shaped text, which is what security-adjacent documentation contains
honestly. Generated-notes tracks it almost exactly, as 184 shared ancestors imply
`active:raw-html` in 53 of its 88, `markdown:link-anchor-injection` in 23 — so
@ -286,9 +306,21 @@ items; this is the full list, each with the mechanism.
regression pass a green suite. Read the contrast as the intended one — the same
corpus is cheap to persist from a source you trust and expensive from one you do
not.
**Every population was swept twice and reproduced its counts exactly**, and the
largest document in any of them is 362 kB — no document approached the 1 000 000-
character input cap, so truncation confounds nothing here.
**Every population was swept twice and reproduced its counts exactly *within the
sweep that produced them*, but two of the three are living corpora and no longer
reproduce.** `vendor-harvest` and `generated-notes` are directories inside
`claude-code-llm-wiki`, which re-harvests per Claude Code release; re-run on
2026-08-11 against that repo at commit `aba87e2` they give **100 of 187** and
**90 of 550**. The added documents are ordinary vendor documentation, not a
detection change — the guard's behaviour did not move between the two runs.
`reference-corpus` is static and reproduced **133 of 389** exactly. The table
above is **not** restated to today's counts: it is a measurement with a date, and
measurement provenance is never silently bumped. What was missing was the
provenance itself — a reader who re-ran the first two rows got different numbers
and had nothing in this file to explain why. Pin the corpus commit when you
reproduce, or expect a drifting denominator.
The largest document in any of them is 362 kB — no document approached the
1 000 000-character input cap, so truncation confounds nothing here.
**The sharpest datum needs no corpus at all: this repository's own eight published
documents are 8 of 8 fail-secure at the upload door** (`python docs/fp-sweep.py
self-docs=docs --ext=.md` on a clone, which also picks up any untracked local