docs(limitations): the raw-html FP rate was blamed on the wrong bullet
The "Measured, document by document" bullet attributed `active:raw-html` in 52 of vendor-harvest's 98 to the relative-URL-attribute over-reach. Measured by re-running the gate with that branch narrowed to external targets only, it frees 1 document there, 1 in generated-notes, 3 in reference-corpus. The over-reach is real and nearly free; the rate is driven by the NAME branch (`<a>` 298, `<frame>` 94, `<img>` 63 per wiki corpus; `<base>` 113 in reference-corpus). Two further corrections the census forced: - The over-reach is not only MDX. In reference-corpus it lands on Azure APIM policy XML (`<set-header>` 50, `<ip-filter>` 3, `<set-query-parameter>` 2). - New bullet: APIM's attribute-less `<base />` collides with HTML's `<base>`, whose whole affordance is `href`. 30 of reference-corpus's 389 documents. Also pins corpus provenance. "Every population was swept twice and reproduced its counts exactly" is false for two of the three: vendor-harvest and generated-notes live in claude-code-llm-wiki, which re-harvests per Claude Code release. At its commit aba87e2 they now give 100/187 and 90/550; reference-corpus is static and reproduced 133/389 exactly. The published table keeps its original counts — measurement provenance is never silently bumped — and gains the provenance it was missing. README limitations count 33 -> 34.
This commit is contained in:
parent
0dce50fb05
commit
6e87a0ff16
2 changed files with 39 additions and 7 deletions
|
|
@ -242,7 +242,7 @@ a green scan means safe content. The highest-impact items:
|
|||
exact constructs the gate grades. Budget human review, or run a source you actually
|
||||
trust as trusted. Re-run it yourself with [`docs/fp-sweep.py`](docs/fp-sweep.py).
|
||||
|
||||
**Full list — 33 items, each with the mechanism, plus the out-of-scope boundary:**
|
||||
**Full list — 34 items, each with the mechanism, plus the out-of-scope boundary:**
|
||||
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
|
||||
consumer corpora, including the false positives the URL-shape rule actually produces.
|
||||
|
||||
|
|
|
|||
|
|
@ -228,6 +228,27 @@ items; this is the full list, each with the mechanism.
|
|||
framesets, which appear in essentially no modern documentation, while `Frame` is a
|
||||
common component name. Case is not an available discriminator: HTML is
|
||||
case-insensitive, so PascalCase cannot be treated as "component, not tag".
|
||||
**Correction (2026-08-11): this bullet was named as the cause of the raw-html
|
||||
false-positive rate, and it is not.** The bullet below used to attribute
|
||||
`active:raw-html` in *52 of vendor-harvest's 98* to this over-reach. Measured by
|
||||
re-running the gate with the URL-attribute branch narrowed to *external* targets
|
||||
only, it frees **1 document in vendor-harvest, 1 in generated-notes, 3 in
|
||||
reference-corpus** — the over-reach is real and it costs almost nothing, because
|
||||
the documents carrying it almost always also carry a tag that is active by *name*,
|
||||
and one raw-html finding per document is all it takes. What actually drives the
|
||||
rate is the **name branch**: per wiki corpus `<a>` 298 occurrences, `<frame>` 94,
|
||||
`<img>` 63; in reference-corpus `<base>` 113.
|
||||
**And the over-reach is not only MDX.** In reference-corpus it lands on Azure APIM
|
||||
policy XML — `<set-header>` 50, `<ip-filter>` 3, `<set-query-parameter>` 2 — which
|
||||
shares nothing with the `<Card href="/…">` shape beyond having a URL attribute.
|
||||
- **`<base />` in Azure APIM policy XML grades HIGH on the name branch alone.**
|
||||
APIM's `<base />` means "run the inherited policy" and carries no attributes; it
|
||||
collides with HTML's `<base>`, whose entire affordance is its `href`. It appears in
|
||||
**30 of reference-corpus's 389** documents. An attribute-less `<base>` has no
|
||||
affordance in any renderer, and a `<base href="…">` would still be caught by the
|
||||
URL-attribute branch — so the name-branch entry is doing no work here that the
|
||||
attribute branch does not already do. This is a distinct over-reach class from the
|
||||
MDX/APIM one above, and it was unnamed until the 2026-08-11 census.
|
||||
- **Raw-HTML findings count end tags.** `</a>` is active by name on its own, so a
|
||||
corpus census that counts only opening tags understates what this detector reports
|
||||
by roughly the ratio of closing to opening active tags (measured at 1.6× on one
|
||||
|
|
@ -261,9 +282,8 @@ items; this is the full list, each with the mechanism.
|
|||
**What moved them is mostly residuals this document already concedes**, counted by
|
||||
the labels at each document's *worst* severity (a histogram of every label present
|
||||
would credit the over-block to whatever else happened to be in the document). In
|
||||
vendor-harvest, `active:raw-html` is a top driver in **52 of the 98** — the MDX
|
||||
component / relative-attribute over-reach two bullets up — `markdown:link-anchor-
|
||||
injection` in 23, and only about ten documents are moved by genuinely
|
||||
vendor-harvest, `active:raw-html` is a top driver in **52 of the 98**,
|
||||
`markdown:link-anchor-injection` in 23, and only about ten documents are moved by genuinely
|
||||
injection-shaped text, which is what security-adjacent documentation contains
|
||||
honestly. Generated-notes tracks it almost exactly, as 184 shared ancestors imply
|
||||
— `active:raw-html` in 53 of its 88, `markdown:link-anchor-injection` in 23 — so
|
||||
|
|
@ -286,9 +306,21 @@ items; this is the full list, each with the mechanism.
|
|||
regression pass a green suite. Read the contrast as the intended one — the same
|
||||
corpus is cheap to persist from a source you trust and expensive from one you do
|
||||
not.
|
||||
**Every population was swept twice and reproduced its counts exactly**, and the
|
||||
largest document in any of them is 362 kB — no document approached the 1 000 000-
|
||||
character input cap, so truncation confounds nothing here.
|
||||
**Every population was swept twice and reproduced its counts exactly *within the
|
||||
sweep that produced them*, but two of the three are living corpora and no longer
|
||||
reproduce.** `vendor-harvest` and `generated-notes` are directories inside
|
||||
`claude-code-llm-wiki`, which re-harvests per Claude Code release; re-run on
|
||||
2026-08-11 against that repo at commit `aba87e2` they give **100 of 187** and
|
||||
**90 of 550**. The added documents are ordinary vendor documentation, not a
|
||||
detection change — the guard's behaviour did not move between the two runs.
|
||||
`reference-corpus` is static and reproduced **133 of 389** exactly. The table
|
||||
above is **not** restated to today's counts: it is a measurement with a date, and
|
||||
measurement provenance is never silently bumped. What was missing was the
|
||||
provenance itself — a reader who re-ran the first two rows got different numbers
|
||||
and had nothing in this file to explain why. Pin the corpus commit when you
|
||||
reproduce, or expect a drifting denominator.
|
||||
The largest document in any of them is 362 kB — no document approached the
|
||||
1 000 000-character input cap, so truncation confounds nothing here.
|
||||
**The sharpest datum needs no corpus at all: this repository's own eight published
|
||||
documents are 8 of 8 fail-secure at the upload door** (`python docs/fp-sweep.py
|
||||
self-docs=docs --ext=.md` on a clone, which also picks up any untracked local
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue