1
0
Fork 0
llm-ingestion-pipeline-secu.../docs
Kjell Tore Guttormsen 0df7e87c2f test(docs-scripts): the census re-derived the URL reader it measures, and nothing would have caught the drift
`docs/fp-sweep.py` and `docs/rawhtml-census.py` produce the numbers published in
`docs/LIMITATIONS.md`, and both reach past the public API into private module
state -- `active_content._ACTIVE_TAGS`, `_EVENT_ATTR_RE`, `_URL_ATTR_RE`,
`_has_external_target`, `calibration.RISK_RANK`. A rename inside `src/` broke
them while the suite stayed green, and the breakage would have surfaced months
later, at the moment someone tried to re-measure a published claim. This was the
last uncovered contract in the repo.

The coverage found a live one. `rawhtml-census.has_external_url_attr` parsed
attributes with its own pattern instead of calling the shipped reader, and the
copy had drifted on two shapes:

  - a URL attribute reached through a prefix. `_URL_ATTR_RE` matches `src=`
    inside `data-src=` on a word boundary, so the shipped predicate reads the
    value and blocks; the census's own name table saw `data-src` and skipped it.
  - a multi-candidate `srcset`. The shipped reader splits on `[,\s]+` so a
    relative first candidate cannot mask an external one behind it; the census
    tested the whole attribute value as a single URL.

Both made the `A` candidate rows free documents the shipped predicate keeps --
under-counting against the `PRODUCTION` row printed directly beside them, which
that row exists to expose. The census now delegates to
`active_content._url_attr_is_external`, so there is one reader, not two. This
repo already carries the general form of that lesson in `docs/URL-SHAPE.md`:
three consumers reconstructed a predicate from prose and each got a different
wrong answer.

NO PUBLISHED NUMBER MOVED. Re-measured against all three live populations after
the fix, in one session each: reference-corpus 389 docs (A frees 3, base-url 13,
both 25 -- the numbers in `docs/LIMITATIONS.md`, unchanged), vendor-harvest 187,
generated-notes 550. `PRODUCTION` equals `A + base-url` in all three (108/108,
98/98, 88/88), and vendor-harvest exercises the corrected branch for real (8
external `Card` attributes). The defect was latent, not published.

The tests are scoped to what a test here can honestly hold. The corpora live
outside this repo in private consumer repos, so neither script can be run end to
end from the suite and a stand-in corpus would only pin a fiction. What is
pinned: every imported name still exists with the shape used; `fp-sweep`'s
metric guard fails when the action map is re-mapped (a guard that cannot fail
protects nothing); `measure` still reads `.disposition` and `.assessment` and
excludes empty files from the denominator; the census's in-process patch point
still moves the gate, so a census patching a dead symbol cannot print six
identical rows and read as a finding; `PRODUCTION` equals `A + base-url` across
every branch of the predicate; and both scripts still refuse an argument-less
run rather than measuring nothing.

759 passing (was 736). Coverage matrix unchanged at 128/128 with 6/6 documented
gaps holding. The `736` in `docs/ADOPTION-BRIEF.md` is scoped "as of v0.6.1" and
is correct for that tag; it moves to 759 at the next version bump.
2026-08-11 23:01:58 +02:00
..
ADOPTION-BRIEF.md release(0.6.1): the ZWJ fix gets a tag a consumer can pin 2026-08-11 22:30:32 +02:00
BRIEF.md release(0.6.0): the raw-html narrowing ships, and the version claims it already made become true 2026-08-11 18:01:36 +02:00
fp-sweep.py measure(fp): the upload door costs a human on 16-53% of benign documents 2026-08-10 21:32:56 +02:00
LIMITATIONS.md fix(zwj): the zero-width check tested identity, so every emoji-composed document was hard-blocked 2026-08-11 22:14:47 +02:00
OKF-INGESTION-BRIEF.md docs: version-sync + SECURITY/CONTRIBUTING + honest-limits (Session E) 2026-07-15 10:08:24 +02:00
PLAN-v1.md docs(plan): the v1.0 gate rested on okf's old pin, and its file list on four surfaces 2026-08-11 06:46:29 +02:00
PLAN.md fix(output): 19 quadratic regex runs on the output path, worst ~5.7h at the cap 2026-07-31 18:31:58 +02:00
rawhtml-census.py test(docs-scripts): the census re-derived the URL reader it measures, and nothing would have caught the drift 2026-08-11 23:01:58 +02:00
redos-sweep.py fix(sanitize,okf,active_content): three quadratic patterns, two on the input path 2026-08-01 20:06:36 +02:00
review-2026-07.md docs: reframe novelty claim to composite write-time contract (review MAJOR #3) 2026-07-15 09:22:38 +02:00
URL-SHAPE.md docs(url-shape): make the rule reconstructable, and record what three corpora measured 2026-07-27 08:56:24 +02:00