test(fixtures): replace sector-specific example material with generic, fictitious examples — green

Every fixture, test document, tool example and document now uses an invented
kitchen-and-baking handbook series, written in this repository. The package's
behaviour is unchanged; src/ changes are comments and help text only.

- Generated fixtures are regenerated from their generators. Their structural
  counts are identical before and after: elements, images, rows, cells,
  headings, bookmarks and the witness inventory's per-document totals. The
  image-inbox and accounting documents are renamed kapittel-84-*.
- tools/okf_accounting_gate.py: the two options that named one real corpus
  each are replaced by a generic, repeatable --corpus PATH with no default.
  Row 5 compares the PDF pair alone. Gate verdict unchanged: RED rows 2, 3, 6.
- tools/okf_witness.py: the STS JSON reader for one publisher's delivery is
  removed, along with its three twins and five tests. The mutation harness
  loses W09.
- docs/: 13 dated reports that documented runs on a retired reference corpus
  are removed, and 40 are neutralized. Dead links are removed, and no new
  dangling path is introduced.
- The synthetic MCP-gate corpus and the residual probe words are neutral.

Valgt: keep the `okf quality --fasit` bar value (the measured fraction, one corpus) and
rewrite only its provenance, because the verdict stays unchanged and the
number names nothing.

Term check with the local list: 0 of 411 tracked files, 0 file names, 0 of
27 binary fixtures. Suite after git add: 2457 passed, 1 skipped. The base
tree had 2460 passed and 2 skipped; five tests went with the JSON reader and
four were added by the term check. ruff, ruff format and mypy --strict src/
are clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 13:54:57 +02:00
commit 9d1f4b14ed
174 changed files with 1889 additions and 6512 deletions

View file

@ -1,12 +1,12 @@
"""The soft-hyphen normalisation door, in front of the persist gate.
Operator decision 2026-09-18. R761 Prosesskoden:2025 is refused WHOLE by
`llm-ingestion-guard` 1.4.0 -- `output:zero-width-present`, HIGH, an any-tier
Operator decision 2026-09-18. A 701-page reference standard was refused WHOLE
by `llm-ingestion-guard` 1.4.0 -- `output:zero-width-present`, HIGH, an any-tier
carrier and therefore `fail_secure` at every trust level -- because the
publisher's source carries 71 U+00AD SOFT HYPHEN and 0 real zero-width
characters (U+200B, U+200C, U+200D, U+FEFF, U+2060 all 0, measured by PM on
both deliveries). Those 71 are Norwegian hyphenation points inside words:
`ar[SHY]beider`, `bitu[SHY]men`, `asfalt[SHY]betong`. The verdict is formally
both deliveries). Those 71 are Norwegian hyphenation points inside words, of
the kind `ar[SHY]beider` and `surdeigs[SHY]bakst`. The verdict is formally
right and materially a false positive.
Of the three ways out, the operator chose this one: okf removes U+00AD BEFORE
@ -17,9 +17,9 @@ before -- which is the known-negative every test here carries.
The three alternatives and why they are not here: weakening the guard is the
security repo's call and weakens every consumer's screen; delivering a cleaned
corpus fixes one document and no other; and doing nothing leaves a 701-page
process code unreadable for the whole chain.
reference standard unreadable for the whole chain.
U+00A0 NBSP is NOT touched (6 633 of them in R761). The guard has no rule
U+00A0 NBSP is NOT touched (6 633 of them in that document). The guard has no rule
about it -- `_ZERO_WIDTH_CPS` is exactly {200B, 200C, 200D, FEFF, 00AD} on
1.4.0, measured -- so nothing here has to.
"""
@ -41,7 +41,7 @@ ZERO_WIDTH = "​"
#: Every character the door must leave alone, each one a way for it to reach
#: further than its own name. The four real zero-width carriers (the guard's
#: set minus U+00AD) and U+2060, which the guard does not screen for at all;
#: U+00A0 NBSP, which R761 ships 6 633 of; the three hyphens a reader would
#: U+00A0 NBSP, which that document ships 6 633 of; the three hyphens a reader would
#: confuse with a soft one -- U+002D HYPHEN-MINUS, U+2010 HYPHEN, U+2011
#: NON-BREAKING HYPHEN; and a combining accent, because the door is the one
#: place in the chain that does NOT normalise to NFC.
@ -60,7 +60,7 @@ UNTOUCHED = (
_MIXED = (
"".join(f"ar{{s}}beider {mark}{{s}}{mark} paa linje {n}\n" for n, mark in enumerate(UNTOUCHED))
+ "asfalt{s}betong\n"
+ "surdeigs{s}bakst\n"
)
@ -69,7 +69,7 @@ def test_the_door_removes_one_character_and_leaves_every_other_one_where_it_was(
README and CLAUDE.md repeat it. Until this test the sentence was held by
nothing: PM's mutant P6 -- a door that ALSO eats U+00A0 -- passed the whole
suite green (2 171 passed, RC 0), and it would have eaten all 6 633 NBSP
in R761 while the log went on claiming otherwise.
in that document while the log went on claiming otherwise.
The invariant is exact rather than a share, and it is stated as the
stronger of the two equivalent forms: instead of putting the removed
@ -134,7 +134,7 @@ def _build(inbox: Path, out: Path, accounting: Path) -> tuple[int, str]:
_SOURCE = (
"# Pro{s}sess 84\n\nAr{s}beider med bitu{s}men og asfalt{s}betong.\n\n"
"# Kapit{s}tel 84\n\nAr{s}beider med sjoko{s}lade og surdeigs{s}bakst.\n\n"
"# Krav\n\nTilsvar{s}ende krav gjelder.\n"
)
@ -209,12 +209,12 @@ def test_the_pdf_page_table_is_measured_over_the_normalised_text(
readings of one document, and a locator built against the other rendering
points a consumer at the wrong page with full confidence."""
pages = (
extract._PdfPage(1, f"Ar{SHY}beider med bitu{SHY}men."),
extract._PdfPage(2, f"Asfalt{SHY}betong."),
extract._PdfPage(1, f"Ar{SHY}beider med sjoko{SHY}lade."),
extract._PdfPage(2, f"Surdeigs{SHY}bakst."),
)
monkeypatch.setattr(extract, "_pdf_pages", lambda *a, **k: iter(pages))
table = extract._pdf_units(b"", False, False)
joined = extract._PDF_PAGE_SEPARATOR.join(extract._pdf_page_text(p) for p in pages)
text, removed = extract.normalise_extracted(joined)
assert removed == 3
assert table.starts[1] == text.index("Asfaltbetong")
assert table.starts[1] == text.index("Surdeigsbakst")