test(fixtures): replace sector-specific example material with generic, fictitious examples — green
Every fixture, test document, tool example and document now uses an invented kitchen-and-baking handbook series, written in this repository. The package's behaviour is unchanged; src/ changes are comments and help text only. - Generated fixtures are regenerated from their generators. Their structural counts are identical before and after: elements, images, rows, cells, headings, bookmarks and the witness inventory's per-document totals. The image-inbox and accounting documents are renamed kapittel-84-*. - tools/okf_accounting_gate.py: the two options that named one real corpus each are replaced by a generic, repeatable --corpus PATH with no default. Row 5 compares the PDF pair alone. Gate verdict unchanged: RED rows 2, 3, 6. - tools/okf_witness.py: the STS JSON reader for one publisher's delivery is removed, along with its three twins and five tests. The mutation harness loses W09. - docs/: 13 dated reports that documented runs on a retired reference corpus are removed, and 40 are neutralized. Dead links are removed, and no new dangling path is introduced. - The synthetic MCP-gate corpus and the residual probe words are neutral. Valgt: keep the `okf quality --fasit` bar value (the measured fraction, one corpus) and rewrite only its provenance, because the verdict stays unchanged and the number names nothing. Term check with the local list: 0 of 411 tracked files, 0 file names, 0 of 27 binary fixtures. Suite after git add: 2457 passed, 1 skipped. The base tree had 2460 passed and 2 skipped; five tests went with the JSON reader and four were added by the term check. ruff, ruff format and mypy --strict src/ are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
1e7345a401
commit
9d1f4b14ed
174 changed files with 1889 additions and 6512 deletions
|
|
@ -1,12 +1,12 @@
|
|||
"""The soft-hyphen normalisation door, in front of the persist gate.
|
||||
|
||||
Operator decision 2026-09-18. R761 Prosesskoden:2025 is refused WHOLE by
|
||||
`llm-ingestion-guard` 1.4.0 -- `output:zero-width-present`, HIGH, an any-tier
|
||||
Operator decision 2026-09-18. A 701-page reference standard was refused WHOLE
|
||||
by `llm-ingestion-guard` 1.4.0 -- `output:zero-width-present`, HIGH, an any-tier
|
||||
carrier and therefore `fail_secure` at every trust level -- because the
|
||||
publisher's source carries 71 U+00AD SOFT HYPHEN and 0 real zero-width
|
||||
characters (U+200B, U+200C, U+200D, U+FEFF, U+2060 all 0, measured by PM on
|
||||
both deliveries). Those 71 are Norwegian hyphenation points inside words:
|
||||
`ar[SHY]beider`, `bitu[SHY]men`, `asfalt[SHY]betong`. The verdict is formally
|
||||
both deliveries). Those 71 are Norwegian hyphenation points inside words, of
|
||||
the kind `ar[SHY]beider` and `surdeigs[SHY]bakst`. The verdict is formally
|
||||
right and materially a false positive.
|
||||
|
||||
Of the three ways out, the operator chose this one: okf removes U+00AD BEFORE
|
||||
|
|
@ -17,9 +17,9 @@ before -- which is the known-negative every test here carries.
|
|||
The three alternatives and why they are not here: weakening the guard is the
|
||||
security repo's call and weakens every consumer's screen; delivering a cleaned
|
||||
corpus fixes one document and no other; and doing nothing leaves a 701-page
|
||||
process code unreadable for the whole chain.
|
||||
reference standard unreadable for the whole chain.
|
||||
|
||||
U+00A0 NBSP is NOT touched (6 633 of them in R761). The guard has no rule
|
||||
U+00A0 NBSP is NOT touched (6 633 of them in that document). The guard has no rule
|
||||
about it -- `_ZERO_WIDTH_CPS` is exactly {200B, 200C, 200D, FEFF, 00AD} on
|
||||
1.4.0, measured -- so nothing here has to.
|
||||
"""
|
||||
|
|
@ -41,7 +41,7 @@ ZERO_WIDTH = ""
|
|||
#: Every character the door must leave alone, each one a way for it to reach
|
||||
#: further than its own name. The four real zero-width carriers (the guard's
|
||||
#: set minus U+00AD) and U+2060, which the guard does not screen for at all;
|
||||
#: U+00A0 NBSP, which R761 ships 6 633 of; the three hyphens a reader would
|
||||
#: U+00A0 NBSP, which that document ships 6 633 of; the three hyphens a reader would
|
||||
#: confuse with a soft one -- U+002D HYPHEN-MINUS, U+2010 HYPHEN, U+2011
|
||||
#: NON-BREAKING HYPHEN; and a combining accent, because the door is the one
|
||||
#: place in the chain that does NOT normalise to NFC.
|
||||
|
|
@ -60,7 +60,7 @@ UNTOUCHED = (
|
|||
|
||||
_MIXED = (
|
||||
"".join(f"ar{{s}}beider {mark}{{s}}{mark} paa linje {n}\n" for n, mark in enumerate(UNTOUCHED))
|
||||
+ "asfalt{s}betong\n"
|
||||
+ "surdeigs{s}bakst\n"
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -69,7 +69,7 @@ def test_the_door_removes_one_character_and_leaves_every_other_one_where_it_was(
|
|||
README and CLAUDE.md repeat it. Until this test the sentence was held by
|
||||
nothing: PM's mutant P6 -- a door that ALSO eats U+00A0 -- passed the whole
|
||||
suite green (2 171 passed, RC 0), and it would have eaten all 6 633 NBSP
|
||||
in R761 while the log went on claiming otherwise.
|
||||
in that document while the log went on claiming otherwise.
|
||||
|
||||
The invariant is exact rather than a share, and it is stated as the
|
||||
stronger of the two equivalent forms: instead of putting the removed
|
||||
|
|
@ -134,7 +134,7 @@ def _build(inbox: Path, out: Path, accounting: Path) -> tuple[int, str]:
|
|||
|
||||
|
||||
_SOURCE = (
|
||||
"# Pro{s}sess 84\n\nAr{s}beider med bitu{s}men og asfalt{s}betong.\n\n"
|
||||
"# Kapit{s}tel 84\n\nAr{s}beider med sjoko{s}lade og surdeigs{s}bakst.\n\n"
|
||||
"# Krav\n\nTilsvar{s}ende krav gjelder.\n"
|
||||
)
|
||||
|
||||
|
|
@ -209,12 +209,12 @@ def test_the_pdf_page_table_is_measured_over_the_normalised_text(
|
|||
readings of one document, and a locator built against the other rendering
|
||||
points a consumer at the wrong page with full confidence."""
|
||||
pages = (
|
||||
extract._PdfPage(1, f"Ar{SHY}beider med bitu{SHY}men."),
|
||||
extract._PdfPage(2, f"Asfalt{SHY}betong."),
|
||||
extract._PdfPage(1, f"Ar{SHY}beider med sjoko{SHY}lade."),
|
||||
extract._PdfPage(2, f"Surdeigs{SHY}bakst."),
|
||||
)
|
||||
monkeypatch.setattr(extract, "_pdf_pages", lambda *a, **k: iter(pages))
|
||||
table = extract._pdf_units(b"", False, False)
|
||||
joined = extract._PDF_PAGE_SEPARATOR.join(extract._pdf_page_text(p) for p in pages)
|
||||
text, removed = extract.normalise_extracted(joined)
|
||||
assert removed == 3
|
||||
assert table.starts[1] == text.index("Asfaltbetong")
|
||||
assert table.starts[1] == text.index("Surdeigsbakst")
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue