test(extract): the door's promise "no other character is touched" is held
`log.md` writes that sentence on every run and README and CLAUDE.md repeat
it. Until now nothing held it: PM's mutant P6 -- a door that ALSO removes
U+00A0 -- passed the entire suite green (2 171 passed, RC 0) while it would
have eaten all 6 633 NBSP in R761 with the log still claiming otherwise.
Ten characters, each a way for the door to reach past its own name: the four
real zero-width carriers, U+2060, U+00A0, the three hyphens a reader confuses
with a soft one (U+002D, U+2010, U+2011) and a combining accent (the door is
the one step that does NOT normalise to NFC). Each appears at least twice in
the fixture -- asserted, so the loop cannot pass over an empty set.
The invariant is the stronger of the two equivalent forms: rather than
reinserting the removed characters at their booked positions, the test builds
the expected string with its OWN filter over the source. That pins ORDER as
well as multiset, and needs no positions -- which the door does not return.
All three numbers (21 removed, the expected string, every surviving count)
come from the test, never from the door. The second test repeats it through
`extract_document`, the one place the door is applied, so a second remover
beside it is red too.
Red proven in a scratch copy of HEAD (`/tmp/shy-mut`, verified that the
scratch `src/` is the code that runs), control green at 7 passed:
P6 door also removes U+00A0 -> 2 failed
`assert text == expected` / `AssertionError: assert 'arbeider pa...'
== 'arbeider \xa...'`
P2010 door also removes U+2010 -> 2 failed
P6b door collapses U+00A0 to " " -> 2 failed
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
d1de1a6a9d
commit
0da282894f
1 changed files with 74 additions and 0 deletions
|
|
@ -38,6 +38,80 @@ from llm_ingestion_okf import cli, extract
|
|||
SHY = ""
|
||||
ZERO_WIDTH = ""
|
||||
|
||||
#: Every character the door must leave alone, each one a way for it to reach
|
||||
#: further than its own name. The four real zero-width carriers (the guard's
|
||||
#: set minus U+00AD) and U+2060, which the guard does not screen for at all;
|
||||
#: U+00A0 NBSP, which R761 ships 6 633 of; the three hyphens a reader would
|
||||
#: confuse with a soft one -- U+002D HYPHEN-MINUS, U+2010 HYPHEN, U+2011
|
||||
#: NON-BREAKING HYPHEN; and a combining accent, because the door is the one
|
||||
#: place in the chain that does NOT normalise to NFC.
|
||||
UNTOUCHED = (
|
||||
"\u00a0",
|
||||
"\u200b",
|
||||
"\u200c",
|
||||
"\u200d",
|
||||
"\ufeff",
|
||||
"\u2060",
|
||||
"-",
|
||||
"\u2010",
|
||||
"\u2011",
|
||||
"\u0301",
|
||||
)
|
||||
|
||||
_MIXED = (
|
||||
"".join(f"ar{{s}}beider {mark}{{s}}{mark} paa linje {n}\n" for n, mark in enumerate(UNTOUCHED))
|
||||
+ "asfalt{s}betong\n"
|
||||
)
|
||||
|
||||
|
||||
def test_the_door_removes_one_character_and_leaves_every_other_one_where_it_was() -> None:
|
||||
"""`log.md` says "No other character is touched" on every single run, and
|
||||
README and CLAUDE.md repeat it. Until this test the sentence was held by
|
||||
nothing: PM's mutant P6 -- a door that ALSO eats U+00A0 -- passed the whole
|
||||
suite green (2 171 passed, RC 0), and it would have eaten all 6 633 NBSP
|
||||
in R761 while the log went on claiming otherwise.
|
||||
|
||||
The invariant is exact rather than a share, and it is stated as the
|
||||
stronger of the two equivalent forms: instead of putting the removed
|
||||
characters back at their booked positions and comparing, the test builds
|
||||
the expected string with its OWN filter over the source. That fixes not
|
||||
only the multiset of surviving characters but their ORDER, so a door that
|
||||
removed and re-inserted elsewhere is caught too -- and it needs no
|
||||
positions, which the door does not return.
|
||||
|
||||
All three counts come from the test, never from the door.
|
||||
"""
|
||||
source = _MIXED.format(s=SHY)
|
||||
for mark in UNTOUCHED:
|
||||
assert source.count(mark) >= 2, mark
|
||||
assert len(UNTOUCHED) == 10
|
||||
|
||||
removed_here = sum(1 for character in source if character == SHY)
|
||||
expected = "".join(character for character in source if character != SHY)
|
||||
assert removed_here == 21
|
||||
|
||||
text, removed = extract.normalise_extracted(source)
|
||||
assert removed == removed_here
|
||||
assert text == expected
|
||||
assert SHY not in text
|
||||
for mark in UNTOUCHED:
|
||||
assert text.count(mark) == source.count(mark), mark
|
||||
|
||||
|
||||
def test_the_promise_holds_through_the_one_door_and_not_only_in_it() -> None:
|
||||
"""The same invariant one layer out, where the sentence is actually
|
||||
published: `extract_document` is the single place the door is applied, and
|
||||
a second remover sitting beside it would leave this test red while the
|
||||
function above stayed green. `.txt` is the extractor that returns the
|
||||
source verbatim, so the comparison is against the bytes that went in."""
|
||||
source = _MIXED.format(s=SHY)
|
||||
expected = "".join(character for character in source if character != SHY)
|
||||
document = extract.extract_document("notat.txt", source.encode("utf-8"))
|
||||
assert document.text == expected
|
||||
assert document.soft_hyphens == sum(1 for character in source if character == SHY)
|
||||
for mark in UNTOUCHED:
|
||||
assert document.text.count(mark) == source.count(mark), mark
|
||||
|
||||
|
||||
def _build(inbox: Path, out: Path, accounting: Path) -> tuple[int, str]:
|
||||
err = io.StringIO()
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue