feat(extract): one normalisation door removes U+00AD before the guard

Operator decision 2026-09-18, and the whole of it: okf removes the soft
hyphen, the guard is untouched, and the number is BOOKED.

`normalise_extracted(text) -> (text, removed)` in `extract`, applied
once at the end of `extract_document`, so both entry points and both
sides of a plan's `text_sha256` see one string. `_pdf_units` measures
its page offsets through the same door, because that table is
CHARACTER offsets rebuilt from the pages while the text it indexes
comes back normalised -- two readings of one document, and a table
built against the other names the wrong page with full confidence.
Removing a character removes no newline, so every line-indexed rule
downstream is unmoved by construction.

`ExtractedDocument.soft_hyphens` carries the count out;
`InboxResult.normalised` carries it per document; `account_run` reads
it off the RUN rather than recounting the source, because a second
count would be a second reader. It reaches the accounting JSON as
`normalised_soft_hyphen` at both levels and `log.md` as a
`**Normalisation**` bullet naming the count, the documents touched and
that no other character is moved.

EXPOSURE, measured and not assumed: 0 of the 78 readable documents of
the K2 reference corpus carry U+00AD or any of the four real
zero-width characters (the 8 unreadable ones raise before extraction),
0 in the pinned K2 bundle's concept bodies, and 0 across
`tests/fixtures`, `examples`, `skills`, `docs`, `src`, README and
CHANGELOG. The door cannot have moved a byte anyone here has measured.

Suite 2179 passed, 1 skipped; ruff and `mypy --strict src/` clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 06:10:53 +02:00
commit eebaf534fa
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
3 changed files with 103 additions and 2 deletions

View file

@ -643,6 +643,11 @@ class InboxResult:
# ones that were found and not carried, with their codes. The content
# accounting books a document's images from this, never from the bundle.
document_assets: tuple[DocumentAssets, ...] = ()
# Per document the run READ: how many U+00AD the normalisation door
# removed before the persist gate saw the text. One entry per document
# that carried at least one, so a run over a corpus with none of them
# carries an empty tuple and says `0` rather than nothing.
normalised: tuple[DocumentNormalisation, ...] = ()
@dataclass(frozen=True)
@ -654,6 +659,19 @@ class DocumentAssets:
rejected: tuple[AssetRejection, ...]
@dataclass(frozen=True)
class DocumentNormalisation:
"""What the normalisation door removed from one document.
Recorded where the removal HAPPENED rather than counted again off the
source afterwards: a second count would be a second reader, and the number
the accounting publishes has to be the number the run acted on.
"""
source_file: str
soft_hyphens: int
def relative_source(path: Path, inbox: Path) -> str:
"""A dropped file's name as the provenance layer records it.
@ -1108,6 +1126,7 @@ def process_inbox(
refused_assets: list[AssetRejection] = []
carried_files: set[str] = set()
document_assets: list[DocumentAssets] = []
normalised: list[DocumentNormalisation] = []
# Phase 1: name every file BEFORE any gate call or write, so an intra-run
# collision is caught while both files can still be refused together. Under
@ -1417,6 +1436,12 @@ def process_inbox(
posixpath.normpath((directory / reference).as_posix())
for reference in document.files
)
if document.soft_hyphens:
normalised.append(
DocumentNormalisation(
source_file=source_name(path), soft_hyphens=document.soft_hyphens
)
)
if outputs:
document_assets.append(
DocumentAssets(
@ -1485,6 +1510,7 @@ def process_inbox(
assets_rejected=tuple(refused_assets),
carried_files=tuple(sorted(carried_files)),
document_assets=tuple(document_assets),
normalised=tuple(normalised),
)