feat(extract): one normalisation door removes U+00AD before the guard
Operator decision 2026-09-18, and the whole of it: okf removes the soft hyphen, the guard is untouched, and the number is BOOKED. `normalise_extracted(text) -> (text, removed)` in `extract`, applied once at the end of `extract_document`, so both entry points and both sides of a plan's `text_sha256` see one string. `_pdf_units` measures its page offsets through the same door, because that table is CHARACTER offsets rebuilt from the pages while the text it indexes comes back normalised -- two readings of one document, and a table built against the other names the wrong page with full confidence. Removing a character removes no newline, so every line-indexed rule downstream is unmoved by construction. `ExtractedDocument.soft_hyphens` carries the count out; `InboxResult.normalised` carries it per document; `account_run` reads it off the RUN rather than recounting the source, because a second count would be a second reader. It reaches the accounting JSON as `normalised_soft_hyphen` at both levels and `log.md` as a `**Normalisation**` bullet naming the count, the documents touched and that no other character is moved. EXPOSURE, measured and not assumed: 0 of the 78 readable documents of the K2 reference corpus carry U+00AD or any of the four real zero-width characters (the 8 unreadable ones raise before extraction), 0 in the pinned K2 bundle's concept bodies, and 0 across `tests/fixtures`, `examples`, `skills`, `docs`, `src`, README and CHANGELOG. The door cannot have moved a byte anyone here has measured. Suite 2179 passed, 1 skipped; ruff and `mypy --strict src/` clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
68cfa905a3
commit
eebaf534fa
3 changed files with 103 additions and 2 deletions
|
|
@ -643,6 +643,11 @@ class InboxResult:
|
|||
# ones that were found and not carried, with their codes. The content
|
||||
# accounting books a document's images from this, never from the bundle.
|
||||
document_assets: tuple[DocumentAssets, ...] = ()
|
||||
# Per document the run READ: how many U+00AD the normalisation door
|
||||
# removed before the persist gate saw the text. One entry per document
|
||||
# that carried at least one, so a run over a corpus with none of them
|
||||
# carries an empty tuple and says `0` rather than nothing.
|
||||
normalised: tuple[DocumentNormalisation, ...] = ()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
|
@ -654,6 +659,19 @@ class DocumentAssets:
|
|||
rejected: tuple[AssetRejection, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DocumentNormalisation:
|
||||
"""What the normalisation door removed from one document.
|
||||
|
||||
Recorded where the removal HAPPENED rather than counted again off the
|
||||
source afterwards: a second count would be a second reader, and the number
|
||||
the accounting publishes has to be the number the run acted on.
|
||||
"""
|
||||
|
||||
source_file: str
|
||||
soft_hyphens: int
|
||||
|
||||
|
||||
def relative_source(path: Path, inbox: Path) -> str:
|
||||
"""A dropped file's name as the provenance layer records it.
|
||||
|
||||
|
|
@ -1108,6 +1126,7 @@ def process_inbox(
|
|||
refused_assets: list[AssetRejection] = []
|
||||
carried_files: set[str] = set()
|
||||
document_assets: list[DocumentAssets] = []
|
||||
normalised: list[DocumentNormalisation] = []
|
||||
|
||||
# Phase 1: name every file BEFORE any gate call or write, so an intra-run
|
||||
# collision is caught while both files can still be refused together. Under
|
||||
|
|
@ -1417,6 +1436,12 @@ def process_inbox(
|
|||
posixpath.normpath((directory / reference).as_posix())
|
||||
for reference in document.files
|
||||
)
|
||||
if document.soft_hyphens:
|
||||
normalised.append(
|
||||
DocumentNormalisation(
|
||||
source_file=source_name(path), soft_hyphens=document.soft_hyphens
|
||||
)
|
||||
)
|
||||
if outputs:
|
||||
document_assets.append(
|
||||
DocumentAssets(
|
||||
|
|
@ -1485,6 +1510,7 @@ def process_inbox(
|
|||
assets_rejected=tuple(refused_assets),
|
||||
carried_files=tuple(sorted(carried_files)),
|
||||
document_assets=tuple(document_assets),
|
||||
normalised=tuple(normalised),
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue