feat(extract): one normalisation door removes U+00AD before the guard

Operator decision 2026-09-18, and the whole of it: okf removes the soft
hyphen, the guard is untouched, and the number is BOOKED.

`normalise_extracted(text) -> (text, removed)` in `extract`, applied
once at the end of `extract_document`, so both entry points and both
sides of a plan's `text_sha256` see one string. `_pdf_units` measures
its page offsets through the same door, because that table is
CHARACTER offsets rebuilt from the pages while the text it indexes
comes back normalised -- two readings of one document, and a table
built against the other names the wrong page with full confidence.
Removing a character removes no newline, so every line-indexed rule
downstream is unmoved by construction.

`ExtractedDocument.soft_hyphens` carries the count out;
`InboxResult.normalised` carries it per document; `account_run` reads
it off the RUN rather than recounting the source, because a second
count would be a second reader. It reaches the accounting JSON as
`normalised_soft_hyphen` at both levels and `log.md` as a
`**Normalisation**` bullet naming the count, the documents touched and
that no other character is moved.

EXPOSURE, measured and not assumed: 0 of the 78 readable documents of
the K2 reference corpus carry U+00AD or any of the four real
zero-width characters (the 8 unreadable ones raise before extraction),
0 in the pinned K2 bundle's concept bodies, and 0 across
`tests/fixtures`, `examples`, `skills`, `docs`, `src`, README and
CHANGELOG. The door cannot have moved a byte anyone here has measured.

Suite 2179 passed, 1 skipped; ruff and `mypy --strict src/` clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

src/llm_ingestion_okf/extract.py

@@ -381,6 +381,46 @@ _REMOTE_SOURCE = re.compile(r"^(?:[a-zA-Z][a-zA-Z0-9+.-]*:|//)")
 _DATA_URI = re.compile(r"^data:(?P<media>[^;,]*)(?P<base64>;base64)?,(?P<payload>.*)$", re.DOTALL)
 
 
+#: The ONE character the normalisation door removes, and the reason it is one
+#: character and not a class. `llm-ingestion-guard` 1.4.0 puts U+00AD in
+#: `_ZERO_WIDTH_CPS` beside U+200B, U+200C, U+200D and U+FEFF, and
+#: `output:zero-width-present` is an any-tier carrier: `fail_secure` at every
+#: trust level, with no sanitisation and no exception. Measured by PM
+#: 2026-09-18, R761 Prosesskoden:2025 carries 71 U+00AD and 0 of the other
+#: four, so a 701-page process code is unreadable for the whole chain over
+#: Norwegian hyphenation points inside words -- `ar[SHY]beider`,
+#: `bitu[SHY]men`, `asfalt[SHY]betong`. The verdict is formally right and
+#: materially a false positive, and the operator's answer (2026-09-18) is that
+#: okf removes this character before the guard and SAYS SO in the accounting.
+#:
+#: The other four stay. They carry no typographic job in running text, so
+#: removing one would be a decision about what the guard screens for, taken in
+#: the wrong repository. U+00A0 NBSP is not in the guard's set at all and is
+#: not touched either -- R761 ships 6 633 of them.
+SOFT_HYPHEN = "\u00ad"
+
+
+def normalise_extracted(text: str) -> tuple[str, int]:
+    """The normalisation door: the text without U+00AD, and how many were removed.
+
+    Applied ONCE, at the end of :func:`extract_document`, so every caller of
+    either entry point gets the same string and `propose` and Door B cannot
+    disagree about the text a plan's `text_sha256` indexes. The only other
+    place that has to know about it is :func:`_pdf_units`, which rebuilds a
+    table of CHARACTER offsets from the pages rather than from the returned
+    text: two readings of one document, and a table built against the other
+    one would name the wrong page with full confidence.
+
+    Removing a character never removes a newline, so every LINE-indexed rule
+    downstream -- the proposer's grammars, `xml_outline`, `pdf_outline`'s
+    per-page line check -- is unmoved by construction.
+    """
+    removed = text.count(SOFT_HYPHEN)
+    if not removed:
+        return text, 0
+    return text.replace(SOFT_HYPHEN, ""), removed
+
+
 @dataclass(frozen=True)
 class ExtractedDocument:
     """One dropped file's text, and the images that stand inside that text.
@@ -400,6 +440,10 @@ class ExtractedDocument:
     #: unpointed file with the same bytes as a carried one was carried through
     #: nothing (R761 ships eight such duplicates).
     files: tuple[str, ...] = ()
+    #: How many U+00AD the normalisation door removed from this text. Zero for
+    #: every document that carried none, which is 0 of the 78 readable
+    #: documents of the K2 reference corpus (measured 2026-09-19).
+    soft_hyphens: int = 0
 
 
 class _AssetCollector:
@@ -2586,7 +2630,11 @@ def _pdf_units(data: bytes, headings: bool, ocr: bool, assets: bool = False) ->
         # The page as it reaches the text, pointers included: a locator built
         # from the body alone would drift by two lines per carried image and
         # would name the wrong page from the first one onwards.
-        offset += len(_pdf_page_text(page)) + len(_PDF_PAGE_SEPARATOR)
+        # Through the normalisation door for the same reason one step smaller:
+        # the text this table indexes has had its soft hyphens removed, so
+        # measuring the page before the door would drift by one character per
+        # hyphen and name the wrong page.
+        offset += len(normalise_extracted(_pdf_page_text(page))[0]) + len(_PDF_PAGE_SEPARATOR)
     return SourceUnits("pages", tuple(starts), tuple(numbers))
 
 
@@ -2776,9 +2824,11 @@ def extract_document(
         text = _ASSET_READERS[suffix](data, collector)
     else:
         text = extractor(data)
+    rendered, soft_hyphens = normalise_extracted(renderer(text) if renderer is not None else text)
     return ExtractedDocument(
-        text=renderer(text) if renderer is not None else text,
+        text=rendered,
         images=tuple(collector.images) if collector is not None else (),
         rejected=tuple(collector.rejected) if collector is not None else (),
         files=tuple(collector.files) if collector is not None else (),
+        soft_hyphens=soft_hyphens,
     )