open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(accounting): the log states where the soft-hyphen count comes from

Browse source 
Chose the stated provenance over a second, independent counter, because a
second counter would be a number built to disagree. The door acts on the
EXTRACTED text; a counter inside `accounting.py` would have to re-read the
SOURCE bytes, and for every type extraction does not carry verbatim (pdf,
xml, html, docx) those are counts over two different strings. The gate would
then have to decide which difference is a loss and which is extraction doing
its job -- inventing a discrepancy where there is none. `accounting.py` does
not hold the source bytes at that point either; being a second reader of a
document is what `tools/okf_witness.py` exists for, outside the package.

Where an independent denominator DOES exist the tests already pin it from
outside the door: the fixture writes its own five, and R761's 71 is PM's
count over the delivery.

The `**Normalisation**` bullet now ends: "The count is the door's own, read
off the run and not recounted from the source."

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 08:42:15 +02:00
commit 7509b6b6bc
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
1 changed files with 7 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

9
src/llm_ingestion_okf/accounting.py
View file

@ -848,7 +848,8 @@ class Accounting:
lines.append(
f"* **Normalisation**: {self.normalised_soft_hyphen} soft hyphen(s) (U+00AD) "
f"removed from {touched} of {len(self.documents)} document(s) before the persist "
"gate. No other character is touched."
"gate. No other character is touched. The count is the door's own, read off the "
"run and not recounted from the source."
)
for doc in self.documents:
if doc.status == REJECTED:
@ -1007,7 +1008,11 @@ def account_run(inbox: Path, walked: Sequence[Path], result: InboxResult) -> Acc
)
# Read off the run, never recounted from the source: a second count would
# be a second reader, and the number this publishes has to be the number
# the normalisation door acted on.
# the normalisation door acted on. It is also the one number here with no
# independent denominator behind it, and the door acts on the EXTRACTED
# text -- so a second counter over the source bytes would disagree by
# construction on every type extraction does not carry verbatim. The
# provenance is stated in `log_lines` instead of hidden.
removed = {item.source_file: item.soft_hyphens for item in result.normalised}
for document in documents:
document.normalised_soft_hyphen = removed.get(document.source_file, 0)