feat(accounting): okf build accounts for every source element

okf build --accounting PATH inventories every source document before
extraction, in the gate's per-format vocabulary, and after the run gives
each element one fate (carried / pointer / coded rejection), written as
JSON and summarised in log.md. "carried" is checked against the written
concept bodies, so a gate that drops a line is found (test). Exit 1 on
anything unaccounted or double-booked. Opt-in: +744 s (+19 %) on the
43-document reference corpus, and that corpus fails the check on 24 real
losses (22 images on text-less PDF pages, 2 docx Title paragraphs).

Changed without the flag:
- okf build exits 1 when it extracted documents and persisted none.
  Door B and corpus.measure are unchanged. One test relied on exit 0.
- An image file carried through a persisted document is its own K1b
  column, no longer also extractor_unknown. The set is what the resolver
  actually carried (ExtractedDocument.files), never a byte match.

tools/okf_accounting_gate.py (checks untouched) is green on all six rows,
R761 110 of 110 under both gates.

Report: docs/2026-09-17-innholdsregnskapet-bygget.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-17 18:35:20 +02:00
commit 864570b320
13 changed files with 1751 additions and 59 deletions

View file

@ -386,6 +386,12 @@ class ExtractedDocument:
text: str
images: tuple[ExtractedImage, ...] = ()
rejected: tuple[AssetRejection, ...] = ()
#: The references, relative to the document's own directory, whose bytes
#: the resolver returned and that were CARRIED as images. Recorded where
#: the resolution happened rather than inferred from bytes afterwards: an
#: unpointed file with the same bytes as a carried one was carried through
#: nothing (R761 ships eight such duplicates).
files: tuple[str, ...] = ()
class _AssetCollector:
@ -402,6 +408,7 @@ class _AssetCollector:
self._resolve = resolve
self.images: list[ExtractedImage] = []
self.rejected: list[AssetRejection] = []
self.files: list[str] = []
def carry(self, data: bytes, *, name: str, label: str | None = None) -> str:
"""Bytes the reader already holds, as the block that stands in the text."""
@ -445,8 +452,10 @@ class _AssetCollector:
label=label,
href=source,
)
found = source
data = self._resolve(source) if self._resolve is not None else None
if data is None and sibling is not None and sibling != source and self._resolve is not None:
found = sibling
data = self._resolve(sibling)
if data is None:
return self.reject(
@ -455,7 +464,11 @@ class _AssetCollector:
reason="the file the document points at was not found beside it",
label=label,
)
return self.carry(data, name=source, label=label)
carried = len(self.images)
block = self.carry(data, name=source, label=label)
if len(self.images) > carried:
self.files.append(found)
return block
def _data_uri(self, match: re.Match[str], *, label: str | None) -> str:
payload = match.group("payload")
@ -2408,4 +2421,5 @@ def extract_document(
text=renderer(text) if renderer is not None else text,
images=tuple(collector.images) if collector is not None else (),
rejected=tuple(collector.rejected) if collector is not None else (),
files=tuple(collector.files) if collector is not None else (),
)