feat(accounting): okf build accounts for every source element
okf build --accounting PATH inventories every source document before extraction, in the gate's per-format vocabulary, and after the run gives each element one fate (carried / pointer / coded rejection), written as JSON and summarised in log.md. "carried" is checked against the written concept bodies, so a gate that drops a line is found (test). Exit 1 on anything unaccounted or double-booked. Opt-in: +744 s (+19 %) on the 43-document reference corpus, and that corpus fails the check on 24 real losses (22 images on text-less PDF pages, 2 docx Title paragraphs). Changed without the flag: - okf build exits 1 when it extracted documents and persisted none. Door B and corpus.measure are unchanged. One test relied on exit 0. - An image file carried through a persisted document is its own K1b column, no longer also extractor_unknown. The set is what the resolver actually carried (ExtractedDocument.files), never a byte match. tools/okf_accounting_gate.py (checks untouched) is green on all six rows, R761 110 of 110 under both gates. Report: docs/2026-09-17-innholdsregnskapet-bygget.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
0b00de4408
commit
864570b320
13 changed files with 1751 additions and 59 deletions
|
|
@ -386,6 +386,12 @@ class ExtractedDocument:
|
|||
text: str
|
||||
images: tuple[ExtractedImage, ...] = ()
|
||||
rejected: tuple[AssetRejection, ...] = ()
|
||||
#: The references, relative to the document's own directory, whose bytes
|
||||
#: the resolver returned and that were CARRIED as images. Recorded where
|
||||
#: the resolution happened rather than inferred from bytes afterwards: an
|
||||
#: unpointed file with the same bytes as a carried one was carried through
|
||||
#: nothing (R761 ships eight such duplicates).
|
||||
files: tuple[str, ...] = ()
|
||||
|
||||
|
||||
class _AssetCollector:
|
||||
|
|
@ -402,6 +408,7 @@ class _AssetCollector:
|
|||
self._resolve = resolve
|
||||
self.images: list[ExtractedImage] = []
|
||||
self.rejected: list[AssetRejection] = []
|
||||
self.files: list[str] = []
|
||||
|
||||
def carry(self, data: bytes, *, name: str, label: str | None = None) -> str:
|
||||
"""Bytes the reader already holds, as the block that stands in the text."""
|
||||
|
|
@ -445,8 +452,10 @@ class _AssetCollector:
|
|||
label=label,
|
||||
href=source,
|
||||
)
|
||||
found = source
|
||||
data = self._resolve(source) if self._resolve is not None else None
|
||||
if data is None and sibling is not None and sibling != source and self._resolve is not None:
|
||||
found = sibling
|
||||
data = self._resolve(sibling)
|
||||
if data is None:
|
||||
return self.reject(
|
||||
|
|
@ -455,7 +464,11 @@ class _AssetCollector:
|
|||
reason="the file the document points at was not found beside it",
|
||||
label=label,
|
||||
)
|
||||
return self.carry(data, name=source, label=label)
|
||||
carried = len(self.images)
|
||||
block = self.carry(data, name=source, label=label)
|
||||
if len(self.images) > carried:
|
||||
self.files.append(found)
|
||||
return block
|
||||
|
||||
def _data_uri(self, match: re.Match[str], *, label: str | None) -> str:
|
||||
payload = match.group("payload")
|
||||
|
|
@ -2408,4 +2421,5 @@ def extract_document(
|
|||
text=renderer(text) if renderer is not None else text,
|
||||
images=tuple(collector.images) if collector is not None else (),
|
||||
rejected=tuple(collector.rejected) if collector is not None else (),
|
||||
files=tuple(collector.files) if collector is not None else (),
|
||||
)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue