llm-ingestion-okf/docs/2026-09-17-innholdsregnskapet-rod-gate.md

Content accounting in okf build: the gate, written red

Capability loop, step 3. This session adds no capability. It adds the gate a later capability has to turn green, together with the fasit that gate judges against.

• Command: python3 tools/okf_accounting_gate.py [--json] [--r761 DIR] [--consume]
• Exit codes: 0 when every failing row (1–6) is green, 1 otherwise, 2 when the gate did not run.
• Fasit: tools/okf_witness.py, committed as tests/fixtures/accounting/*inventory.json.
• Tests: tests/test_accounting_gate.py, 42 tests, green.

Why a gate at all

okf build's conservation identity, merged + coded rejections == N, counts FILES. Two things happen that the identity cannot see, and both were reproduced here on bc39e80 (v0.10.0) before any code was written:

• A file can be merged while content inside it is gone. No inventory of the source exists, so nothing can state what a merged file lost.
• A file can be carried and rejected at once (finding B). The fixture corpus has a graphics/ directory next to its documents. Its two PNGs are carried into assets/ through the documents that point at them, and the same two files are also counted as extractor_unknown rejections. Each has two fates.
• A rejected document is logged as if it held nothing (finding C). The fixture document the guard refuses produces Images: 0 carried of 0 found, although its source declares one image.

The fasit is independent

The witness imports no llm_ingestion_okf module. A test runs every witness in a fresh process and then reads sys.modules. A control run of the same check, with the package pre-loaded, shows the check fires.

The witness counts each format by the format's own rules:

• STS: ElementTree, and the publisher's JSON node tree.
• docx, pptx, xlsx and odt: the zip members' own XML.
• PDF: pdfplumber objects, and poppler (pdfinfo, pdfimages -list).
• HTML: html.parser.
• md, txt, csv, json and rtf: stdlib readers, and a control-word scan for rtf.

Four documents are also pinned to hand counts.

Two witnesses disagreed by tag, and both disagreements were about placement rather than content (R761, measured). A section's label is sec/label on 7 714 sections in the XML. The JSON puts it at sec/label on 4 954 of them and at sec/title/label on the other 2 760. A table's label is table-wrap/label in the XML (10) and table-wrap/table/caption in the JSON (10). The witness therefore counts ROLES, and the mapping is written in okf_witness._sts_role together with these numbers. After the mapping, the two R761 witnesses agree on all ten roles.

The PM's figures were re-measured and all hold: p 12 528, title 2 761, td 197, list-item 111, graphic 50, 109 other files in the zip (50 pointed at, 59 not). The PDF witnesses agree at 701 pages and 50 images; pdfplumber takes 50 s for that and poppler 1 s.

The door the capability must open

The gate reads okf build --accounting PATH: one JSON object with a per-document inventory taken before extraction and before the gate, per-element fates (carried / pointer / rejected{code}), and exactly one fate per inbox file that is not a document. It also reads one line in log.md per rejected document:

<source_file>: <M> elements found in the source, 0 carried: document rejected `<code>`

The full contract is in the module docstring. Until the flag exists, row 2 is 0 of 13, and row 3 books every element as unaccounted. For a file, it derives the file's fates from the bundle bytes (assets are named by content) and from K1b.

Finding A: why the guard refuses R761, and the decision

The cause is measured. The R761 XML carries 71 U+00AD SOFT HYPHEN and no other invisible codepoint, and the extracted text carries the same 71. Guard 1.4.0 lists U+00AD in _ZERO_WIDTH_CPS (output.py:250). One soft hyphen is enough to fail_secure at both tiers; the same string without it is clean. Image pointers are not the cause: --no-assets gives the same refusal, as the PM also measured.

This is a question for the guard. It was sent to llm-ingestion-pipeline-security with the repro, and nothing was built there.

Decision on exit 0 when nothing was persisted: okf build should NOT exit 0 when at least one document was extracted and none was persisted. The conservation identity holding is necessary, not sufficient.

• This repository already refuses the analogous case: a run with zero plans fails (exit 2) rather than emitting a flat bundle and calling it success.
• okf build is the trusted-source path, an operator pointing at their own folder, where "everything refused" is a finding and not a normal outcome.
• A script running okf build && okf skill would otherwise wire a skill to an empty bundle.

Door B's library function keeps its behaviour, because for a hostile inbox "all rejected" is correct. The change is an exit code, which this order forbids, so it belongs to the capability order and not here.

Exceptions proposed, none approved

The gate lists three exceptions in every run. None of them lowers a denominator:

• PDF headings, paragraphs and tables. A PDF without a structure tree declares none of them, so no witness can count them.
• xlsx images. The reader deliberately does not carry them.
• Images in md, txt, csv, json, odt and rtf. No reader for these types carries image bytes.

Approving any of them is the operator's decision.

Output at this commit

See the order's closing message for the full text. In short:

Row	Result	Status
1	13 of 13	GREEN
2	0 of 13	RED
3	0 of 15, u = 248, d = 2	RED
4	0 of 1	RED
5	4 of 4	GREEN
6	59 of 110 (R761, two builds)	RED

Row 6 runs two builds:

• Default gate: the document is refused, so every element is unaccounted.
• --gate none: the document is persisted, and the 50 pictures it points at are both carried and rejected, which is finding B at scale (d = 50). A first run reported 57: 7 of the 59 unpointed files share bytes with pointed ones, and bytes in assets/ had been credited to them. A file nothing points at is never carried; a test holds that.

Row 7 (--consume) is a diagnostic and never fails.

Limits

• The witness vocabulary is ours. A format element the witness does not name is invisible to the gate.
• The fixture corpus is constructed, one document per type. The real-corpus row is N = 1 (R761).
• Row 6 needs the R761 source on this machine. Without it the row is RED locally and SKIPPED when CI is set, never green.
• The --r761 default points into another repository's data directory, and the gate only reads from it.