open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(accounting): okf build accounts for every source element

Browse source 
okf build --accounting PATH inventories every source document before
extraction, in the gate's per-format vocabulary, and after the run gives
each element one fate (carried / pointer / coded rejection), written as
JSON and summarised in log.md. "carried" is checked against the written
concept bodies, so a gate that drops a line is found (test). Exit 1 on
anything unaccounted or double-booked. Opt-in: +744 s (+19 %) on the
43-document reference corpus, and that corpus fails the check on 24 real
losses (22 images on text-less PDF pages, 2 docx Title paragraphs).

Changed without the flag:
- okf build exits 1 when it extracted documents and persisted none.
  Door B and corpus.measure are unchanged. One test relied on exit 0.
- An image file carried through a persisted document is its own K1b
  column, no longer also extractor_unknown. The set is what the resolver
  actually carried (ExtractedDocument.files), never a byte match.

tools/okf_accounting_gate.py (checks untouched) is green on all six rows,
R761 110 of 110 under both gates.

Report: docs/2026-09-17-innholdsregnskapet-bygget.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-17 18:35:20 +02:00
commit 864570b320
13 changed files with 1751 additions and 59 deletions
Download patch file Download diff file Expand all files Collapse all files

177
docs/2026-09-17-innholdsregnskapet-bygget.md Normal file
View file

@ -0,0 +1,177 @@
# Content accounting in `okf build`: built
Capability loop, step 5. This work carries out order
`20260917T134442Z-3174112963`. The judge is `tools/okf_accounting_gate.py`,
written red at `0b00de4`. Its checks are unchanged here.
## Result
The gate is GREEN on all six rows (exit 0):
- rows 1 to 5 on the fixtures;
- row 6 on R761 Prosesskoden:2025, 110 of 110 units under both builds.
None of the three proposed exceptions was needed for these corpora. They are
still not approved, and the build does not apply them. A workbook carrying an
image, which the reader does not carry, is reported as one unaccounted element
and exits 1. `tests/test_content_accounting.py` holds that.
## What changed
1. **`okf build --accounting PATH`** (`src/llm_ingestion_okf/accounting.py`).
- **Before extraction:** an inventory of every source document in the
gate's per-format vocabulary.
- **After the run:** one fate per element — `carried`, `pointer` or a coded
rejection — written as JSON to PATH and summarised in `log.md`.
- **Exit:** 1 when anything is unaccounted or double-booked.
2. **`carried` is checked, not declared.** Every piece of an element's text
must be present in the concept bodies written for that document. Both sides
are compared as letters and digits only, case-folded. A container is
carried when its contents are.
- **Images** are booked from what the reader did with each placement:
carried, `asset_remote` (booked as `pointer`), or the reader's rejection
code.
- **A refused document** books every element under the gate's code.
3. **One fate per file.**
- An image file that a persisted document carried is no longer also an
`extractor_unknown` rejection. It has its own column in the conservation
identity.
- The set of carried files comes from what the resolver actually resolved
and carried (`ExtractedDocument.files`), never from byte equality.
- Why not byte equality: the gate's own first run on R761 credited 7
unpointed files that share bytes with pointed ones (d = 57, not 50).
4. **Exit 1 when documents were extracted and none persisted.** This is the
decision recorded in `docs/2026-09-17-innholdsregnskapet-rod-gate.md`.
- Door B's library function and `corpus.measure` are unchanged.
- One test in this repository relied on exit 0 here:
`tests/test_cli_gate.py::test_build_refuses_a_document_the_real_guard_refuses`.
- No script relies on it.
- `okf project` calls the build as a function, so it does not see the new
exit code. A project over a folder where every document was refused
still writes a skill. This is noted, not changed.
5. **`log.md`.**
- With `--accounting`, the `Images` bullet counts what the sources declare,
and each refused document gets its own line.
- The K1b line names the carried column only when that column is
non-zero. The pre-change line is therefore byte-identical for a corpus
without such files, which is tested.
## The check can fail, and where it is weak
- **Known-positive.** A gate that silently drops one line of `notat.md`
leaves K1b intact. It is reported as `paragraph 1 unaccounted`, and the
build exits 1 (test).
- **Control on R761, measured.** The document was built with `--gate none`,
as one concept, and then its written text was cut.
| Written text | Result |
|---|---|
| whole | 0 unaccounted over 31 096 elements |
| cut to half | 4 823 paragraphs, 3 621 sections and 55 list items unaccounted — but only 3 titles and 16 section labels |
| first 200 characters only | 12 475 of 12 528 paragraphs and 2 761 of 2 761 titles unaccounted, but 150 of 197 cells |
- **The limit.** The check proves that a string is present, not where it is.
A short element (a label such as `84.1`, a one-word title, a one-number
cell) is often present elsewhere in the same document, so its loss can go
unseen. Long elements are well covered.
- **Why not stricter ordered matching.** It would close part of this, but the
inventory's order is not always the written order: the pptx inventory lists
a slide's tables before its shapes. It is not built.
## Independence, stated precisely
`tools/okf_witness.py` imports nothing from this package; a test proves that on
the live import graph. The package's inventory implements the SAME
definitions a second time, and for markdown it reuses the proposer's own fence
rule (`propose.fenced_lines`), where the witness has its own. Agreement on the
fixtures (row 2, 13 of 13) is evidence that the definitions can be implemented
as written. It is not an independent count of the source. The independent
count is the witness's, and it is committed as data.
Known divergence, not exercised by any fixture: two fenced blocks with no
line between them are one `code_block` here and two in the witness.
## Cost
Measured on the 43-document reference corpus (`K2/trinn1`), built twice
from one snapshot of `src/`, each time as one run under `/usr/bin/time -l`:
| | wall | peak RSS | result |
|---|---:|---:|---|
| without `--accounting` | 3 902 s | 8.83 GB | exit 0 |
| with `--accounting` | 4 646 s | 9.36 GB | exit 1, 28 unaccounted |
- **The difference is +744 s (+19 %) and +0.53 GB.** The run without the flag
shared the machine with the test suite and two gate runs, so its wall time
is inflated and the true difference is likely larger. The 0.10.0 figure for
the same build was 3 088 s.
- **Most of the cost is the PDF inventory**, which opens every PDF a second
time with pdfplumber to list image placements.
- **Every concept file is the same in both builds except two,** and the
reason is not the flag. It is a 0.10.0 determinism defect (see Findings).
`log.md` differs by the accounting lines, as it should.
**Why the door is not the default.** It is opt-in for two reasons:
1. It costs about a fifth of the build time.
2. On the reference corpus, the check FAILS. A default-on flag would turn
every existing `okf build` of that corpus into exit 1 until the defects
below are fixed, or an exception is approved.
That second point is the door working as intended, and it is also a breaking
change nobody has decided on.
## Findings on the reference corpus
The door reported 28 unaccounted elements. Four of them were false positives
in the check itself, and both mechanisms are now fixed, each with a test:
- **A converter attribute between carried words:** `[Sted]{.mark}` for a
highlighted placeholder (1 element).
- **A `w:br` inside a table cell,** which the converter writes on two
grid-table rows (3 elements: the paragraph, its cell
and its table).
Recomputed on the same bundle after those fixes, **24 remain, and all 24 are
real losses:**
1. **22 images on PDF pages that have no text layer.**
- Affected: `Bilag 9.1` (16), `RIG-R02` (4), `Bilag 9.2` (2).
- Cause: `_pdf_pages` drops a page with no text, and its images with it.
They are neither carried nor rejected.
- pdfplumber lists 212 / 102 / 6 placements where the reader handled
196 / 98 / 4. The difference equals the placements on text-less pages,
exactly, in all three documents.
- These are plausibly the most important pictures in those files, because
a page with no text is a drawing or a scan.
2. **2 docx headings in the `Tittel` (Title) style.**
- Affected: Vedlegg 2 and Vedlegg 3.
- Cause: the converter moves a Title paragraph into document metadata, so
the body has an empty `# ` where the title stood.
Neither is fixed here. Both change bundle bytes, so each is a separate order.
The 24 combine two measurements: 22 image losses from the door build, and 2
text losses from recomputing the text accounting over that same bundle with
the fixed check. A fresh door build after the fixes was not run.
**A determinism defect in 0.10.0, found by the same comparison.**
- Inline PDF images (`BI … EI`) get their name from pdfminer, which derives
it from `id()` of a Python object. The pointer line
`Image: page-3-6344095824 (not carried: …)` therefore changes between runs.
- Affected: two concept files of `K2/trinn1` (`bilag-6-teknisk-oppsett`,
`bilag-9-1-...`).
- This breaks the bit-exact rebuild invariant. It is not caused by this work.
It is reported here and not fixed, for the same reason as above.
## Proposals, not carried out
- **Version:** `0.11.0`. The change adds a flag and changes an exit code for
existing users. The PM wants an independent review before any tag.
- **Not a default:** see Cost.
- **Separate orders:** carry the images of text-less PDF pages; keep a docx
Title paragraph in the body; make inline PDF image names deterministic.
- **A cheaper PDF inventory:** count image placements from the pages the reader
already parsed, and keep an independent count only in the gate.