llm-ingestion-okf/docs
Kjell Tore Guttormsen bc39e8091f feat(assets): a bundle carries the images its sources declare (0.10.0)
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 10:01:31 +02:00
..
plan feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
2026-08-21-g2-pdf-extraction-measurement.md docs(extract): measure one Vegnormalene PDF page against the extraction registry 2026-08-21 19:16:15 +02:00
2026-08-29-konverteringsarmen-arm-a.md docs(measure): run the conversion arm on K2 - pandoc wins both office formats 2026-08-30 00:41:48 +02:00
2026-08-29-telling-offentlig-prosjektkorpus.md docs(measure): count the public project corpus - K2 measured, K1 blocked 2026-08-29 07:03:50 +02:00
2026-09-02-k3-k4-k5-baseline.md docs(measurement): K3, K4 and K5 numbers against the committed method 2026-09-02 15:55:19 +02:00
2026-09-02-k3-k4-k5-metode.md docs(measurement): K3, K4 and K5 method with n declared before any verdict 2026-09-02 15:46:03 +02:00
2026-09-02-korpuskjoring-k1-k2.md docs(measurement): K1 and K2 over the K2 corpus 2026-09-02 15:43:45 +02:00
2026-09-03-k2-bundle-rebuild.md test(corpus): a rebuild with log.md present reproduces the bundle exactly 2026-09-03 05:16:25 +02:00
2026-09-04-cid-andel-k2.md docs(measure): CID-glyph share across K2 -- Bilag 9.1 alone, not a K1 arm 2026-09-04 20:23:41 +02:00
2026-09-04-k2-pptx-odt-rtf.md docs(measure): K2 on pptx/odt/rtf -- denominator 0 in both named corpora 2026-09-04 21:12:41 +02:00
2026-09-04-k3-arm-c.md docs(measure): K3 with Arm C beside the baseline, two numbers, no threshold 2026-09-04 17:58:50 +02:00
2026-09-07-k2-pptx-odt-rtf-fixtures.md test(fidelity): K2 on pptx, odt and rtf -- 59/59 and 20/20, on a set we wrote 2026-09-07 05:22:12 +02:00
2026-09-07-k3-arm-d.md docs(measure): K3 Arm D beside a re-rated Arm B -- reach and no threshold 2026-09-07 03:04:01 +02:00
2026-09-07-k3-arm-e.md docs(measure): K3 Arm E beside Arm B and Arm D -- three rows, no threshold 2026-09-07 11:48:58 +02:00
2026-09-07-k3-enhetsark-metode.md docs(measure): K3 unit worksheet delivered for operator, no code changed 2026-09-07 16:36:16 +02:00
2026-09-07-okf-konsumskill-maaling.md docs(consume): refutation sweep — smoothed-IDF rarity weighting cannot zero both known-negatives 2026-09-07 09:56:36 +02:00
2026-09-08-blindsone-below-k-k2.md feat(consume): measure the below-k blind spot, add one flag-gated vocabulary bridge 2026-09-08 04:55:45 +02:00
2026-09-08-blindsone-laas2-budsjett-k2.md feat(consume): measure the budget lock, add one flag-gated top-rank reservation 2026-09-08 05:18:30 +02:00
2026-09-08-claude-code-skill-vilkaarlig-bundle.md docs(consume): the Claude Code recipe, measured end to end on two bundles 2026-09-08 15:32:32 +02:00
2026-09-08-eksakt-oppslag.md fix(consume): match an identifier by equality, deliver the concept a question names 2026-09-08 13:27:59 +02:00
2026-09-08-k3-arm-f-mot-enhetsarket.md feat(propose): Arm F, one unit fold behind a flag, measured against the operator's worksheet [skip-docs] 2026-09-08 16:28:41 +02:00
2026-09-08-k3-runde2-per-filtype.md feat(cli,propose): reach the arms from okf build, keep a sheet heading behind a flag, fix two PDF contents mechanisms 2026-09-08 18:21:25 +02:00
2026-09-08-k3-runde3-per-filtype.md docs(cli): the new default's K2 digest, measured on the bundle it produces 2026-09-08 21:14:51 +02:00
2026-09-08-k3-runde4-pdf-skrift-og-ocr.md feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off 2026-09-08 23:10:47 +02:00
2026-09-08-k3-runde5-hitat8-og-skriftakse.md feat(propose,cli): typography as a reserve, and the two of our own numbers it took to measure it 2026-09-09 00:25:51 +02:00
2026-09-08-kravnummer-tokenisering.md feat(propose,cli): the sheet's rows, the wrapped sentence, and a default that moved 2026-09-08 20:34:18 +02:00
2026-09-08-o5-okf-project.md feat(cli): okf project/consume/check/skill, and a generated skill with no path into a checkout 2026-09-08 21:47:38 +02:00
2026-09-08-prisform-og-loggen-k2.md fix(extract,build): write a spreadsheet as pipe tables, stop linking the run log from the index 2026-09-08 10:06:58 +02:00
2026-09-08-proveniens-k2.md feat(inbox): point every concept at the document it came from, with a locator per format 2026-09-08 14:39:24 +02:00
2026-09-08-rangeringsbom-sammensatte-ord.md fix(consume): the compound-word miss is a degenerate signal's tie-break, behind a flag 2026-09-08 18:53:18 +02:00
2026-09-08-sjeldenhetsvekt.md feat(propose,cli): the sheet's rows, the wrapped sentence, and a default that moved 2026-09-08 20:34:18 +02:00
2026-09-09-k3-runde6-outline-gaten-og-prioren.md feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment 2026-09-09 14:17:00 +02:00
2026-09-09-k3-runde9-restene.md feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
2026-09-09-k3-runde10-rtf-prefiks-shim-ruff.md docs(k3): round 10, and the fasit that held a working rule off the default [skip-docs] 2026-09-09 23:27:03 +02:00
2026-09-09-o6-forsteskjerm-og-skillmodi.md feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md feat(cli,consume): the first span, and the fusion that punished fine-graining 2026-09-09 15:40:01 +02:00
2026-09-10-k3-runde11-html-og-dominans.md docs(k3): round 11, and the corpus that rebuilt byte-identical [skip-docs] 2026-09-10 01:10:59 +02:00
2026-09-10-k3-runde12-pdf-outlines.md feat(propose): the PDF shipped a structure index and the build discarded it unopened 2026-09-10 02:27:41 +02:00
2026-09-10-k3-runde14-deklarert-struktur-tar-ruten.md docs(k3): round 14, the declared-structure route at the shipped defaults 2026-09-10 07:53:22 +02:00
2026-09-10-k3-runde15-bundle-mismatch.md feat(check): a skill and a payload naming different bundles is a finding 2026-09-10 23:42:06 +02:00
2026-09-10-k3-runde16-hele-tittelen-tar-ruten.md docs(k3): round 16, the whole title takes the route precision could not 2026-09-10 22:26:57 +02:00
2026-09-10-r1-release-v080.md docs(release): R1, v0.8.0 tagged locally and installed from the tag 2026-09-10 21:14:19 +02:00
2026-09-10-r3-release-v081.md docs(release): R3, v0.8.1 tagged locally on 3daf983 2026-09-10 23:10:30 +02:00
2026-09-11-k3-runde8-tabellblokk-og-siste-spenn.md feat(propose,cli): the coverage gap had one cause, and round 7's own decomposition did not reproduce 2026-09-09 17:43:56 +02:00
2026-09-11-k3-runde13-xml-sts.md docs(k3): round 13, and the two proposer rules between a reader and its ceiling [skip-docs] 2026-09-10 04:13:56 +02:00
2026-09-11-k3-runde17-dekningen-stopper-ved-en-bredere-tittel.md docs(k3): round 17, a covered title stops beneath a title that answers more 2026-09-11 01:45:53 +02:00
2026-09-11-k3-runde18-konsumskillen-regenerert.md fix(skill): regenerate skills/okf-consume from the golden bundle it ships a payload for 2026-09-11 02:21:35 +02:00
2026-09-11-k3-runde19-dokumentidentitet-og-frontmatter.md docs(k3): round 19, a document's own identity, per-run frontmatter, and a description from the source 2026-09-11 03:59:34 +02:00
2026-09-11-k3-runde20-delt-katalog-og-arvet-kontekst.md docs(k3): round 20, a directory every concept shares and a section whose body is its heading 2026-09-11 05:38:35 +02:00
2026-09-11-k3-runde21-parent-til-leseren.md docs(k3): round 21, parent all the way to the reader 2026-09-11 14:28:42 +02:00
2026-09-11-k3-runde22-yaml-lesbar-frontmatter.md docs(k3): round 22, frontmatter a YAML reader reads back the same 2026-09-11 11:47:40 +02:00
2026-09-11-r4-release-v082.md docs(release): R4, v0.8.2 installed and measured from the tag 2026-09-11 02:40:31 +02:00
2026-09-11-r5-release-v083.md docs(release): R5, v0.8.3 installed and measured from the tag 2026-09-11 06:21:31 +02:00
2026-09-11-r6-release-v084.md docs(release): R6, v0.8.4 installed and measured from the tag 2026-09-11 15:04:21 +02:00
2026-09-12-g37-terskler.md feat(quality): okf quality --fasit, boundary recall against a declared structure 2026-09-13 07:27:33 +02:00
2026-09-12-k3-runde23-stien-i-kroppssignalet.md docs(k3): round 25, the default reading drops the door's link line 2026-09-12 19:07:55 +02:00
2026-09-12-k3-runde24-blokkform-til-de-flate-leserne.md docs(k3): round 24, a reason that became false is struck and the emitter stays flow 2026-09-12 16:50:26 +02:00
2026-09-12-k3-runde25-standardlesingen-uten-lenkelinja.md docs(release): R7, v0.8.5 tagged locally and measured from the tag 2026-09-12 19:47:30 +02:00
2026-09-12-r7-release-v085.md docs(release): R7, v0.8.5 tagged locally and measured from the tag 2026-09-12 19:47:30 +02:00
2026-09-15-f1-f2-gaten-og-kodefencen.md fix(gate,propose): okf build runs a real guard; a code fence declares no structure 2026-09-16 00:19:50 +02:00
2026-09-17-bilder-i-bundlen-trinn1.md feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
consumption-contract.md feat(consume): --follow-parent carries the enclosing section's text, from room the cut left 2026-09-11 13:11:24 +02:00
okf-nokkelinventar.md build(deps): pin llm-ingestion-guard v1.3.0 so the gate reads our own goldens 2026-09-03 20:41:03 +02:00
phase-3-split-table.md docs(phase-3): keep the index-entry grammar on one line 2026-08-04 12:37:02 +02:00
upstream-okf-upgrade-runbook.md feat(watch): weekly OKF upstream watch that can prove it found nothing 2026-08-23 20:38:37 +02:00