feat(assets): a bundle carries the images its sources declare (0.10.0)

Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-17 10:01:31 +02:00
commit bc39e8091f
33 changed files with 3638 additions and 64 deletions

View file

@ -195,6 +195,56 @@ one boundary rule:
PROJECTION recomputed from the whole bundle each round, which is what makes
rebuild-from-scratch equal an incremental update byte for byte. `DEFAULT` is
untouched and byte-identical. Record: `docs/plan/structure-derivation.md`.
**SINCE 0.10.0 DOOR B CARRIES THE IMAGES ITS SOURCES DECLARE.** Until then no
reader here fetched, named or copied one -- `<img>`'s `attrs` were never
read, an STS `<graphic>` was walked past, a PDF was opened for text alone,
the converter's markdown writer dropped every picture, and the only writer
into a bundle was `materialize.write_bytes(..., content: str)`. Measured on
R761 Prosesskoden:2025: the process text is carried in full while **12
`Tabell N-N` and 9 `Figur N-N`** captions stand over nothing, because that
publisher ships those tables as raster pictures in BOTH the PDF and the
NISO-STS delivery -- process 84's "toleranseklasse ... er gitt i tabell 84-2"
points at empty space. **Five readers PLACE and one module DECIDES:**
`assets.py` owns what an image is (sniffed from the bytes, never from the
claimed extension), what it is called (`<sha256[:12]>-<the source's own
BASENAME>`, so one image reached by two paths is one file) and how it is
pointed at (one two-line block, one regex, `IMAGE_POINTER`, which is what
`okf describe` will find its work with). The pointer is a markdown image at
`/assets/<name>` -- bundle-absolute, because a segmented bundle puts concepts
at different depths -- followed by one line carrying the source's own file
name and the size in px. **`.xlsx` is deliberately NOT a row**: its converter
writes one pipe table per sheet and a two-line block inside one would break
the `source_rows` locator read back out of it; 0 of 4 K2 workbooks hold any
media, so it is a stated limit and not a loss taken. **A PDF stream that is
already a file is carried VERBATIM** (`DCTDecode`, `JPXDecode` -- 29 of
R761's 50 objects), and raw samples are encoded to PNG with stdlib `zlib`.
Rendering the page region was the alternative and was FELLED on
determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the installed
rasteriser -- the one property `OCR_DPI`'s docstring already admits OCR text
cannot have. What the encoder cannot express EXACTLY (stencil mask, `Decode`
array, CMYK, anything but 8-bit samples) is refused with a code and counted,
never approximated. **NO SIZE FLOOR, and that is a measurement:** over the
4 828 image objects of the K2 corpus the distribution is 149 / 162 / 92 /
406 / 498 / 590 / 2 931 across the size buckets -- a broad spread with no
gap, unlike `OCR_CID_SHARE`'s bimodal one, so a threshold would be a number
this package chose. **ON by default; `--no-assets` reproduces the pre-0.10.0
bytes.** Cost measured on the 43-document reference corpus, two builds of one
commit: **453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets (2 964
carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ.** The ONE new
concept has a measured cause and not a guessed one: the pointers are body
text, so a section holding 146 of that document's images grew from **19.0 %
to 30.6 %** of the extracted text and crossed `--outline-gate`'s 0.20 share
clause. **THE PLAN AND THE RUN MUST AGREE**: a plan records `text_sha256` of
the exact string it was proposed against, so `propose` and the door take the
same `assets` value and each computes the SAME resolver root independently --
the document's own directory, containment by `connectors.safe_resolve`. A
reference above it is refused (`asset_unresolved`), a remote one is never
fetched (`asset_remote`, extraction opens no socket) and both leave a line in
the concept saying what was there. **THE IMAGE BYTES ARE NOT SCREENED** --
the guard is text-only, the pointer block passes the gate as body text, the
picture beside it passes nothing -- and `log.md` says so on every run.
- **Door C — external bundle import:** third-party OKF bundles are assessed
per concept via the guard's `okf.import_bundle`; only concepts clearing the
guard's non-blocking floor are merged/indexed here. Two invariants, both
@ -204,6 +254,16 @@ one boundary rule:
data and persist bytes the guard never screened), and ownership is therefore
proven by **content identity** — an occupied target name is re-used only
when the bytes there are already identical, never overwritten otherwise.
**Since 0.10.0 it also carries the ASSETS its merged concepts point at**, by
that same content-identity rule. Measured before the repair: a bundle built
with `--assets` imported as **6 of 6** concepts and **no `assets/` at all**,
so every pointer in the imported bundle named a missing file — the same
"complete and not" defect one door over. POINTED AT, never every file in the
sender's `assets/`: an asset belonging to a concept the gate refused must not
ride in on the back of one it cleared, and an asset nothing names is a file
no retirement pass reaches. A pointer whose asset the sender did not ship is
left alone, because SPEC §6.1 requires a consumer to tolerate a broken link
and a pointer recording a figure nobody holds is information.
**`okf build` RUNS a real guard and NAMES it in the bundle (F1, 2026-09-15).**
From the day the command was packaged until then, `corpus.measure` wired an
@ -666,6 +726,18 @@ and fixtures, never code.
`docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`,
`docs/2026-09-08-k3-runde2-per-filtype.md` and
`docs/2026-09-08-k3-runde3-per-filtype.md`.
- **`--assets` / `--no-assets` (0.10.0) is not a segmentation flag either**,
and it is the first flag here that writes a NON-MARKDOWN file. ON by default.
It adds no boundary rule; it changes what the extracted text SAYS, so it sits
with the three PDF reader flags rather than with the twelve arms — and like
them it must be given the same value on both sides of a plan. The full
measurement, the layout and the refusal codes are in the Door B paragraph
above; the spec proposal for the layout is
`docs/plan/okf-assets-section-6-4.md`. `okf project` does not take it: it
owns no flag that moves a bundle's bytes, so it gets the default. The corpus
harness takes it with the SAME default, for the reason `--gate` does — a test
holds the two paths byte-equal, and two defaults would make that equality
depend on which command you ran.
- **`--frontmatter KEY=VALUE` (K3-19, repeatable) is not a segmentation flag**
and moves no byte unless given: it stamps a key on every concept of the run,
split on the FIRST `=` and written on ONE line -- a block-form