feat(assets): a bundle carries the images its sources declare (0.10.0)
Until now no reader in this package fetched, named, described or copied a single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>` was walked past, a PDF was opened for its text alone, the converter's markdown writer dropped every picture, and the only writer into a bundle took `content: str`. The two lossiness warnings said so on every run, which made the loss honest and did not make it smaller. Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N` and 9 `Figur N-N` captions stand over nothing, because that publisher ships those tables as raster pictures in both. Process 84's "toleranseklasse ... er gitt i tabell 84-2" points at empty space. THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its denominator out of the source (`page.images`, `word/media/`, `ppt/media/`, `<img`, `<graphic`), never from a constant here. Measured at332961a, built from `git archive` and not from the editable tree: carried 0 of 8 local images across 5 documents (9 declared), and no `assets/` at all. After: 8 of 8, with the ninth a remote source carried as a pointer without a file. FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is (sniffed from the bytes, never from the claimed extension), what it is called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed at (one two-line block, one regex). `.xlsx` is deliberately not a row: a block inside its pipe tables would break the `source_rows` locator, and 0 of 4 K2 workbooks hold media. A PDF stream that is already a file is carried VERBATIM (29 of R761's 50 objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so no new dependency. Rendering the page region was the alternative and was felled on determinism: a rasterised crop's bytes, and therefore the asset's content-addressed name and the bundle's digest, would depend on the installed rasteriser. What the encoder cannot express exactly is refused with a code and counted, never approximated. NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of the K2 corpus the size distribution is a broad spread with no gap, unlike OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose. ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document reference corpus at332961aversus rebuilt at HEAD with `--no-assets`: 865 files on both sides, `diff -rq` reports ONE difference, the added `Images: NOT CARRIED` line in log.md. Every concept byte-identical. Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets (2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s -> 3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new concept has a measured cause: the pointers are body text, so a section holding 146 of that document's images grew from 19.0 % to 30.6 % of the extracted text and crossed `--outline-gate`'s 0.20 share clause. THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block passes the gate as body text, the picture beside it passes nothing, and log.md says so on every run. Also fixed, both found by measuring rather than by reading: - a markdown image is no longer read as a cross-reference. `structure._LINK` never looked at the character in front of the bracket, so every pointer would have arrived in the index as an edge to a concept that cannot exist. - Door C carries the assets its merged concepts point at. Before this, importing a bundle built with `--assets` merged 6 of 6 concepts and wrote no `assets/` at all, so every pointer named a missing file. Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md Spec proposal: docs/plan/okf-assets-section-6-4.md Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
332961a19c
commit
bc39e8091f
33 changed files with 3638 additions and 64 deletions
64
CHANGELOG.md
64
CHANGELOG.md
|
|
@ -5,10 +5,72 @@ All notable changes to this project will be documented in this file.
|
|||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [Unreleased]
|
||||
## [0.10.0] — 2026-09-17
|
||||
|
||||
### Added
|
||||
|
||||
- **A bundle carries the images its sources declare (0.10.0).** Until now no
|
||||
reader in this package fetched, named, described or copied a single image:
|
||||
`<img>`'s attributes were never read, a NISO-STS `<graphic>` was walked past,
|
||||
a PDF was opened for its text alone, the converter's markdown writer dropped
|
||||
every picture, and the only writer into a bundle took `content: str`. The two
|
||||
lossiness warnings said so on every run, which made the loss honest and did
|
||||
not make it smaller. Measured on R761 Prosesskoden:2025: the process text is
|
||||
carried in full while 12 `Tabell N-N` and 9 `Figur N-N` captions stand over
|
||||
nothing, so process 84's "toleranseklasse ... er gitt i tabell 84-2" points
|
||||
at empty space.
|
||||
**Five readers place, one module decides.** `pdf` (embedded image XObjects),
|
||||
`docx`/`pptx`/`odt`/`rtf` (the converter's media, through `--extract-media`),
|
||||
`html`/`htm` (`<img src alt>`, local paths and inline data URIs) and `xml`
|
||||
(`<graphic xlink:href>`, resolved against the href and then against a sibling
|
||||
`graphics/`). `llm_ingestion_okf.assets` decides what an image IS, what it is
|
||||
called and how it is pointed at, so "carried N of M" means one thing across
|
||||
all five. `.xlsx` is deliberately excluded: a two-line block inside its pipe
|
||||
tables would break the row locator read back out of them.
|
||||
**The bytes go to `assets/`** at the bundle root under
|
||||
`<sha256[:12]>-<the source's own base name>`, and the concept carries a
|
||||
two-line pointer where the picture stood -- a markdown image, then the
|
||||
source's own file name and the size in pixels. A PDF stream that is already a
|
||||
file (`DCTDecode`, `JPXDecode`) is carried VERBATIM; raw samples are encoded
|
||||
to PNG with `zlib` from the stdlib, so no new dependency and no rasteriser
|
||||
version enters an asset's bytes or its content-addressed name. What this
|
||||
encoder cannot express exactly -- a stencil mask, a `Decode` array, CMYK,
|
||||
anything but 8-bit samples -- is refused with a code and counted, never
|
||||
approximated.
|
||||
**ON by default, with `--no-assets` reproducing the pre-0.10.0 bytes.**
|
||||
Measured over the 43-document reference corpus, two builds of one commit:
|
||||
453 -> 454 concepts, 865 -> 867 markdown files, 0 -> 2 964 assets (2 964
|
||||
carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
|
||||
3 088 s, peak RSS 6.26 -> 8.74 GB, and 422 of 865 markdown files differ. The
|
||||
one new concept has a measured cause: the pointers are body text, so a
|
||||
section holding 146 of that document's images grew from 19.0 % to 30.6 % of
|
||||
the extracted text and crossed `--outline-gate`'s 0.20 share clause.
|
||||
**`log.md` states it either way** -- "N carried of M found", or `NOT CARRIED`
|
||||
under `--no-assets`, so a bundle nobody looked for figures in cannot be
|
||||
mistaken for a bundle of documents that had none. A concept on this
|
||||
repository's own profiles also carries `images: N`, conditional, counted out
|
||||
of the concept's own text.
|
||||
**The image bytes are NOT screened**, and the log says so: the guard is
|
||||
text-only, the pointer block passes the gate as body text, and the picture
|
||||
beside it passes nothing.
|
||||
**Door C carries them too.** Measured before the repair: importing a bundle
|
||||
built with `--assets` merged 6 of 6 concepts and wrote no `assets/` at all,
|
||||
so every pointer in the imported bundle named a missing file. Only the assets
|
||||
a MERGED concept points at are carried -- an asset belonging to a refused
|
||||
concept must not ride in on the back of a cleared one.
|
||||
A proposed SPEC section 6.4 for the layout is in
|
||||
`docs/plan/okf-assets-section-6-4.md`; `_okf-canonical` is not edited from
|
||||
here.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A markdown image is no longer read as a cross-reference.**
|
||||
`structure._LINK` reads `[...](target)` and never looked at the character in
|
||||
front of the bracket, so an asset pointer would have arrived in the index as
|
||||
a `references` edge to a concept that cannot exist -- and the digits in an
|
||||
asset's file name would have been read as a document number. The link's span
|
||||
is still masked, so the number scan cannot see it either.
|
||||
|
||||
- **`okf build` now runs a real guard, and the bundle says which one (F1).**
|
||||
From the day the command was packaged until 2026-09-15, `corpus.measure`
|
||||
wired an unconditional approve-everything stub into `process_inbox` and no
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue