open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(assets): a bundle carries the images its sources declare (0.10.0)

Browse source 
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-17 10:01:31 +02:00
commit bc39e8091f
33 changed files with 3638 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

92
README.md
View file

@ -10,7 +10,7 @@ types](#supported-file-types) lists each one with the evidence behind it.
Python 3.10+ and [uv](https://docs.astral.sh/uv/). One line:
```sh
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.9.0"
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.0"
```
## Use it
@ -77,6 +77,16 @@ that test red.
| `.odt` | `_extract_office` | `[extract]`: pypandoc-binary | constructed | N = 1 document. 1 of 1 declared headings recovered, 1 concept, 0 characters in no segment. |
| `.rtf` | `_extract_office` | `[extract]`: pypandoc-binary | constructed | N = 1 document, and the weakest row here. 0 declared headings: the container carries no heading style, so the author's title is bold text and the document lands as one concept — content preserved, structure zero. The `--bold-title` flag reads that bold line and is off by default. |
**Since 0.10.0 five of those rows also carry IMAGES** — `.pdf`, `.docx`,
`.pptx`, `.html`/`.htm` and `.xml`. The bytes go to `assets/` at the bundle
root under a content-addressed name, and the concept carries a pointer where
the picture stood. `.xlsx` is deliberately not among them: its converter writes
one pipe table per sheet and a two-line block inside one would break the row
locator `source_rows` is read back out of; measured 2026-09-16, 0 of 4 K2
workbooks hold any media at all, so the row is a stated limit and not a loss
taken. `.csv`, `.json`, `.md`, `.txt`, `.odt` and `.rtf` are absent because
nothing has measured an image reaching them.
The three classes are the code's own and are not interchangeable. `measured`
means real documents someone wrote for their own purposes, counted against a
fasit written before the lookup. `constructed` means the row has been through
@ -110,11 +120,11 @@ Neither this package nor the guard it depends on is on a package index yet, so
both install by direct reference. With uv, one command resolves both:
```sh
uv pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.9.0"
uv pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.0"
```
uv resolves the guard on its own, because it reads the `[tool.uv.sources]`
entry in the `pyproject.toml` **of the tag it is installing**, and `v0.9.0`
entry in the `pyproject.toml` **of the tag it is installing**, and `v0.10.0`
points that entry at `llm-ingestion-guard` `v1.4.0`. Use `uv tool install`
instead of `uv pip install` when you want the `okf` command on `PATH` without an
active virtualenv — that is the form the first screen shows.
@ -125,10 +135,10 @@ With plain pip, the transitive git dependency does not resolve on its own —
```sh
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.4.0"
pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.9.0"
pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.0"
```
The guard tag is paired to the okf tag, not to this branch. `v0.9.0` declares
The guard tag is paired to the okf tag, not to this branch. `v0.10.0` declares
`llm-ingestion-guard>=1.2,<2.0`, which `v1.4.0` satisfies; the pairing above is
read off that tag's own `[tool.uv.sources]`, not off this branch. Reading a pin
off `main` and installing it against an older okf tag is the one combination
@ -139,7 +149,15 @@ that fails.
These are not install lines. They record what each earlier tag was, so a reader
who meets one in an older document knows what they are looking at.
- `v0.9.0` — the current tag: `okf quality <bundle>`, a per-file-type
- `v0.10.0` — the current tag: a bundle carries the IMAGES its sources
declare. Five readers place them (`pdf`, the converter's office rows,
`html`, `xml`), `assets/` at the bundle root holds the bytes under a
content-addressed name, and the concept carries a two-line pointer where the
picture stood. ON by default; `--no-assets` reproduces the pre-0.10.0 bytes,
measured on the 43-document reference corpus as a one-line difference in
`log.md`. The image bytes are not screened — the gate reads text — and the
log says so. Door C carries the assets its merged concepts point at.
- `v0.9.0` — `okf quality <bundle>`, a per-file-type
verdict on a bundle with the denominator on every line. Three verdicts
(`PASS` / `FAIL` / `UNMEASURED`) and a type with no measured threshold is
never `PASS`; two bars ship, `.pdf` 8/32 and `.docx` 2/5, both regression
@ -231,6 +249,66 @@ than the clock, so two builds of the same folder are byte-identical — a
wall-clock default would break rebuild-equals-incremental for every caller who
did not pass them.
### Images: what the source draws, carried (0.10.0)
`okf build` carries the images its sources declare. The bytes go to `assets/`
at the bundle root, named `<sha256[:12]>-<the source's own base name>`, and the
concept carries a two-line pointer where the picture stood:
```markdown
![Tabell 84-2 Toleranseklasser](/assets/e54e5f5da0e8-tabell-84-2.png)
Image: graphics/tabell-84-2.png (120x90 px) -- Tabell 84-2 Toleranseklasser
```
The first line is markdown, so a reader that renders the concept sees the
picture. The second states what the first cannot — the name the SOURCE gave the
file and the size in pixels — which are the two facts a person checking the
bundle against the original needs.
**Why it exists.** Measured on R761 Prosesskoden:2025, a process code published
as a 701-page PDF and as a NISO-STS delivery: the process text is carried in
full, and 12 `Tabell N-N` and 9 `Figur N-N` captions stand over nothing,
because the publisher ships those tables as raster images in both deliveries.
Process 84 says "toleranseklasse ... er gitt i tabell 84-2" and table 84-2 is a
JPEG. A bundle like that reads as complete and is not.
**What it costs, measured on the 43-document reference corpus** (`K2/trinn1`,
two builds of the same commit, 2026-09-17):
| | `--no-assets` | default |
|---|---|---|
| concept files | 453 | 454 |
| markdown files | 865 | 867 |
| assets | 0 | 2 964 |
| bundle size | 4.7 MB | 115 MB |
| wall time | 2 414 s | 3 088 s |
| peak RSS | 6.26 GB | 8.74 GB |
2 964 carried of 3 145 found; 4 622 pointers, so content de-duplication folds
1 658 repeats into the files they already are. 422 of the 865 markdown files
differ. **One concept appears, and the mechanism is measured rather than
guessed:** the pointers are body text, so a section holding 146 of that
document's images grew from 19.0 % to 30.6 % of the extracted text and crossed
`--outline-gate`'s 0.20 share clause.
**`--no-assets` reproduces the pre-0.10.0 bytes**, and `log.md` then says
`NOT CARRIED` rather than falling silent — a bundle nobody looked for figures
in must not read like a bundle of documents that had none.
**The image bytes are not screened.** The gate reads text; a picture is not
text. The pointer block passes the gate like any other body line, and the file
beside it passes nothing. `log.md` says so on every run that carries one.
**No size floor, and that is a measurement too.** The obvious filter is "ignore
anything under N pixels", and the distribution offers no N: over the 4 828
image objects in that corpus, 149 declare no size, 162 are under 32x32, 92
under 64x64, 406 under 128x128, 498 under 256x256, 590 under 512x512 and 2 931
are larger — a broad spread with no gap, unlike `OCR_CID_SHARE`'s, which is
bimodal with nothing between the modes. A threshold read off no gap is a number
this package chose, and it would silently drop somebody's small table.
<!-- cli-default-assets: on -->
`--frontmatter KEY=VALUE` (repeatable) stamps a key on every concept of the
run, for what the operator knows and the document does not say — an edition,
a publisher's address. It splits on the first `=` and writes the value
@ -729,7 +807,7 @@ not need this repository — the first line installs the command, the second
builds the bundle and writes a skill beside it, the third asks.
```sh
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.9.0"
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.0"
okf project ~/my-documents
claude
```