open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(assets): a bundle carries the images its sources declare (0.10.0)

Browse source 
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-17 10:01:31 +02:00
commit bc39e8091f
33 changed files with 3638 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

64
CHANGELOG.md
View file

@ -5,10 +5,72 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [0.10.0] — 2026-09-17
### Added
- **A bundle carries the images its sources declare (0.10.0).** Until now no
reader in this package fetched, named, described or copied a single image:
`<img>`'s attributes were never read, a NISO-STS `<graphic>` was walked past,
a PDF was opened for its text alone, the converter's markdown writer dropped
every picture, and the only writer into a bundle took `content: str`. The two
lossiness warnings said so on every run, which made the loss honest and did
not make it smaller. Measured on R761 Prosesskoden:2025: the process text is
carried in full while 12 `Tabell N-N` and 9 `Figur N-N` captions stand over
nothing, so process 84's "toleranseklasse ... er gitt i tabell 84-2" points
at empty space.
**Five readers place, one module decides.** `pdf` (embedded image XObjects),
`docx`/`pptx`/`odt`/`rtf` (the converter's media, through `--extract-media`),
`html`/`htm` (`<img src alt>`, local paths and inline data URIs) and `xml`
(`<graphic xlink:href>`, resolved against the href and then against a sibling
`graphics/`). `llm_ingestion_okf.assets` decides what an image IS, what it is
called and how it is pointed at, so "carried N of M" means one thing across
all five. `.xlsx` is deliberately excluded: a two-line block inside its pipe
tables would break the row locator read back out of them.
**The bytes go to `assets/`** at the bundle root under
`<sha256[:12]>-<the source's own base name>`, and the concept carries a
two-line pointer where the picture stood -- a markdown image, then the
source's own file name and the size in pixels. A PDF stream that is already a
file (`DCTDecode`, `JPXDecode`) is carried VERBATIM; raw samples are encoded
to PNG with `zlib` from the stdlib, so no new dependency and no rasteriser
version enters an asset's bytes or its content-addressed name. What this
encoder cannot express exactly -- a stencil mask, a `Decode` array, CMYK,
anything but 8-bit samples -- is refused with a code and counted, never
approximated.
**ON by default, with `--no-assets` reproducing the pre-0.10.0 bytes.**
Measured over the 43-document reference corpus, two builds of one commit:
453 -> 454 concepts, 865 -> 867 markdown files, 0 -> 2 964 assets (2 964
carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, and 422 of 865 markdown files differ. The
one new concept has a measured cause: the pointers are body text, so a
section holding 146 of that document's images grew from 19.0 % to 30.6 % of
the extracted text and crossed `--outline-gate`'s 0.20 share clause.
**`log.md` states it either way** -- "N carried of M found", or `NOT CARRIED`
under `--no-assets`, so a bundle nobody looked for figures in cannot be
mistaken for a bundle of documents that had none. A concept on this
repository's own profiles also carries `images: N`, conditional, counted out
of the concept's own text.
**The image bytes are NOT screened**, and the log says so: the guard is
text-only, the pointer block passes the gate as body text, and the picture
beside it passes nothing.
**Door C carries them too.** Measured before the repair: importing a bundle
built with `--assets` merged 6 of 6 concepts and wrote no `assets/` at all,
so every pointer in the imported bundle named a missing file. Only the assets
a MERGED concept points at are carried -- an asset belonging to a refused
concept must not ride in on the back of a cleared one.
A proposed SPEC section 6.4 for the layout is in
`docs/plan/okf-assets-section-6-4.md`; `_okf-canonical` is not edited from
here.
### Fixed
- **A markdown image is no longer read as a cross-reference.**
`structure._LINK` reads `[...](target)` and never looked at the character in
front of the bracket, so an asset pointer would have arrived in the index as
a `references` edge to a concept that cannot exist -- and the digits in an
asset's file name would have been read as a document number. The link's span
is still masked, so the number scan cannot see it either.
- **`okf build` now runs a real guard, and the bundle says which one (F1).**
From the day the command was packaged until 2026-09-15, `corpus.measure`
wired an unconditional approve-everything stub into `process_inbox` and no

72
CLAUDE.md
View file

@ -195,6 +195,56 @@ one boundary rule:
PROJECTION recomputed from the whole bundle each round, which is what makes
rebuild-from-scratch equal an incremental update byte for byte. `DEFAULT` is
untouched and byte-identical. Record: `docs/plan/structure-derivation.md`.
**SINCE 0.10.0 DOOR B CARRIES THE IMAGES ITS SOURCES DECLARE.** Until then no
reader here fetched, named or copied one -- `<img>`'s `attrs` were never
read, an STS `<graphic>` was walked past, a PDF was opened for text alone,
the converter's markdown writer dropped every picture, and the only writer
into a bundle was `materialize.write_bytes(..., content: str)`. Measured on
R761 Prosesskoden:2025: the process text is carried in full while **12
`Tabell N-N` and 9 `Figur N-N`** captions stand over nothing, because that
publisher ships those tables as raster pictures in BOTH the PDF and the
NISO-STS delivery -- process 84's "toleranseklasse ... er gitt i tabell 84-2"
points at empty space. **Five readers PLACE and one module DECIDES:**
`assets.py` owns what an image is (sniffed from the bytes, never from the
claimed extension), what it is called (`<sha256[:12]>-<the source's own
BASENAME>`, so one image reached by two paths is one file) and how it is
pointed at (one two-line block, one regex, `IMAGE_POINTER`, which is what
`okf describe` will find its work with). The pointer is a markdown image at
`/assets/<name>` -- bundle-absolute, because a segmented bundle puts concepts
at different depths -- followed by one line carrying the source's own file
name and the size in px. **`.xlsx` is deliberately NOT a row**: its converter
writes one pipe table per sheet and a two-line block inside one would break
the `source_rows` locator read back out of it; 0 of 4 K2 workbooks hold any
media, so it is a stated limit and not a loss taken. **A PDF stream that is
already a file is carried VERBATIM** (`DCTDecode`, `JPXDecode` -- 29 of
R761's 50 objects), and raw samples are encoded to PNG with stdlib `zlib`.
Rendering the page region was the alternative and was FELLED on
determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the installed
rasteriser -- the one property `OCR_DPI`'s docstring already admits OCR text
cannot have. What the encoder cannot express EXACTLY (stencil mask, `Decode`
array, CMYK, anything but 8-bit samples) is refused with a code and counted,
never approximated. **NO SIZE FLOOR, and that is a measurement:** over the
4 828 image objects of the K2 corpus the distribution is 149 / 162 / 92 /
406 / 498 / 590 / 2 931 across the size buckets -- a broad spread with no
gap, unlike `OCR_CID_SHARE`'s bimodal one, so a threshold would be a number
this package chose. **ON by default; `--no-assets` reproduces the pre-0.10.0
bytes.** Cost measured on the 43-document reference corpus, two builds of one
commit: **453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets (2 964
carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ.** The ONE new
concept has a measured cause and not a guessed one: the pointers are body
text, so a section holding 146 of that document's images grew from **19.0 %
to 30.6 %** of the extracted text and crossed `--outline-gate`'s 0.20 share
clause. **THE PLAN AND THE RUN MUST AGREE**: a plan records `text_sha256` of
the exact string it was proposed against, so `propose` and the door take the
same `assets` value and each computes the SAME resolver root independently --
the document's own directory, containment by `connectors.safe_resolve`. A
reference above it is refused (`asset_unresolved`), a remote one is never
fetched (`asset_remote`, extraction opens no socket) and both leave a line in
the concept saying what was there. **THE IMAGE BYTES ARE NOT SCREENED** --
the guard is text-only, the pointer block passes the gate as body text, the
picture beside it passes nothing -- and `log.md` says so on every run.
- **Door C — external bundle import:** third-party OKF bundles are assessed
per concept via the guard's `okf.import_bundle`; only concepts clearing the
guard's non-blocking floor are merged/indexed here. Two invariants, both
@ -204,6 +254,16 @@ one boundary rule:
data and persist bytes the guard never screened), and ownership is therefore
proven by **content identity** — an occupied target name is re-used only
when the bytes there are already identical, never overwritten otherwise.
**Since 0.10.0 it also carries the ASSETS its merged concepts point at**, by
that same content-identity rule. Measured before the repair: a bundle built
with `--assets` imported as **6 of 6** concepts and **no `assets/` at all**,
so every pointer in the imported bundle named a missing file — the same
"complete and not" defect one door over. POINTED AT, never every file in the
sender's `assets/`: an asset belonging to a concept the gate refused must not
ride in on the back of one it cleared, and an asset nothing names is a file
no retirement pass reaches. A pointer whose asset the sender did not ship is
left alone, because SPEC §6.1 requires a consumer to tolerate a broken link
and a pointer recording a figure nobody holds is information.
**`okf build` RUNS a real guard and NAMES it in the bundle (F1, 2026-09-15).**
From the day the command was packaged until then, `corpus.measure` wired an
@ -666,6 +726,18 @@ and fixtures, never code.
`docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`,
`docs/2026-09-08-k3-runde2-per-filtype.md` and
`docs/2026-09-08-k3-runde3-per-filtype.md`.
- **`--assets` / `--no-assets` (0.10.0) is not a segmentation flag either**,
and it is the first flag here that writes a NON-MARKDOWN file. ON by default.
It adds no boundary rule; it changes what the extracted text SAYS, so it sits
with the three PDF reader flags rather than with the twelve arms — and like
them it must be given the same value on both sides of a plan. The full
measurement, the layout and the refusal codes are in the Door B paragraph
above; the spec proposal for the layout is
`docs/plan/okf-assets-section-6-4.md`. `okf project` does not take it: it
owns no flag that moves a bundle's bytes, so it gets the default. The corpus
harness takes it with the SAME default, for the reason `--gate` does — a test
holds the two paths byte-equal, and two defaults would make that equality
depend on which command you ran.
- **`--frontmatter KEY=VALUE` (K3-19, repeatable) is not a segmentation flag**
and moves no byte unless given: it stamps a key on every concept of the run,
split on the FIRST `=` and written on ONE line -- a block-form

92
README.md
View file

@ -10,7 +10,7 @@ types](#supported-file-types) lists each one with the evidence behind it.
Python 3.10+ and [uv](https://docs.astral.sh/uv/). One line:
```sh
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.9.0"
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.0"
```
## Use it
@ -77,6 +77,16 @@ that test red.
| `.odt` | `_extract_office` | `[extract]`: pypandoc-binary | constructed | N = 1 document. 1 of 1 declared headings recovered, 1 concept, 0 characters in no segment. |
| `.rtf` | `_extract_office` | `[extract]`: pypandoc-binary | constructed | N = 1 document, and the weakest row here. 0 declared headings: the container carries no heading style, so the author's title is bold text and the document lands as one concept — content preserved, structure zero. The `--bold-title` flag reads that bold line and is off by default. |
**Since 0.10.0 five of those rows also carry IMAGES** — `.pdf`, `.docx`,
`.pptx`, `.html`/`.htm` and `.xml`. The bytes go to `assets/` at the bundle
root under a content-addressed name, and the concept carries a pointer where
the picture stood. `.xlsx` is deliberately not among them: its converter writes
one pipe table per sheet and a two-line block inside one would break the row
locator `source_rows` is read back out of; measured 2026-09-16, 0 of 4 K2
workbooks hold any media at all, so the row is a stated limit and not a loss
taken. `.csv`, `.json`, `.md`, `.txt`, `.odt` and `.rtf` are absent because
nothing has measured an image reaching them.
The three classes are the code's own and are not interchangeable. `measured`
means real documents someone wrote for their own purposes, counted against a
fasit written before the lookup. `constructed` means the row has been through
@ -110,11 +120,11 @@ Neither this package nor the guard it depends on is on a package index yet, so
both install by direct reference. With uv, one command resolves both:
```sh
uv pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.9.0"
uv pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.0"
```
uv resolves the guard on its own, because it reads the `[tool.uv.sources]`
entry in the `pyproject.toml` **of the tag it is installing**, and `v0.9.0`
entry in the `pyproject.toml` **of the tag it is installing**, and `v0.10.0`
points that entry at `llm-ingestion-guard` `v1.4.0`. Use `uv tool install`
instead of `uv pip install` when you want the `okf` command on `PATH` without an
active virtualenv — that is the form the first screen shows.
@ -125,10 +135,10 @@ With plain pip, the transitive git dependency does not resolve on its own —
```sh
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.4.0"
pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.9.0"
pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.0"
```
The guard tag is paired to the okf tag, not to this branch. `v0.9.0` declares
The guard tag is paired to the okf tag, not to this branch. `v0.10.0` declares
`llm-ingestion-guard>=1.2,<2.0`, which `v1.4.0` satisfies; the pairing above is
read off that tag's own `[tool.uv.sources]`, not off this branch. Reading a pin
off `main` and installing it against an older okf tag is the one combination
@ -139,7 +149,15 @@ that fails.
These are not install lines. They record what each earlier tag was, so a reader
who meets one in an older document knows what they are looking at.
- `v0.9.0` — the current tag: `okf quality <bundle>`, a per-file-type
- `v0.10.0` — the current tag: a bundle carries the IMAGES its sources
declare. Five readers place them (`pdf`, the converter's office rows,
`html`, `xml`), `assets/` at the bundle root holds the bytes under a
content-addressed name, and the concept carries a two-line pointer where the
picture stood. ON by default; `--no-assets` reproduces the pre-0.10.0 bytes,
measured on the 43-document reference corpus as a one-line difference in
`log.md`. The image bytes are not screened — the gate reads text — and the
log says so. Door C carries the assets its merged concepts point at.
- `v0.9.0` — `okf quality <bundle>`, a per-file-type
verdict on a bundle with the denominator on every line. Three verdicts
(`PASS` / `FAIL` / `UNMEASURED`) and a type with no measured threshold is
never `PASS`; two bars ship, `.pdf` 8/32 and `.docx` 2/5, both regression
@ -231,6 +249,66 @@ than the clock, so two builds of the same folder are byte-identical — a
wall-clock default would break rebuild-equals-incremental for every caller who
did not pass them.
### Images: what the source draws, carried (0.10.0)
`okf build` carries the images its sources declare. The bytes go to `assets/`
at the bundle root, named `<sha256[:12]>-<the source's own base name>`, and the
concept carries a two-line pointer where the picture stood:
```markdown
![Tabell 84-2 Toleranseklasser](/assets/e54e5f5da0e8-tabell-84-2.png)
Image: graphics/tabell-84-2.png (120x90 px) -- Tabell 84-2 Toleranseklasser
```
The first line is markdown, so a reader that renders the concept sees the
picture. The second states what the first cannot — the name the SOURCE gave the
file and the size in pixels — which are the two facts a person checking the
bundle against the original needs.
**Why it exists.** Measured on R761 Prosesskoden:2025, a process code published
as a 701-page PDF and as a NISO-STS delivery: the process text is carried in
full, and 12 `Tabell N-N` and 9 `Figur N-N` captions stand over nothing,
because the publisher ships those tables as raster images in both deliveries.
Process 84 says "toleranseklasse ... er gitt i tabell 84-2" and table 84-2 is a
JPEG. A bundle like that reads as complete and is not.
**What it costs, measured on the 43-document reference corpus** (`K2/trinn1`,
two builds of the same commit, 2026-09-17):
| | `--no-assets` | default |
|---|---|---|
| concept files | 453 | 454 |
| markdown files | 865 | 867 |
| assets | 0 | 2 964 |
| bundle size | 4.7 MB | 115 MB |
| wall time | 2 414 s | 3 088 s |
| peak RSS | 6.26 GB | 8.74 GB |
2 964 carried of 3 145 found; 4 622 pointers, so content de-duplication folds
1 658 repeats into the files they already are. 422 of the 865 markdown files
differ. **One concept appears, and the mechanism is measured rather than
guessed:** the pointers are body text, so a section holding 146 of that
document's images grew from 19.0 % to 30.6 % of the extracted text and crossed
`--outline-gate`'s 0.20 share clause.
**`--no-assets` reproduces the pre-0.10.0 bytes**, and `log.md` then says
`NOT CARRIED` rather than falling silent — a bundle nobody looked for figures
in must not read like a bundle of documents that had none.
**The image bytes are not screened.** The gate reads text; a picture is not
text. The pointer block passes the gate like any other body line, and the file
beside it passes nothing. `log.md` says so on every run that carries one.
**No size floor, and that is a measurement too.** The obvious filter is "ignore
anything under N pixels", and the distribution offers no N: over the 4 828
image objects in that corpus, 149 declare no size, 162 are under 32x32, 92
under 64x64, 406 under 128x128, 498 under 256x256, 590 under 512x512 and 2 931
are larger — a broad spread with no gap, unlike `OCR_CID_SHARE`'s, which is
bimodal with nothing between the modes. A threshold read off no gap is a number
this package chose, and it would silently drop somebody's small table.
<!-- cli-default-assets: on -->
`--frontmatter KEY=VALUE` (repeatable) stamps a key on every concept of the
run, for what the operator knows and the document does not say — an edition,
a publisher's address. It splits on the first `=` and writes the value
@ -729,7 +807,7 @@ not need this repository — the first line installs the command, the second
builds the bundle and writes a skill beside it, the third asks.
```sh
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.9.0"
uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.10.0"
okf project ~/my-documents
claude
```

270
docs/2026-09-17-bilder-i-bundlen-trinn1.md Normal file
View file

@ -0,0 +1,270 @@
# Bilder i OKF-bundles, trinn 1: de bæres (0.10.0)
Ordre `20260916T050910Z-1628427832-from-.claude`, trinn 1 av 2. Trinn 2
(`okf describe`, Claude vision) er ikke i denne leveransen og ikke i denne
rapporten.
Utgangspunktet er operatørens premiss, ordrett: «det som ender opp i en bundle
etter en prosess med å konvertere X antall kilder MÅ være 100 % riktig».
---
## § 0 Premissene målt først
Ordren oppgir hva PM målte i dette repoet og ber om at det gjentas
(Verifiseringsloven, ansikt 3). Målt på `332961a`, 2026-09-16:
| Påstand | Målt her | Status |
| --- | --- | --- |
| Ingen leser henter, navngir eller kopierer et bilde | `page.images` og `extract_table`: **0 treff** i `src/`. `handle_starttag` leser aldri `attrs` (`extract.py`). `<graphic>` forekommer ikke i XML-leseren. `page.to_image` finnes kun inne i OCR-grenen | **Bekreftet** |
| Eneste skriver er `write_bytes(..., content: str)` | Ja, UTF-8, ingen binær skrivesti | **Bekreftet** |
| 108 grep-treff over 23 filer | Målt her: **127 treff over 12 filer** med `grep -rIEn` over `src/*.py` | **Avviker** — PMs kommando er ikke oppgitt, så tallene er ikke sammenliknbare. Substansen (ingen treff er en bildeleser) er bekreftet ved gjennomlesing av alle 127 |
| SPEC er taus om binære filer | `_okf-canonical` `ad30107`: § 3 «a directory tree of markdown files», § 11 punkt 1 scoper til `.md`, § 6.3 er en konvensjon | **Bekreftet** |
To premisser i ordren er **ikke** reprodusert og er merket som det: «84 filer i
kildezip-ens `graphics/`» — katalogen jeg har lesetilgang til
(`~/repos/vegnormal-okf/build/860019-html/graphics`) holder **109 filer**, og
XML-en refererer **50** av dem. Det er en annen artefakt enn zip-en ordren
siterer, ikke en motsigelse.
R761-målingen som utløste ordren er ikke etterprøvd her i sin helhet; det jeg
målte selv er at side 496 i PDF-en bærer **2 DCTDecode-bilder** rett under
teksten «Tabell 84-2:», og at hele dokumentet bærer **50 bildeobjekter på 38 av
701 sider**, fordelt **29 DCTDecode / 21 FlateDecode** — samme antall som
NISO-STS-leveransens 50 `<graphic>`.
---
## § 1 Gaten, skrevet rød først
`tests/test_asset_gate.py`, skrevet før én linje kapabilitetskode. Nevneren
leses ut av **kilden** (`page.images`, `word/media/`, `ppt/media/`, `<img`,
`<graphic`), aldri fra en konstant i dette repoet — en konstant er repoet som
påstår sin egen forventning, og den blir gal i det en fixture regenereres.
Målt på `332961a`, bygget fra `git archive` og ikke fra arbeidstreet (et
editable install leser `src/` live, så en «før»-kjøring i dette treet ville målt
endringen den skulle gå forut for):
```
carried 0 of 2 local (2 declared) prosess-84-tabell.pdf
carried 0 of 1 local (1 declared) prosess-84-notat.docx
carried 0 of 1 local (1 declared) prosess-84-presentasjon.pptx
carried 0 of 2 local (3 declared) prosess-84-web.html
carried 0 of 2 local (2 declared) prosess-84-sts.xml
---------------------------------------------------------------
carried 0 of 8 local images across 5 documents (9 declared),
and the bundle held no assets/ directory at all.
```
Etter trinn 1: **8 av 8**, og det niende (en `https://`-kilde) er en peker uten
fil, talt som funnet-og-ikke-båret.
**En fixture-defekt gaten fant selv:** de fem dokumentene het først
`prosess-84.{pdf,docx,pptx,html,xml}`. Dørens egen § 3-kollisjonsregel refuserte
to av dem (`inbox_slug_collision: 2/7`), så to lesere ble aldri kjørt og gaten
rapporterte en bæredefekt som i virkeligheten var en fixturedefekt. Fem
forskjellige stammer nå.
---
## § 2 Hva som ble bygget
**Fem lesere PLASSERER, én modul BESTEMMER.** `llm_ingestion_okf.assets` eier
hva et bilde er, hva det heter og hvordan det pekes på; leserne vet bare hvor i
sitt eget dokument bildet står og hva kilden kaller det.
| Rad | Hvor bildet hentes | Etikett |
| --- | --- | --- |
| `.pdf` | bilde-XObjects på siden (`page.images`) | ingen — PDF har intet captionsfelt |
| `.docx` `.pptx` `.odt` `.rtf` | konverterens `--extract-media` | `descr`/alt fra containeren |
| `.html` `.htm` | `<img src alt>`, lokal sti eller `data:`-URI | `alt` |
| `.xml` | `<graphic xlink:href>`, href-en og så `graphics/<navn>` | ingen — STS har intet captionsfelt her |
`.xlsx` er **bevisst ikke** en rad: konverteren skriver én pipe-tabell per ark,
og en toradersblokk inne i en slik tabell ville brutt rad-lokatoren
`source_rows` leses tilbake ut av. Målt 2026-09-16: **0 av 4** K2-arbeidsbøker
bærer media i det hele tatt, så raden er en uttalt grense og ikke et tap.
**Etiketten gjettes ikke.** To av de fire formatene har intet captionselement —
verken et PDF-bildeobjekt eller en STS-`<graphic>` bærer ett, og «Figur 11.1
…»-linja et menneske leser er en søsken-`<p>` leseren allerede emitterer på egen
linje. Å utlede en etikett fra nærmeste linje ville vært en umerket heuristikk.
**Layouten.** `assets/` i bundle-rota,
`<sha256[:12]>-<kildens eget BASENAVN><snuset suffiks>`. I konseptet, der bildet
sto:
```markdown
![Tabell 84-2 Toleranseklasser](/assets/e54e5f5da0e8-tabell-84-2.png)
Image: graphics/tabell-84-2.png (120x90 px) -- Tabell 84-2 Toleranseklasser
```
Basenavnet og ikke stien: målt på fixture-innboksen ble ett bilde skrevet
**to ganger under to navn i én kjøring**, fordi HTML-dokumentet peker på
`graphics/figur-84-1.png` og STS-dokumentet på `figur-84-1.png` — med digesten i
begge navnene som annonserte at bytene var like. Stien er en egenskap ved
pekeren, ikke ved bildet, og hele originalen overlever på pekerens egen linje.
**Typen snuses, aldri påstås.** En `.jpg` som i virkeligheten er en PNG bæres som
PNG under et `.png`-navn; alternativet er en bundle hvis filnavn er uenige med
sitt eget innhold.
---
## § 3 PDF: to ruter, og hvorfor rasterisering ble felt
`get_data()` kjører hver filter pdfminer kjenner og stopper ved bildekodekene, så
en `DCTDecode`-strøm kommer tilbake som en ferdig JPEG og en `FlateDecode`-strøm
som rå sampler. **Ruten velges av BYTENE, ikke av filternavnet:** snus resultatet
som et bildeformat, bæres det ordrett; ellers kodes samplene til PNG med
stdlib-`zlib`.
Måling som begrunner det: R761 har **29 av 50** DCTDecode og **21** FlateDecode.
Over det 33-dokumenters K2-korpuset er populasjonen **4 828 objekter**, og
filtrene er blandet nok (`FlateDecode`, `DCTDecode`, `JPXDecode`,
`ASCII85Decode`-kjeder, `CCITTFaxDecode`) til at en gjetning fra filternavnet
ville vært gal på flere hundre.
**Alternativet ordren nevnte — rendret bbox ved 200 dpi — ble felt på
determinisme.** Et rasterisert utsnitt ville vært én kodesti og håndtert hver
filter, men bytene, og dermed assetens innholdsadresserte navn og hele bundlens
digest, ville vært avhengige av hvilken versjon av rasteriseren som var
installert. Det er nøyaktig egenskapen `OCR_DPI` sin egen docstring allerede
innrømmer at OCR-tekst ikke kan ha. En innebygd strøm har ingen slik avhengighet.
**Det koderen ikke kan uttrykke EKSAKT, nekter den for:** stencilmaske,
`Decode`-array, CMYK, alt annet enn 8-bits sampler, en `SMask` som ikke lar seg
bære. Koden er `asset_pdf_unsupported`, den telles, og den skriver en linje i
konseptet. Et bilde som er plausibelt feil farge er feil på en måte ingen
konsument kan oppdage.
**Ingen størrelsesgulv, og det er også en måling.** Det opplagte filteret er
«ignorer alt under N piksler», og fordelingen tilbyr ingen N. Over de 4 828
objektene: **149** uten oppgitt størrelse, **162** under 32x32, **92** under
64x64, **406** under 128x128, **498** under 256x256, **590** under 512x512,
**2 931** større. Et bredt spenn uten gap — motsatt av `OCR_CID_SHARE`, som er
bimodal med ingenting mellom modene. En terskel lest av ingen gap er et tall
dette repoet valgte, og det ville stille droppet noens lille tabell.
---
## § 4 Kontrollen på bytene
To hele bygg av det 43-dokumenters referansekorpuset (`K2/trinn1`), og `diff -r`
mellom dem. En eksponeringstelling er ikke en kontroll.
**Kontroll 1 — flytter opt-outen noe?** `332961a` bygget fra `git archive` mot
HEAD med `--no-assets`:
```
865 filer på begge sider. diff -rq: ÉN forskjell.
14a15
> * **Images**: NOT CARRIED — this run did not look for images, ...
```
Hvert eneste konsept er byte-identisk. Den ene forskjellen er den nye
`log.md`-linja, og den er med vilje: en bundle ingen lette etter figurer i må
ikke kunne forveksles med en bundle av dokumenter som ikke hadde noen.
**Kontroll 2 — hva koster defaulten?** Samme commit, `--no-assets` mot default:
| | `--no-assets` | default |
| --- | --- | --- |
| konsepter | 453 | **454** |
| markdown-filer | 865 | **867** |
| assets | 0 | **2 964** |
| bundle-størrelse | 4,7 MB | **115 MB** |
| veggtid | 2 414 s | **3 088 s** |
| topp-RSS | 6,26 GB | **8,74 GB** |
| md-filer som skiller seg | — | **422 av 865** |
`log.md`: **2 964 båret av 3 145 funnet** (181 nektet, 5,8 %). **4 622 pekere**
mot 2 964 filer — innholds-dedupen folder 1 658 gjentakelser inn i filene de
allerede er.
**Det ene nye konseptet har en MÅLT årsak.** Kandidaten
`- 20 …torv ødometerapparat …` i `Del II Bilag 3.2.1 - RIG-R01 Datarapport.pdf`
er en `rule:outline`-kandidat som `--outline-gate` slipper inn når ett gjenfunnet
overskriftsspenn dekker `OUTLINE_SHARE = 0.20` av teksten. Målt:
| | tekst | spennet | andel | gaten |
| --- | --- | --- | --- | --- |
| `--no-assets` | 71 255 | 13 566 | **0,190** | droppet |
| default | 90 854 | 27 757 | **0,306** | sluppet inn |
Seksjonen holder **146** av dokumentets bilder. Pekerne er kroppstekst, så
spennet vokste og krysset terskelen. Det er ikke en segmenteringsregel som
endret seg — det er den samme regelen som leser en lengre tekst.
---
## § 5 Konsumentflatene på en bundle MED `assets/`
§ 11 punkt 1 scoper konformans til `.md`-filer, så en `.png` i `assets/` deltar
ikke. Målt, ikke antatt, på en bundle bygget fra fixture-innboksen (6 assets,
6 konsepter):
| Flate | Resultat |
| --- | --- |
| `okf check --skill … --payload …` | `conformant: 17 rules over 4 excerpts and 2 withheld entries, 0 findings`, rc **0** |
| `okf skill` | rc **0**, SKILL.md skrevet |
| `okf consume` | rc **0**, 4 utdrag; pekerne reiser med utdragsteksten |
| `okf quality` | rc **3** (ingenting kunne dømmes — hver filtype har 1 dokument, under gulvet på 5). Ingen falsk `PASS`, ingen krasj |
| guard 1.4.0 `okf.import_bundle` (Dør C) | **6 av 6** konsepter slått sammen; pekerblokkene passerer gaten som kroppstekst |
**Guarden avviser ikke binære filer** — den ser dem ikke, fordi importøren går
over `.md`. Ingen `coord-send` til `llm-ingestion-pipeline-security` er derfor
nødvendig for trinn 1.
**Men Dør C bar dem ikke.** Målt 2026-09-17, før reparasjonen: importen slo
sammen **6 av 6** konsepter og skrev **ingen `assets/`-katalog i det hele tatt**,
så hver `![…](/assets/…)` i den importerte bundlen pekte på en fil som ikke var
der — samme «komplett og ikke»-defekt én dør bortenfor. Dør C bærer nå de
assetene et SAMMENSLÅTT konsept peker på, etter samme innholdsidentitetsregel den
allerede eier. Aldri hele avsenderens `assets/`: et bilde som hører til et
konsept gaten nektet, skal ikke sitte på ryggen av ett den slapp gjennom.
---
## § 6 Hva dette IKKE dekker
- **Trinn 2 er ikke bygget.** `okf describe`, transkripsjon med vision,
verifisering mot bildet — ingenting av det finnes. Invarianten «no model calls
anywhere in the run path» er uberørt: `assets.py` ser aldri på et bilde.
- **`.png`/`.jpg` som EGNE innboksfiler er fortsatt utenfor scope**
(`extractor_unknown`), som ordren sier. Fixture-innboksens to PNG-er
rapporteres som `extractor_unknown: 2/7` på begge commits.
- **R761 er ikke bygget her.** Tallene over er K2 og fixture-innboksen. En
R761-bygging hører hjemme i `vegnormal-okf` og er deres ordre, ikke denne.
- **`--no-assets`-kontrollen er kjørt på ETT korpus.** N = 1 korpus, 43
dokumenter. Den sier ingenting om et korpus med andre filtyper.
- **181 av 3 145 bilder ble nektet** og kodene er talt, men ingen har sett på
hva de 181 var. «5,8 % nektet» er et tall, ikke en diagnose.
- **Kostnaden er publisert, ikke forsvart.** 4,7 MB -> 115 MB på 43 dokumenter
er en 24x bundle. Om defaulten skal stå er operatørens, og tallene over er hva
den avgjørelsen skal tas på.
---
## § 7 Reproduksjon
```bash
# gaten
uv run pytest tests/test_asset_gate.py -q
# baselinen, fra git archive og aldri fra arbeidstreet
git archive 332961a | tar -x -C /tmp/base332961a
PYTHONPATH=/tmp/base332961a/src python3 -m llm_ingestion_okf.cli build \
~/corpora/okf-telling-20260829/K2/trinn1 --bundle /tmp/k2-base \
--bundle-id k2-trinn1-20260903 --okf-version 0.2
# de to byggene
okf build ~/corpora/okf-telling-20260829/K2/trinn1 --bundle /tmp/k2-off \
--bundle-id k2-trinn1-20260903 --okf-version 0.2 --no-assets
okf build ~/corpora/okf-telling-20260829/K2/trinn1 --bundle /tmp/k2-on \
--bundle-id k2-trinn1-20260903 --okf-version 0.2
diff -rq /tmp/k2-base /tmp/k2-off # ett avvik: log.md
diff -rq /tmp/k2-off /tmp/k2-on # 422 md-filer + 2 964 assets
```

96
docs/plan/okf-assets-section-6-4.md Normal file
View file

@ -0,0 +1,96 @@
# Proposed SPEC § 6.4: `assets/`, the bytes a concept points at
Status: **a proposal, raised from a consumer**. Written in this repository
because this repository implements the shape; the wording belongs upstream and
`_okf-canonical` is not edited from here. Pinned commit read while writing:
`ad30107` (OKF v0.2).
## Why it is needed
OKF v0.2 is silent about non-markdown files. § 3 says "A bundle is a directory
tree of markdown files"; § 11's conformance list scopes every clause to `.md`
files; § 6.3 makes `references/` a convention for external material carried as
concepts. So a picture is neither permitted nor forbidden — it is unaddressed,
and a producer that carries one is guessing about where it goes and what a
consumer may assume.
The need is not hypothetical. Measured on R761 Prosesskoden:2025, a Norwegian
road-construction process code published both as a 701-page PDF and as a
NISO-STS XML delivery: the process text is carried in full, and 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because the publisher ships
those tables as raster images in **both** deliveries. Process 84 says
"toleranseklasse ... er gitt i tabell 84-2" and table 84-2 is a JPEG. A bundle
built from that document reads as complete and is not.
## The proposed wording
> ### 6.4 The `assets/` convention
>
> A bundle MAY carry non-markdown files that its concepts point at — images
> extracted from a source document, and anything else a concept embeds rather
> than describes. An `assets/` directory at the bundle root conventionally
> holds them.
>
> A concept points at an asset with a standard markdown image or link whose
> target is a path-valued reference under § 6.2 — the bundle-relative form
> (`/assets/<name>`) is recommended, for the same reason § 6.1 recommends it
> for links between concepts: it is stable when a concept moves within its
> subdirectory.
>
> Asset file names are the producer's. A content-addressed name (for example a
> prefix of the file's SHA-256 followed by a readable remnant of the source's
> own name) is RECOMMENDED, because it makes the same bytes dropped twice one
> file and makes a rebuild of one corpus produce one bundle.
>
> An asset is not a concept. It carries no frontmatter, it is not enumerated by
> § 8's index files, and § 11's conformance clauses do not apply to it — they
> are scoped to `.md` files, and this section does not widen them.
>
> Consumers MUST NOT reject a bundle because it carries files they do not
> recognise, and MUST tolerate an asset pointer whose target is absent, for the
> same reason § 6.1 requires them to tolerate a broken link: the pointer may
> record that the source had a figure this bundle does not hold.
## What it does NOT propose
- **No screening claim.** Whether the bytes of an asset were examined is
outside this section and outside the format. This library states it per run
in `log.md` because its own gate is text-only; a picture is not text and did
not pass it.
- **No required directory.** `assets/` is a convention, exactly as
`references/` is. A producer that puts its images elsewhere and points at
them correctly is conformant.
- **No new frontmatter family.** This library writes a count (`images: N`) on
its own profiles, and that is a local key, not a proposal. § 11 already tells
consumers not to reject a concept over an unknown key.
## Conformance measured, not assumed
The claim "existing consumers do not break" is § 11 item 1 scoping to `.md`
files, plus the consumer-side MUST NOTs. Measured on a bundle WITH `assets/`,
built by `okf build` from the fixture inbox:
| Surface | Result |
| --- | --- |
| `okf check` (17 rules) | `conformant: 17 rules over 4 excerpts and 2 withheld entries, 0 findings`, rc 0 |
| `okf skill` | rc 0 |
| `okf consume` | rc 0, 4 excerpts; the pointers travel with the excerpt text |
| `okf quality` | rc 3 — "nothing could be judged", because each file type has one document and the floor is five. No false `PASS`, no crash |
| guard 1.4.0 `okf.import_bundle` | 6 of 6 concepts merged; the pointer blocks pass the gate as body text |
The guard does not reject a bundle carrying binary files — it does not see
them, because the importer walks `.md`. So no coordination message to
`llm-ingestion-pipeline-security` is needed for this step.
The measurement that did NOT pass first time is in the report: Door C merged
the concepts and wrote no `assets/` at all, so every pointer in the imported
bundle named a missing file. Fixed here, by the content-identity rule that door
already owns. The run record is
`docs/2026-09-17-bilder-i-bundlen-trinn1.md` § 5.
## Route
Raised through `portfolio-optimiser-commons`, which owns the ingest-spec this
library implements, and from there upstream. Not edited into `_okf-canonical`
from here: that mirror is a read-only pin, and a spec change written by its
implementer is not a spec change.

2
pyproject.toml
View file

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "llm-ingestion-okf"
version = "0.9.0"
version = "0.10.0"
description = "Shared OKF (Open Knowledge Format) ingestion library: spec-based connectors, bundle inbox, and external-bundle import, with security delegated to llm-ingestion-guard."
readme = "README.md"
license = "MIT"

2
src/llm_ingestion_okf/__init__.py
View file

@ -76,7 +76,7 @@ from .manifest import (
)
from .materialize import IngestResult, materialize_bundle
__version__ = "0.9.0"
__version__ = "0.10.0"
__all__ = [
"BlockedFile",

482
src/llm_ingestion_okf/assets.py Normal file
View file

@ -0,0 +1,482 @@
"""Binary assets: what an image IS, what it is called, and how it is pointed at.
Until 0.10.0 this package had no image path at all. Every reader recovered
text, every warning said so, and the only writer into a bundle was
:func:`materialize.write_bytes`, whose signature is ``(bundle_dir, name,
content: str)`` -- UTF-8 in, text out, no binary route anywhere. A document
whose table is a raster picture therefore reached a concept as an absence with
no denominator. Measured on R761 Prosesskoden:2025, the publisher's own
NISO-STS delivery: the process text is carried in full, and 12 ``Tabell N-N``
and 9 ``Figur N-N`` captions stand over nothing, so process 84's
"toleranseklasse ... er gitt i tabell 84-2" points at empty space.
THIS MODULE IS THE ONE PLACE THAT DECIDES WHAT AN IMAGE IS. A reader hands it
bytes and a name; it returns a carried image or raises a coded rejection. That
is what makes "N carried of M found" mean the same thing for ``pdf``, ``docx``,
``html`` and ``xml``, and it is what keeps each format's quirks out of the
bundle layout.
THREE RULES, and each one exists because the alternative is a silent lie:
- **The type is SNIFFED, never claimed.** Measured on the R761 delivery, the
graphics directory holds ``.bmp``, ``.jpg`` and ``.png`` side by side and the
document's ``xlink:href`` values are whatever the publisher's tool wrote. A
name is a claim; the magic bytes are the fact. A consumer dispatching on the
extension of a name that lies reads the file wrong with full confidence.
- **The name is CONTENT-ADDRESSED** -- ``<sha256[:12]>-<reduced original>``.
Two drops of one image are one file, a rebuild of one corpus is one bundle,
and the digest carries the uniqueness so the readable tail can be shortened
without any risk of collision. That is the byte-determinism rule this package
already holds for text, extended to the bytes beside it.
- **The pointer is ONE GRAMMAR**, owned here. ``okf describe`` (step 2) has to
find every pointer mechanically in order to write a transcription under it,
so the block is a regex this module ships beside the writer rather than a
shape each reader invents and each consumer re-derives.
WHAT THIS MODULE DOES NOT DO: it never looks at a picture. Classifying an image
as a table or a figure, and reading what it says, is a model call, and the
invariant "no model calls anywhere in the run path" is not negotiated here --
step 2 is a separate command, outside the build path, and this module is
importable without it.
**The image BYTES are not screened.** The guard is text-only (its own boundary,
not ours), so what passes a persist gate is the pointer block, as body text,
like every other line. The bytes of a carried image are written to the bundle
unscreened. Stated here rather than implied, because a consumer weighing an
untrusted drop needs to know which half of the concept was looked at.
"""
from __future__ import annotations
import hashlib
import re
import struct
import unicodedata
import zlib
from dataclasses import dataclass
from .errors import ExtractionError
#: The one directory a bundle keeps its binary assets in, at the bundle root.
#: Fixed rather than configurable: a consumer resolving `/assets/...` out of a
#: concept has only the bundle, and a per-profile directory would make that
#: pointer unresolvable without also shipping the profile that wrote it.
ASSETS_DIR = "assets"
#: How much of the digest names the file. 12 hex characters is 48 bits; over
#: the largest asset population measured here (4 828 image objects in one
#: 33-document corpus) the birthday probability of a collision is about
#: 4e-11. A collision would be caught anyway -- an occupied name is re-used
#: only when the bytes there are already identical, the same content-identity
#: rule Door C proves ownership with.
DIGEST_PREFIX = 12
#: How much of the original name survives into the asset name. The digest
#: carries uniqueness, so this is decoration and truncating it is safe -- which
#: is the opposite of `materialize.check_filename_length`'s situation, where the
#: name IS the identity and truncation would silently merge two documents.
NAME_TAIL_MAX = 60
#: Magic bytes -> (media type, suffix). Sniffed in this order; the first match
#: wins, and nothing here overlaps.
_MAGIC: tuple[tuple[bytes, str, str], ...] = (
(b"\x89PNG\r\n\x1a\n", "image/png", ".png"),
(b"\xff\xd8\xff", "image/jpeg", ".jpg"),
(b"GIF87a", "image/gif", ".gif"),
(b"GIF89a", "image/gif", ".gif"),
(b"BM", "image/bmp", ".bmp"),
(b"II\x2a\x00", "image/tiff", ".tiff"),
(b"MM\x00\x2a", "image/tiff", ".tiff"),
)
#: JPEG 2000, in both the forms a PDF `JPXDecode` stream hands back: the JP2
#: container and a bare codestream.
_JP2_SIGNATURE = b"\x00\x00\x00\x0cjP \r\n\x87\n"
_J2K_SIGNATURE = b"\xff\x4f\xff\x51"
#: The frame markers that carry a JPEG's dimensions. Every SOF except the four
#: that are not frame headers at all (`DHT` 0xC4, `JPG` 0xC8, `DAC` 0xCC).
_JPEG_SOF = frozenset(range(0xC0, 0xD0)) - {0xC4, 0xC8, 0xCC}
#: `materialize.reduce_to_id_grammar`'s rule, restated. Not imported: this
#: module is reached from `extract.py`, whose registry must not import the
#: contract layer, and `materialize` pulls in `manifest` and `profiles`.
#: `tests/test_assets.py` holds the two forms equal on the same inputs, so the
#: restatement cannot drift into a second grammar.
_SEPARATOR_RUN = re.compile(r"[^a-z0-9]+")
#: One pointer block, as this module writes it. Group 1 is the label, group 2
#: is the asset file name, group 3 is the whole second line. `okf describe`
#: finds its work with this and writes under the match; a consumer wanting to
#: strip pointers uses the same expression, so there is one definition of what
#: a pointer looks like rather than one per reader.
IMAGE_POINTER = re.compile(
r"^!\[(?P<label>[^\]\n]*)\]\(/" + ASSETS_DIR + r"/(?P<asset>[^)\s]+)\)\n"
r"(?P<detail>Image: [^\n]*)$",
re.MULTILINE,
)
@dataclass(frozen=True)
class ExtractedImage:
"""One image a document carries, as this package will write it.
`name` is what the SOURCE calls the file -- a `xlink:href`, an `<img src>`,
a media entry inside an OOXML container, or a synthesised name for a PDF
image object, which has none. It is carried verbatim into the pointer's
second line and reduced (never trusted) into the asset's own name.
`label` is a caption or alt text where the format HAS one, and `None`
where it does not. Two of the four readers have none: a PDF image object
and an STS `<graphic>` carry no caption element, and the caption a human
sees is running text the extractor already emitted on its own line.
Deriving a label from the nearest line would be an unmarked heuristic,
which this package treats as worse than no heuristic at all.
"""
data: bytes
name: str
media_type: str
suffix: str
width: int | None
height: int | None
label: str | None = None
@dataclass(frozen=True)
class AssetRejection:
"""An image that was FOUND and not carried, with the reason as a code.
A rejection is a row in the run log, never a failed document: one
unreadable picture must not cost the 3 000 concepts of text around it. The
denominator is what makes the log worth reading -- "51 carried" says
nothing without "of 53 found".
"""
name: str
code: str
reason: str
def sniff(data: bytes) -> tuple[str, str] | None:
"""`(media type, suffix)` read from the bytes, or `None` if not an image.
The claimed extension never participates. A `.jpg` that is really a PNG is
carried as a PNG under a `.png` name, because the alternative is a bundle
whose file names disagree with their contents.
"""
if data.startswith(_JP2_SIGNATURE) or data.startswith(_J2K_SIGNATURE):
return "image/jp2", ".jp2"
for magic, media_type, suffix in _MAGIC:
if data.startswith(magic):
return media_type, suffix
return None
def dimensions(data: bytes) -> tuple[int, int] | None:
"""`(width, height)` in pixels from the file's own header, or `None`.
`None` is a MEASUREMENT: this reader does not read every format's geometry
(JPEG 2000 and TIFF are absent), and a concept that printed `0x0 px` would
be stating a number nobody took. The pointer says "dimensions unknown"
instead.
"""
kind = sniff(data)
if kind is None:
return None
suffix = kind[1]
try:
if suffix == ".png":
if len(data) < 24 or data[12:16] != b"IHDR":
return None
width, height = struct.unpack(">II", data[16:24])
return (width, height) if width and height else None
if suffix == ".jpg":
return _jpeg_dimensions(data)
if suffix == ".gif":
if len(data) < 10:
return None
width, height = struct.unpack("<HH", data[6:10])
return (width, height) if width and height else None
if suffix == ".bmp":
return _bmp_dimensions(data)
except (struct.error, IndexError):
return None
return None
def _jpeg_dimensions(data: bytes) -> tuple[int, int] | None:
position = 2
end = len(data)
while position + 3 < end:
if data[position] != 0xFF:
position += 1
continue
marker = data[position + 1]
if marker in (0xD8, 0x01) or 0xD0 <= marker <= 0xD7:
position += 2
continue
if marker == 0xFF:
position += 1
continue
length = struct.unpack(">H", data[position + 2 : position + 4])[0]
if marker in _JPEG_SOF:
if position + 9 > end:
return None
height, width = struct.unpack(">HH", data[position + 5 : position + 9])
return (width, height) if width and height else None
position += 2 + length
return None
def _bmp_dimensions(data: bytes) -> tuple[int, int] | None:
if len(data) < 26:
return None
header_size = struct.unpack("<I", data[14:18])[0]
if header_size == 12:
width, height = struct.unpack("<hh", data[18:22])
else:
width, height = struct.unpack("<ii", data[18:26])
# A negative height is a top-down BMP; the magnitude is the pixel count.
return (abs(width), abs(height)) if width and height else None
def read_image(data: bytes, *, name: str, label: str | None = None) -> ExtractedImage:
"""One image, typed by its bytes, or a coded refusal.
Raises :class:`ExtractionError` with `asset_type_unknown` when the bytes
are not an image this package recognises. The caller records that as an
:class:`AssetRejection` and keeps going -- a document is not lost over one
picture.
"""
kind = sniff(data)
if kind is None:
raise ExtractionError(
f"the bytes behind {name!r} are not an image format this package "
f"recognises (first bytes {data[:8]!r})",
code="asset_type_unknown",
)
media_type, suffix = kind
size = dimensions(data)
return ExtractedImage(
data=data,
name=name,
media_type=media_type,
suffix=suffix,
width=size[0] if size else None,
height=size[1] if size else None,
label=label,
)
def digest(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def _reduce(text: str) -> str:
return _SEPARATOR_RUN.sub("-", unicodedata.normalize("NFC", text).lower()).strip("-")
def asset_name(image: ExtractedImage) -> str:
"""`<sha256[:12]>-<reduced original BASENAME><sniffed suffix>`.
DEDUP IS ON CONTENT, and the digest is what makes it so: the same bytes
arriving twice are one file, whichever document dropped them. The residual
is stated rather than claimed away -- identical bytes shipped under two
different base names are two files in the bundle, both correct and both
holding the same picture. Measured on the fixture inbox and on the R761
delivery, that case occurs 0 times; a rule that could not produce it at all
would have to drop the readable tail entirely, and the tail is what makes
`assets/` legible to the person checking a bundle by hand.
The suffix comes from the bytes and the stem from the name, reduced to the
same grammar every generated filename in this package uses. A link target
sits inside `](...)`, which a space or a closing parenthesis terminates --
and the R761 delivery's own hrefs carry both (`25-0143 - Tabeller -
Prosesskoden (R761-R762).jpg`), so reducing is what makes the pointer
followable rather than merely tidy.
"""
# The BASENAME, never the path the document reached it through. Measured
# on the fixture inbox: one image pointed at as `graphics/figur-84-1.png`
# from an HTML document and as `figur-84-1.png` from an STS one was written
# twice, under two names, in one run -- with the digest in both announcing
# that the bytes were identical. The path is a property of the pointer, not
# of the picture, and the full original survives on the pointer's own line.
base = image.name.rsplit("/", 1)[-1]
stem = _reduce(base.rsplit(".", 1)[0] if "." in base else base)
head = digest(image.data)[:DIGEST_PREFIX]
if not stem:
return f"{head}{image.suffix}"
return f"{head}-{stem[:NAME_TAIL_MAX].rstrip('-')}{image.suffix}"
def asset_href(image: ExtractedImage) -> str:
"""The bundle-absolute path SPEC SS 6.2 allows.
Absolute rather than relative because a segmented bundle puts concepts at
different depths: `assets/x.png` resolves to two different places from two
concepts of one document, and `/assets/x.png` to one place from every
concept in the bundle.
"""
return f"/{ASSETS_DIR}/{asset_name(image)}"
def _inline(value: str) -> str:
"""A label, made safe for the one line it is written on.
`[` and `]` are the link grammar's own delimiters and a newline would open
a third line in a two-line block, so both are replaced rather than escaped:
Door B refuses a title containing a bracket outright (`inbox_title_invalid`)
and this text reaches a title through no route, but the pointer is body text
a proposer reads, and a half-open link there is a pointer that resolves
nowhere.
"""
collapsed = " ".join(value.split())
return collapsed.replace("[", "(").replace("]", ")")
def render_block(image: ExtractedImage) -> str:
"""The two lines that stand where the image stands.
Line one is markdown, so a reader that renders the concept sees the picture
and a reader that does not sees the label. Line two states what the first
line cannot: the name the SOURCE gave the file, and the size in pixels --
the two facts a person checking the bundle against the original needs, and
the two a transcription in step 2 has to be judged against.
"""
label = image.label or image.name or asset_name(image)
size = (
f"{image.width}x{image.height} px"
if image.width is not None and image.height is not None
else "dimensions unknown"
)
detail = f"Image: {_inline(image.name or asset_name(image))} ({size})"
if image.label:
detail += f" -- {_inline(image.label)}"
return f"![{_inline(label)}]({asset_href(image)})\n{detail}"
def encode_png(
width: int,
height: int,
samples: bytes,
*,
channels: int,
palette: bytes | None = None,
alpha: bytes | None = None,
) -> bytes:
"""8-bit samples as a PNG, using nothing but `zlib`.
A PDF image is usually not a file: `FlateDecode` hands back raw samples
with the colour model in the dictionary beside them, so carrying one at all
means encoding it. Doing that with the stdlib rather than with a renderer
is what keeps the `pdf` row free of a new dependency AND keeps an asset's
bytes -- and therefore its content-addressed name -- independent of which
version of a rasteriser happened to be installed. `OCR_DPI`'s docstring
states the opposite property for OCR text, and the difference is deliberate.
`channels` is 1 (grey, or indexed when `palette` is given) or 3 (RGB).
`alpha` is one byte per pixel, from a PDF `SMask`; absent means opaque.
"""
if channels not in (1, 3):
raise ExtractionError(
f"PNG encoding supports 1 or 3 channels, not {channels}",
code="asset_samples_invalid",
)
if width <= 0 or height <= 0:
raise ExtractionError(
f"PNG encoding needs positive dimensions, got {width}x{height}",
code="asset_samples_invalid",
)
expected = width * height * channels
if len(samples) < expected:
raise ExtractionError(
f"the sample buffer holds {len(samples)} bytes where {width}x{height} "
f"at {channels} channels needs {expected} -- refusing to pad, because a "
"short buffer means the image dictionary was read wrong",
code="asset_samples_invalid",
)
if alpha is not None and len(alpha) < width * height:
raise ExtractionError(
f"the alpha buffer holds {len(alpha)} bytes where {width}x{height} needs "
f"{width * height}",
code="asset_samples_invalid",
)
if palette is not None:
if channels != 1:
raise ExtractionError(
"a palette applies to single-channel samples only",
code="asset_samples_invalid",
)
if not palette or len(palette) % 3:
raise ExtractionError(
f"a palette must be whole RGB triples, got {len(palette)} bytes",
code="asset_samples_invalid",
)
colour_type = 3
elif alpha is not None:
colour_type = 6 if channels == 3 else 4
else:
colour_type = 2 if channels == 3 else 0
rows = bytearray()
for row in range(height):
# Filter type 0 (None) on every row. A filter would shrink the file and
# would make the bytes depend on a heuristic; this encoder's output has
# to be reproducible from the samples alone, for as long as the bundle
# is quoted by its digest.
rows.append(0)
start = row * width * channels
line = samples[start : start + width * channels]
if alpha is None:
rows += line
else:
for pixel in range(width):
rows += line[pixel * channels : (pixel + 1) * channels]
rows.append(alpha[row * width + pixel])
def chunk(kind: bytes, payload: bytes) -> bytes:
return (
len(payload).to_bytes(4, "big")
+ kind
+ payload
+ zlib.crc32(kind + payload).to_bytes(4, "big")
)
ihdr = struct.pack(">IIBBBBB", width, height, 8, colour_type, 0, 0, 0)
body = b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", ihdr)
if palette is not None:
body += chunk(b"PLTE", palette)
# Level 9 with no filtering: `zlib` is stdlib and its output for a given
# level is stable within a Python build, which is the same guarantee the
# rest of this package's byte-determinism rests on.
body += chunk(b"IDAT", zlib.compress(bytes(rows), 9))
return body + chunk(b"IEND", b"")
def render_missing(
name: str,
*,
reason: str,
label: str | None = None,
href: str | None = None,
) -> str:
"""A pointer to an image the bundle does NOT carry, and why.
An image this package found and could not carry is stated in the concept,
not dropped. The reader of the bundle is the person who has to decide
whether the missing picture mattered, and they cannot decide about an
absence they were never shown -- which is precisely the shape of the defect
this whole capability exists to close.
The href is kept when the source had one and it survives the link grammar,
so a remote figure says WHERE it was. Extraction never opens a socket: the
network gate is an explicit per-run opt-in and extraction is not on that
path, so a remote source is carried as a name and never as bytes.
"""
shown = _inline(name or "image")
detail = f"Image: {shown} (not carried: {_inline(reason)})"
if href and not re.search(r"[\s()\[\]]", href):
return f"![{_inline(label or name or 'image')}]({href})\n{detail}"
return detail

38
src/llm_ingestion_okf/cli.py
View file

@ -75,6 +75,7 @@ from functools import partial
from pathlib import Path
from .corpus import (
DEFAULT_ASSETS,
GATE_GUARD_TRUSTED_SOURCE,
GATE_NAMES,
LOG_NAME,
@ -231,6 +232,7 @@ DEFAULT_SHELL_PARENT = False
#: reproduces published numbers with it; what changed is that reaching it is an
#: act, and `log.md` records which gate screened the bundle either way.
DEFAULT_GATE = GATE_GUARD_TRUSTED_SOURCE
#: Round 9. Clause 1 asked whether a title ENDED in a number, which is a
#: question about the number rather than about the title: a drawing's dimension
#: chain, a P&ID's schematic labels, a door schedule and a borehole log's
@ -413,6 +415,7 @@ def _propose_plans(
ocr: bool = False,
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
shell_parent: bool = DEFAULT_SHELL_PARENT,
assets: bool = False,
) -> tuple[int, int, int]:
"""Propose a plan per dropped file. Returns (written, nothing, failed).
@ -454,6 +457,7 @@ def _propose_plans(
ocr=ocr,
pdf_outline=pdf_outline,
shell_parent=shell_parent,
assets=assets,
)
except ProposerError as exc:
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
@ -495,6 +499,7 @@ def build(
shell_parent: bool = DEFAULT_SHELL_PARENT,
frontmatter: Mapping[str, str] | None = None,
gate: str = DEFAULT_GATE,
assets: bool = DEFAULT_ASSETS,
) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing.
@ -535,6 +540,7 @@ def build(
ocr=ocr,
concept_frontmatter_values=concept_values,
gate=gate,
assets=assets,
)
_write_log(bundle, report, profile=STRUCTURED_V1)
return report
@ -576,6 +582,12 @@ def build(
ocr=ocr,
pdf_outline=pdf_outline,
shell_parent=shell_parent,
# THE SAME VALUE ON BOTH SIDES, and it cannot be otherwise: a plan
# records `text_sha256` of the exact string it was proposed
# against, and carrying an image inserts lines into that string.
# Proposing with the images off and replaying with them on makes
# `assert_plan_applies` refuse every document that has a figure.
assets=assets,
)
print(
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
@ -595,6 +607,7 @@ def build(
ocr=ocr,
concept_frontmatter_values=concept_values,
gate=gate,
assets=assets,
)
_write_log(bundle, report, profile=SEGMENTED_OKF_V0_2)
return report
@ -897,6 +910,30 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"pre-2026-09-11 spans byte for byte"
),
)
build_parser.add_argument(
"--assets",
action="store_true",
default=DEFAULT_ASSETS,
help=(
"carry the images the sources declare into the bundle's assets/ "
"directory and point at them from the concept text where they "
"stand. ON since 0.10.0. Measured on the R761 process code: 12 "
"Tabell N-N and 9 Figur N-N captions stood over nothing, because "
"the publisher ships those tables as raster pictures. The image "
"BYTES are not screened -- the gate reads text -- and log.md says so"
),
)
build_parser.add_argument(
"--no-assets",
action="store_false",
dest="assets",
help=(
"the explicit opt-out, reproducing the pre-0.10.0 bytes for a "
"consumer pinned to them. log.md then says NOT CARRIED, so a "
"bundle nobody looked for figures in cannot be mistaken for a "
"bundle of documents that had none"
),
)
build_parser.add_argument(
"--gate",
choices=GATE_NAMES,
@ -1084,6 +1121,7 @@ def main(argv: list[str] | None = None) -> int:
pdf_outline=args.pdf_outline,
shell_parent=args.shell_parent,
gate=args.gate,
assets=args.assets,
frontmatter=frontmatter_from_flags(args.frontmatter or ()),
)
except (IngestError, OSError, ValueError) as exc:

86
src/llm_ingestion_okf/corpus.py
View file

@ -40,6 +40,7 @@ from collections.abc import Callable, Mapping
from dataclasses import dataclass, replace
from pathlib import Path
from .assets import ASSETS_DIR
from .errors import IngestError
from .extract import extract_text
from .inbox import (
@ -87,6 +88,33 @@ LOG_TITLE = "Corpus run history"
#: numbers with it and a caller measuring segmentation alone has a legitimate
#: reason to take the gate out of the picture -- but reaching it is now an act,
#: and the bundle's own log says so.
#: Whether a build carries the images its sources declare (0.10.0).
#:
#: ON, and the reason is the premise the capability was ordered on: what ends
#: up in a bundle after converting N sources has to BE the sources. Measured on
#: R761 Prosesskoden:2025 -- 2 726 of 2 726 processes carry their text and 12
#: `Tabell N-N` and 9 `Figur N-N` captions stand over nothing, because the
#: publisher ships those tables as raster pictures in both the PDF and the
#: NISO-STS delivery. A bundle that reads as complete and answers "toleranse-
#: klasse ... er gitt i tabell 84-2" with an empty space is the defect; it is
#: not a preference about bundle size.
#:
#: WHAT THIS COSTS IS PUBLISHED RATHER THAN IMPLIED. Measured over the K2
#: reference corpus, 2026-09-16: 32 of its 33 PDFs carry image objects and the
#: population is 4 828, of which 2 931 are 512x512 pixels or larger, 1 307 are
#: smaller and 149 declare no size at all. A drawing set contributes 2 524 on
#: its own. So this is a large change for a drawing-heavy corpus and none at
#: all for a prose one, and `--no-assets` reproduces the pre-0.10.0 bytes
#: exactly -- measured by `diff`, not asserted.
#:
#: NO SIZE FLOOR, and that is a measurement too. The obvious filter would be
#: "ignore anything under N pixels", and the distribution above does not offer
#: an N: it is a broad spread with no gap, unlike `OCR_CID_SHARE`'s, which is
#: bimodal with nothing between the modes. A threshold read off no gap is a
#: number this package chose, and it would silently drop somebody's small
#: table. Stated as a limit instead of guessed at.
DEFAULT_ASSETS = True
GATE_NONE = "none"
#: `okf build`'s default, set by the operator 2026-09-15 on a measurement:
#: over the 453 concept bodies of the pinned default bundle, this tier returns
@ -200,6 +228,14 @@ class CorpusReport:
#: stub so every existing construction site stays source-compatible; the
#: two commands that build bundles both pass it explicitly.
gate: str = GATE_NONE
#: THE ASSET DENOMINATOR (0.10.0). `assets_carried` is what reached the
#: bundle, `assets_found` is what the readers saw. Both, always: "51
#: carried" is a measurement only beside "of 53 found", and a run whose
#: figures were all refused must not read like a run over documents that had
#: none. `assets` says whether they were looked for at all.
assets: bool = False
assets_carried: int = 0
assets_found: int = 0
@property
def merged(self) -> int:
@ -291,11 +327,37 @@ class CorpusReport:
f"* **Gate**: {self.gate} "
f"({_GATE_DESCRIPTIONS.get(self.gate, 'unrecognised gate name')}). "
"Every persisted byte of this bundle passed it.",
self._assets_line(),
]
if self.unaccounted:
lines.append("* **Unaccounted**: " + ", ".join(self.unaccounted) + " — K1b FAILED.")
return "\n".join(lines) + "\n"
def _assets_line(self) -> str:
"""What this run did about the images in its sources, stated either way.
An absent bullet would make "this corpus had no figures" and "this run
never looked" the same artifact, which is the trust trap `--gate none`
already has its own sentence for. So the OFF case is written out in
capitals rather than left to an inference.
THE IMAGE BYTES ARE NOT SCREENED, and the log is where a consumer finds
that out. The guard is text-only, so the pointer block passed the gate
as body text and the picture beside it did not pass anything.
"""
if not self.assets:
return (
"* **Images**: NOT CARRIED — this run did not look for images, so a "
"document whose table is a picture reached the bundle as text with a "
"gap in it. Absence of an image here is not evidence the sources had none."
)
return (
f"* **Images**: {self.assets_carried} carried of {self.assets_found} found, "
f"written to `{ASSETS_DIR}/` and pointed at from the concepts they stand in. "
"The image BYTES were not screened: the gate above reads text, and a "
"picture is not text."
)
def load_plans(plans_dir: Path) -> dict[str, SegmentationPlan]:
"""Every proposal artifact in a directory, keyed by filename.
@ -357,6 +419,7 @@ def measure(
ocr: bool = False,
concept_frontmatter_values: Mapping[str, str] | None = None,
gate: str = GATE_NONE,
assets: bool = False,
) -> CorpusReport:
"""Run the corpus through the door and count what happened.
@ -381,6 +444,7 @@ def measure(
pdf_headings=pdf_headings,
heading_reserve=heading_reserve,
ocr=ocr,
assets=assets,
concept_frontmatter_values=concept_frontmatter_values,
)
elapsed = time.monotonic() - started
@ -416,6 +480,9 @@ def measure(
codes=tuple(sorted(counts.items())),
unaccounted=unaccounted_names(dropped=dropped, merged=merged_names, coded=coded_names),
gate=gate,
assets=assets,
assets_carried=len(result.assets),
assets_found=len(result.assets) + len(result.assets_rejected),
)
@ -456,6 +523,24 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"would make that equality depend on which command you ran"
),
)
parser.add_argument(
"--assets",
action="store_true",
default=DEFAULT_ASSETS,
help=(
"carry the images the sources declare into the bundle. The same "
"name and the same default as `okf build --assets`, for the same "
"reason the gate carries them: the two paths are held byte-equal "
"by a test, and two different defaults would make that equality "
"depend on which command you ran"
),
)
parser.add_argument(
"--no-assets",
action="store_false",
dest="assets",
help="the explicit opt-out, reproducing the pre-0.10.0 bytes",
)
parser.add_argument(
"--okf-version",
default=None,
@ -532,6 +617,7 @@ def main(argv: list[str] | None = None) -> int:
report = measure(
args.corpus,
bundle,
assets=args.assets,
ingested_at=args.ingested_at,
plans=plans,
profile=profile,

31
src/llm_ingestion_okf/errors.py
View file

@ -108,6 +108,31 @@ class ExtractionError(IngestError):
`extractor_extra_missing`: the `[extract]` extra can be fully installed
and the document parsed, with only the OCR engine absent, and one code
for both would send an operator to reinstall what they already have
ASSET codes (0.10.0). None of these fails a document: an image a reader
could not carry is a ROW in the run log and a line in the concept saying
what was there, because one unreadable picture must not cost the three
thousand concepts of text around it.
- `asset_type_unknown` — the bytes behind a pointer are not an image format
this package recognises. Sniffed from the bytes, never from the claimed
extension: a `.jpg` that is really a PNG would otherwise be written under
a name whose extension lies
- `asset_unresolved` — the file a document points at was not found beside
it, or an inline data URI could not be decoded. Containment is the
document's own directory, so a reference above it lands here rather than
being followed
- `asset_remote` — the source is off this machine. Extraction opens no
socket: network access is an explicit per-run opt-in and extraction is not
on that path, so a remote image is carried as a pointer and never as bytes
- `asset_pdf_unsupported` — a PDF image whose samples this encoder does not
express: a stencil mask, a `Decode` array, a colour space with no exact
PNG form, a soft mask that cannot be carried, or anything but 8-bit
samples. Refused rather than approximated, because a picture that is
plausibly the wrong colour is wrong in a way no consumer can detect
- `asset_samples_invalid` — the sample buffer does not fit the dimensions
the image dictionary declares. Refused rather than padded: a short buffer
means the dictionary was read wrong
"""
@ -130,6 +155,12 @@ class MaterializationError(IngestError):
- `ingested_at_invalid` — ingested_at is not ISO-8601 UTC with a Z suffix
- `collision_unstamped` — the §3 collision gate: a generated name is
occupied by a file without the ingest stamp
- `asset_collision` — two different images reduce to one asset name in one
run, or a name in `assets/` is occupied by different bytes (Doors B and
C, 0.10.0). An asset name carries the digest of its own contents, so this
is a `sha256[:12]` collision; refused rather than resolved, because
resolving it silently means one of two pictures is lost and every pointer
to it shows the other
- `source_reference_unquotable` — a manifest source's id or locator
contains a character that would restructure the `sources` flow mapping
(Door A, v0.2 profiles); refused rather than emitted, because the

775
src/llm_ingestion_okf/extract.py
View file

@ -23,6 +23,8 @@ registry's. No guard call and no model call anywhere in this module.
from __future__ import annotations
import base64
import binascii
import collections
import csv
import functools
@ -30,6 +32,7 @@ import io
import re
import statistics
import tempfile
import urllib.parse
import warnings
import zipfile
from collections.abc import Callable, Sequence
@ -39,7 +42,17 @@ from pathlib import Path
from xml.etree import ElementTree
from xml.etree.ElementTree import Element
from .errors import ExtractionError, ExtractionWarning
from .assets import (
AssetRejection,
ExtractedImage,
encode_png,
read_image,
render_block,
render_missing,
sniff,
)
from .connectors import safe_resolve
from .errors import ExtractionError, ExtractionWarning, SourceError
from .render import render_fenced_block, render_table
# Binary types gated behind the optional `[extract]` extra that it ships no
@ -329,6 +342,174 @@ _XML_INLINE_TAGS = frozenset(
_ATX_MAX_LEVEL = 6
# --- assets: the images a document carries beside its text ------------------
#
# ADDED IN 0.10.0, and off unless the caller asks. Until then no reader here
# fetched, named or copied an image: `<img>`'s attributes were never read, an
# STS `<graphic>` was walked past, a PDF was opened for `extract_text` alone,
# and the converter's markdown writer dropped every picture. The two lossiness
# warnings above said so on every run, which made the loss honest and did not
# make it smaller.
#
# THE READER PLACES, THE COLLECTOR DECIDES. Each reader knows where in its own
# document an image stands and what the document calls it; nothing else. What
# an image IS, what it is named in the bundle and how it is pointed at is
# `assets.py`'s, so the four readers cannot drift into four grammars, and
# "carried N of M" means one thing across all of them.
#: How a reader asks for bytes the document only POINTS at. The inbox supplies
#: one rooted at the dropped file's own directory and refuses to leave the drop
#: -- an `<img src="../../../etc/passwd">` is a path traversal written by
#: whoever wrote the document, which is exactly the class of input this package
#: treats as untrusted. `None` means the caller gave no resolver, and every
#: pointer then resolves to nothing rather than to a guess.
Resolver = Callable[[str], bytes | None]
#: A scheme this package will not open. Extraction opens no socket at all --
#: network access here is an explicit per-run opt-in and extraction is not on
#: that path -- so a remote image is carried as a POINTER and never as bytes.
_REMOTE_SOURCE = re.compile(r"^(?:[a-zA-Z][a-zA-Z0-9+.-]*:|//)")
_DATA_URI = re.compile(r"^data:(?P<media>[^;,]*)(?P<base64>;base64)?,(?P<payload>.*)$", re.DOTALL)
@dataclass(frozen=True)
class ExtractedDocument:
"""One dropped file's text, and the images that stand inside that text.
The text is what it always was when `images` and `rejected` are empty --
which is what the `assets=False` default guarantees byte for byte, so every
golden, every pinned bundle and every published digest is unmoved by this
capability existing.
"""
text: str
images: tuple[ExtractedImage, ...] = ()
rejected: tuple[AssetRejection, ...] = ()
class _AssetCollector:
"""What a reader hands an image to, and the only place a rejection is made.
A rejection is a ROW, never a failed document: one unreadable picture must
not cost the text around it. Every rejection also leaves a line in the
concept saying what was there and why it is not here -- an absence a reader
is never shown is an absence they cannot weigh, which is the defect this
whole capability closes one level up.
"""
def __init__(self, resolve: Resolver | None = None) -> None:
self._resolve = resolve
self.images: list[ExtractedImage] = []
self.rejected: list[AssetRejection] = []
def carry(self, data: bytes, *, name: str, label: str | None = None) -> str:
"""Bytes the reader already holds, as the block that stands in the text."""
try:
image = read_image(data, name=name, label=label)
except ExtractionError as exc:
return self.reject(name, code=exc.code, reason=str(exc), label=label)
self.images.append(image)
return render_block(image)
def reject(
self,
name: str,
*,
code: str,
reason: str,
label: str | None = None,
href: str | None = None,
) -> str:
self.rejected.append(AssetRejection(name=name, code=code, reason=reason))
return render_missing(name, reason=reason, label=label, href=href)
def local(self, source: str, *, label: str | None = None, sibling: str | None = None) -> str:
"""An href the DOCUMENT points at: a data URI, a local path, or remote.
`sibling` is a SECOND path to try, and it exists because one format's
convention is not another's: a NISO-STS delivery writes a bare file
name and ships the files in a `graphics/` directory beside the
document. Passing the candidate rather than teaching this method about
STS keeps the convention with the reader that has it, and the
containment rule with the resolver that owns it.
"""
data_uri = _DATA_URI.match(source)
if data_uri is not None:
return self._data_uri(data_uri, label=label)
if _REMOTE_SOURCE.match(source):
return self.reject(
source,
code="asset_remote",
reason="the source is off this machine and extraction opens no socket",
label=label,
href=source,
)
data = self._resolve(source) if self._resolve is not None else None
if data is None and sibling is not None and sibling != source and self._resolve is not None:
data = self._resolve(sibling)
if data is None:
return self.reject(
source,
code="asset_unresolved",
reason="the file the document points at was not found beside it",
label=label,
)
return self.carry(data, name=source, label=label)
def _data_uri(self, match: re.Match[str], *, label: str | None) -> str:
payload = match.group("payload")
name = f"data-uri{Path(match.group('media').split('/')[-1] or 'bin').suffix or ''}"
try:
raw = (
base64.b64decode(payload, validate=True)
if match.group("base64")
else urllib.parse.unquote_to_bytes(payload)
)
except (binascii.Error, ValueError) as exc:
return self.reject(
name,
code="asset_unresolved",
reason=f"the inline data URI could not be decoded: {exc}",
label=label,
)
return self.carry(raw, name=name, label=label)
def directory_resolver(root: Path) -> Resolver:
"""A resolver rooted at ONE directory, fail-closed, reading nothing else.
Containment is against the DOCUMENT'S OWN directory rather than against the
whole drop, and that is not caution for its own sake: the proposer reads a
file straight off disk and the door reads it out of the inbox, and a plan
indexes the exact string it was proposed against. One root both sides can
compute from the document alone is what makes the two renderings identical
without threading a second path through either. A reference above the
document's directory is refused (`asset_unresolved`) rather than followed;
the limit is stated in the concept, like every other rejection.
`safe_resolve` is Door A's own rule, reused verbatim: `..` traversal, an
absolute path, a symlink escape and a prefix-collision sibling all fail
closed. An `<img src="../../../etc/passwd">` is a path written by whoever
wrote the document, which is exactly the class of input this package treats
as untrusted.
"""
def resolve(relative: str) -> bytes | None:
try:
target = safe_resolve(root, relative)
except SourceError:
return None
try:
if not target.is_file():
return None
return target.read_bytes()
except OSError:
return None
return resolve
def decode_text(data: bytes) -> str:
"""Decode file bytes as UTF-8 (BOM-stripping), typed on failure.
@ -384,12 +565,13 @@ class _HTMLTextExtractor(HTMLParser):
one-line form. Nothing is ever dropped here beyond `_SKIP_TAGS`.
"""
def __init__(self) -> None:
def __init__(self, collector: _AssetCollector | None = None) -> None:
super().__init__(convert_charrefs=True)
self._lines: list[str] = []
self._current: list[str] = []
self._prefix = ""
self._skip_depth = 0
self._collector = collector
def _break(self, prefix: str = "") -> None:
"""Close the line being accumulated and open the next one."""
@ -399,6 +581,26 @@ class _HTMLTextExtractor(HTMLParser):
self._lines.append(f"{self._prefix}{line}")
self._prefix = prefix
def _image(self, attrs: list[tuple[str, str | None]]) -> None:
"""An `<img>`, in the place it stands.
`attrs` was read by nothing here until 0.10.0, so `src` and `alt` were
both dropped -- a document's figures left no trace in the concept at
all, not even their alt text. The block goes on its own lines because
markdown wants a paragraph and because every boundary grammar in
`propose.py` is line-anchored.
"""
if self._collector is None or self._skip_depth:
return
values = {key: value or "" for key, value in attrs}
source = values.get("src", "").strip()
if not source:
return
self._break()
for line in self._collector.local(source, label=values.get("alt") or None).split("\n"):
self._lines.append(line)
self._break()
def _open(self, tag: str) -> bool:
"""Break for a block or heading tag; report whether it was one."""
level = _HEADING_TAGS.get(tag)
@ -415,10 +617,14 @@ class _HTMLTextExtractor(HTMLParser):
self._current.append(" ")
if tag in _SKIP_TAGS:
self._skip_depth += 1
if tag == "img":
self._image(attrs)
def handle_startendtag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
if not self._open(tag):
self._current.append(" ")
if tag == "img":
self._image(attrs)
def handle_endtag(self, tag: str) -> None:
if tag in _SKIP_TAGS and self._skip_depth > 0:
@ -435,9 +641,9 @@ class _HTMLTextExtractor(HTMLParser):
return "\n".join(self._lines)
def _extract_html(data: bytes) -> str:
def _extract_html(data: bytes, collector: _AssetCollector | None = None) -> str:
"""`html`/`htm`: text via `html.parser`, script/style stripped (spec B3)."""
parser = _HTMLTextExtractor()
parser = _HTMLTextExtractor(collector)
parser.feed(decode_text(data))
parser.close()
return parser.text()
@ -472,8 +678,9 @@ class _XmlTextExtractor:
identical to `"".join(root.itertext())`.
"""
def __init__(self, *, sts: bool) -> None:
def __init__(self, *, sts: bool, collector: _AssetCollector | None = None) -> None:
self._sts = sts
self._collector = collector
self._lines: list[str] = []
self._current: list[str] = []
self._prefix = ""
@ -572,8 +779,42 @@ class _XmlTextExtractor:
self._lines.extend(render_table(rows[0], rows[1:]).rstrip("\n").split("\n"))
return True
def _graphic(self, element: Element) -> bool:
"""A `<graphic>`, in the place it stands. Reports whether it was one.
Measured on the R761 delivery, 2026-09-16: 50 `<graphic>` elements, all
50 direct children of a `<sec>`, none inside a `<table-wrap>`, none
carrying a caption element of any kind -- the "Figur 11.1 ..." line a
human reads is a sibling `<p>` this reader already emits on its own
line. So the label falls back to the file name rather than being
guessed from the neighbourhood.
TWO RESOLUTION ROUTES, and the second is the delivery's own convention:
the href as written, and then `graphics/<name>`, because that publisher
writes a BARE file name and ships the files in a sibling directory.
Both are tried through the caller's resolver, which is what keeps the
containment rule in one place.
"""
if self._collector is None:
return False
href = next(
(value for key, value in element.attrib.items() if _local_name(key) == "href"),
None,
)
if not href:
return False
block = self._collector.local(href, sibling=f"graphics/{Path(href).name}")
self._emit_lines(block.split("\n"))
return True
def _emit_lines(self, lines: list[str]) -> None:
for line in lines:
self._emit(line)
def _walk(self, element: Element, depth: int) -> None:
tag = _local_name(element.tag)
if tag in ("graphic", "inline-graphic") and self._graphic(element):
return
if self._sts and tag == "table-wrap" and self._table(element):
return
skip: set[int] = set()
@ -627,7 +868,9 @@ class _XmlTextExtractor:
return "\n".join(self._lines)
def _xml_document(data: bytes) -> tuple[str, tuple[OutlineMark, ...]]:
def _xml_document(
data: bytes, collector: _AssetCollector | None = None
) -> tuple[str, tuple[OutlineMark, ...]]:
"""`xml`: NISO-STS structure as markdown, any other schema as its text.
A DTD IS REFUSED RATHER THAN PARSED, and that is a guarantee about this
@ -643,7 +886,7 @@ def _xml_document(data: bytes) -> tuple[str, tuple[OutlineMark, ...]]:
holds on every interpreter.
"""
root = _parse_xml(data)
reader = _XmlTextExtractor(sts=_is_sts(root))
reader = _XmlTextExtractor(sts=_is_sts(root), collector=collector)
return reader.text(root), tuple(reader.marks)
@ -736,11 +979,13 @@ def _child_text(element: Element, name: str) -> str | None:
return None
def _extract_xml(data: bytes) -> str:
return _xml_document(data)[0]
def _extract_xml(data: bytes, collector: _AssetCollector | None = None) -> str:
return _xml_document(data, collector)[0]
def xml_outline(name: str, data: bytes) -> tuple[OutlineMark, ...]:
def xml_outline(
name: str, data: bytes, *, assets: bool = False, resolve: Resolver | None = None
) -> tuple[OutlineMark, ...]:
"""`xml`: the sections the document DECLARES, as marks on the extracted text.
The counterpart of `pdf_outline`, and the difference between them is the
@ -762,7 +1007,12 @@ def xml_outline(name: str, data: bytes) -> tuple[OutlineMark, ...]:
list as "leave every rule untouched", never as a route.
"""
del name # the registry decides which reader runs; kept for `pdf_outline`'s shape
return _xml_document(data)[1]
# `assets` and `resolve` are NOT options of this arm, exactly as
# `pdf_headings` is not one of `pdf_outline`'s: carrying an image inserts
# lines into the extracted text, so marks computed with the images off name
# the right sections at the wrong line numbers. They are threaded so both
# sides of the plan can be computed against ONE rendering.
return _xml_document(data, _AssetCollector(resolve) if assets else None)[1]
def _local_name(tag: str) -> str:
@ -966,11 +1216,251 @@ def _mark_headings(lines: list[tuple[str, float, str]], levels: dict[float, int]
_PDF_PAGE_SEPARATOR = "\n\n"
@dataclass(frozen=True)
class _PdfPage:
"""One page's text, and the images drawn on it, kept APART on purpose.
The pointer blocks are appended after the page's own lines by
`_pdf_page_text`, and the body is kept separately because `pdf_outline`
compares the page's line splitting against `page.extract_text_lines()` --
a per-page check that ships and that decides whether the primary bridge
route may be used at all. Appended lines are not in that geometry, so a
joined string would fail the check on every page carrying an image and
silently demote 2 762 bookmarks to the fallback route.
"""
number: int
text: str
images: tuple[ExtractedImage, ...] = ()
rejected: tuple[AssetRejection, ...] = ()
def _pdf_page_text(page: _PdfPage) -> str:
"""A page as it reaches the extracted text: its lines, then its pointers.
END OF PAGE, not the image's y position, and the reason is stated rather
than hidden: a PDF image has a bounding box and no place in the reading
order, so "where it stands" is the page. Inserting by y would reorder the
page's own lines against the geometry `pdf_outline` checks itself against,
and would put a pointer inside a sentence. A caption printed above a figure
therefore keeps its own line where the document put it, and the pointer
follows the page it was drawn on.
"""
blocks = [render_block(image) for image in page.images]
blocks += [
render_missing(rejection.name, reason=rejection.reason) for rejection in page.rejected
]
if not blocks:
return page.text
joined = "\n\n".join(blocks)
return f"{page.text}\n\n{joined}" if page.text else joined
#: The bits-per-component this encoder expresses. A PDF may store 1, 2, 4, 8 or
#: 16, and everything but 8 is REFUSED with a code rather than rescaled --
#: rescaling a 1-bit stencil to 8 bits is a decision about what black means, and
#: a wrong one looks exactly like a right one.
_PDF_SAMPLE_BITS = 8
def _pdf_colour(space: object) -> tuple[int, bytes | None] | None:
"""A PDF colour space as `(channels, palette)`, or `None` if not expressible.
`None` is the honest answer for CMYK, for a separation space and for
anything with a transfer function: converting those needs a colour model
this package does not carry, and a guess would be a picture that is
plausibly the wrong colour. It is counted and stated, never approximated.
"""
from pdfminer.pdftypes import PDFStream, resolve1
space = resolve1(space)
name = getattr(space, "name", None)
if name in ("DeviceGray", "CalGray", "G"):
return 1, None
if name in ("DeviceRGB", "CalRGB", "RGB"):
return 3, None
if not isinstance(space, list) or not space:
return None
head = getattr(resolve1(space[0]), "name", None)
if head == "ICCBased" and len(space) > 1:
profile = resolve1(space[1])
components = resolve1(profile.attrs.get("N")) if isinstance(profile, PDFStream) else None
return (int(components), None) if components in (1, 3) else None
if head in ("CalGray",):
return 1, None
if head in ("CalRGB", "Lab"):
return 3, None
if head in ("Indexed", "I") and len(space) >= 4:
base = _pdf_colour(space[1])
if base is None:
return None
lookup = resolve1(space[3])
if isinstance(lookup, PDFStream):
lookup = lookup.get_data()
if not isinstance(lookup, bytes):
return None
if base[0] == 3:
palette = lookup[: (len(lookup) // 3) * 3]
else:
# PNG's PLTE is RGB triples only, so a grey palette is widened
# rather than refused. Widening a grey to r=g=b is exact, not an
# approximation -- which is why this branch exists and the CMYK one
# does not.
palette = b"".join(bytes([value, value, value]) for value in lookup)
return (1, palette) if palette else None
return None
def _pdf_alpha(attrs: dict[str, object], width: int, height: int) -> bytes | None | bool:
"""A soft mask as one alpha byte per pixel, `None` for none, `False` to refuse.
An `SMask` this encoder cannot express is a REFUSAL rather than a dropped
channel: an image whose transparency is thrown away is composited against
nothing and reads as a black or white rectangle over the page, which is a
picture that is wrong in a way no consumer can detect.
"""
from pdfminer.pdftypes import PDFStream, resolve1
mask = resolve1(attrs.get("SMask"))
if mask is None:
return None
if not isinstance(mask, PDFStream):
return False
shape = mask.attrs
if (
resolve1(shape.get("Width")) != width
or resolve1(shape.get("Height")) != height
or resolve1(shape.get("BitsPerComponent")) != _PDF_SAMPLE_BITS
):
return False
try:
alpha = mask.get_data()
except Exception:
return False
return alpha if len(alpha) >= width * height else False
def _pdf_image(stream: object, name: str) -> ExtractedImage:
"""One image XObject, carried verbatim where it already is a file.
TWO ROUTES, and which one runs is decided by the BYTES rather than by the
filter name. `get_data()` applies every filter pdfminer knows and stops at
the image codecs, so a `DCTDecode` stream comes back as a finished JPEG and
a `FlateDecode` one comes back as raw samples. Sniffing the result is what
makes the first route exact: an embedded JPEG is written to the bundle as
the publisher's own bytes, unre-encoded, and its content-addressed name is
therefore stable for as long as the document is.
Measured on R761 (2026-09-16): 29 of 50 image objects are `DCTDecode` and
take the verbatim route; 21 are `FlateDecode` and are encoded here. Over
the 33-document K2 reference corpus the population is 4 828 objects, and
the filters are mixed enough (`FlateDecode`, `DCTDecode`, `JPXDecode`,
`ASCII85Decode` chains, `CCITTFaxDecode`) that guessing from the filter
name would have been wrong on several hundred.
RENDERING THE PAGE REGION WAS THE ALTERNATIVE AND IT WAS NOT TAKEN. A
rasterised crop would be one code path and would handle every filter, but
its bytes -- and therefore the asset's name and the bundle's digest --
would depend on the version of the rasteriser installed, which is the one
property `OCR_DPI`'s docstring already admits OCR text cannot have. An
embedded stream has no such dependency.
"""
from pdfminer.pdftypes import resolve1
try:
data = stream.get_data() # type: ignore[attr-defined]
except Exception as exc:
raise ExtractionError(
f"the PDF image stream behind {name!r} could not be decoded: {exc}",
code="asset_pdf_unsupported",
) from exc
if data and sniff(data) is not None:
return read_image(data, name=name)
attrs = dict(getattr(stream, "attrs", {}))
width = resolve1(attrs.get("Width"))
height = resolve1(attrs.get("Height"))
bits = resolve1(attrs.get("BitsPerComponent"))
if not isinstance(width, int) or not isinstance(height, int):
raise ExtractionError(
f"the PDF image {name!r} declares no usable size",
code="asset_pdf_unsupported",
)
if resolve1(attrs.get("ImageMask")):
raise ExtractionError(
f"the PDF image {name!r} is a stencil mask, which paints the current "
"fill colour rather than carrying one of its own",
code="asset_pdf_unsupported",
)
if bits != _PDF_SAMPLE_BITS:
raise ExtractionError(
f"the PDF image {name!r} stores {bits}-bit samples; this encoder writes "
f"{_PDF_SAMPLE_BITS}-bit ones and will not rescale, because rescaling a "
"stencil is a decision about what black means",
code="asset_pdf_unsupported",
)
if attrs.get("Decode") is not None:
raise ExtractionError(
f"the PDF image {name!r} carries a Decode array, which remaps every "
"sample; carrying it unmapped would invert the picture",
code="asset_pdf_unsupported",
)
colour = _pdf_colour(attrs.get("ColorSpace"))
if colour is None:
raise ExtractionError(
f"the PDF image {name!r} uses a colour space this encoder does not "
f"express ({attrs.get('ColorSpace')!r})",
code="asset_pdf_unsupported",
)
alpha = _pdf_alpha(attrs, width, height)
if alpha is False:
raise ExtractionError(
f"the PDF image {name!r} has a soft mask this encoder cannot express; "
"dropping transparency would composite the picture against nothing",
code="asset_pdf_unsupported",
)
channels, palette = colour
encoded = encode_png(
width,
height,
data,
channels=channels,
palette=palette,
alpha=alpha if isinstance(alpha, bytes) else None,
)
return read_image(encoded, name=name)
def _pdf_images(page: object) -> tuple[tuple[ExtractedImage, ...], tuple[AssetRejection, ...]]:
"""Every image drawn on one page, with the failures kept beside them."""
carried: list[ExtractedImage] = []
rejected: list[AssetRejection] = []
number = getattr(page, "page_number", 0)
for index, drawn in enumerate(getattr(page, "images", []) or [], start=1):
# The name a PDF image does NOT have. An XObject is reached through a
# resource name local to one page's dictionary, so it is not an
# identifier -- the page number in front of it is what makes the string
# readable, and the content-addressed digest is what makes it unique.
label = str(drawn.get("name") or index).lstrip("/")
name = f"page-{number}-{label}"
stream = drawn.get("stream")
if stream is None:
rejected.append(
AssetRejection(name, "asset_pdf_unsupported", "the image object has no stream")
)
continue
try:
carried.append(_pdf_image(stream, name))
except ExtractionError as exc:
rejected.append(AssetRejection(name, exc.code, str(exc)))
return tuple(carried), tuple(rejected)
@functools.lru_cache(maxsize=1)
def _pdf_pages(
data: bytes, headings: bool = False, ocr: bool = False
) -> tuple[tuple[int, str], ...]:
"""Every page that produced text, as `(page number, text)`, in page order.
data: bytes, headings: bool = False, ocr: bool = False, assets: bool = False
) -> tuple[_PdfPage, ...]:
"""Every page that produced content, as a `_PdfPage`, in page order.
The page NUMBER is 1-based and comes from the document, so a page that
yielded nothing removes itself from the sequence without renumbering the
@ -991,6 +1481,13 @@ def _pdf_pages(
from its lines. Measured, the two agree on 11 of 11 pages of a real tender
PDF -- but "agree on the document I tried" is not a contract, so the
default does not depend on it.
`assets` is the third, and with it off not one line below it runs: no
stream is decoded, no sample buffer is allocated, and the emitted pages are
the objects they always were. A page that produced no TEXT is still
dropped even when it carries an image, because `_extract_pdf` refuses a
document with no text at all (`extractor_empty_pdf`) and an image-only
document is `--ocr`'s question, not this one's.
"""
try:
import pdfplumber
@ -1005,8 +1502,12 @@ def _pdf_pages(
# cannot supply it. A title page is 100 % heading, and a per-page
# median would compare it with itself and mark nothing.
recovered: list[str | list[tuple[str, float, str]]] = []
numbers: list[int] = []
drawn: list[tuple[tuple[ExtractedImage, ...], tuple[AssetRejection, ...]]] = []
for page in pdf.pages:
flat = (page.extract_text() or "").rstrip()
numbers.append(page.page_number)
drawn.append(_pdf_images(page) if assets else ((), ()))
if read is not None and needs_ocr(flat):
# The page's own text is unusable, so it is replaced
# WHOLESALE rather than merged with: a page of `(cid:N)`
@ -1047,7 +1548,11 @@ def _pdf_pages(
raise ExtractionError(
f"the PDF parser failed on this file: {exc}", code="extractor_pdf_error"
) from exc
return tuple((number, page) for number, page in enumerate(pages, start=1) if page)
return tuple(
_PdfPage(number=number, text=text, images=images, rejected=rejected)
for number, text, (images, rejected) in zip(numbers, pages, drawn)
if text
)
@dataclass(frozen=True)
@ -1138,7 +1643,12 @@ def _outline_page_and_top(doc: object, dest: object, action: object) -> tuple[ob
def pdf_outline(
name: str, data: bytes, *, pdf_headings: bool = False, ocr: bool = False
name: str,
data: bytes,
*,
pdf_headings: bool = False,
ocr: bool = False,
assets: bool = False,
) -> PdfOutline:
"""`pdf`: the file's own `/Outlines` tree, as marks on the extracted text.
@ -1177,14 +1687,21 @@ def pdf_outline(
from pdfminer.pdfdocument import PDFNoOutlines
from pdfminer.pdfpage import PDFPage
rendered = _pdf_pages(data, pdf_headings, ocr)
rendered = _pdf_pages(data, pdf_headings, ocr, assets)
starts: dict[int, int] = {}
page_lines: dict[int, list[str]] = {}
# The page's OWN lines, without the pointer blocks appended after them.
# `starts` has to count the appended lines (they are in the text a mark
# indexes) while the geometry check must not see them, because
# `extract_text_lines` reports the page and knows nothing about what this
# package added underneath it.
body_lines: dict[int, list[str]] = {}
offset = 0
for number, page_text in rendered:
starts[number] = offset
page_lines[number] = page_text.split("\n")
offset += len(page_lines[number]) + 1
for rendered_page in rendered:
starts[rendered_page.number] = offset
page_lines[rendered_page.number] = _pdf_page_text(rendered_page).split("\n")
body_lines[rendered_page.number] = rendered_page.text.split("\n")
offset += len(page_lines[rendered_page.number]) + 1
unresolved = 0
collided = 0
@ -1225,7 +1742,7 @@ def pdf_outline(
lines = page_lines[number]
tops: list[float] | None = None
geometry = page.extract_text_lines()
if [str(entry["text"]) for entry in geometry] == lines:
if [str(entry["text"]) for entry in geometry] == body_lines[number]:
tops = [float(entry["top"]) for entry in geometry]
height = float(page.height)
for level, title, top in group:
@ -1268,7 +1785,9 @@ def _normalise_outline(value: str) -> str:
return re.sub(r"\s+", "", value).lower()
def _extract_pdf(data: bytes, *, headings: bool = False, ocr: bool = False) -> str:
def _extract_pdf(
data: bytes, *, headings: bool = False, ocr: bool = False, assets: bool = False
) -> str:
"""`pdf`: page text via `pdfplumber`, in page order, pages separated by a
blank line.
@ -1284,8 +1803,8 @@ def _extract_pdf(data: bytes, *, headings: bool = False, ocr: bool = False) -> s
and pymupdf each emit all labels then all values. Re-pairing those is
guesswork, and in a requirements document a wrong pairing looks right.
"""
pages = _pdf_pages(data, headings, ocr)
text = _PDF_PAGE_SEPARATOR.join(page for _, page in pages)
pages = _pdf_pages(data, headings, ocr, assets)
text = _PDF_PAGE_SEPARATOR.join(_pdf_page_text(page) for page in pages)
if not text:
raise ExtractionError(
"the PDF yielded no text on any page; a scanned or image-only "
@ -1332,7 +1851,94 @@ def _convert_bytes(source: bytes, to: str, format: str, extra_args: Sequence[str
)
def _extract_office(suffix: str, data: bytes) -> str:
#: A markdown image as the converter's own writer emits it. Measured against
#: pandoc 3.10.2 on hand-laid fixtures: a `.docx` picture arrives as
#: `![<descr>](<path>){width="..." height="..."}` and a `.pptx` one as
#: `![<descr>](<path> "<name>")`, so the title form and the attribute form are
#: both real and a regex written for one of them silently leaves the other's
#: link in the text.
_MEDIA_LINK = re.compile(
r"!\[(?P<alt>[^\]]*)\]\("
r"(?:<(?P<angle>[^>]*)>|(?P<plain>[^)\s]*))"
r'(?:\s+"(?P<title>[^"]*)")?\)'
r"(?P<attrs>\{[^}]*\})?"
)
def _convert_with_media(
source: bytes, to: str, format: str, extra_args: Sequence[str]
) -> tuple[str, dict[str, bytes]]:
"""The converter call again, with `--extract-media` and the files read back.
A separate function rather than a flag on `_convert_bytes` because the
media must be READ INSIDE the temporary directory's lifetime: the directory
is removed on every path, and a caller handed a rewritten markdown string
pointing into it would hold links to files that no longer exist. Returning
the bytes is what makes the seam closed.
The staging path is absolute, so the converter writes absolute links. That
is deliberate: every one of them is replaced below, and a link that somehow
survived would carry a temporary directory name into a concept -- a string
that differs on every run, which a byte-determinism rule would catch loudly
rather than never.
"""
import pypandoc
from ._pandoc import converter_path
with tempfile.TemporaryDirectory() as staging:
staged = Path(staging) / f"input.{format}"
staged.write_bytes(source)
media_root = Path(staging) / "extracted"
with converter_path():
text = str(
pypandoc.convert_file(
str(staged),
to,
format=format,
extra_args=[*extra_args, f"--extract-media={media_root}"],
)
)
media: dict[str, bytes] = {}
if media_root.is_dir():
for path in sorted(media_root.rglob("*")):
if path.is_file():
media[str(path)] = path.read_bytes()
return text, media
def _rewrite_media_links(text: str, media: dict[str, bytes], collector: _AssetCollector) -> str:
"""Every converter image link, replaced by this package's own pointer block.
UNCONDITIONAL, including the links that cannot be resolved. The converter
already emitted a markdown image before this existed -- measured on a
hand-laid `.docx`, today's output carries
`![Tabell 84-2](media/tabell-84-2.png)` with no such file anywhere, which
`structure._scan_references` reads as a cross-reference to a concept that
cannot exist. Leaving an unresolvable link in place would keep that defect
and add a temporary directory name to it.
"""
def replace(match: re.Match[str]) -> str:
target = match.group("angle") or match.group("plain") or ""
label = match.group("alt") or match.group("title") or None
data = media.get(target)
if data is not None:
# The name the CONTAINER gave it, not the staging path: pandoc
# preserves the part name under its own media directory, so
# `word/media/tabell-84-2.png` arrives as `media/tabell-84-2.png`.
inside = target.split("/extracted/", 1)[-1]
return collector.carry(data, name=inside, label=label)
if not target:
return collector.reject(
"image", code="asset_unresolved", reason="the converter emitted no target"
)
return collector.local(target, label=label)
return _MEDIA_LINK.sub(replace, text)
def _extract_office(suffix: str, data: bytes, collector: _AssetCollector | None = None) -> str:
"""The five office rows, converted through the vendored binary.
Shaped after `_extract_pdf`: the gate is an import probe rather than a
@ -1349,7 +1955,11 @@ def _extract_office(suffix: str, data: bytes) -> str:
writer = _SPREADSHEET_WRITER if spreadsheet else _PANDOC_WRITER
args = _SPREADSHEET_ARGS if spreadsheet else _PANDOC_ARGS
try:
text = _convert_bytes(data, writer, _PANDOC_FORMATS[suffix], args)
if collector is None:
text = _convert_bytes(data, writer, _PANDOC_FORMATS[suffix], args)
else:
text, media = _convert_with_media(data, writer, _PANDOC_FORMATS[suffix], args)
text = _rewrite_media_links(text, media, collector)
except ExtractionError:
raise
except Exception as exc:
@ -1594,14 +2204,17 @@ def _line_units(text: str) -> SourceUnits:
return SourceUnits("lines", tuple(starts), tuple(range(1, len(starts) + 1)))
def _pdf_units(data: bytes, headings: bool, ocr: bool) -> SourceUnits:
def _pdf_units(data: bytes, headings: bool, ocr: bool, assets: bool = False) -> SourceUnits:
starts: list[int] = []
numbers: list[int] = []
offset = 0
for number, page in _pdf_pages(data, headings, ocr):
for page in _pdf_pages(data, headings, ocr, assets):
starts.append(offset)
numbers.append(number)
offset += len(page) + len(_PDF_PAGE_SEPARATOR)
numbers.append(page.number)
# The page as it reaches the text, pointers included: a locator built
# from the body alone would drift by two lines per carried image and
# would name the wrong page from the first one onwards.
offset += len(_pdf_page_text(page)) + len(_PDF_PAGE_SEPARATOR)
return SourceUnits("pages", tuple(starts), tuple(numbers))
@ -1646,7 +2259,13 @@ def _spreadsheet_units(text: str) -> SourceUnits | None:
def source_units(
filename: str, data: bytes, text: str, *, pdf_headings: bool = False, ocr: bool = False
filename: str,
data: bytes,
text: str,
*,
pdf_headings: bool = False,
ocr: bool = False,
assets: bool = False,
) -> SourceUnits | None:
"""The unit table for one dropped file, or `None` when it has none.
@ -1660,7 +2279,7 @@ def source_units(
"""
suffix = Path(filename).suffix.lower()
if suffix == ".pdf":
return _pdf_units(data, pdf_headings, ocr)
return _pdf_units(data, pdf_headings, ocr, assets)
if suffix == ".xlsx":
return _spreadsheet_units(text)
if suffix in _CORE_EXTRACTORS or suffix in _PANDOC_FORMATS:
@ -1675,6 +2294,8 @@ def extract_text(
renderer: Callable[[str], str] | None = None,
pdf_headings: bool = False,
ocr: bool = False,
assets: bool = False,
resolve: Resolver | None = None,
) -> str:
"""Convert one dropped file's bytes to OKF concept text, dispatched by type.
@ -1704,17 +2325,87 @@ def extract_text(
passing either argument gets today's behaviour, silently, which is correct
-- the options describe a reader, not a policy for the run.
"""
return extract_document(
filename,
data,
renderer=renderer,
pdf_headings=pdf_headings,
ocr=ocr,
assets=assets,
resolve=resolve,
).text
#: The types that can carry an image, and the reader that places it. Kept apart
#: from `_CORE_EXTRACTORS` and `_OPTIONAL_EXTRACTORS` rather than folded into
#: them, and that separation is the byte-identity guarantee: with `assets=False`
#: not one entry below is consulted and the dispatch is the one every golden,
#: every pinned bundle and every published digest was measured on. `.csv`,
#: `.json`, `.md` and `.txt` are absent because the formats carry no image;
#: `.xlsx` is absent because its converter writes one pipe table per sheet and
#: a two-line block inside one would break the row locator `_spreadsheet_units`
#: reads back out of it -- measured 2026-09-16, 0 of 4 K2 workbooks hold any
#: media at all, so the row is a limit stated rather than a loss taken.
_ASSET_READERS: dict[str, Callable[[bytes, _AssetCollector], str]] = {
".html": _extract_html,
".htm": _extract_html,
".xml": _extract_xml,
**{
suffix: functools.partial(_extract_office, suffix)
for suffix in _PANDOC_FORMATS
if suffix != ".xlsx"
},
}
def extract_document(
filename: str,
data: bytes,
*,
renderer: Callable[[str], str] | None = None,
pdf_headings: bool = False,
ocr: bool = False,
assets: bool = False,
resolve: Resolver | None = None,
) -> ExtractedDocument:
"""One dropped file as text PLUS the images that stand inside that text.
The entry point :func:`extract_text` keeps for the eight callers that want
a string, and the one Door B uses since 0.10.0. With `assets=False` -- the
default, everywhere -- this runs exactly the dispatch that existed before
the asset layer did, and returns an :class:`ExtractedDocument` whose text is
byte-identical and whose two image tuples are empty.
`resolve` answers for the formats that POINT at a file instead of embedding
it (`html`, `xml`). Without one every pointer resolves to nothing and is
stated as such; with one, containment is that resolver's rule and not this
module's. `pdf` and the office rows embed their images and never consult it.
"""
suffix = Path(filename).suffix.lower()
extractor = _CORE_EXTRACTORS.get(suffix) or _OPTIONAL_EXTRACTORS.get(suffix)
if extractor is not None:
if suffix == ".pdf" and (pdf_headings or ocr):
text = _extract_pdf(data, headings=pdf_headings, ocr=ocr)
if extractor is None:
if suffix in _UNPARSED_OPTIONAL_EXTENSIONS:
raise _extra_missing(suffix)
raise ExtractionError(
f"no extractor is registered for file extension {suffix!r} ({filename!r})",
code="extractor_unknown",
)
collector = _AssetCollector(resolve) if assets else None
if suffix == ".pdf":
if pdf_headings or ocr or assets:
text = _extract_pdf(data, headings=pdf_headings, ocr=ocr, assets=assets)
else:
text = extractor(data)
return renderer(text) if renderer is not None else text
if suffix in _UNPARSED_OPTIONAL_EXTENSIONS:
raise _extra_missing(suffix)
raise ExtractionError(
f"no extractor is registered for file extension {suffix!r} ({filename!r})",
code="extractor_unknown",
if collector is not None:
for page in _pdf_pages(data, pdf_headings, ocr, True):
collector.images.extend(page.images)
collector.rejected.extend(page.rejected)
elif collector is not None and suffix in _ASSET_READERS:
text = _ASSET_READERS[suffix](data, collector)
else:
text = extractor(data)
return ExtractedDocument(
text=renderer(text) if renderer is not None else text,
images=tuple(collector.images) if collector is not None else (),
rejected=tuple(collector.rejected) if collector is not None else (),
)

59
src/llm_ingestion_okf/importer.py
View file

@ -27,11 +27,13 @@ obeys the verdict it returns.
from __future__ import annotations
from collections.abc import Mapping
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Protocol
from .assets import ASSETS_DIR, IMAGE_POINTER
from .connectors import safe_resolve
from .errors import IngestError, MaterializationError, SourceError
from .extract import decode_text
from .materialize import (
@ -355,6 +357,48 @@ def _read_bundle(
return documents, failed
def _carry_assets(source: Path, bundle: Path, merged: Sequence[MergedConcept]) -> None:
"""Copy each asset a merged concept points at, by content identity.
The sender's bytes verbatim, exactly as the concept beside them: this door
may not rewrite a merged concept, so it may not rewrite the pointer either,
and the file therefore has to land under the name the pointer already
names. An occupied name is re-used only when the bytes there are already
identical -- Door C's ownership rule, and here the name carries the digest
of those bytes, so a mismatch is a `sha256[:12]` collision and is refused
rather than resolved.
A pointer whose asset the sender did not ship is left alone. SPEC SS 6.1
requires a consumer to tolerate a broken link, and a pointer recording that
the source had a figure nobody holds is information, not corruption.
"""
for entry in merged:
try:
text = entry.path.read_text(encoding="utf-8")
except OSError:
continue
for match in IMAGE_POINTER.finditer(text):
name = match.group("asset")
try:
origin = safe_resolve(source / ASSETS_DIR, name)
target = safe_resolve(bundle / ASSETS_DIR, name)
except SourceError:
continue
if not origin.is_file():
continue
data = origin.read_bytes()
if target.exists():
if target.read_bytes() != data:
raise MaterializationError(
f"the asset {name!r} already exists here with different bytes; "
"refusing to overwrite content this import did not write",
code="asset_collision",
)
continue
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(data)
def import_bundle(
source_dir: Path,
bundle_dir: Path,
@ -544,6 +588,19 @@ def import_bundle(
MergedConcept(concept_path=concept_path, path=path, reasons=verdict.reasons)
)
# THE ASSETS THE MERGED CONCEPTS POINT AT (0.10.0), read off the files
# that actually landed. Measured before this existed: a bundle built
# with `--assets` imported as 6 of 6 concepts and NO `assets/`
# directory, so every pointer in the imported bundle named a file that
# was not there -- the same "complete and not" defect the capability
# exists to close, one door over.
#
# POINTED AT, never "every file in the sender's assets/". An asset
# belonging to a concept the gate refused must not ride in on the back
# of one it cleared, and an asset nothing names is a file no retirement
# pass ever reaches.
_carry_assets(source, bundle, merged)
# §10 pointers, surfaced over what actually landed. Read AFTER the merge
# decision and never before it: this door's tolerance is structural —
# it writes the sender's bytes verbatim and judges no shape — and a

118
src/llm_ingestion_okf/inbox.py
View file

@ -27,8 +27,23 @@ from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass, replace
from pathlib import Path, PurePosixPath
from .assets import (
ASSETS_DIR,
IMAGE_POINTER,
AssetRejection,
ExtractedImage,
asset_name,
)
from .connectors import safe_resolve
from .errors import IngestError, MaterializationError, SegmentationError, SourceError
from .extract import DeclaredIdentity, SourceUnits, declared_identity, extract_text, source_units
from .extract import (
DeclaredIdentity,
SourceUnits,
declared_identity,
directory_resolver,
extract_document,
source_units,
)
from .materialize import (
_render_root_frontmatter,
check_filename_length,
@ -39,6 +54,7 @@ from .materialize import (
write_bytes,
)
from .profiles import (
ASSET_COUNT_KEY,
DEFAULT,
BundleProfile,
IndexEntry,
@ -210,6 +226,19 @@ def render_inbox_concept(
code="inbox_source_file_invalid",
)
# HOW MANY ASSET POINTERS THIS CONCEPT CARRIES, counted out of the concept's
# own text rather than threaded down from the extraction. Two reasons, and
# the second is the load-bearing one: a segmented document's images belong
# to the segments whose spans hold them, so a document-level total would be
# written onto every segment and be wrong on all but one of them; and a
# count a reader can verify from the file in front of them is a different
# kind of fact from a count only the producer could have known.
#
# Named on this repository's own profiles only, so a bundle written under
# `DEFAULT` or `STRICT_V1` keeps exactly the key set its contract names.
assets_carried = (
len(IMAGE_POINTER.findall(text)) if ASSET_COUNT_KEY in profile.frontmatter.order else 0
)
frontmatter = {
"type": okf_type,
"title": title,
@ -287,6 +316,11 @@ def render_inbox_concept(
title=source_title,
)
)
if assets_carried:
# Conditional, like `req_number`: absent is "this concept carries no
# image", which is what every bundle written before 0.10.0 says, so a
# corpus without pictures is byte-identical across the move.
frontmatter[ASSET_COUNT_KEY] = str(assets_carried)
if concept_frontmatter_values:
# LAST, and the position is the precedence: a value the caller states
# for the run beats what the document declares, which beats the file
@ -592,6 +626,13 @@ class InboxResult:
# consumer's four buckets keep their meaning: a skipped directory holds no
# dropped FILE outcome, it explains a set of files that were never dropped.
skipped: tuple[SkippedPath, ...] = ()
# THE ASSET DENOMINATOR (0.10.0). `assets` is what reached the bundle;
# `assets_rejected` is what was found and could not be. Both, or neither
# number means anything: "51 carried" is a measurement only beside "of 53
# found", and a run whose figures were all refused would otherwise look
# exactly like a run over documents that had none.
assets: tuple[str, ...] = ()
assets_rejected: tuple[AssetRejection, ...] = ()
def relative_source(path: Path, inbox: Path) -> str:
@ -970,6 +1011,7 @@ def process_inbox(
pdf_headings: bool = False,
heading_reserve: Callable[[str], bool] | None = None,
ocr: bool = False,
assets: bool = False,
concept_frontmatter_values: Mapping[str, str] | None = None,
) -> InboxResult:
"""Convert every file dropped in `inbox_dir` into an OKF concept.
@ -1037,6 +1079,14 @@ def process_inbox(
quarantined: list[BlockedFile] = []
rejected: list[BlockedFile] = []
failed: list[FailedFile] = []
# Keyed by asset name, so one image dropped by two documents is one entry
# and the bundle holds one file. The bytes are kept until the write, which
# happens per document AFTER that document's gate decision -- an image
# belonging to a document the guard refused must not be left behind in
# `assets/`, where nothing would ever point at it and nothing would ever
# retire it.
carried_assets: dict[str, bytes] = {}
refused_assets: list[AssetRejection] = []
# Phase 1: name every file BEFORE any gate call or write, so an intra-run
# collision is caught while both files can still be refused together. Under
@ -1184,13 +1234,23 @@ def process_inbox(
continue
outputs: list[tuple[str, str, tuple[str, ...]]] = []
try:
text = extract_text(
# The resolver is rooted at the DOCUMENT's own directory, which is
# the same root `propose.propose_segments` computes from the file it
# reads off disk. One root both sides derive independently is what
# makes the two renderings identical -- and a plan indexes the exact
# string it was proposed against, so a resolver that disagreed would
# turn every document carrying a pointer into a coded rejection.
resolve = directory_resolver(path.parent) if assets else None
document = extract_document(
source_name(path),
source_bytes,
renderer=_resolve_renderer(profile, path.name),
pdf_headings=pdf_headings,
ocr=ocr,
assets=assets,
resolve=resolve,
)
text = document.text
# The heading RESERVE, supplied as a predicate rather than decided
# here: the condition is the proposer's outline grammar, and the
# door does not own that grammar. A callable keeps the dependency
@ -1201,13 +1261,16 @@ def process_inbox(
reading_fonts = pdf_headings
if heading_reserve is not None and not pdf_headings and heading_reserve(text):
reading_fonts = True
text = extract_text(
document = extract_document(
source_name(path),
source_bytes,
renderer=_resolve_renderer(profile, path.name),
pdf_headings=True,
ocr=ocr,
assets=assets,
resolve=resolve,
)
text = document.text
# Computed from the SAME text the plan's offsets index, so the
# locator and the offset can never disagree about which rendering
# they describe. `None` when the profile names no provenance:
@ -1220,6 +1283,7 @@ def process_inbox(
text,
pdf_headings=reading_fonts,
ocr=ocr,
assets=assets,
)
if profile.provenance is not None
else None
@ -1321,6 +1385,12 @@ def process_inbox(
continue
bundle.mkdir(parents=True, exist_ok=True)
if outputs and (document.images or document.rejected):
# AFTER the gate, and only where the document actually produced
# concepts. An asset written for a refused document would be an
# orphan no pointer names and no retirement pass reaches.
_write_assets(bundle, document.images, carried_assets)
refused_assets.extend(document.rejected)
for target_name, content, reasons in outputs:
# `write_bytes` resolves a subpath through `safe_resolve` but never
# creates one. Without this the very first hierarchical write fails.
@ -1377,6 +1447,8 @@ def process_inbox(
failed=tuple(sorted(failed, key=lambda entry: entry.source_file)),
concepts=tuple(concepts),
skipped=skipped,
assets=tuple(sorted(carried_assets)),
assets_rejected=tuple(refused_assets),
)
@ -1397,6 +1469,46 @@ ADJUDICATION_STATES = (ADJUDICATION_PROPOSED, ADJUDICATION_ADJUDICATED)
ADJUDICATION_COMPANION_KEYS = ("adjudicated_by", "adjudicated_at", "adjudication_dwell_s")
def _write_assets(bundle: Path, images: Sequence[ExtractedImage], seen: dict[str, bytes]) -> None:
"""Put one document's images in the bundle's `assets/` directory.
OWNERSHIP IS PROVEN BY CONTENT IDENTITY, which is Door C's rule reused
verbatim: an occupied name is re-used only when the bytes there are already
identical, and never overwritten otherwise. Here the name carries the
digest of those very bytes, so an occupied name with different contents is
a `sha256[:12]` collision -- refused loudly rather than resolved silently,
because silently resolving it would mean one of two pictures is lost and
every pointer to it shows the other.
A binary write, and the only one in this package. `materialize.write_bytes`
takes `content: str` and encodes UTF-8, which is correct for every text
guarantee it holds and cannot carry a JPEG.
"""
directory = bundle / ASSETS_DIR
for image in images:
name = asset_name(image)
known = seen.get(name)
if known is not None:
if known != image.data:
raise MaterializationError(
f"two different images reduce to the asset name {name!r} in one run; "
"refusing to overwrite the first, because every pointer to it would "
"then show the second",
code="asset_collision",
)
continue
directory.mkdir(parents=True, exist_ok=True)
target = safe_resolve(directory, name)
if target.exists() and target.read_bytes() != image.data:
raise MaterializationError(
f"the asset {name!r} already exists in the bundle with different bytes; "
"refusing to overwrite content this run did not write",
code="asset_collision",
)
target.write_bytes(image.data)
seen[name] = image.data
def _validate_facets(structure: DocumentStructure, profile: BundleProfile) -> None:
"""Refuse a document whose values cannot be rendered as index facets.

24
src/llm_ingestion_okf/profiles.py
View file

@ -1414,10 +1414,23 @@ _STRUCTURE_KEYS = (
# 2026-08-26 arm that lost on trap exposure was 6 031 characters against
# 21 879 for the head-scan it lost to - the headroom for carrying the metadata
# back into the index is most of that gap.
#: How many binary assets a concept carries (0.10.0). NOT a facet: it is a fact
#: about the extraction, not about the document's own structure, and projecting
#: it into the index would put a count where a consumer expects a subject.
#:
#: Named on the profiles this repository owns and on no others. `DEFAULT` states
#: commons' ingest-spec SS 5 layer and `STRICT_V1` the wiki's ratified contract,
#: so naming it in either from here is this repository editing someone else's
#: contract (O2) -- the same reason `sources` sits on the segmented v0.2 profile
#: alone. Under those profiles Door B still CARRIES the images and still writes
#: the pointer; only the count is absent, which is a smaller loss than a key
#: appearing in a contract its owner never agreed to.
ASSET_COUNT_KEY = "images"
STRUCTURED_V1 = BundleProfile(
types=DEFAULT.types,
frontmatter=FrontmatterSchema(
order=(*DEFAULT.frontmatter.order, *_STRUCTURE_KEYS),
order=(*DEFAULT.frontmatter.order, *_STRUCTURE_KEYS, ASSET_COUNT_KEY),
collapsed_keys=DEFAULT.frontmatter.collapsed_keys,
),
paths=DEFAULT.paths,
@ -1610,7 +1623,14 @@ assert _SEGMENTED_FACETS is not None
# release.
SEGMENTED_OKF_V0_2 = BundleProfile(
types=OKF_V0_2.types,
frontmatter=OKF_V0_2.frontmatter,
# OKF_V0_2's schema plus the asset count, and the key is appended HERE for
# the same reason `sources` is: `OKF_V0_2` is Door A's profile, where no
# dropped binary document is ever read, and widening it would name a key in
# a contract that has no producer for it.
frontmatter=replace(
OKF_V0_2.frontmatter,
order=(*OKF_V0_2.frontmatter.order, ASSET_COUNT_KEY),
),
paths=SEGMENTED_V1.paths,
# The facet tuple is EXTENDED here rather than shared, and that is the
# discriminator doing its job: `SEGMENTED_V1.index.facets` is one object

46
src/llm_ingestion_okf/propose.py
View file

@ -62,7 +62,14 @@ from pathlib import Path
from typing import Any
from .errors import IngestError
from .extract import OutlineMark, extract_text, strip_converter_attribute, xml_outline
from .extract import (
OutlineMark,
Resolver,
directory_resolver,
extract_text,
strip_converter_attribute,
xml_outline,
)
from .extract import pdf_outline as extract_pdf_outline
from .materialize import reduce_to_id_grammar
from .segmentation import heading_only, observed_extractor_version
@ -1613,6 +1620,8 @@ def build_plan(
bold_title: bool = False,
outline_marks: Sequence[OutlineMark] | None = None,
shell_parent: bool = False,
assets: bool = False,
resolve: Resolver | None = None,
) -> dict[str, Any]:
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set()
@ -1626,7 +1635,12 @@ def build_plan(
# is the document.
outline_rule = RULE_PDF_OUTLINE
if extractor_id in DECLARED_STRUCTURE_IDS:
outline_marks = xml_outline(source.name, source_bytes)
# The SAME rendering the text was extracted under. A mark is a line
# index into that exact string, and carrying an image inserts lines:
# marks read with the images off would name the right sections at the
# wrong line numbers, silently, on every document with a figure above a
# heading.
outline_marks = xml_outline(source.name, source_bytes, assets=assets, resolve=resolve)
outline_rule = RULE_XML_SECTION
entries: list[dict[str, Any]] = []
candidates = find_candidates(
@ -1755,6 +1769,7 @@ def run(
ocr: bool = False,
pdf_outline: bool = False,
shell_parent: bool = False,
assets: bool = False,
) -> int:
if max_segment_chars < 0:
raise ProposerError(
@ -1798,7 +1813,19 @@ def run(
# exact string -- a plan proposed against one rendering and replayed
# against another is refused by `assert_plan_applies`, which is the
# right outcome and a confusing one to debug.
text = extract_text(source.name, source_bytes, pdf_headings=pdf_headings, ocr=ocr)
# Rooted at the document's own directory, which is what Door B derives
# for the same file. The two sides never exchange the root; they compute
# it, so a plan and the run that replays it read the same bytes for
# every `<img src>` and every `<graphic xlink:href>`.
resolve = directory_resolver(source.parent) if assets else None
text = extract_text(
source.name,
source_bytes,
pdf_headings=pdf_headings,
ocr=ocr,
assets=assets,
resolve=resolve,
)
reading_fonts = pdf_headings
# The reserve, and the reason it re-extracts rather than post-processes:
# the font reader works on the PDF's glyph geometry, which the joined
@ -1807,7 +1834,14 @@ def run(
# never a pair to combine.
if pdf_headings_reserve and not pdf_headings:
if heading_reserve_applies(text, outline_run=outline_run):
text = extract_text(source.name, source_bytes, pdf_headings=True, ocr=ocr)
text = extract_text(
source.name,
source_bytes,
pdf_headings=True,
ocr=ocr,
assets=assets,
resolve=resolve,
)
reading_fonts = True
# LAST, and against the text that is final: a plan indexes one exact
# string, so marks bridged onto the pre-reserve rendering would name
@ -1815,7 +1849,7 @@ def run(
# reserve may have changed, and the marks follow it.
marks = (
extract_pdf_outline(
source.name, source_bytes, pdf_headings=reading_fonts, ocr=ocr
source.name, source_bytes, pdf_headings=reading_fonts, ocr=ocr, assets=assets
).marks
if pdf_outline
else ()
@ -1844,6 +1878,8 @@ def run(
bold_title=bold_title,
outline_marks=marks,
shell_parent=shell_parent,
assets=assets,
resolve=resolve,
)
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
# plan cannot be replayed -- `process_inbox` refuses one, because a plan

11
src/llm_ingestion_okf/structure.py
View file

@ -235,6 +235,17 @@ def _scan_references(body: str, offset: int, own_number: str | None) -> tuple[st
start, end = match.span(1)
for position in range(start, end):
masked[position] = " "
# AN IMAGE IS NOT A CROSS-REFERENCE. `![alt](target)` embeds a resource
# where a link points at a subject, and `_LINK` cannot tell them apart
# because it never looks at the character in front of the bracket. Left
# unhandled, every asset pointer written since 0.10.0 would arrive in
# the index as an edge to a concept that cannot exist -- the same shape
# as K3-21's `Enclosing section:` line becoming a second, unresolved
# edge. Masked and then skipped, never merely skipped: the file name
# carries digits (`tabell-84-2.png`) that the number scan below would
# otherwise read as a document this one refers to.
if match.start() > 0 and body[match.start() - 1] == "!":
continue
# A fragment-only target points inside THIS document, and a target
# carrying a brace is a template placeholder from prose ABOUT links
# (`reduce_to_id_grammar` cannot emit a brace). Neither can ever

BIN
tests/fixtures/image-inbox/graphics/figur-84-1.png vendored Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 94 B

BIN
tests/fixtures/image-inbox/graphics/tabell-84-2.png vendored Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 122 B

BIN
tests/fixtures/image-inbox/prosess-84-notat.docx vendored Normal file
View file

Binary file not shown.

BIN
tests/fixtures/image-inbox/prosess-84-presentasjon.pptx vendored Normal file
View file

Binary file not shown.

18
tests/fixtures/image-inbox/prosess-84-sts.xml vendored Normal file
View file

@ -0,0 +1,18 @@
<?xml version="1.0" encoding="UTF-8"?>
<standard xmlns:xlink="http://www.w3.org/1999/xlink">
<front><std-ident><doc-number>R761</doc-number></std-ident></front>
<body>
<sec>
<label>84</label>
<title>Konstruksjoner av betong</title>
<sec>
<label>84.1</label>
<title>Toleranser</title>
<p>Toleranseklasse er gitt i tabell 84-2.</p>
<graphic xlink:href="graphics/tabell-84-2.png"/>
<p>Figur 84-1 viser prinsippet.</p>
<graphic xlink:href="figur-84-1.png"/>
</sec>
</sec>
</body>
</standard>

BIN
tests/fixtures/image-inbox/prosess-84-tabell.pdf vendored Normal file
View file

Binary file not shown.

14
tests/fixtures/image-inbox/prosess-84-web.html vendored Normal file
View file

@ -0,0 +1,14 @@
<!doctype html>
<html><head><title>Prosess 84</title></head>
<body>
<h1>84 Konstruksjoner av betong</h1>
<p>Toleranseklasse for de enkelte konstruksjonsdeler er gitt i tabell 84-2.</p>
<img src="graphics/tabell-84-2.png" alt="Tabell 84-2 Toleranseklasser">
<p>Figuren under viser prinsippet.</p>
<figure>
<img src="graphics/figur-84-1.png" alt="Figur 84-1 Prinsippskisse">
<figcaption>Figur 84-1 Prinsippskisse</figcaption>
</figure>
<p>Og en som ligger et annet sted:</p>
<img src="https://example.invalid/ekstern.png" alt="Ekstern figur">
</body></html>

327
tests/fixtures/make_image_fixtures.py vendored Normal file
View file

@ -0,0 +1,327 @@
"""Regenerate the image-bearing fixtures for the asset path (0.10.0).
Five documents, one per reader, each carrying a KNOWN number of images so a
gate can state "carried N of M" with M read out of the source rather than out
of this package. Written in a second file rather than appended to
`make_fixtures.py` for one reason: every fixture that file emits is byte-pinned
by a test, and the object numbering of the PDF builders is part of those bytes.
Adding an XObject to a shared builder would regenerate files whose whole value
is that they have not moved.
The same policy holds here as there: no generator library. The PNG is written
out with `zlib` from the stdlib, the JPEG as a header sequence (the readers
copy JPEG bytes through and read nothing but the frame marker, so a decodable
photograph would test nothing extra and could not be hand-audited), and the
containers are assembled part by part.
Run from the repository root: python3 tests/fixtures/make_image_fixtures.py
"""
from __future__ import annotations
import io
import struct
import zipfile
import zlib
from pathlib import Path
HERE = Path(__file__).parent
IMAGES = HERE / "image-inbox"
_XML = '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
_ZIP_DATE = (2020, 1, 1, 0, 0, 0)
def png(width: int, height: int, value: int = 0x40) -> bytes:
"""A real, single-channel PNG of a flat grey."""
def chunk(kind: bytes, payload: bytes) -> bytes:
return (
len(payload).to_bytes(4, "big")
+ kind
+ payload
+ zlib.crc32(kind + payload).to_bytes(4, "big")
)
ihdr = struct.pack(">IIBBBBB", width, height, 8, 0, 0, 0, 0)
raw = b"".join(b"\x00" + bytes([value] * width) for _ in range(height))
return (
b"\x89PNG\r\n\x1a\n"
+ chunk(b"IHDR", ihdr)
+ chunk(b"IDAT", zlib.compress(raw, 9))
+ chunk(b"IEND", b"")
)
def jpeg_header(width: int, height: int) -> bytes:
"""A JPEG's marker sequence: SOI, JFIF, a baseline frame header, EOI.
Not a decodable photograph, and that is the point of it. The `pdf` reader
passes `DCTDecode` bytes through untouched and reads only the frame header
for the size, which is precisely what R761's own table images need -- 29 of
its 50 image objects are `DCTDecode`. A fixture that also carried entropy
data would exercise no additional line of this package and could not be read
byte by byte by a person.
"""
frame = bytes([8, height >> 8, height & 0xFF, width >> 8, width & 0xFF, 1, 1, 0x11, 0])
return (
b"\xff\xd8"
b"\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00"
+ b"\xff\xc0"
+ (len(frame) + 2).to_bytes(2, "big")
+ frame
+ b"\xff\xd9"
)
# --- pdf ---------------------------------------------------------------------
#
# TWO images on one page, and they are deliberately of the two kinds the
# measurement on R761 found: 29 `DCTDecode` objects, which arrive as a finished
# JPEG file, and 21 `FlateDecode` ones, which arrive as raw samples with the
# colour model in the dictionary beside them and have to be encoded to be
# carried at all. A fixture with only one kind would leave half the reader
# unexercised, and it is the encoded half that can be silently wrong.
PDF_GRAY_WIDTH, PDF_GRAY_HEIGHT = 4, 3
PDF_GRAY_SAMPLES = bytes([0, 60, 120, 180, 20, 80, 140, 200, 40, 100, 160, 255])
PDF_JPEG_WIDTH, PDF_JPEG_HEIGHT = 360, 269
PDF_CONTENT = (
b"BT /F1 12 Tf 20 170 Td (Toleranseklasse er gitt i tabell 84-2) Tj ET\n"
b"q 80 0 0 60 20 90 cm /ImFlate Do Q\n"
b"q 80 0 0 60 20 20 cm /ImJpeg Do Q\n"
)
def build_image_pdf() -> bytes:
"""A one-page PDF with a Flate image and a DCT image in its resources."""
flate = zlib.compress(PDF_GRAY_SAMPLES, 9)
jpeg = jpeg_header(PDF_JPEG_WIDTH, PDF_JPEG_HEIGHT)
objects = [
b"<< /Type /Catalog /Pages 2 0 R >>",
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 4 0 R "
b"/Resources << /Font << /F1 5 0 R >> "
b"/XObject << /ImFlate 6 0 R /ImJpeg 7 0 R >> >> >>",
b"<< /Length "
+ str(len(PDF_CONTENT)).encode()
+ b" >>\nstream\n"
+ PDF_CONTENT
+ b"endstream",
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>",
b"<< /Type /XObject /Subtype /Image /Width "
+ str(PDF_GRAY_WIDTH).encode()
+ b" /Height "
+ str(PDF_GRAY_HEIGHT).encode()
+ b" /ColorSpace /DeviceGray /BitsPerComponent 8 /Filter /FlateDecode /Length "
+ str(len(flate)).encode()
+ b" >>\nstream\n"
+ flate
+ b"\nendstream",
b"<< /Type /XObject /Subtype /Image /Width "
+ str(PDF_JPEG_WIDTH).encode()
+ b" /Height "
+ str(PDF_JPEG_HEIGHT).encode()
+ b" /ColorSpace /DeviceGray /BitsPerComponent 8 /Filter /DCTDecode /Length "
+ str(len(jpeg)).encode()
+ b" >>\nstream\n"
+ jpeg
+ b"\nendstream",
]
out = bytearray(b"%PDF-1.4\n")
offsets = []
for number, body in enumerate(objects, start=1):
offsets.append(len(out))
out += str(number).encode() + b" 0 obj\n" + body + b"\nendobj\n"
xref_at = len(out)
size = str(len(objects) + 1).encode()
out += b"xref\n0 " + size + b"\n0000000000 65535 f \n"
for offset in offsets:
out += ("%010d 00000 n \n" % offset).encode()
out += b"trailer\n<< /Size " + size + b" /Root 1 0 R >>\n"
out += b"startxref\n" + str(xref_at).encode() + b"\n%%EOF\n"
return bytes(out)
# --- html --------------------------------------------------------------------
#
# THREE `<img>` and only two of them can be carried. The remote one is the
# boundary written as a fixture: this package never opens a socket during
# extraction (the network gate is an explicit per-run opt-in and extraction is
# not on that path), so a remote source becomes a pointer WITHOUT a file, and
# the gate counts it as found-and-not-carried rather than as absent.
HTML_DOCUMENT = """<!doctype html>
<html><head><title>Prosess 84</title></head>
<body>
<h1>84 Konstruksjoner av betong</h1>
<p>Toleranseklasse for de enkelte konstruksjonsdeler er gitt i tabell 84-2.</p>
<img src="graphics/tabell-84-2.png" alt="Tabell 84-2 Toleranseklasser">
<p>Figuren under viser prinsippet.</p>
<figure>
<img src="graphics/figur-84-1.png" alt="Figur 84-1 Prinsippskisse">
<figcaption>Figur 84-1 Prinsippskisse</figcaption>
</figure>
<p>Og en som ligger et annet sted:</p>
<img src="https://example.invalid/ekstern.png" alt="Ekstern figur">
</body></html>
"""
# --- niso-sts ----------------------------------------------------------------
#
# The shape the R761 delivery actually has, measured 2026-09-16: 50 `<graphic>`
# elements, every one a direct child of a `<sec>`, none inside a `<table-wrap>`,
# each carrying a bare file name in `xlink:href` that resolves against a sibling
# `graphics/` directory. No `<caption>` anywhere near them -- the caption a
# human reads is a `<p>` the extractor already emits on its own line.
STS_DOCUMENT = """<?xml version="1.0" encoding="UTF-8"?>
<standard xmlns:xlink="http://www.w3.org/1999/xlink">
<front><std-ident><doc-number>R761</doc-number></std-ident></front>
<body>
<sec>
<label>84</label>
<title>Konstruksjoner av betong</title>
<sec>
<label>84.1</label>
<title>Toleranser</title>
<p>Toleranseklasse er gitt i tabell 84-2.</p>
<graphic xlink:href="graphics/tabell-84-2.png"/>
<p>Figur 84-1 viser prinsippet.</p>
<graphic xlink:href="figur-84-1.png"/>
</sec>
</sec>
</body>
</standard>
"""
def build_docx() -> bytes:
"""A `.docx` with one embedded image, its alt text on the drawing."""
image = png(40, 30, value=0x30)
parts: dict[str, str | bytes] = {
"[Content_Types].xml": _XML
+ '<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
+ '<Default Extension="xml" ContentType="application/xml"/>'
+ '<Default Extension="rels" ContentType="application/vnd.openxmlformats-package'
+ '.relationships+xml"/>'
+ '<Default Extension="png" ContentType="image/png"/>'
+ '<Override PartName="/word/document.xml" ContentType="application/vnd'
+ '.openxmlformats-officedocument.wordprocessingml.document.main+xml"/>'
+ "</Types>",
"_rels/.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006'
+ '/relationships/officeDocument" Target="word/document.xml"/></Relationships>',
"word/_rels/document.xml.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rIdImg" Type="http://schemas.openxmlformats.org/officeDocument'
+ '/2006/relationships/image" Target="media/tabell-84-2.png"/></Relationships>',
"word/media/tabell-84-2.png": image,
"word/document.xml": _XML
+ '<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"'
+ ' xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships"'
+ ' xmlns:wp="http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing"'
+ ' xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main"'
+ ' xmlns:pic="http://schemas.openxmlformats.org/drawingml/2006/picture"><w:body>'
+ "<w:p><w:r><w:t>Toleranseklasse er gitt i tabell 84-2.</w:t></w:r></w:p>"
+ '<w:p><w:r><w:drawing><wp:inline><wp:extent cx="381000" cy="285750"/>'
+ '<wp:docPr id="1" name="Bilde 1" descr="Tabell 84-2 Toleranseklasser"/>'
+ '<a:graphic><a:graphicData uri="http://schemas.openxmlformats.org/drawingml/2006'
+ '/picture"><pic:pic><pic:nvPicPr><pic:cNvPr id="1" name="tabell-84-2.png"'
+ ' descr="Tabell 84-2 Toleranseklasser"/><pic:cNvPicPr/></pic:nvPicPr>'
+ '<pic:blipFill><a:blip r:embed="rIdImg"/><a:stretch><a:fillRect/></a:stretch>'
+ '</pic:blipFill><pic:spPr><a:xfrm><a:off x="0" y="0"/>'
+ '<a:ext cx="381000" cy="285750"/></a:xfrm>'
+ '<a:prstGeom prst="rect"><a:avLst/></a:prstGeom></pic:spPr></pic:pic>'
+ "</a:graphicData></a:graphic></wp:inline></w:drawing></w:r></w:p>"
+ "<w:p><w:r><w:t>Etter tabellen gjelder NS-EN 13670.</w:t></w:r></w:p>"
+ "</w:body></w:document>",
}
return build_container(parts)
def build_pptx() -> bytes:
"""A `.pptx` with one titled slide and one embedded image on it."""
image = png(48, 36, value=0x70)
parts: dict[str, str | bytes] = {
"[Content_Types].xml": _XML
+ '<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
+ '<Default Extension="xml" ContentType="application/xml"/>'
+ '<Default Extension="rels" ContentType="application/vnd.openxmlformats-package'
+ '.relationships+xml"/>'
+ '<Default Extension="png" ContentType="image/png"/>'
+ '<Override PartName="/ppt/presentation.xml" ContentType="application/vnd'
+ '.openxmlformats-officedocument.presentationml.presentation.main+xml"/>'
+ '<Override PartName="/ppt/slides/slide1.xml" ContentType="application/vnd'
+ '.openxmlformats-officedocument.presentationml.slide+xml"/>'
+ "</Types>",
"_rels/.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006'
+ '/relationships/officeDocument" Target="ppt/presentation.xml"/></Relationships>',
"ppt/_rels/presentation.xml.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rId2" Type="http://schemas.openxmlformats.org/officeDocument/2006'
+ '/relationships/slide" Target="slides/slide1.xml"/></Relationships>',
"ppt/presentation.xml": _XML
+ '<p:presentation xmlns:p="http://schemas.openxmlformats.org/presentationml/2006/main"'
+ ' xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">'
+ '<p:sldIdLst><p:sldId id="256" r:id="rId2"/></p:sldIdLst></p:presentation>',
"ppt/slides/_rels/slide1.xml.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rIdImg" Type="http://schemas.openxmlformats.org/officeDocument'
+ '/2006/relationships/image" Target="../media/skisse.png"/></Relationships>',
"ppt/media/skisse.png": image,
"ppt/slides/slide1.xml": _XML
+ '<p:sld xmlns:p="http://schemas.openxmlformats.org/presentationml/2006/main"'
+ ' xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main"'
+ ' xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">'
+ "<p:cSld><p:spTree>"
+ '<p:nvGrpSpPr><p:cNvPr id="1" name=""/><p:cNvGrpSpPr/><p:nvPr/></p:nvGrpSpPr>'
+ "<p:grpSpPr/>"
+ '<p:sp><p:nvSpPr><p:cNvPr id="2" name="Tittel 1"/><p:cNvSpPr/>'
+ '<p:nvPr><p:ph type="title"/></p:nvPr></p:nvSpPr><p:spPr/>'
+ "<p:txBody><a:bodyPr/><a:p><a:r><a:t>Toleranser</a:t></a:r></a:p></p:txBody></p:sp>"
+ '<p:pic><p:nvPicPr><p:cNvPr id="3" name="skisse.png" descr="Prinsippskisse"/>'
+ "<p:cNvPicPr/><p:nvPr/></p:nvPicPr>"
+ '<p:blipFill><a:blip r:embed="rIdImg"/><a:stretch><a:fillRect/></a:stretch>'
+ '</p:blipFill><p:spPr><a:xfrm><a:off x="0" y="0"/>'
+ '<a:ext cx="457200" cy="342900"/></a:xfrm>'
+ '<a:prstGeom prst="rect"><a:avLst/></a:prstGeom></p:spPr></p:pic>'
+ "</p:spTree></p:cSld></p:sld>",
}
return build_container(parts)
def build_container(parts: dict[str, str | bytes]) -> bytes:
out = io.BytesIO()
with zipfile.ZipFile(out, "w", compression=zipfile.ZIP_DEFLATED) as archive:
for name, payload in parts.items():
info = zipfile.ZipInfo(name, date_time=_ZIP_DATE)
info.compress_type = zipfile.ZIP_DEFLATED
archive.writestr(info, payload)
return out.getvalue()
if __name__ == "__main__":
(IMAGES / "graphics").mkdir(parents=True, exist_ok=True)
# FIVE DISTINCT STEMS. One stem across five extensions is refused by the
# door's own SS 3 collision rule -- measured while building this fixture:
# `prosess-84.{pdf,docx,pptx}` gave `inbox_slug_collision: 2/7` and two of
# the five readers were never exercised at all, with the gate reporting a
# carrying defect that was really a fixture defect.
written: list[tuple[str, bytes]] = [
("graphics/tabell-84-2.png", png(120, 90, value=0x20)),
("graphics/figur-84-1.png", png(64, 48, value=0x80)),
("prosess-84-web.html", HTML_DOCUMENT.encode("utf-8")),
("prosess-84-sts.xml", STS_DOCUMENT.encode("utf-8")),
("prosess-84-tabell.pdf", build_image_pdf()),
("prosess-84-notat.docx", build_docx()),
("prosess-84-presentasjon.pptx", build_pptx()),
]
for name, payload in written:
(IMAGES / name).write_bytes(payload)
print(f"wrote image-inbox/{name}")

181
tests/test_asset_bundle.py Normal file
View file

@ -0,0 +1,181 @@
"""Door B writes the images beside the text, and says how many (0.10.0).
The gate (`test_asset_gate.py`) asks whether the bytes arrive. This file pins
the four properties that decide whether their arrival is trustworthy.
- **A concept says how many images it carries.** `images: N`, conditional, so a
document without figures pays nothing and every bundle built before this
existed is byte-identical. Without the count a consumer cannot tell "this
document had no figures" from "this build dropped them", which is the
distinction the whole capability exists to restore.
- **The key belongs to the profiles this repository owns.** `DEFAULT` states
commons' ingest-spec SS 5 layer and `STRICT_V1` the wiki's ratified contract;
naming a key in either from here is this repository editing someone else's
contract (O2), the same reason `sources` sits on the segmented v0.2 profile
alone. Under those profiles the images are still CARRIED and still POINTED
at -- only the count is absent.
- **`--no-assets` reproduces the pre-move bytes.** Every default this package
has ever moved carries an opt-out that does, and the claim is measured on a
whole bundle rather than asserted.
- **The run log states the denominator.** SS 9's `log.md` already carries the
gate's name and the file counts because they are the facts about a run that
the bundle cannot otherwise recover. "51 images carried of 53 found" is the
same class of fact, and without it a bundle whose figures were all refused
looks exactly like a bundle of documents that had none.
"""
from __future__ import annotations
import filecmp
import warnings
from pathlib import Path
import pytest
from llm_ingestion_okf import cli, corpus
from llm_ingestion_okf.assets import ASSETS_DIR
from llm_ingestion_okf.profiles import DEFAULT, SEGMENTED_OKF_V0_2, STRICT_V1, STRUCTURED_V1
FIXTURES = Path(__file__).parent / "fixtures" / "image-inbox"
BUNDLE_ID = "asset-bundle-fixture"
OKF_VERSION = "0.2"
def _inbox(root: Path) -> Path:
inbox = root / "inbox"
(inbox / "graphics").mkdir(parents=True)
for source in sorted(FIXTURES.rglob("*")):
if source.is_file():
(inbox / source.relative_to(FIXTURES)).write_bytes(source.read_bytes())
return inbox
def _build(inbox: Path, bundle: Path, *extra: str) -> int:
with warnings.catch_warnings():
warnings.simplefilter("ignore")
return cli.main(
[
"build",
str(inbox),
"--bundle",
str(bundle),
"--bundle-id",
BUNDLE_ID,
"--okf-version",
OKF_VERSION,
*extra,
]
)
def _concepts(bundle: Path) -> list[Path]:
return [
path
for path in sorted(bundle.rglob("*.md"))
if path.name not in {"index.md", corpus.LOG_NAME}
]
# --- the count -------------------------------------------------------------
def test_a_concept_that_carries_images_counts_them(tmp_path: Path) -> None:
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
bundle = tmp_path / "bundle"
assert _build(_inbox(tmp_path), bundle) == 0
counted = [path for path in _concepts(bundle) if "\nimages: " in path.read_text("utf-8")]
assert counted, "no concept declared an image count"
for path in counted:
text = path.read_text("utf-8")
line = next(row for row in text.splitlines() if row.startswith("images: "))
assert int(line.split(":", 1)[1]) == text.count("](/assets/")
def test_a_concept_without_images_declares_no_count(tmp_path: Path) -> None:
"""Conditional, like `req_number`: absent is the document saying nothing."""
pytest.importorskip("llm_ingestion_guard")
inbox = tmp_path / "inbox"
inbox.mkdir()
(inbox / "ren.md").write_text("# Kostnader\n\nIngen figurer her.\n", encoding="utf-8")
bundle = tmp_path / "bundle"
assert _build(inbox, bundle) == 0
for path in _concepts(bundle):
assert "images:" not in path.read_text("utf-8")
def test_the_key_is_named_by_the_profiles_this_repository_owns() -> None:
assert "images" in STRUCTURED_V1.frontmatter.order
assert "images" in SEGMENTED_OKF_V0_2.frontmatter.order
# O2: naming a key in either of these is editing a contract owned elsewhere.
assert "images" not in DEFAULT.frontmatter.order
assert "images" not in STRICT_V1.frontmatter.order
# --- the opt-out reproduces the pre-move bytes -----------------------------
def test_no_assets_writes_no_assets_directory(tmp_path: Path) -> None:
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
bundle = tmp_path / "bundle"
assert _build(_inbox(tmp_path), bundle, "--no-assets") == 0
assert not (bundle / ASSETS_DIR).exists()
for path in _concepts(bundle):
assert "](/assets/" not in path.read_text("utf-8")
def test_the_two_runs_differ_in_the_asset_layer_and_nowhere_else(tmp_path: Path) -> None:
"""A document with NO images is byte-identical under both settings.
The measurement that matters for every consumer who has already built a
bundle: turning the capability on must move nothing in a corpus that has no
pictures in it. Asserted on files rather than on a count, because a count is
insensitive to exactly the change this would hide.
"""
pytest.importorskip("llm_ingestion_guard")
inbox = tmp_path / "inbox"
inbox.mkdir()
(inbox / "ren.md").write_text(
"# Kostnader\n\nIngen figurer.\n\n## Pris\n\nEn tabell uten bilde.\n", encoding="utf-8"
)
on, off = tmp_path / "on", tmp_path / "off"
assert _build(inbox, on) == 0
assert _build(inbox, off, "--no-assets") == 0
names = sorted(path.relative_to(on) for path in on.rglob("*") if path.is_file())
assert names == sorted(path.relative_to(off) for path in off.rglob("*") if path.is_file())
for name in names:
if name.name == corpus.LOG_NAME:
continue # the log names the setting on purpose
assert filecmp.cmp(on / name, off / name, shallow=False), name
# --- the log states the denominator ----------------------------------------
def test_the_log_carries_images_found_and_carried(tmp_path: Path) -> None:
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
bundle = tmp_path / "bundle"
assert _build(_inbox(tmp_path), bundle) == 0
log = (bundle / corpus.LOG_NAME).read_text("utf-8")
assert "**Images**:" in log
assert " of " in log.split("**Images**:", 1)[1].splitlines()[0]
def test_the_log_names_the_capability_as_off_when_it_is(tmp_path: Path) -> None:
"""`--no-assets` is stated, never implied by a missing line.
The same rule `--gate none` follows: a bundle built without screening says
`NOTHING WAS SCREENED` rather than leaving a reader to infer it from an
absent bullet. A bundle whose figures were never looked for must not read
as a bundle of documents that had none.
"""
pytest.importorskip("llm_ingestion_guard")
inbox = tmp_path / "inbox"
inbox.mkdir()
(inbox / "ren.md").write_text("# Kostnader\n\nIngen figurer.\n", encoding="utf-8")
bundle = tmp_path / "bundle"
assert _build(inbox, bundle, "--no-assets") == 0
log = (bundle / corpus.LOG_NAME).read_text("utf-8")
assert "NOT CARRIED" in log

244
tests/test_asset_gate.py Normal file
View file

@ -0,0 +1,244 @@
"""THE GATE: does a bundle carry the images its sources declare? (0.10.0)
Written BEFORE the capability, and red on purpose. Its whole job is to state
one number per reader with the DENOMINATOR read out of the source document
rather than out of this package -- "carried 0 of 9" is a measurement; "no image
support" is a sentence. The denominators here are computed by opening the
fixture bytes and counting what the FORMAT says is there (`page.images`,
`word/media/`, `ppt/media/`, `<img`, `<graphic`), so the gate keeps working
when the fixtures change and cannot drift into asserting our own output back at
us.
**Measured at `332961a`**, built from `git archive` rather than from the
editable tree -- an editable install reads `src/` live, so a "before" run taken
in this working tree would have been measuring the change it was supposed to
predate:
carried 0 of 2 local (2 declared) prosess-84-tabell.pdf
carried 0 of 1 local (1 declared) prosess-84-notat.docx
carried 0 of 1 local (1 declared) prosess-84-presentasjon.pptx
carried 0 of 2 local (3 declared) prosess-84-web.html
carried 0 of 2 local (2 declared) prosess-84-sts.xml
---------------------------------------------------------------
carried 0 of 8 local images across 5 documents (9 declared),
and the bundle held no `assets/` directory at all.
Two of the seven files in the fixture inbox are the PNGs the HTML and STS
documents point at, and the run reports them as `extractor_unknown: 2/7` at
both commits. That is deliberate and unchanged: `.png` as a DROPPED FILE is a
separate question with its own order, and an image reached through a document
is this one.
WHY THE GATE IS A BUNDLE-LEVEL COUNT and not a per-function assertion: the
failure this exists to stop is not "the reader returned nothing". It is a
bundle that looks complete and is not -- R761's process 84 saying "toleranse-
klasse ... er gitt i tabell 84-2" over an empty space, with `okf check` green
and `okf quality` unable to see it. So the gate runs the real command, walks
the written bundle, and asks whether the bytes are there.
THE REMOTE IMAGE IS PART OF THE DENOMINATOR AND NOT PART OF THE TARGET. The
HTML fixture carries three `<img>`; one points at `https://example.invalid/`.
Extraction opens no socket -- network access here is an explicit per-run opt-in
and extraction is not on that path -- so that image is FOUND, counted, and
carried as a pointer without a file. A gate that quietly dropped it from the
denominator would report 2 of 2 and hide the one case a reader most needs told.
"""
from __future__ import annotations
import re
import warnings
import zipfile
from pathlib import Path
import pytest
from llm_ingestion_okf import cli, corpus
from llm_ingestion_okf.assets import ASSETS_DIR, IMAGE_POINTER
FIXTURES = Path(__file__).parent / "fixtures" / "image-inbox"
BUNDLE_ID = "asset-gate-fixture"
OKF_VERSION = "0.2"
#: Every document in the fixture inbox, with the reader it exercises.
DOCUMENTS = (
"prosess-84-tabell.pdf",
"prosess-84-notat.docx",
"prosess-84-presentasjon.pptx",
"prosess-84-web.html",
"prosess-84-sts.xml",
)
# --- denominators, read out of the source ----------------------------------
def declared_images(path: Path) -> int:
"""How many images the SOURCE says it holds, by the format's own rule.
Never a constant: a constant is this package asserting its own expectation,
and it goes stale the moment a fixture is regenerated. Each branch reads the
container the way the format defines it, which is also the number a person
checking the bundle by hand would arrive at.
"""
suffix = path.suffix.lower()
data = path.read_bytes()
if suffix == ".pdf":
pdfplumber = pytest.importorskip("pdfplumber")
with warnings.catch_warnings():
warnings.simplefilter("ignore")
with pdfplumber.open(path) as pdf:
return sum(len(page.images) for page in pdf.pages)
if suffix in (".docx", ".pptx", ".xlsx"):
with zipfile.ZipFile(path) as archive:
return sum(1 for name in archive.namelist() if "/media/" in name)
text = data.decode("utf-8")
if suffix in (".html", ".htm"):
return len(re.findall(r"<img\b", text))
if suffix == ".xml":
return len(re.findall(r"<(?:inline-)?graphic\b", text))
raise AssertionError(f"no declared-image rule for {path.name}")
def local_images(path: Path) -> int:
"""The declared images MINUS the ones whose source is off this machine.
Separate from :func:`declared_images` and both are reported: the difference
between them is exactly the network boundary, and collapsing the two would
turn a boundary into an absence.
"""
declared = declared_images(path)
if path.suffix.lower() in (".html", ".htm"):
remote = len(re.findall(r'<img\b[^>]*src="(?:https?:)?//', path.read_text("utf-8")))
return declared - remote
return declared
# --- the run ---------------------------------------------------------------
def _inbox(tmp_path: Path) -> Path:
inbox = tmp_path / "inbox"
(inbox / "graphics").mkdir(parents=True)
for source in sorted(FIXTURES.rglob("*")):
if source.is_file():
target = inbox / source.relative_to(FIXTURES)
target.write_bytes(source.read_bytes())
return inbox
def _build(inbox: Path, bundle: Path, *extra: str) -> int:
return cli.main(
[
"build",
str(inbox),
"--bundle",
str(bundle),
"--bundle-id",
BUNDLE_ID,
"--okf-version",
OKF_VERSION,
*extra,
]
)
def _concepts(bundle: Path) -> list[Path]:
return [
path
for path in sorted(bundle.rglob("*.md"))
if path.name not in {"index.md", corpus.LOG_NAME}
]
def carried_per_source(bundle: Path) -> dict[str, int]:
"""Pointers in the bundle whose asset file is actually present, per source.
A pointer whose bytes are missing is NOT carried. Counting pointers alone
would let an empty `assets/` pass the gate, which is the same defect one
level down -- a bundle that looks complete and is not.
"""
counts: dict[str, int] = {}
for concept in _concepts(bundle):
text = concept.read_text("utf-8")
source = ""
for line in text.splitlines():
if line.startswith("source_file:"):
source = line.split(":", 1)[1].strip().strip("'\"")
break
for match in IMAGE_POINTER.finditer(text):
if (bundle / ASSETS_DIR / match.group("asset")).is_file():
counts[source] = counts.get(source, 0) + 1
return counts
@pytest.fixture(scope="module")
def built(tmp_path_factory: pytest.TempPathFactory) -> tuple[Path, dict[str, int]]:
pytest.importorskip("pdfplumber")
pytest.importorskip("pypandoc")
pytest.importorskip("llm_ingestion_guard")
root = tmp_path_factory.mktemp("asset-gate")
inbox = _inbox(root)
bundle = root / "bundle"
with warnings.catch_warnings():
warnings.simplefilter("ignore")
assert _build(inbox, bundle) == 0
return bundle, carried_per_source(bundle)
# --- the gate --------------------------------------------------------------
@pytest.mark.parametrize("document", DOCUMENTS)
def test_every_local_image_reaches_the_bundle(
document: str, built: tuple[Path, dict[str, int]]
) -> None:
"""carried == local, per source document, with both numbers printed."""
bundle, carried = built
want = local_images(FIXTURES / document)
got = carried.get(document, 0)
assert got == want, (
f"{document}: carried {got} of {want} local images "
f"({declared_images(FIXTURES / document)} declared by the source)"
)
def test_the_bundle_carries_every_local_image_of_every_document(
built: tuple[Path, dict[str, int]],
) -> None:
"""The whole-bundle row, so the gate reports one number a person can quote."""
bundle, carried = built
want = sum(local_images(FIXTURES / document) for document in DOCUMENTS)
got = sum(carried.values())
assert got == want, f"carried {got} of {want} local images across {len(DOCUMENTS)} documents"
def test_a_remote_image_is_a_pointer_without_a_file_never_a_silent_drop(
built: tuple[Path, dict[str, int]],
) -> None:
"""The network boundary, stated in the artifact rather than implied by absence."""
bundle, _ = built
html = [path for path in _concepts(bundle) if "prosess-84-web.html" in path.read_text("utf-8")]
assert html, "the html document produced no concept at all"
body = "\n".join(path.read_text("utf-8") for path in html)
assert "https://example.invalid/ekstern.png" in body, (
"a remote image must leave a pointer naming where it was, so a reader "
"learns the document had a figure this bundle does not hold"
)
assert "not carried" in body
def test_the_asset_bytes_are_the_sources_own_bytes(
built: tuple[Path, dict[str, int]],
) -> None:
"""A carried image is byte-identical to the file the source shipped.
Only the two files the fixture carries as real files can be checked this
way; the PDF and office images arrive inside a container. That is enough to
pin the property that matters -- nothing re-encodes an image that already
is one.
"""
bundle, _ = built
written = {path.read_bytes() for path in (bundle / ASSETS_DIR).glob("*")}
for original in sorted((FIXTURES / "graphics").glob("*.png")):
assert original.read_bytes() in written, original.name

128
tests/test_asset_import.py Normal file
View file

@ -0,0 +1,128 @@
"""Door C carries the assets its merged concepts point at (0.10.0).
Measured on 2026-09-17, before this was fixed: importing a bundle built with
`--assets` merged 6 of 6 concepts -- the guard cleared every pointer block as
ordinary body text -- and wrote **no `assets/` directory at all**. Every
`![Tabell 84-2](/assets/...)` in the imported bundle pointed at a file that was
not there.
That is the same defect one door over: a bundle that reads as complete and is
not. Door C's two invariants make the repair obvious rather than a new
mechanism -- a merged concept is written VERBATIM, so its pointer cannot be
rewritten to point somewhere else, and ownership is proven by CONTENT IDENTITY,
which is exactly the rule an asset name already encodes.
WHAT IS NOT CARRIED IS AS IMPORTANT: an asset no merged concept points at stays
behind. A concept the guard refused takes its pictures with it, or the import
would persist bytes the gate never cleared a reader for -- and an orphan in
`assets/` is a file nothing names and nothing retires.
"""
from __future__ import annotations
import warnings
from pathlib import Path
import pytest
from llm_ingestion_okf import cli
from llm_ingestion_okf.assets import ASSETS_DIR, IMAGE_POINTER
from llm_ingestion_okf.importer import import_bundle
FIXTURES = Path(__file__).parent / "fixtures" / "image-inbox"
INGESTED_AT = "2026-09-17T00:00:00Z"
def _source_bundle(tmp_path: Path) -> Path:
inbox = tmp_path / "inbox"
(inbox / "graphics").mkdir(parents=True)
for source in sorted(FIXTURES.rglob("*")):
if source.is_file():
(inbox / source.relative_to(FIXTURES)).write_bytes(source.read_bytes())
bundle = tmp_path / "source-bundle"
with warnings.catch_warnings():
warnings.simplefilter("ignore")
assert (
cli.main(
[
"build",
str(inbox),
"--bundle",
str(bundle),
"--bundle-id",
"asset-import-fixture",
"--okf-version",
"0.2",
]
)
== 0
)
return bundle
def _imported(tmp_path: Path) -> Path:
from llm_ingestion_okf.guard_adapter import import_gate
target = tmp_path / "imported"
import_bundle(
_source_bundle(tmp_path),
target,
INGESTED_AT,
origin="external",
channel="manual",
gate=import_gate,
)
return target
def test_every_pointer_a_merged_concept_carries_resolves(tmp_path: Path) -> None:
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
target = _imported(tmp_path)
pointers = 0
for concept in sorted(target.rglob("*.md")):
for match in IMAGE_POINTER.finditer(concept.read_text("utf-8")):
pointers += 1
asset = target / ASSETS_DIR / match.group("asset")
assert asset.is_file(), f"{concept.name} points at a missing {asset.name}"
assert pointers, "the fixture bundle carried no pointers at all"
def test_the_imported_bytes_are_the_senders_own(tmp_path: Path) -> None:
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
source = _source_bundle(tmp_path)
from llm_ingestion_okf.guard_adapter import import_gate
target = tmp_path / "imported"
import_bundle(
source, target, INGESTED_AT, origin="external", channel="manual", gate=import_gate
)
carried = sorted((target / ASSETS_DIR).glob("*"))
# The known-positive in the same test: a loop over an empty directory is
# green and proves nothing, which is exactly how a repair can empty the set
# a test iterates over and stay passing.
assert carried, "nothing was carried, so the comparison below ran over nothing"
for asset in carried:
assert asset.read_bytes() == (source / ASSETS_DIR / asset.name).read_bytes()
def test_an_asset_nothing_points_at_stays_behind(tmp_path: Path) -> None:
"""The negative control, and it is the security half of the rule.
An asset belonging to a concept the gate refused must not ride in on the
back of one it cleared.
"""
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
source = _source_bundle(tmp_path)
(source / ASSETS_DIR / "deadbeefdead-ingen-peker.png").write_bytes(
(source / ASSETS_DIR).glob("*.png").__next__().read_bytes() + b"\x00"
)
from llm_ingestion_okf.guard_adapter import import_gate
target = tmp_path / "imported"
import_bundle(
source, target, INGESTED_AT, origin="external", channel="manual", gate=import_gate
)
assert not (target / ASSETS_DIR / "deadbeefdead-ingen-peker.png").exists()

55
tests/test_asset_structure.py Normal file
View file

@ -0,0 +1,55 @@
"""An image is not a cross-reference, and a carried image is counted (0.10.0).
Two door-level properties that the asset layer would otherwise get wrong in
opposite directions.
**A markdown image is not a pointer to another concept.** `structure._LINK`
reads `[...](target)` and does not care what stands in front of the bracket, so
every pointer this capability writes would arrive in the index as a `references`
edge to a concept that cannot exist. That is the same defect K3-21 hit when the
`Enclosing section:` line was read as body text and became a second, unresolved
edge; the repair there was to append after derivation, and the repair here is
that an image link is a different construct from a link.
**A concept says how many images it carries.** A bundle that holds an image and
says nothing about it is only half an improvement: the count is what lets a
consumer, or `okf quality`, tell "this document had no figures" from "this
build dropped them". The key is CONDITIONAL -- written only where the count is
non-zero -- so a bundle of documents without images is byte-identical to one
built before this existed.
"""
from __future__ import annotations
from llm_ingestion_okf.structure import derive_document_structure
POINTER = (
"# 84 Konstruksjoner\n\n"
"Toleranseklasse er gitt i tabell 84-2, jf. N500.\n\n"
"![Tabell 84-2](/assets/e54e5f5da0e8-tabell-84-2.png)\n"
"Image: graphics/tabell-84-2.png (120x90 px) -- Tabell 84-2\n\n"
"Se ellers [N100](n100.md).\n"
)
def test_an_image_pointer_is_not_a_reference() -> None:
structure = derive_document_structure(POINTER, source_file="r761.xml")
assert "/assets/e54e5f5da0e8-tabell-84-2.png" not in structure.references
def test_the_links_beside_it_still_are() -> None:
"""The known-positive in the same text: masking must not eat real edges."""
structure = derive_document_structure(POINTER, source_file="r761.xml")
assert "n100.md" in structure.references
def test_the_asset_name_does_not_leak_a_number_into_references() -> None:
"""A masked span is still masked: `84-2` inside the file name is not a subject.
The span is blanked rather than deleted for exactly this reason -- deleting
would move every later offset and break first-appearance order, and leaving
it would let the number scan read an identifier out of a file name.
"""
structure = derive_document_structure(POINTER, source_file="r761.xml")
assert all("ASSETS" not in subject.upper() for subject in structure.references)
assert "N500" in structure.references

336
tests/test_assets.py Normal file
View file

@ -0,0 +1,336 @@
"""Binary assets: the layer that lets a bundle carry an image at all.
Until 0.10.0 no reader in this package fetched, named, described or copied a
single image, and the only writer into a bundle was
`materialize.write_bytes(bundle_dir, name, content: str)` -- UTF-8, text, no
binary path anywhere. A document whose table is a raster picture therefore
reached a concept as an absence with no denominator: measured on R761
Prosesskoden:2025, the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, and process 84 says "toleranse-
klasse ... er gitt i tabell 84-2" above an empty space.
THIS MODULE IS THE ONE PLACE THAT DECIDES WHAT AN IMAGE IS. Every reader hands
it bytes and gets back either a carried image or a coded rejection, so a
format's own quirks never reach the bundle layout, and "N images carried of M
found" has one definition for `pdf`, `docx`, `html` and `xml` alike.
Three properties are pinned here because a bundle is downstream of all of them:
- **the type is SNIFFED, never claimed.** A `.jpg` that is really a PNG would
otherwise be written under a name whose extension lies, and a consumer that
dispatches on the extension reads it wrong with full confidence.
- **the name is CONTENT-ADDRESSED.** `<sha256[:12]>-<reduced original name>`
makes two drops of one image one file, and makes a rebuild of the same
corpus produce the same bundle -- the byte-determinism rule this package
already holds for text.
- **the pointer is one GRAMMAR.** Step 2 (`okf describe`) has to find every
pointer mechanically in order to write a transcription under it, so the
block is a regex this module owns rather than a shape each reader invents.
"""
from __future__ import annotations
import hashlib
import re
import zlib
import pytest
from llm_ingestion_okf import assets
from llm_ingestion_okf.errors import ExtractionError
# --- hand-laid image bytes -------------------------------------------------
#
# Written out here rather than committed as files: every byte is visible in the
# test that depends on it, and the readers below are header readers, so a
# header is the whole input they have. The PNG is a REAL image (zlib from the
# stdlib); the JPEG is a structurally valid header sequence and not a decodable
# photograph, which is exactly what the JPEG path needs -- it copies the bytes
# through and reads nothing but the SOF marker.
def _png(width: int, height: int) -> bytes:
def chunk(kind: bytes, payload: bytes) -> bytes:
return (
len(payload).to_bytes(4, "big")
+ kind
+ payload
+ zlib.crc32(kind + payload).to_bytes(4, "big")
)
ihdr = width.to_bytes(4, "big") + height.to_bytes(4, "big") + bytes([8, 0, 0, 0, 0])
raw = b"".join(b"\x00" + bytes([0x40] * width) for _ in range(height))
return (
b"\x89PNG\r\n\x1a\n"
+ chunk(b"IHDR", ihdr)
+ chunk(b"IDAT", zlib.compress(raw, 9))
+ chunk(b"IEND", b"")
)
def _jpeg(width: int, height: int) -> bytes:
sof = bytes([8, height >> 8, height & 0xFF, width >> 8, width & 0xFF, 1, 1, 0x11, 0])
return (
b"\xff\xd8"
b"\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00"
+ b"\xff\xc0"
+ (len(sof) + 2).to_bytes(2, "big")
+ sof
+ b"\xff\xd9"
)
def _gif(width: int, height: int) -> bytes:
return (
b"GIF89a"
+ width.to_bytes(2, "little")
+ height.to_bytes(2, "little")
+ b"\x00\x00\x00"
+ b";"
)
def _bmp(width: int, height: int) -> bytes:
header = b"BM" + (54).to_bytes(4, "little") + b"\x00" * 4 + (54).to_bytes(4, "little")
dib = (
(40).to_bytes(4, "little")
+ width.to_bytes(4, "little", signed=True)
+ height.to_bytes(4, "little", signed=True)
+ b"\x01\x00\x18\x00"
+ b"\x00" * 24
)
return header + dib
# --- the type is sniffed, never claimed ------------------------------------
@pytest.mark.parametrize(
("data", "media_type", "suffix"),
[
(_png(4, 3), "image/png", ".png"),
(_jpeg(360, 269), "image/jpeg", ".jpg"),
(_gif(7, 5), "image/gif", ".gif"),
(_bmp(9, 2), "image/bmp", ".bmp"),
],
)
def test_sniff_reads_the_bytes(data: bytes, media_type: str, suffix: str) -> None:
assert assets.sniff(data) == (media_type, suffix)
def test_sniff_refuses_what_is_not_an_image() -> None:
assert assets.sniff(b"%PDF-1.7\n") is None
assert assets.sniff(b"") is None
def test_the_claimed_extension_never_decides() -> None:
"""A PNG named `.jpg` is carried as a PNG, under a `.png` name."""
image = assets.read_image(_png(4, 3), name="tabell-84-2.jpg")
assert image.media_type == "image/png"
assert assets.asset_name(image).endswith(".png")
def test_unknown_bytes_are_a_coded_rejection_not_a_crash() -> None:
with pytest.raises(ExtractionError) as excinfo:
assets.read_image(b"not an image at all", name="x.png")
assert excinfo.value.code == "asset_type_unknown"
# --- dimensions, from the header of each format ----------------------------
@pytest.mark.parametrize(
("data", "size"),
[
(_png(4, 3), (4, 3)),
(_jpeg(360, 269), (360, 269)),
(_gif(7, 5), (7, 5)),
(_bmp(9, 2), (9, 2)),
],
)
def test_dimensions_from_the_header(data: bytes, size: tuple[int, int]) -> None:
assert assets.dimensions(data) == size
def test_dimensions_absent_is_absent_never_zero() -> None:
"""A format whose size this module does not read says so, rather than 0x0.
`0x0 px` in a concept is a measurement nobody took, printed as a fact.
"""
tiff = b"II\x2a\x00" + b"\x00" * 16
assert assets.sniff(tiff) == ("image/tiff", ".tiff")
assert assets.dimensions(tiff) is None
image = assets.read_image(tiff, name="scan.tiff")
assert image.width is None and image.height is None
assert "dimensions unknown" in assets.render_block(image)
# --- the name is content-addressed -----------------------------------------
def test_asset_name_is_digest_plus_a_reduced_original() -> None:
data = _jpeg(360, 269)
image = assets.read_image(data, name="25-0143 - Tabeller - Prosesskoden (R761-R762).jpg")
name = assets.asset_name(image)
assert name.startswith(hashlib.sha256(data).hexdigest()[: assets.DIGEST_PREFIX] + "-")
assert name.endswith(".jpg")
# The link target sits inside `](...)`, which a space or a parenthesis
# terminates -- and `structure._LINK` reads exactly that grammar. A name
# carrying either would produce a pointer no reader can follow.
assert re.fullmatch(r"[a-z0-9][a-z0-9.-]*", name), name
def test_two_drops_of_one_image_are_one_asset() -> None:
data = _png(4, 3)
first = assets.read_image(data, name="figur.png")
second = assets.read_image(data, name="figur.png")
assert assets.asset_name(first) == assets.asset_name(second)
def test_one_name_over_two_contents_stays_two_assets() -> None:
a = assets.read_image(_png(4, 3), name="figur.png")
b = assets.read_image(_png(5, 3), name="figur.png")
assert assets.asset_name(a) != assets.asset_name(b)
def test_a_nameless_image_still_gets_a_name() -> None:
image = assets.read_image(_png(4, 3), name="")
assert re.fullmatch(r"[0-9a-f]{12}\.png", assets.asset_name(image))
# --- the pointer is one grammar --------------------------------------------
def test_block_names_the_original_the_size_and_the_label() -> None:
image = assets.read_image(
_jpeg(360, 269),
name="25-0143 - Tabeller - Prosesskoden (R761-R762).jpg",
label="Tabell 84-2",
)
block = assets.render_block(image)
first, second = block.split("\n")
assert first == f"![Tabell 84-2]({assets.asset_href(image)})"
assert second.startswith("Image: 25-0143 - Tabeller - Prosesskoden (R761-R762).jpg")
assert "360x269 px" in second
assert second.endswith("Tabell 84-2")
def test_the_label_falls_back_to_the_original_name() -> None:
"""Two of the four readers have no caption element at all.
A PDF image object and an STS `<graphic>` carry no caption: the caption on
the page is running text the extractor already emitted. Inventing one from
the nearest line would be an unmarked heuristic, so the alt slot carries
the name the source gave the file.
"""
image = assets.read_image(_png(4, 3), name="graphic_0003.jpg")
assert assets.render_block(image).startswith("![graphic_0003.jpg](")
def test_a_bracket_in_a_label_cannot_break_the_link() -> None:
image = assets.read_image(_png(4, 3), name="f.png", label="Tabell [84-2] jf. pkt (3)")
first = assets.render_block(image).split("\n")[0]
assert first == f"![Tabell (84-2) jf. pkt (3)]({assets.asset_href(image)})"
def test_a_newline_in_a_label_cannot_open_a_third_line() -> None:
image = assets.read_image(_png(4, 3), name="f.png", label="Tabell\n84-2")
assert len(assets.render_block(image).split("\n")) == 2
def test_the_pointer_regex_finds_every_shipped_block() -> None:
"""Step 2 has to find these mechanically; the finder ships with the writer."""
images = [
assets.read_image(_png(4, 3), name="a.png", label="Figur 1"),
assets.read_image(_jpeg(9, 9), name="b.jpg"),
]
text = "Prosess 84\n\n" + "\n\n".join(assets.render_block(i) for i in images) + "\n\nSlutt\n"
found = assets.IMAGE_POINTER.findall(text)
assert [match[1] for match in found] == [assets.asset_name(i) for i in images]
def test_href_is_bundle_absolute() -> None:
"""SPEC SS 6.2 allows a bundle-relative path with a leading `/`.
A concept can sit at any depth under a segmented bundle, so a relative
`assets/...` would resolve differently from two concepts of one document.
"""
image = assets.read_image(_png(4, 3), name="f.png")
assert assets.asset_href(image) == f"/{assets.ASSETS_DIR}/{assets.asset_name(image)}"
# --- PNG encoding, for samples that arrive without a container -------------
#
# A PDF image is usually not a file: `FlateDecode` hands back raw samples with
# the colour model in the dictionary beside them. Encoding those is the only
# way to carry them at all, and it is stdlib (`zlib`), so the `pdf` row does
# not gain a dependency and the output is not bound to a renderer's version.
def test_png_from_gray_samples_round_trips_the_header() -> None:
encoded = assets.encode_png(3, 2, bytes([0, 64, 128, 192, 255, 32]), channels=1)
assert assets.sniff(encoded) == ("image/png", ".png")
assert assets.dimensions(encoded) == (3, 2)
def test_png_from_rgb_samples_round_trips_the_header() -> None:
encoded = assets.encode_png(2, 1, bytes(range(6)), channels=3)
assert assets.dimensions(encoded) == (2, 1)
def test_png_from_indexed_samples_carries_the_palette() -> None:
palette = bytes([255, 0, 0, 0, 255, 0])
encoded = assets.encode_png(2, 1, bytes([0, 1]), channels=1, palette=palette)
assert assets.dimensions(encoded) == (2, 1)
assert b"PLTE" in encoded
def test_png_refuses_a_sample_count_that_does_not_fit() -> None:
"""Refused rather than padded: a short buffer is a misread dictionary."""
with pytest.raises(ExtractionError) as excinfo:
assets.encode_png(4, 4, b"\x00\x01", channels=1)
assert excinfo.value.code == "asset_samples_invalid"
def test_png_encoding_is_byte_stable() -> None:
first = assets.encode_png(3, 2, bytes([0, 64, 128, 192, 255, 32]), channels=1)
second = assets.encode_png(3, 2, bytes([0, 64, 128, 192, 255, 32]), channels=1)
assert first == second
def test_the_restated_reduction_is_materializes_own() -> None:
"""`assets` cannot import `materialize`, so the rule is held equal instead.
`extract.py`'s registry must not import the contract layer (its own
docstring says so, and `materialize` reaches `manifest` and `profiles`), so
the id-grammar reduction is written out a second time. This is what stops
the second copy becoming a second grammar.
"""
from llm_ingestion_okf.materialize import reduce_to_id_grammar
for value in (
"25-0143 - Tabeller - Prosesskoden (R761-R762)",
"Figur 11.1 Toleransekrav",
"grafikk_med_æøå",
"---",
"",
):
assert assets._reduce(value) == reduce_to_id_grammar(value), value
def test_one_image_reached_by_two_paths_is_one_asset() -> None:
"""The asset name reads the BASENAME, never the path the document used.
Measured on the fixture inbox before this rule existed: the HTML document
points at `graphics/figur-84-1.png` and the STS document at
`figur-84-1.png` (resolved through the `graphics/` sibling convention), so
one image was written twice, under two names, from one run -- with the
digest in both of them announcing that the bytes were identical. The path a
document happened to use is not a property of the picture.
"""
data = _png(6, 4)
through_directory = assets.read_image(data, name="graphics/figur-84-1.png")
bare = assets.read_image(data, name="figur-84-1.png")
assert assets.asset_name(through_directory) == assets.asset_name(bare)
# The full original is not lost -- it moves to the line a person reads.
assert "graphics/figur-84-1.png" in assets.render_block(through_directory)

38
tests/test_docs_promises.py
View file

@ -317,3 +317,41 @@ def test_the_readme_names_every_gate_the_command_accepts() -> None:
text = README.read_text(encoding="utf-8")
for name in GATE_NAMES:
assert f"`{name}`" in text, f"README does not name the gate {name}"
# --- the asset default is published and pinned (0.10.0) --------------------
#
# Same obligation as the gate marker above it, for the same reason: 0.10.0
# changes what `okf build` writes for every consumer whose sources carry
# pictures, and a documented default that can drift from the applied one is how
# F1 survived for months.
_ASSETS_LINE = re.compile(r"^<!-- cli-default-assets: (on|off) -->$", re.MULTILINE)
def test_the_readme_names_the_asset_default_the_build_command_applies() -> None:
from llm_ingestion_okf import cli
match = _ASSETS_LINE.search(README.read_text(encoding="utf-8"))
assert match is not None, (
"README.md carries no `<!-- cli-default-assets: ... -->` marker; without it "
"the documented default can drift from the one the command applies"
)
assert (match.group(1) == "on") is cli.DEFAULT_ASSETS
def test_the_readme_names_the_opt_out_that_reproduces_the_old_bytes() -> None:
text = README.read_text(encoding="utf-8")
assert "`--no-assets`" in text
assert "NOT CARRIED" in text
def test_the_readme_states_that_image_bytes_are_not_screened() -> None:
"""The boundary, published rather than left to be discovered.
The guard is text-only. A consumer weighing an untrusted drop has to be
able to learn which half of a concept was looked at without reading this
package's source.
"""
text = README.read_text(encoding="utf-8")
assert "image bytes are not screened" in text.lower()

91
tests/test_error_codes.py
View file

@ -13,7 +13,9 @@ import importlib.util
import json
import sqlite3
import sys
import tempfile
import urllib.error
import warnings
from pathlib import Path
from typing import Any
@ -700,3 +702,92 @@ def test_segmentation_plan_unmatched(tmp_path: Path) -> None:
root_frontmatter_values={"bundle_id": "b-1"},
)
assert code_of(excinfo) == "segmentation_plan_unmatched"
# --- asset codes (0.10.0) --------------------------------------------------
#
# One test per code, like every code above it. These five are the only codes in
# the registry that a caller is expected to COUNT rather than to act on: an
# image a reader could not carry becomes a row in the run log and a line in the
# concept, never a failed document.
def test_asset_type_unknown() -> None:
from llm_ingestion_okf import assets
with pytest.raises(ExtractionError) as excinfo:
assets.read_image(b"%PDF-1.7\n", name="figur.png")
assert excinfo.value.code == "asset_type_unknown"
def test_asset_samples_invalid() -> None:
from llm_ingestion_okf import assets
with pytest.raises(ExtractionError) as excinfo:
assets.encode_png(8, 8, b"\x00", channels=1)
assert excinfo.value.code == "asset_samples_invalid"
def test_asset_remote() -> None:
from llm_ingestion_okf.extract import extract_document
document = extract_document(
"side.html",
b'<html><body><img src="https://example.invalid/x.png" alt="x"></body></html>',
assets=True,
)
assert [item.code for item in document.rejected] == ["asset_remote"]
def test_asset_unresolved() -> None:
from llm_ingestion_okf.extract import extract_document
document = extract_document(
"side.html",
b'<html><body><img src="mangler.png" alt="x"></body></html>',
assets=True,
)
assert [item.code for item in document.rejected] == ["asset_unresolved"]
def test_asset_pdf_unsupported() -> None:
"""A PDF image this encoder refuses rather than approximates.
A 1-bit stencil: carrying it at 8 bits would be a decision about what black
means, and a wrong one is indistinguishable from a right one in the output.
"""
pytest.importorskip("pdfplumber")
from llm_ingestion_okf.extract import extract_document
data = (
Path(__file__).parent / "fixtures" / "image-inbox" / "prosess-84-tabell.pdf"
).read_bytes()
stencil = data.replace(
b"/BitsPerComponent 8 /Filter /FlateDecode", b"/BitsPerComponent 1 /Filter /FlateDecode", 1
)
assert stencil != data
with warnings.catch_warnings():
warnings.simplefilter("ignore")
document = extract_document("krav.pdf", stencil, assets=True)
assert "asset_pdf_unsupported" in [item.code for item in document.rejected]
def test_asset_collision() -> None:
"""Two different pictures reducing to one asset name, refused in the run.
Constructed rather than found: the name carries 12 hex of the digest of its
own bytes, so reaching this by accident is a 48-bit collision. The code
exists because resolving it silently would lose one of the two pictures
while every pointer to it kept showing the other.
"""
from llm_ingestion_okf.assets import ExtractedImage, asset_name
from llm_ingestion_okf.inbox import _write_assets
image = ExtractedImage(b"AAAA", "f.png", "image/png", ".png", 1, 1)
# One name already holding DIFFERENT bytes, which is what a 48-bit digest
# collision would look like from inside the run.
seen = {asset_name(image): b"BBBB"}
with tempfile.TemporaryDirectory() as root:
with pytest.raises(MaterializationError) as excinfo:
_write_assets(Path(root) / "bundle", [image], seen)
assert excinfo.value.code == "asset_collision"

2
uv.lock generated
View file

@ -547,7 +547,7 @@ source = { git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-sec
[[package]]
name = "llm-ingestion-okf"
version = "0.9.0"
version = "0.10.0"
source = { editable = "." }
dependencies = [
{ name = "llm-ingestion-guard" },