open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(assets): every carried image is one a model can be shown

Browse source 
Chosen: a stdlib BMP reader, because `read_image` is on the CORE path and an
asset's name is its content digest. Measured first, as the order requires:
Pillow 12.3.0 IS in this tree (transitively under `pdfplumber`) and it DOES
decode RLE8 correctly -- a hand-written stdlib decoder and Pillow agree on
19 of 19 of R761's real files, RGB per pixel. So the choice does not rest on
capability. It rests on two properties of this package: `.html` and `.xml`
carry images with no `[extract]` extra installed, so a Pillow converter
either makes a core path depend on an optional binary wheel or buys the
second runtime dependency; and encoding through an installed library would
make a bundle's identity move with that library's version, which is the
property 0.10.0 felled page rasterisation over and `encode_png`'s docstring
already defends. Pillow keeps the job it is good for: the INDEPENDENT decoder
in the tests, on neither side of the conversion.

The defect, measured over the frozen R761 delivery's `assets/`, denominator
50: 29 JPEG, 2 PNG and 19 RLE8 BMP. The 19 are byte-correct files nothing
reads, so 19 figures were present and invisible while `images: N` reported
that they had arrived.

- `VIEWABLE_MEDIA_TYPES` is tested against every asset's SNIFFED type, so it
  is a property and not a list of formats we met. WebP is on it and `sniff`
  does not recognise one; the limit is stated, not implied.
- `bmp_to_png`: 8-bit uncompressed, 8-bit RLE8, 24-bit uncompressed. All five
  RLE8 opcodes. 19 of 19 real files convert with RGB identical to Pillow's
  decoding of the source, 2 366 365 pixels compared.
- `asset_not_viewable` and `asset_bmp_unsupported`, both published, both
  leaving the concept's "not carried" line.
- Traceability on the pointer's second line, where the rest of the asset
  metadata already lives: original media type, original sha256 in full, new
  sha256 in full. A converted asset is ONE asset.
- The ceiling is paid on the DECLARATION before a row is allocated, and an
  RLE run is one clipped slice -- painting pixel by pixel leaves the memory
  bounded and the CPU unbounded.

Two repairs the change forced, each measured rather than assumed:

- `tests/test_assets.py`'s "dimensions absent is absent" used a TIFF, which
  is now refused before `read_image` returns. The property still has a
  reachable case -- a JPEG whose frame header never arrives -- and uses it.
- `asset_holds` in the accounting gate proved a carry by hashing the SOURCE
  file, which a converted image's bundle cannot satisfy. It now also reads
  the two digests the bundle states and HASHES THE ASSET ITSELF, so a bundle
  claiming a conversion it did not perform still fails.

`tools/okf_asset_census.py` is the committed instrument for the
known-positive: one row per image, from two pinned trees. It was caught by
the rule it serves -- its first version handed `_pdf_images` the wrong page
object and reported 0 images over 67 PDFs with exit 0. The attribute is
asserted now and a known-positive runs before the sweep.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 08:07:04 +02:00
commit b0b5e71658
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
10 changed files with 826 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

45
CHANGELOG.md
View file

@ -9,6 +9,51 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- **Every carried image is now one a model can be SHOWN, and the ones that
cannot be are refused out loud.** Until this round the asset path carried
whatever format a publisher shipped. Measured 2026-09-19 over the frozen
R761 delivery's own `assets/` (denominator 50): 29 JPEG, 2 PNG and **19 "PC
bitmap, Windows 3.x, 8-bit, compression 1"** — RLE8 BMP. The 19 are
byte-correct files nothing reads, so 19 of that document's figures were
present and invisible at once, with `images: N` reporting that they had
arrived.
- `assets.VIEWABLE_MEDIA_TYPES` states the set (`image/png`, `image/jpeg`,
`image/gif`, `image/webp`) and `read_image` tests every asset's SNIFFED
type against it. It is a property, not a list of formats we happened to
meet: a format nobody here has seen is refused by the same rule that
refuses TIFF.
- **BMP is converted losslessly to PNG** — 8-bit uncompressed, 8-bit RLE8
and 24-bit uncompressed. The reader is stdlib (`struct` + the existing
`zlib` PNG writer) and adds NO dependency. Pillow was measured first and
rejected on two counts: `read_image` is on the CORE path (`.html` and
`.xml` carry images with no `[extract]` extra), and an asset's name is its
content digest, so encoding through an installed library would make a
bundle's identity move with that library's version — the property 0.10.0
felled page rasterisation over. Pillow is the INDEPENDENT decoder in the
tests instead.
- **Lossless, measured on the real files:** all **19 of 19** R761 RLE8
assets convert with RGB identical to Pillow's decoding of the source,
**2 366 365 pixels** compared.
- **Traceability per converted asset**, on the pointer line where the rest
of the asset metadata already lives: the original media type, the original
sha256 in full, and the new one. A converted asset is ONE asset — one file
in `assets/`, one pointer, one row in the accounting.
- **The ceiling is paid before the pixels exist.** The BMP reader bounds the
DECLARED size through the same `check_size` the rest of the image path
uses, before a row is allocated, and an RLE run is written as one clipped
slice — painting pixel by pixel would leave the memory bounded and the CPU
unbounded, since a megabyte of `FF` runs is a hundred million paint steps
against a 32-pixel frame.
- **Two new codes.** `asset_not_viewable` — a real image in a format no
model can be shown and with no lossless conversion here (TIFF, JPEG 2000).
`asset_bmp_unsupported` — a BMP variant this reader does not express
(RLE4, BITFIELDS, 16/32-bit, BITMAPCOREHEADER, over 256 palette entries).
Both leave a "not carried" line in the concept and a row in the run log.
- **The cost, measured with a committed script** (`tools/okf_asset_census.py`,
one row per image from two pinned trees): see the round's report for the
before/after counts, including the **16 of 9 714** JPEG 2000 objects that
stop being carried.
- **One normalisation door in front of the persist gate: U+00AD is removed and
COUNTED** (operator decision 2026-09-18). `llm-ingestion-guard` 1.4.0 keeps
the soft hyphen in `_ZERO_WIDTH_CPS`, and `output:zero-width-present` is an

30
CLAUDE.md
View file

@ -1010,6 +1010,36 @@ and fixtures, never code.
pointer-only concept is substantive.
- Reports: `docs/2026-09-17-bildestien-0-10-1.md` and
`docs/2026-09-18-bildestien-holder-0-10-1.md`.
- **EVERY CARRIED IMAGE IS ONE A MODEL CAN BE SHOWN (0.10.1, 2026-09-19).**
Until this round the path carried whatever a publisher shipped. Measured over
the frozen R761 delivery's own `assets/`, denominator 50: 29 JPEG, 2 PNG and
**19 "PC bitmap, Windows 3.x, 8-bit, compression 1"** -- byte-correct RLE8
BMP files that nothing reads, so 19 figures were present and invisible at
once while `images: N` said they had arrived. `assets.VIEWABLE_MEDIA_TYPES`
(`png`, `jpeg`, `gif`, `webp`) is tested against every asset's SNIFFED type,
so it is a PROPERTY and not a list of formats we met; WebP is on it and
`sniff` does not recognise one, which is stated rather than implied. BMP is
converted losslessly to PNG -- 8-bit uncompressed, 8-bit RLE8 and 24-bit
uncompressed -- and everything else outside the set is refused with
`asset_not_viewable`, a BMP variant this reader does not express with
`asset_bmp_unsupported`, both leaving the concept's own "not carried" line.
**THE READER IS STDLIB AND PILLOW WAS MEASURED AND FELLED**, though the tree
already carries it transitively under `pdfplumber`: `read_image` is on the
CORE path (`.html` and `.xml` carry images with no `[extract]`), and an
asset's name is its content digest, so encoding through an installed library
would make a bundle's identity move with that library's version -- 0.10.0's
own reason for felling page rasterisation. Pillow is the INDEPENDENT decoder
in the tests instead, and against it **19 of 19** real RLE8 assets convert
with identical RGB, **2 366 365 pixels** compared. The ceiling is paid on the
DECLARATION before a row is allocated, and an RLE run is one CLIPPED slice --
painting pixel by pixel leaves the memory bounded and the CPU unbounded, a
megabyte of `FF` runs being a hundred million steps against a 32-pixel frame.
Traceability is on the pointer's second line where the rest of the asset
metadata already lives (original media type, original sha256 in full, new
sha256), and a converted asset is ONE asset: one file, one pointer, one
accounting row. Exposure measured with a committed script
(`tools/okf_asset_census.py`, one row per image, run from two pinned trees).
Report: `docs/2026-09-19-hvert-baaret-bilde-kan-vises.md`.
- **Account for content: `okf build --accounting PATH` and its judge
`python3 tools/okf_accounting_gate.py` (2026-09-17).**
- The gate was written RED at `0b00de4` and turned GREEN on all six rows by

29
README.md
View file

@ -308,6 +308,35 @@ in must not read like a bundle of documents that had none.
text. The pointer block passes the gate like any other body line, and the file
beside it passes nothing. `log.md` says so on every run that carries one.
**Every carried image is one a model can be SHOWN.** A bundle that holds a
picture nothing can read is worse than one that says the picture is missing:
the count reports that it arrived. Measured over the frozen R761 delivery's own
`assets/` (denominator 50): 29 JPEG, 2 PNG and **19 RLE8 BMP** — correct files
that no model decodes. Every asset's type is read off its bytes and tested
against the viewable set; a BMP is converted losslessly to PNG (8-bit
uncompressed, 8-bit RLE8, 24-bit uncompressed), and anything else outside the
set is refused with `asset_not_viewable` and a line in the concept saying what
stood there. A BMP variant this reader does not express — RLE4, BITFIELDS,
16- or 32-bit samples, a 12-byte BITMAPCOREHEADER — is `asset_bmp_unsupported`,
a different fact about the document and a different thing to go and fix.
The reader is stdlib and adds no dependency. Pillow, which this tree already
carries transitively under `pdfplumber`, was measured first and rejected on two
counts: images are carried on the CORE path, where `.html` and `.xml` need no
`[extract]` extra, and an asset's name is its content digest — encoding through
an installed library would make a bundle's identity move with that library's
version, which is the property page rasterisation was felled over. Pillow is
the independent decoder in the tests instead, and against it **19 of 19** of
R761's real RLE8 assets convert with identical RGB, 2 366 365 pixels compared.
A converted asset is ONE asset: one file in `assets/`, one pointer, one row in
the accounting. The pointer's second line — where the source's own file name
and the size in pixels already live — states the original media type, the
original sha256 in full and the new one, so a reader can take the original
delivery, run `shasum -a 256` and find the row.
<!-- asset-viewable-media-types: image/gif,image/jpeg,image/png,image/webp -->
**A size CEILING, read off the same corpora (0.10.1).** An image over
`MAX_IMAGE_PIXELS` (40 000 000 pixels) or `MAX_IMAGE_BYTES` (256 MiB) is
refused with `asset_too_large`, counted like every other refusal. The largest

156
docs/2026-09-19-hvert-baaret-bilde-kan-vises.md Normal file
View file

@ -0,0 +1,156 @@
# Every carried image is one a model can be shown
`llm-ingestion-okf`, 2026-09-19. Order `20260919T033624Z-4535320370`, raised
from PM's measurement of the frozen R761 package's `assets/` directory.
Nothing is tagged, pushed or released here. No new version: the entries go
under the untagged `[0.10.1]`.
## 1. The defect
`file` over the frozen R761 delivery's own `assets/`, denominator **50**:
| Format | Count | A model can show it |
|---|---|---|
| JPEG | 29 | yes |
| PNG | 2 | yes |
| PC bitmap, Windows 3.x, 8-bit, compression 1 (RLE8 BMP) | **19** | **no** |
The 19 are byte-correct files. They are also files nothing decodes, so 19 of
that document's figures were present and invisible at the same time — and the
concept's own `images: N` reported that they had arrived. An absence a reader
is shown is information; a picture that is there and unreadable is worse than
either, because the count says the opposite.
Reproduced here before anything was written, with the committed census
(§ 5): `19 of 50 CARRY image/bmp.bmp`, `29 of 50 CARRY image/jpeg.jpg`,
`2 of 50 CARRY image/png.png`.
## 2. The decoder: measured, then chosen
The order requires the choice to rest on a measurement of what the tree
already has. Pillow 12.3.0 is present, transitively under `pdfplumber` in the
`[extract]` extra, and `PIL` occurs 0 times in `src/`, `tests/` and `tools/`.
**It decodes RLE8 correctly.** A hand-written stdlib decoder and Pillow were
compared on all 19 real files, RGB per pixel through the palette: **19 of 19
identical**. So the measurement's answer is yes, and the choice is not made on
capability.
**Chosen: a stdlib reader. Why, and both reasons are properties of this
package rather than preferences.**
1. `read_image` is on the CORE path. `.html` and `.xml` are stdlib file types
that carry images with no `[extract]` extra installed, so a Pillow-based
converter either makes a core path depend on an optional binary wheel or
buys this package its second runtime dependency — against the "exactly one
runtime dependency, ever" rule a packaging test enforces.
2. An asset's name is its content digest, so the bytes the converter emits are
part of the bundle's identity. Encoding through an installed library would
make that identity move with the library's version. This is the property
0.10.0 felled page rasterisation over, stated in `encode_png`'s own
docstring, and it would be odd to defend it there and give it away here.
`encode_png` already writes a PNG from samples with `zlib` alone; this round
adds the reader in front of it. **Pillow keeps the job it is good for: the
INDEPENDENT decoder in the tests**, on neither side of the conversion.
## 3. What was built
- `assets.VIEWABLE_MEDIA_TYPES` — `image/png`, `image/jpeg`, `image/gif`,
`image/webp`. `read_image` tests every asset's SNIFFED type against it. It
is a property, not a list of formats we happened to meet. WebP is on the set
and `sniff` does not recognise one, so a WebP is refused one step earlier as
`asset_type_unknown`; the limit is stated rather than left implied.
- `assets.bmp_to_png` — 8-bit uncompressed, 8-bit RLE8, 24-bit uncompressed.
All five RLE8 opcodes: encoded run, absolute run with its pad byte, delta,
end-of-line, end-of-bitmap. Bottom-up unless the declared height is
negative. Palette read BGRA → RGB.
- Two published codes. `asset_not_viewable` — a real image in a format no
model can be shown, with no lossless conversion here (TIFF, JPEG 2000).
`asset_bmp_unsupported` — a BMP variant this reader does not express (RLE4,
BITFIELDS, 16/32-bit, BITMAPCOREHEADER, over 256 palette entries). Both
leave the concept's own "not carried" line and a row in the run log.
- `ExtractedImage.converted_from` and `.source_sha256`, written onto the
pointer's second line where the source's file name and pixel size already
live: original media type, original sha256 in full, new sha256 in full. Full
digests because the asset's file name carries only 12 hex characters of the
new one, and a checksum a reader cannot paste into `shasum -a 256` is
decoration.
**A converted asset is ONE asset.** One file in `assets/`, one pointer, one
row in the accounting — the accounting's own columns read
`ExtractedDocument.files`, which is what the resolver resolved, and conversion
does not touch it.
**The ceiling is paid before the pixels exist.** `bmp_to_png` bounds the
DECLARED size through the same `check_size` the rest of the image path uses,
before one row is allocated, and each RLE run is written as one CLIPPED slice.
Painting pixel by pixel would leave the memory bounded and the CPU unbounded:
a megabyte of `FF` runs is a hundred million paint steps against a 32-pixel
frame.
## 4. The eval, red first
Two red commits, because the mutant survey found holes in the first one.
| Commit | What |
|---|---|
| `fea0435` | 12 guards, 8 red on their claim |
| `9e99bb2` | 2 more guards the mutant survey found missing; 9 of 13 red |
The second commit is the interesting one. Walking the five mutants the order
names against the eval as committed, **two survived**:
- *"the format is read from the file extension instead of the bytes"* survived
because every image in the fixture was named after what it is: `.bmp` maps
to `image/bmp` either way. A BMP named `graphics/figur.png` can tell the two
apart.
- *"the ceiling is checked after decoding"* survived because the guard only
asserted that a refusal happens, and `encode_png`'s own `check_size` refuses
too — one frame later, after the memory is spent. The guard already measured
the peak; what it could not do was measure it affordably, since a
50 000 × 50 000 frame is 2.5 GB. At 7 000 × 7 000 the declaration is still
over the 40 MP bound and the unbounded frame is 49 MB.
**The lossless guard needed the same care.** Comparing "the carried asset that
holds these pixels" against the source is green before any conversion exists,
because a BMP carried verbatim trivially holds its own pixels. The holder has
to be required to be VIEWABLE; with that clause the guard reads `0 carried
assets hold these pixels` before, and passes after.
## 5. The measurements
**Lossless, on the real files.** Method: Pillow decodes the SOURCE BMP to RGB;
`read_image` converts it; Pillow decodes the RESULT; the two sizes and two RGB
buffers must be equal. Pillow is on neither side of the conversion.
- **19 of 19** lossless, 0 differing, **2 366 365 pixels** compared.
**Mutants, in a scratch copy of the tree.** Control green first, because
"killed N of N" can also mean the harness cannot invoke pytest at all.
| Mutant | Result |
|---|---|
| M1 the format is read from the file extension | KILLED |
| M2 the conversion skips 8-bit images | KILLED |
| M3 the palette is ignored (grey out) | KILLED |
| M4 the traceability field is omitted | KILLED |
| M5 the ceiling is checked after decoding | KILLED |
| M6 the BMP's BGR order is carried unswapped | KILLED |
| M7 the RLE delta opcode is ignored | KILLED |
**killed 7 of 7, survived 0, not measured 0**, control green at 13 passed.
**The census** (`tools/okf_asset_census.py`) is the committed instrument for
the known-positive. One row per image — the source's own sha256, the format
read off its bytes, and what `read_image` did with it — over every image file
and, with `--pdf`, every image object in every PDF it finds. Run from two
pinned trees and diff the rows.
It was itself caught by the rule it exists to serve. Its first version handed
`_pdf_images` a `page.page_obj` where the function reads `page.images` through
`getattr(..., [])`, so it reported **0 images over 67 PDFs** and exit 0 — a
clean-looking table that had measured nothing. The attribute is asserted now,
and the known-positive (one PDF, 8 images) is run before the sweep.

315
src/llm_ingestion_okf/assets.py
View file

@ -90,6 +90,27 @@ _MAGIC: tuple[tuple[bytes, str, str], ...] = (
(b"MM\x00\x2a", "image/tiff", ".tiff"),
)
#: The formats a model can be SHOWN. Everything a document ships outside this
#: set is converted losslessly to PNG, or refused with a code -- never carried
#: silently, which is what this package did until this round of 0.10.1.
#:
#: MEASURED 2026-09-19 over the frozen R761 delivery's own `assets/`
#: (denominator 50): 29 JPEG, 2 PNG and **19 "PC bitmap, Windows 3.x, 8-bit,
#: compression 1"**. The 19 are byte-correct files that nothing reads, so 19 of
#: that document's figures were present and invisible at the same time -- and
#: the `images: N` count said they had arrived. An absence a reader is shown is
#: information; a picture that is there and unreadable is worse than either.
#:
#: IT IS A PROPERTY, NOT A LIST OF FORMATS WE HAPPENED TO MEET. A carried
#: asset's type is read off its bytes and tested against this set, so a format
#: nobody here has seen is refused by the same rule that refuses TIFF.
#:
#: WebP is on the list and `sniff` does not recognise it: the set states what a
#: model can be shown, not what this package can read. A WebP is therefore
#: refused one step earlier, as `asset_type_unknown`, and never reaches this
#: test. Stating that is cheaper than a set whose name is wider than its reach.
VIEWABLE_MEDIA_TYPES = frozenset({"image/png", "image/jpeg", "image/gif", "image/webp"})
#: JPEG 2000, in both the forms a PDF `JPXDecode` stream hands back: the JP2
#: container and a bare codestream.
_JP2_SIGNATURE = b"\x00\x00\x00\x0cjP \r\n\x87\n"
@ -403,6 +424,16 @@ class ExtractedImage:
sees is running text the extractor already emitted on its own line.
Deriving a label from the nearest line would be an unmarked heuristic,
which this package treats as worse than no heuristic at all.
`converted_from` and `source_sha256` are set when the source was NOT a
format a model can be shown and this package rewrote it (today: a BMP, as
a PNG). They are the whole of the traceability: `converted_from` is the
media type the file had, `source_sha256` is the digest of the bytes the
document actually shipped, and the digest of `data` is what the bundle
holds. With the three of them a reader can take the original delivery,
run `shasum -a 256`, and find the row. Both are `None` for an image
carried verbatim, which is every image this package has ever carried
until now -- so a bundle of JPEGs is byte-identical across the move.
"""
data: bytes
@ -412,6 +443,8 @@ class ExtractedImage:
width: int | None
height: int | None
label: str | None = None
converted_from: str | None = None
source_sha256: str | None = None
@dataclass(frozen=True)
@ -512,6 +545,266 @@ def _bmp_dimensions(data: bytes) -> tuple[int, int] | None:
return (abs(width), abs(height)) if width and height else None
#: `biCompression`: uncompressed, and the 8-bit run-length encoding 19 of
#: R761's 50 assets use. Every other value -- RLE4, BITFIELDS, embedded JPEG or
#: PNG -- is refused by name rather than guessed at.
_BMP_RGB = 0
_BMP_RLE8 = 1
@dataclass(frozen=True)
class _BmpHeader:
"""What the DIB header declares, before anything is decoded."""
header_size: int
width: int
height: int
top_down: bool
bits: int
compression: int
entries: int
offbits: int
def _bmp_header(data: bytes, *, name: str) -> _BmpHeader:
"""The declaration, validated -- never the pixels.
Everything this returns is a CLAIM by an untrusted file. It is read first
precisely so the claim can be bounded before the cost of believing it is
paid: the row buffers below are `width * height` bytes, and that product
comes from these two fields.
"""
if len(data) < 54:
raise ExtractionError(
f"the BMP {name!r} stops inside its own header ({len(data)} bytes)",
code="asset_samples_invalid",
)
offbits = struct.unpack("<I", data[10:14])[0]
header_size = struct.unpack("<I", data[14:18])[0]
if header_size < 40:
raise ExtractionError(
f"the BMP {name!r} carries a {header_size}-byte DIB header; this reader "
"expresses BITMAPINFOHEADER and its extensions, and a 12-byte "
"BITMAPCOREHEADER stores its palette in 3-byte entries, which is a "
"different format wearing the same magic",
code="asset_bmp_unsupported",
)
if len(data) < 14 + header_size:
raise ExtractionError(
f"the BMP {name!r} declares a {header_size}-byte DIB header and holds "
f"{len(data) - 14} bytes after the file header",
code="asset_samples_invalid",
)
width, height = struct.unpack("<ii", data[18:26])
bits = struct.unpack("<H", data[28:30])[0]
compression = struct.unpack("<I", data[30:34])[0]
clr_used = struct.unpack("<I", data[46:50])[0]
if width <= 0 or height == 0:
raise ExtractionError(
f"the BMP {name!r} declares {width}x{height}, which is not a size",
code="asset_size_invalid",
)
top_down = height < 0
entries = clr_used or (1 << bits if bits <= 8 else 0)
if entries > 256:
raise ExtractionError(
f"the BMP {name!r} declares {entries} palette entries; a PNG palette holds at most 256",
code="asset_bmp_unsupported",
)
return _BmpHeader(
header_size=header_size,
width=width,
height=-height if top_down else height,
top_down=top_down,
bits=bits,
compression=compression,
entries=entries,
offbits=offbits,
)
def _bmp_palette(data: bytes, head: _BmpHeader, *, name: str) -> bytes:
"""The colour table as PNG wants it: RGB triples, from the file's BGRA.
The order matters and getting it wrong is invisible in every structural
check: a palette read as RGB gives a picture with red and blue swapped,
the right size, the right number of colours, and the wrong answer.
"""
entries = head.entries or 256
start = 14 + head.header_size
table = data[start : start + entries * 4]
if len(table) < entries * 4:
raise ExtractionError(
f"the BMP {name!r} declares {entries} palette entries and holds {len(table) // 4}",
code="asset_samples_invalid",
)
palette = bytearray()
for index in range(entries):
blue, green, red = table[index * 4], table[index * 4 + 1], table[index * 4 + 2]
palette += bytes((red, green, blue))
return bytes(palette)
def _bmp_flat_rows(data: bytes, head: _BmpHeader, *, per_pixel: int, name: str) -> bytes:
"""Uncompressed rows, unpadded and in top-down order.
A BMP row is padded to a 4-byte boundary and stored bottom-up unless the
declared height is negative. Both are read off the header rather than
assumed, because either mistake produces a picture that is the right size
and is sheared or upside down.
"""
stride = ((head.width * per_pixel + 3) // 4) * 4
needed = stride * head.height
available = len(data) - head.offbits
if available < needed:
raise ExtractionError(
f"the BMP {name!r} needs {needed} bytes of pixel data and holds {available} "
"-- refusing to pad, because a short buffer means the header was read wrong",
code="asset_samples_invalid",
)
rows = [
data[head.offbits + index * stride : head.offbits + index * stride + head.width * per_pixel]
for index in range(head.height)
]
if not head.top_down:
rows.reverse()
return b"".join(rows)
def _bmp_rle8_rows(data: bytes, head: _BmpHeader) -> bytes:
"""The RLE8 opcode stream painted into a frame of the DECLARED size.
Five opcodes, and a decoder implementing only the first is wrong on real
files: an encoded run, an absolute run (whose byte count is padded to an
even length), a delta that SKIPS pixels, end-of-line and end-of-bitmap.
Skipped pixels keep index 0, which is what the format says and what an
independent decoder produces.
THE COST IS THE FRAME, NOT THE STREAM. The rows are allocated once from the
declared size -- already bounded by `check_size` before this is called --
and each run is written as one CLIPPED slice. Painting pixel by pixel would
leave the memory bounded and the CPU unbounded: a megabyte of `FF` runs is
a hundred million paint steps against a frame of 32 pixels.
"""
width, height = head.width, head.height
rows = [bytearray(width) for _ in range(height)]
position = head.offbits
end = len(data)
x = 0
y = 0
while position + 1 < end:
count = data[position]
value = data[position + 1]
position += 2
if count:
if 0 <= y < height and x < width:
stop = min(x + count, width)
rows[y][x:stop] = bytes((value,)) * (stop - x)
x += count
continue
if value == 0:
x = 0
y += 1
elif value == 1:
break
elif value == 2:
if position + 2 > end:
break
x += data[position]
y += data[position + 1]
position += 2
else:
run = data[position : position + value]
position += value + (value & 1)
if 0 <= y < height and x < width:
stop = min(x + len(run), width)
rows[y][x:stop] = run[: stop - x]
x += value
if not head.top_down:
rows.reverse()
return b"".join(bytes(row) for row in rows)
def bmp_to_png(data: bytes, *, name: str) -> bytes:
"""A BMP as a PNG with the same pixels, using nothing but the stdlib.
WHY A READER HERE AND NOT PILLOW, which this tree already carries
transitively under `pdfplumber`. Two reasons, measured rather than
preferred. First, `read_image` is on the CORE path: `.html` and `.xml` are
stdlib file types that carry images with no `[extract]` extra installed, so
a Pillow-based converter would either make a core path depend on an
optional binary wheel or buy this package its second runtime dependency.
Second, and decisive: an asset's name is its content digest, so the bytes
this function emits are part of the bundle's identity. Encoding through an
installed library would make that identity move with the library's version
-- the exact property 0.10.0 felled page rasterisation over. `encode_png`
already writes a PNG from samples with `zlib` alone; this adds the reader in
front of it.
Pillow is still the INDEPENDENT decoder in the tests, which is the job it
is good for here: 19 of 19 of R761's real RLE8 assets decode to identical
RGB through both paths (measured 2026-09-19, before this was written).
"""
head = _bmp_header(data, name=name)
channels = 3 if head.bits == 24 else 1
# THE CEILING FIRST, on the DECLARATION, before one row is allocated. The
# rows below are `width * height` bytes of an untrusted document's claim.
check_size(head.width, head.height, name=name, channels=channels, bits=8)
if head.bits == 8 and head.compression in (_BMP_RGB, _BMP_RLE8):
palette = _bmp_palette(data, head, name=name)
if head.compression == _BMP_RLE8:
if head.top_down:
raise ExtractionError(
f"the BMP {name!r} declares a top-down RLE8 image, which the format "
"does not define",
code="asset_bmp_unsupported",
)
samples = _bmp_rle8_rows(data, head)
else:
samples = _bmp_flat_rows(data, head, per_pixel=1, name=name)
limit = len(palette) // 3
if samples and max(samples) >= limit:
raise ExtractionError(
f"the BMP {name!r} uses palette index {max(samples)} and declares {limit} "
"entries; carrying it would invent a colour",
code="asset_samples_invalid",
)
return encode_png(head.width, head.height, samples, channels=1, palette=palette)
if head.bits == 24 and head.compression == _BMP_RGB:
raw = _bmp_flat_rows(data, head, per_pixel=3, name=name)
swapped = bytearray(raw)
# BGR on disk, RGB in a PNG. Two slices rather than a loop, and read
# from `raw` both times so the first assignment cannot feed the second.
swapped[0::3] = raw[2::3]
swapped[2::3] = raw[0::3]
return encode_png(head.width, head.height, bytes(swapped), channels=3)
raise ExtractionError(
f"the BMP {name!r} stores {head.bits}-bit samples under compression "
f"{head.compression}; this reader expresses 8-bit (uncompressed and RLE8) and "
"24-bit uncompressed, and will not guess at the rest",
code="asset_bmp_unsupported",
)
#: Source media type -> the function that makes a viewable file of it. A map
#: rather than a branch, so what this package can convert is one readable line
#: and adding a format is adding a row.
_CONVERTERS = {"image/bmp": bmp_to_png}
def to_viewable(data: bytes, *, media_type: str, name: str) -> bytes:
"""Bytes a model can be shown, or a coded refusal. Never a silent carry."""
convert = _CONVERTERS.get(media_type)
if convert is None:
raise ExtractionError(
f"the image {name!r} is {media_type}, which no model can be shown, and this "
"package has no lossless conversion for it; refused rather than carried in a "
"format nothing reads",
code="asset_not_viewable",
)
return convert(data, name=name)
def read_image(data: bytes, *, name: str, label: str | None = None) -> ExtractedImage:
"""One image, typed by its bytes, or a coded refusal.
@ -528,6 +821,16 @@ def read_image(data: bytes, *, name: str, label: str | None = None) -> Extracted
code="asset_type_unknown",
)
media_type, suffix = kind
# THE VIEWABILITY GATE, and it stands before the size is read because the
# size that matters is the one the CARRIED file has. A BMP that becomes a
# PNG is measured as the PNG a consumer will open.
converted_from: str | None = None
source_sha256: str | None = None
if media_type not in VIEWABLE_MEDIA_TYPES:
converted_from = media_type
source_sha256 = digest(data)
data = to_viewable(data, media_type=media_type, name=name)
media_type, suffix = "image/png", ".png"
size = dimensions(data)
# THE BOUND HOLDS FOR A FILE CARRIED VERBATIM TOO. This package does not
# decode one, so it pays nothing for it -- but writing a 7 000 x 7 000 PNG
@ -547,6 +850,8 @@ def read_image(data: bytes, *, name: str, label: str | None = None) -> Extracted
width=size[0] if size else None,
height=size[1] if size else None,
label=label,
converted_from=converted_from,
source_sha256=source_sha256,
)
@ -635,6 +940,16 @@ def render_block(image: ExtractedImage) -> str:
detail = f"Image: {_inline(image.name or asset_name(image))} ({size})"
if image.label:
detail += f" -- {_inline(image.label)}"
# WHAT THE CONVERSION DID, on the line the rest of the asset metadata is
# already on. Both digests in full: the asset's file name carries only the
# first 12 hex characters of the new one, and a checksum a reader cannot
# paste into `shasum -a 256` is decoration. Written LAST so a labelled
# image that was not converted keeps the bytes it has today.
if image.converted_from and image.source_sha256:
detail += (
f" -- converted from {image.converted_from} sha256:{image.source_sha256}"
f" to {image.media_type} sha256:{digest(image.data)}"
)
return f"![{_inline(label)}]({asset_href(image)})\n{detail}"

14
src/llm_ingestion_okf/errors.py
View file

@ -151,6 +151,20 @@ class ExtractionError(IngestError):
statistic about the first untrue. Refused before the stream is read: a
negative dimension multiplies to a negative pixel count, under which
every bound reads as satisfied
- `asset_not_viewable` — the bytes are a real image in a format no model
can be SHOWN (TIFF, JPEG 2000), and this package has no lossless
conversion for it. DISTINCT from `asset_type_unknown`, which says the
bytes are not an image at all: this one says they are, and carrying them
would put a file in the bundle that the `images: N` count reports as
arrived and nothing downstream can read. Measured 2026-09-19 on the
frozen R761 delivery: 19 of its 50 assets were carried in exactly that
condition, as RLE8 BMP
- `asset_bmp_unsupported` — a BMP variant this reader does not express
(RLE4, BITFIELDS, 16- or 32-bit samples, a 12-byte BITMAPCOREHEADER, a
palette over 256 entries). DISTINCT from `asset_not_viewable`, which
says there is no conversion route for the format at all: this one says
there is one and this file is outside it, which is a different fact
about the document and a different thing to go and fix
- `asset_pdf_unbounded` — the image is reached through a PDF stream filter
this package has no measured cost ratio for (`LZWDecode`,
`RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything unknown), or

16
tests/test_assets.py
View file

@ -156,11 +156,19 @@ def test_dimensions_absent_is_absent_never_zero() -> None:
"""A format whose size this module does not read says so, rather than 0x0.
`0x0 px` in a concept is a measurement nobody took, printed as a fact.
The vehicle used to be a TIFF. Since the viewable-asset round a TIFF never reaches
`read_image`'s return at all -- it is a format no model can be shown and
there is no lossless conversion for it, so it is refused as
`asset_not_viewable`. The property still has a reachable case, and this is
it: a JPEG whose frame header never arrives. `sniff` types it from the
first three bytes, `_jpeg_dimensions` walks to the end and finds no SOF,
and the pointer has to say so rather than print a size.
"""
tiff = b"II\x2a\x00" + b"\x00" * 16
assert assets.sniff(tiff) == ("image/tiff", ".tiff")
assert assets.dimensions(tiff) is None
image = assets.read_image(tiff, name="scan.tiff")
headless = b"\xff\xd8\xff\xfe\x00\x04ab\xff\xd9"
assert assets.sniff(headless) == ("image/jpeg", ".jpg")
assert assets.dimensions(headless) is None
image = assets.read_image(headless, name="scan.jpg")
assert image.width is None and image.height is None
assert "dimensions unknown" in assets.render_block(image)

23
tests/test_docs_promises.py
View file

@ -357,6 +357,29 @@ def test_the_readme_states_that_image_bytes_are_not_screened() -> None:
assert "image bytes are not screened" in text.lower()
_VIEWABLE_LINE = re.compile(
r"^<!-- asset-viewable-media-types: ([a-z0-9/,+.-]+) -->$", re.MULTILINE
)
def test_the_readme_publishes_the_viewable_set_the_code_applies() -> None:
"""A published set is a test obligation, the same as a published bound.
Sorted on both sides so the marker states a SET and not an order, and
compared as a whole rather than by membership: a README naming three of
four would pass every containment check and still tell a consumer that a
format is refused when it is carried.
"""
from llm_ingestion_okf.assets import VIEWABLE_MEDIA_TYPES
match = _VIEWABLE_LINE.search(README.read_text(encoding="utf-8"))
assert match is not None, (
"README carries no `<!-- asset-viewable-media-types: ... -->` marker; without it "
"the set a consumer reads and the set the code applies can drift apart silently"
)
assert match.group(1).split(",") == sorted(VIEWABLE_MEDIA_TYPES)
_MAX_PIXELS_LINE = re.compile(r"^<!-- asset-max-pixels: (\d+) -->$", re.MULTILINE)

31
tools/okf_accounting_gate.py
View file

@ -361,6 +361,19 @@ def _sha12(path: Path) -> str:
return _sha256(path)[:12]
#: The conversion clause `assets.render_block` writes: the source's media type
#: and sha256, and the media type and sha256 of what the run actually carried.
#: One expression, so the judge has one definition of the claim it verifies.
_CONVERSION = re.compile(
r"converted from \S+ sha256:(?P<before>[0-9a-f]{64}) to \S+ sha256:(?P<after>[0-9a-f]{64})"
)
def _conversions(bundle_text: str) -> dict[str, str]:
"""source digest -> the digest the bundle says it carried instead."""
return {m.group("before"): m.group("after") for m in _CONVERSION.finditer(bundle_text)}
def asset_holds(build: Build, source: Path) -> bool:
"""Did the run carry THESE bytes, placed under their own content address?
@ -377,10 +390,26 @@ def asset_holds(build: Build, source: Path) -> bool:
by construction -- and it would be wrong: measured 2026-09-18 on R761,
whose own hrefs carry spaces, capitals and parentheses, a judge checking
the full name reported 50 of 50 carried images as missing.
A SECOND ROUTE, for an image the build CONVERTS. Since the viewable-asset
round a source in a format no model can be shown reaches the bundle as a
PNG, so its own bytes are not in `assets/` and never will be -- measured,
the day that landed R761 went from 0 to 19 claimed-and-not-found, which is
exactly its RLE8 BMP count. The bundle states both digests on the pointer
line, and this reads them and then HASHES THE ASSET ITSELF: the claim is
accepted only when a file in `assets/` really holds the bytes the bundle
says were written. A bundle claiming a conversion it did not perform still
fails, which is the difference between reading the bundle and believing
the report.
"""
digest = _sha256(source)
return any(
if any(
found == digest and name.startswith(digest[:12]) for name, found in build.assets.items()
):
return True
written = _conversions(build.bundle_text).get(digest)
return written is not None and any(
found == written and name.startswith(written[:12]) for name, found in build.assets.items()
)

172
tools/okf_asset_census.py Normal file
View file

@ -0,0 +1,172 @@
#!/usr/bin/env python3
"""What happens to every image this machine can reach, one row each.
The known-positive instrument for a change to the asset path. A round that
converts one format has to show that it moved NOTHING else, and "nothing else"
is a statement about a corpus, not about a fixture. This script produces the
per-image row that makes the claim checkable: the source's own sha256, the
format read off its bytes, and what `assets.read_image` did with it -- carried
under some digest, or refused with a code.
Run it from two pinned trees and diff the rows. A JPEG whose carried digest
moved is a regression with a name; a BMP whose carried digest moved from a
`.bmp` row to a `.png` row is the round working.
IT IS NOT PART OF THE PACKAGE. It lives outside `src/`, never enters a wheel,
and a consumer's install surface is unchanged by its existence.
Two rules it is built around, both learned in this repository:
- **A count with no denominator is not a measurement.** Every summary line
carries `of N`, and a run that found no image says so rather than printing
an empty table that reads like a clean result.
- **A failed read is never an empty result.** An unreadable file is its own
row with its own reason, never a silent absence from the census.
Usage:
python3 tools/okf_asset_census.py <path> [<path> ...] [--pdf] [--rows]
`<path>` is a file or a directory walked recursively. `--pdf` additionally
opens every `.pdf` found and runs the PDF image reader over it, which is the
slow half and needs the `[extract]` extra. `--rows` prints one line per image;
without it only the summary is printed.
"""
from __future__ import annotations
import argparse
import hashlib
import sys
from collections import Counter
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src"))
from llm_ingestion_okf import assets as assets_module
from llm_ingestion_okf.errors import ExtractionError
#: Suffixes worth opening as a standalone image. The type is still SNIFFED --
#: this only decides which files are picked up off the disk, and a file whose
#: name lies is reported under the format its bytes declare.
IMAGE_SUFFIXES = frozenset(
{".png", ".jpg", ".jpeg", ".gif", ".bmp", ".tif", ".tiff", ".webp", ".jp2", ".j2k"}
)
def _digest(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def _row(origin: str, data: bytes) -> tuple[str, str, str]:
"""`(origin, source digest, outcome)` for one image's bytes."""
before = _digest(data)
sniffed = assets_module.sniff(data)
kind = sniffed[0] if sniffed else "not-an-image"
try:
image = assets_module.read_image(data, name=origin)
except ExtractionError as exc:
return origin, before, f"REJECT {exc.code} in={kind}"
return (
origin,
before,
f"CARRY {image.media_type}{image.suffix} in={kind} out={_digest(image.data)}",
)
def _walk(paths: list[Path]) -> list[Path]:
found: list[Path] = []
for path in paths:
if path.is_file():
found.append(path)
elif path.is_dir():
found += [p for p in path.rglob("*") if p.is_file()]
else:
print(f"census: {path} is neither a file nor a directory", file=sys.stderr)
return sorted(found)
def _pdf_rows(path: Path) -> list[tuple[str, str, str]]:
"""Every image object the PDF reader reaches on this file.
The reader is asked for the images it CARRIES and the ones it refuses, so
a document whose pictures all fail is a set of coded rows and not a blank.
"""
import pdfplumber
from llm_ingestion_okf import extract as extract_module
rows: list[tuple[str, str, str]] = []
with pdfplumber.open(str(path)) as document:
for number, page in enumerate(document.pages, start=1):
# THE PAGE, not `page.page_obj`. `_pdf_images` reads `page.images`
# through `getattr(..., [])`, so handing it the wrong object
# returns an empty tuple and no error: measured 2026-09-19, that
# mistake reported 0 images over 67 PDFs and read like a clean
# result. The attribute is asserted rather than assumed.
if not hasattr(page, "images"):
raise RuntimeError(
f"{path}: the page object exposes no `images` attribute; this census "
"would report an absence it never measured"
)
carried, rejected = extract_module._pdf_images(page)
for image in carried:
rows.append(
(
f"{path}#p{number}:{image.name}",
_digest(image.data),
f"CARRY {image.media_type}{image.suffix} out={_digest(image.data)}",
)
)
for rejection in rejected:
rows.append((f"{path}#p{number}:{rejection.name}", "-", f"REJECT {rejection.code}"))
return rows
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("paths", nargs="+", type=Path)
parser.add_argument("--pdf", action="store_true", help="also open every .pdf found")
parser.add_argument("--rows", action="store_true", help="print one line per image")
args = parser.parse_args(argv)
files = _walk(args.paths)
rows: list[tuple[str, str, str]] = []
pdfs = 0
unreadable = 0
for path in files:
if path.suffix.lower() in IMAGE_SUFFIXES:
try:
data = path.read_bytes()
except OSError as exc:
unreadable += 1
rows.append((str(path), "-", f"UNREADABLE {exc.__class__.__name__}"))
continue
rows.append(_row(str(path), data))
elif args.pdf and path.suffix.lower() == ".pdf":
pdfs += 1
try:
rows += _pdf_rows(path)
except Exception as exc: # a broken PDF is a row, never a blank
unreadable += 1
rows.append((str(path), "-", f"UNREADABLE {exc.__class__.__name__}"))
if args.rows:
for origin, before, outcome in rows:
print(f"{before}\t{outcome}\t{origin}")
total = len(rows)
print(f"files walked: {len(files)}; pdfs opened: {pdfs}; image rows: {total}")
if total == 0:
print("NO IMAGE MEASURED -- this is an absence, not a clean result")
return 3
summary = Counter(outcome.split(" out=")[0] for _, _, outcome in rows)
for outcome, count in sorted(summary.items()):
print(f" {count} of {total}\t{outcome}")
carried = sum(count for outcome, count in summary.items() if outcome.startswith("CARRY"))
print(f"carried: {carried} of {total}; refused: {total - carried} of {total}")
return 0
if __name__ == "__main__":
raise SystemExit(main())