open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(assets): every carried image is one a model can be shown

Browse source 
Chosen: a stdlib BMP reader, because `read_image` is on the CORE path and an
asset's name is its content digest. Measured first, as the order requires:
Pillow 12.3.0 IS in this tree (transitively under `pdfplumber`) and it DOES
decode RLE8 correctly -- a hand-written stdlib decoder and Pillow agree on
19 of 19 of R761's real files, RGB per pixel. So the choice does not rest on
capability. It rests on two properties of this package: `.html` and `.xml`
carry images with no `[extract]` extra installed, so a Pillow converter
either makes a core path depend on an optional binary wheel or buys the
second runtime dependency; and encoding through an installed library would
make a bundle's identity move with that library's version, which is the
property 0.10.0 felled page rasterisation over and `encode_png`'s docstring
already defends. Pillow keeps the job it is good for: the INDEPENDENT decoder
in the tests, on neither side of the conversion.

The defect, measured over the frozen R761 delivery's `assets/`, denominator
50: 29 JPEG, 2 PNG and 19 RLE8 BMP. The 19 are byte-correct files nothing
reads, so 19 figures were present and invisible while `images: N` reported
that they had arrived.

- `VIEWABLE_MEDIA_TYPES` is tested against every asset's SNIFFED type, so it
  is a property and not a list of formats we met. WebP is on it and `sniff`
  does not recognise one; the limit is stated, not implied.
- `bmp_to_png`: 8-bit uncompressed, 8-bit RLE8, 24-bit uncompressed. All five
  RLE8 opcodes. 19 of 19 real files convert with RGB identical to Pillow's
  decoding of the source, 2 366 365 pixels compared.
- `asset_not_viewable` and `asset_bmp_unsupported`, both published, both
  leaving the concept's "not carried" line.
- Traceability on the pointer's second line, where the rest of the asset
  metadata already lives: original media type, original sha256 in full, new
  sha256 in full. A converted asset is ONE asset.
- The ceiling is paid on the DECLARATION before a row is allocated, and an
  RLE run is one clipped slice -- painting pixel by pixel leaves the memory
  bounded and the CPU unbounded.

Two repairs the change forced, each measured rather than assumed:

- `tests/test_assets.py`'s "dimensions absent is absent" used a TIFF, which
  is now refused before `read_image` returns. The property still has a
  reachable case -- a JPEG whose frame header never arrives -- and uses it.
- `asset_holds` in the accounting gate proved a carry by hashing the SOURCE
  file, which a converted image's bundle cannot satisfy. It now also reads
  the two digests the bundle states and HASHES THE ASSET ITSELF, so a bundle
  claiming a conversion it did not perform still fails.

`tools/okf_asset_census.py` is the committed instrument for the
known-positive: one row per image, from two pinned trees. It was caught by
the rule it serves -- its first version handed `_pdf_images` the wrong page
object and reported 0 images over 67 PDFs with exit 0. The attribute is
asserted now and a known-positive runs before the sweep.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 08:07:04 +02:00
commit b0b5e71658
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
10 changed files with 826 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

315
src/llm_ingestion_okf/assets.py
View file

@ -90,6 +90,27 @@ _MAGIC: tuple[tuple[bytes, str, str], ...] = (
(b"MM\x00\x2a", "image/tiff", ".tiff"),
)
#: The formats a model can be SHOWN. Everything a document ships outside this
#: set is converted losslessly to PNG, or refused with a code -- never carried
#: silently, which is what this package did until this round of 0.10.1.
#:
#: MEASURED 2026-09-19 over the frozen R761 delivery's own `assets/`
#: (denominator 50): 29 JPEG, 2 PNG and **19 "PC bitmap, Windows 3.x, 8-bit,
#: compression 1"**. The 19 are byte-correct files that nothing reads, so 19 of
#: that document's figures were present and invisible at the same time -- and
#: the `images: N` count said they had arrived. An absence a reader is shown is
#: information; a picture that is there and unreadable is worse than either.
#:
#: IT IS A PROPERTY, NOT A LIST OF FORMATS WE HAPPENED TO MEET. A carried
#: asset's type is read off its bytes and tested against this set, so a format
#: nobody here has seen is refused by the same rule that refuses TIFF.
#:
#: WebP is on the list and `sniff` does not recognise it: the set states what a
#: model can be shown, not what this package can read. A WebP is therefore
#: refused one step earlier, as `asset_type_unknown`, and never reaches this
#: test. Stating that is cheaper than a set whose name is wider than its reach.
VIEWABLE_MEDIA_TYPES = frozenset({"image/png", "image/jpeg", "image/gif", "image/webp"})
#: JPEG 2000, in both the forms a PDF `JPXDecode` stream hands back: the JP2
#: container and a bare codestream.
_JP2_SIGNATURE = b"\x00\x00\x00\x0cjP \r\n\x87\n"
@ -403,6 +424,16 @@ class ExtractedImage:
sees is running text the extractor already emitted on its own line.
Deriving a label from the nearest line would be an unmarked heuristic,
which this package treats as worse than no heuristic at all.
`converted_from` and `source_sha256` are set when the source was NOT a
format a model can be shown and this package rewrote it (today: a BMP, as
a PNG). They are the whole of the traceability: `converted_from` is the
media type the file had, `source_sha256` is the digest of the bytes the
document actually shipped, and the digest of `data` is what the bundle
holds. With the three of them a reader can take the original delivery,
run `shasum -a 256`, and find the row. Both are `None` for an image
carried verbatim, which is every image this package has ever carried
until now -- so a bundle of JPEGs is byte-identical across the move.
"""
data: bytes
@ -412,6 +443,8 @@ class ExtractedImage:
width: int | None
height: int | None
label: str | None = None
converted_from: str | None = None
source_sha256: str | None = None
@dataclass(frozen=True)
@ -512,6 +545,266 @@ def _bmp_dimensions(data: bytes) -> tuple[int, int] | None:
return (abs(width), abs(height)) if width and height else None
#: `biCompression`: uncompressed, and the 8-bit run-length encoding 19 of
#: R761's 50 assets use. Every other value -- RLE4, BITFIELDS, embedded JPEG or
#: PNG -- is refused by name rather than guessed at.
_BMP_RGB = 0
_BMP_RLE8 = 1
@dataclass(frozen=True)
class _BmpHeader:
"""What the DIB header declares, before anything is decoded."""
header_size: int
width: int
height: int
top_down: bool
bits: int
compression: int
entries: int
offbits: int
def _bmp_header(data: bytes, *, name: str) -> _BmpHeader:
"""The declaration, validated -- never the pixels.
Everything this returns is a CLAIM by an untrusted file. It is read first
precisely so the claim can be bounded before the cost of believing it is
paid: the row buffers below are `width * height` bytes, and that product
comes from these two fields.
"""
if len(data) < 54:
raise ExtractionError(
f"the BMP {name!r} stops inside its own header ({len(data)} bytes)",
code="asset_samples_invalid",
)
offbits = struct.unpack("<I", data[10:14])[0]
header_size = struct.unpack("<I", data[14:18])[0]
if header_size < 40:
raise ExtractionError(
f"the BMP {name!r} carries a {header_size}-byte DIB header; this reader "
"expresses BITMAPINFOHEADER and its extensions, and a 12-byte "
"BITMAPCOREHEADER stores its palette in 3-byte entries, which is a "
"different format wearing the same magic",
code="asset_bmp_unsupported",
)
if len(data) < 14 + header_size:
raise ExtractionError(
f"the BMP {name!r} declares a {header_size}-byte DIB header and holds "
f"{len(data) - 14} bytes after the file header",
code="asset_samples_invalid",
)
width, height = struct.unpack("<ii", data[18:26])
bits = struct.unpack("<H", data[28:30])[0]
compression = struct.unpack("<I", data[30:34])[0]
clr_used = struct.unpack("<I", data[46:50])[0]
if width <= 0 or height == 0:
raise ExtractionError(
f"the BMP {name!r} declares {width}x{height}, which is not a size",
code="asset_size_invalid",
)
top_down = height < 0
entries = clr_used or (1 << bits if bits <= 8 else 0)
if entries > 256:
raise ExtractionError(
f"the BMP {name!r} declares {entries} palette entries; a PNG palette holds at most 256",
code="asset_bmp_unsupported",
)
return _BmpHeader(
header_size=header_size,
width=width,
height=-height if top_down else height,
top_down=top_down,
bits=bits,
compression=compression,
entries=entries,
offbits=offbits,
)
def _bmp_palette(data: bytes, head: _BmpHeader, *, name: str) -> bytes:
"""The colour table as PNG wants it: RGB triples, from the file's BGRA.
The order matters and getting it wrong is invisible in every structural
check: a palette read as RGB gives a picture with red and blue swapped,
the right size, the right number of colours, and the wrong answer.
"""
entries = head.entries or 256
start = 14 + head.header_size
table = data[start : start + entries * 4]
if len(table) < entries * 4:
raise ExtractionError(
f"the BMP {name!r} declares {entries} palette entries and holds {len(table) // 4}",
code="asset_samples_invalid",
)
palette = bytearray()
for index in range(entries):
blue, green, red = table[index * 4], table[index * 4 + 1], table[index * 4 + 2]
palette += bytes((red, green, blue))
return bytes(palette)
def _bmp_flat_rows(data: bytes, head: _BmpHeader, *, per_pixel: int, name: str) -> bytes:
"""Uncompressed rows, unpadded and in top-down order.
A BMP row is padded to a 4-byte boundary and stored bottom-up unless the
declared height is negative. Both are read off the header rather than
assumed, because either mistake produces a picture that is the right size
and is sheared or upside down.
"""
stride = ((head.width * per_pixel + 3) // 4) * 4
needed = stride * head.height
available = len(data) - head.offbits
if available < needed:
raise ExtractionError(
f"the BMP {name!r} needs {needed} bytes of pixel data and holds {available} "
"-- refusing to pad, because a short buffer means the header was read wrong",
code="asset_samples_invalid",
)
rows = [
data[head.offbits + index * stride : head.offbits + index * stride + head.width * per_pixel]
for index in range(head.height)
]
if not head.top_down:
rows.reverse()
return b"".join(rows)
def _bmp_rle8_rows(data: bytes, head: _BmpHeader) -> bytes:
"""The RLE8 opcode stream painted into a frame of the DECLARED size.
Five opcodes, and a decoder implementing only the first is wrong on real
files: an encoded run, an absolute run (whose byte count is padded to an
even length), a delta that SKIPS pixels, end-of-line and end-of-bitmap.
Skipped pixels keep index 0, which is what the format says and what an
independent decoder produces.
THE COST IS THE FRAME, NOT THE STREAM. The rows are allocated once from the
declared size -- already bounded by `check_size` before this is called --
and each run is written as one CLIPPED slice. Painting pixel by pixel would
leave the memory bounded and the CPU unbounded: a megabyte of `FF` runs is
a hundred million paint steps against a frame of 32 pixels.
"""
width, height = head.width, head.height
rows = [bytearray(width) for _ in range(height)]
position = head.offbits
end = len(data)
x = 0
y = 0
while position + 1 < end:
count = data[position]
value = data[position + 1]
position += 2
if count:
if 0 <= y < height and x < width:
stop = min(x + count, width)
rows[y][x:stop] = bytes((value,)) * (stop - x)
x += count
continue
if value == 0:
x = 0
y += 1
elif value == 1:
break
elif value == 2:
if position + 2 > end:
break
x += data[position]
y += data[position + 1]
position += 2
else:
run = data[position : position + value]
position += value + (value & 1)
if 0 <= y < height and x < width:
stop = min(x + len(run), width)
rows[y][x:stop] = run[: stop - x]
x += value
if not head.top_down:
rows.reverse()
return b"".join(bytes(row) for row in rows)
def bmp_to_png(data: bytes, *, name: str) -> bytes:
"""A BMP as a PNG with the same pixels, using nothing but the stdlib.
WHY A READER HERE AND NOT PILLOW, which this tree already carries
transitively under `pdfplumber`. Two reasons, measured rather than
preferred. First, `read_image` is on the CORE path: `.html` and `.xml` are
stdlib file types that carry images with no `[extract]` extra installed, so
a Pillow-based converter would either make a core path depend on an
optional binary wheel or buy this package its second runtime dependency.
Second, and decisive: an asset's name is its content digest, so the bytes
this function emits are part of the bundle's identity. Encoding through an
installed library would make that identity move with the library's version
-- the exact property 0.10.0 felled page rasterisation over. `encode_png`
already writes a PNG from samples with `zlib` alone; this adds the reader in
front of it.
Pillow is still the INDEPENDENT decoder in the tests, which is the job it
is good for here: 19 of 19 of R761's real RLE8 assets decode to identical
RGB through both paths (measured 2026-09-19, before this was written).
"""
head = _bmp_header(data, name=name)
channels = 3 if head.bits == 24 else 1
# THE CEILING FIRST, on the DECLARATION, before one row is allocated. The
# rows below are `width * height` bytes of an untrusted document's claim.
check_size(head.width, head.height, name=name, channels=channels, bits=8)
if head.bits == 8 and head.compression in (_BMP_RGB, _BMP_RLE8):
palette = _bmp_palette(data, head, name=name)
if head.compression == _BMP_RLE8:
if head.top_down:
raise ExtractionError(
f"the BMP {name!r} declares a top-down RLE8 image, which the format "
"does not define",
code="asset_bmp_unsupported",
)
samples = _bmp_rle8_rows(data, head)
else:
samples = _bmp_flat_rows(data, head, per_pixel=1, name=name)
limit = len(palette) // 3
if samples and max(samples) >= limit:
raise ExtractionError(
f"the BMP {name!r} uses palette index {max(samples)} and declares {limit} "
"entries; carrying it would invent a colour",
code="asset_samples_invalid",
)
return encode_png(head.width, head.height, samples, channels=1, palette=palette)
if head.bits == 24 and head.compression == _BMP_RGB:
raw = _bmp_flat_rows(data, head, per_pixel=3, name=name)
swapped = bytearray(raw)
# BGR on disk, RGB in a PNG. Two slices rather than a loop, and read
# from `raw` both times so the first assignment cannot feed the second.
swapped[0::3] = raw[2::3]
swapped[2::3] = raw[0::3]
return encode_png(head.width, head.height, bytes(swapped), channels=3)
raise ExtractionError(
f"the BMP {name!r} stores {head.bits}-bit samples under compression "
f"{head.compression}; this reader expresses 8-bit (uncompressed and RLE8) and "
"24-bit uncompressed, and will not guess at the rest",
code="asset_bmp_unsupported",
)
#: Source media type -> the function that makes a viewable file of it. A map
#: rather than a branch, so what this package can convert is one readable line
#: and adding a format is adding a row.
_CONVERTERS = {"image/bmp": bmp_to_png}
def to_viewable(data: bytes, *, media_type: str, name: str) -> bytes:
"""Bytes a model can be shown, or a coded refusal. Never a silent carry."""
convert = _CONVERTERS.get(media_type)
if convert is None:
raise ExtractionError(
f"the image {name!r} is {media_type}, which no model can be shown, and this "
"package has no lossless conversion for it; refused rather than carried in a "
"format nothing reads",
code="asset_not_viewable",
)
return convert(data, name=name)
def read_image(data: bytes, *, name: str, label: str | None = None) -> ExtractedImage:
"""One image, typed by its bytes, or a coded refusal.
@ -528,6 +821,16 @@ def read_image(data: bytes, *, name: str, label: str | None = None) -> Extracted
code="asset_type_unknown",
)
media_type, suffix = kind
# THE VIEWABILITY GATE, and it stands before the size is read because the
# size that matters is the one the CARRIED file has. A BMP that becomes a
# PNG is measured as the PNG a consumer will open.
converted_from: str | None = None
source_sha256: str | None = None
if media_type not in VIEWABLE_MEDIA_TYPES:
converted_from = media_type
source_sha256 = digest(data)
data = to_viewable(data, media_type=media_type, name=name)
media_type, suffix = "image/png", ".png"
size = dimensions(data)
# THE BOUND HOLDS FOR A FILE CARRIED VERBATIM TOO. This package does not
# decode one, so it pays nothing for it -- but writing a 7 000 x 7 000 PNG
@ -547,6 +850,8 @@ def read_image(data: bytes, *, name: str, label: str | None = None) -> Extracted
width=size[0] if size else None,
height=size[1] if size else None,
label=label,
converted_from=converted_from,
source_sha256=source_sha256,
)
@ -635,6 +940,16 @@ def render_block(image: ExtractedImage) -> str:
detail = f"Image: {_inline(image.name or asset_name(image))} ({size})"
if image.label:
detail += f" -- {_inline(image.label)}"
# WHAT THE CONVERSION DID, on the line the rest of the asset metadata is
# already on. Both digests in full: the asset's file name carries only the
# first 12 hex characters of the new one, and a checksum a reader cannot
# paste into `shasum -a 256` is decoration. Written LAST so a labelled
# image that was not converted keeps the bytes it has today.
if image.converted_from and image.source_sha256:
detail += (
f" -- converted from {image.converted_from} sha256:{image.source_sha256}"
f" to {image.media_type} sha256:{digest(image.data)}"
)
return f"![{_inline(label)}]({asset_href(image)})\n{detail}"

14
src/llm_ingestion_okf/errors.py
View file

@ -151,6 +151,20 @@ class ExtractionError(IngestError):
statistic about the first untrue. Refused before the stream is read: a
negative dimension multiplies to a negative pixel count, under which
every bound reads as satisfied
- `asset_not_viewable` — the bytes are a real image in a format no model
can be SHOWN (TIFF, JPEG 2000), and this package has no lossless
conversion for it. DISTINCT from `asset_type_unknown`, which says the
bytes are not an image at all: this one says they are, and carrying them
would put a file in the bundle that the `images: N` count reports as
arrived and nothing downstream can read. Measured 2026-09-19 on the
frozen R761 delivery: 19 of its 50 assets were carried in exactly that
condition, as RLE8 BMP
- `asset_bmp_unsupported` — a BMP variant this reader does not express
(RLE4, BITFIELDS, 16- or 32-bit samples, a 12-byte BITMAPCOREHEADER, a
palette over 256 entries). DISTINCT from `asset_not_viewable`, which
says there is no conversion route for the format at all: this one says
there is one and this file is outside it, which is a different fact
about the document and a different thing to go and fix
- `asset_pdf_unbounded` — the image is reached through a PDF stream filter
this package has no measured cost ratio for (`LZWDecode`,
`RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything unknown), or