open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(assets): a bundle carries the images its sources declare (0.10.0)

Browse source 
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-17 10:01:31 +02:00
commit bc39e8091f
33 changed files with 3638 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

482
src/llm_ingestion_okf/assets.py Normal file
View file

@ -0,0 +1,482 @@
"""Binary assets: what an image IS, what it is called, and how it is pointed at.
Until 0.10.0 this package had no image path at all. Every reader recovered
text, every warning said so, and the only writer into a bundle was
:func:`materialize.write_bytes`, whose signature is ``(bundle_dir, name,
content: str)`` -- UTF-8 in, text out, no binary route anywhere. A document
whose table is a raster picture therefore reached a concept as an absence with
no denominator. Measured on R761 Prosesskoden:2025, the publisher's own
NISO-STS delivery: the process text is carried in full, and 12 ``Tabell N-N``
and 9 ``Figur N-N`` captions stand over nothing, so process 84's
"toleranseklasse ... er gitt i tabell 84-2" points at empty space.
THIS MODULE IS THE ONE PLACE THAT DECIDES WHAT AN IMAGE IS. A reader hands it
bytes and a name; it returns a carried image or raises a coded rejection. That
is what makes "N carried of M found" mean the same thing for ``pdf``, ``docx``,
``html`` and ``xml``, and it is what keeps each format's quirks out of the
bundle layout.
THREE RULES, and each one exists because the alternative is a silent lie:
- **The type is SNIFFED, never claimed.** Measured on the R761 delivery, the
graphics directory holds ``.bmp``, ``.jpg`` and ``.png`` side by side and the
document's ``xlink:href`` values are whatever the publisher's tool wrote. A
name is a claim; the magic bytes are the fact. A consumer dispatching on the
extension of a name that lies reads the file wrong with full confidence.
- **The name is CONTENT-ADDRESSED** -- ``<sha256[:12]>-<reduced original>``.
Two drops of one image are one file, a rebuild of one corpus is one bundle,
and the digest carries the uniqueness so the readable tail can be shortened
without any risk of collision. That is the byte-determinism rule this package
already holds for text, extended to the bytes beside it.
- **The pointer is ONE GRAMMAR**, owned here. ``okf describe`` (step 2) has to
find every pointer mechanically in order to write a transcription under it,
so the block is a regex this module ships beside the writer rather than a
shape each reader invents and each consumer re-derives.
WHAT THIS MODULE DOES NOT DO: it never looks at a picture. Classifying an image
as a table or a figure, and reading what it says, is a model call, and the
invariant "no model calls anywhere in the run path" is not negotiated here --
step 2 is a separate command, outside the build path, and this module is
importable without it.
**The image BYTES are not screened.** The guard is text-only (its own boundary,
not ours), so what passes a persist gate is the pointer block, as body text,
like every other line. The bytes of a carried image are written to the bundle
unscreened. Stated here rather than implied, because a consumer weighing an
untrusted drop needs to know which half of the concept was looked at.
"""
from __future__ import annotations
import hashlib
import re
import struct
import unicodedata
import zlib
from dataclasses import dataclass
from .errors import ExtractionError
#: The one directory a bundle keeps its binary assets in, at the bundle root.
#: Fixed rather than configurable: a consumer resolving `/assets/...` out of a
#: concept has only the bundle, and a per-profile directory would make that
#: pointer unresolvable without also shipping the profile that wrote it.
ASSETS_DIR = "assets"
#: How much of the digest names the file. 12 hex characters is 48 bits; over
#: the largest asset population measured here (4 828 image objects in one
#: 33-document corpus) the birthday probability of a collision is about
#: 4e-11. A collision would be caught anyway -- an occupied name is re-used
#: only when the bytes there are already identical, the same content-identity
#: rule Door C proves ownership with.
DIGEST_PREFIX = 12
#: How much of the original name survives into the asset name. The digest
#: carries uniqueness, so this is decoration and truncating it is safe -- which
#: is the opposite of `materialize.check_filename_length`'s situation, where the
#: name IS the identity and truncation would silently merge two documents.
NAME_TAIL_MAX = 60
#: Magic bytes -> (media type, suffix). Sniffed in this order; the first match
#: wins, and nothing here overlaps.
_MAGIC: tuple[tuple[bytes, str, str], ...] = (
(b"\x89PNG\r\n\x1a\n", "image/png", ".png"),
(b"\xff\xd8\xff", "image/jpeg", ".jpg"),
(b"GIF87a", "image/gif", ".gif"),
(b"GIF89a", "image/gif", ".gif"),
(b"BM", "image/bmp", ".bmp"),
(b"II\x2a\x00", "image/tiff", ".tiff"),
(b"MM\x00\x2a", "image/tiff", ".tiff"),
)
#: JPEG 2000, in both the forms a PDF `JPXDecode` stream hands back: the JP2
#: container and a bare codestream.
_JP2_SIGNATURE = b"\x00\x00\x00\x0cjP \r\n\x87\n"
_J2K_SIGNATURE = b"\xff\x4f\xff\x51"
#: The frame markers that carry a JPEG's dimensions. Every SOF except the four
#: that are not frame headers at all (`DHT` 0xC4, `JPG` 0xC8, `DAC` 0xCC).
_JPEG_SOF = frozenset(range(0xC0, 0xD0)) - {0xC4, 0xC8, 0xCC}
#: `materialize.reduce_to_id_grammar`'s rule, restated. Not imported: this
#: module is reached from `extract.py`, whose registry must not import the
#: contract layer, and `materialize` pulls in `manifest` and `profiles`.
#: `tests/test_assets.py` holds the two forms equal on the same inputs, so the
#: restatement cannot drift into a second grammar.
_SEPARATOR_RUN = re.compile(r"[^a-z0-9]+")
#: One pointer block, as this module writes it. Group 1 is the label, group 2
#: is the asset file name, group 3 is the whole second line. `okf describe`
#: finds its work with this and writes under the match; a consumer wanting to
#: strip pointers uses the same expression, so there is one definition of what
#: a pointer looks like rather than one per reader.
IMAGE_POINTER = re.compile(
r"^!\[(?P<label>[^\]\n]*)\]\(/" + ASSETS_DIR + r"/(?P<asset>[^)\s]+)\)\n"
r"(?P<detail>Image: [^\n]*)$",
re.MULTILINE,
)
@dataclass(frozen=True)
class ExtractedImage:
"""One image a document carries, as this package will write it.
`name` is what the SOURCE calls the file -- a `xlink:href`, an `<img src>`,
a media entry inside an OOXML container, or a synthesised name for a PDF
image object, which has none. It is carried verbatim into the pointer's
second line and reduced (never trusted) into the asset's own name.
`label` is a caption or alt text where the format HAS one, and `None`
where it does not. Two of the four readers have none: a PDF image object
and an STS `<graphic>` carry no caption element, and the caption a human
sees is running text the extractor already emitted on its own line.
Deriving a label from the nearest line would be an unmarked heuristic,
which this package treats as worse than no heuristic at all.
"""
data: bytes
name: str
media_type: str
suffix: str
width: int | None
height: int | None
label: str | None = None
@dataclass(frozen=True)
class AssetRejection:
"""An image that was FOUND and not carried, with the reason as a code.
A rejection is a row in the run log, never a failed document: one
unreadable picture must not cost the 3 000 concepts of text around it. The
denominator is what makes the log worth reading -- "51 carried" says
nothing without "of 53 found".
"""
name: str
code: str
reason: str
def sniff(data: bytes) -> tuple[str, str] | None:
"""`(media type, suffix)` read from the bytes, or `None` if not an image.
The claimed extension never participates. A `.jpg` that is really a PNG is
carried as a PNG under a `.png` name, because the alternative is a bundle
whose file names disagree with their contents.
"""
if data.startswith(_JP2_SIGNATURE) or data.startswith(_J2K_SIGNATURE):
return "image/jp2", ".jp2"
for magic, media_type, suffix in _MAGIC:
if data.startswith(magic):
return media_type, suffix
return None
def dimensions(data: bytes) -> tuple[int, int] | None:
"""`(width, height)` in pixels from the file's own header, or `None`.
`None` is a MEASUREMENT: this reader does not read every format's geometry
(JPEG 2000 and TIFF are absent), and a concept that printed `0x0 px` would
be stating a number nobody took. The pointer says "dimensions unknown"
instead.
"""
kind = sniff(data)
if kind is None:
return None
suffix = kind[1]
try:
if suffix == ".png":
if len(data) < 24 or data[12:16] != b"IHDR":
return None
width, height = struct.unpack(">II", data[16:24])
return (width, height) if width and height else None
if suffix == ".jpg":
return _jpeg_dimensions(data)
if suffix == ".gif":
if len(data) < 10:
return None
width, height = struct.unpack("<HH", data[6:10])
return (width, height) if width and height else None
if suffix == ".bmp":
return _bmp_dimensions(data)
except (struct.error, IndexError):
return None
return None
def _jpeg_dimensions(data: bytes) -> tuple[int, int] | None:
position = 2
end = len(data)
while position + 3 < end:
if data[position] != 0xFF:
position += 1
continue
marker = data[position + 1]
if marker in (0xD8, 0x01) or 0xD0 <= marker <= 0xD7:
position += 2
continue
if marker == 0xFF:
position += 1
continue
length = struct.unpack(">H", data[position + 2 : position + 4])[0]
if marker in _JPEG_SOF:
if position + 9 > end:
return None
height, width = struct.unpack(">HH", data[position + 5 : position + 9])
return (width, height) if width and height else None
position += 2 + length
return None
def _bmp_dimensions(data: bytes) -> tuple[int, int] | None:
if len(data) < 26:
return None
header_size = struct.unpack("<I", data[14:18])[0]
if header_size == 12:
width, height = struct.unpack("<hh", data[18:22])
else:
width, height = struct.unpack("<ii", data[18:26])
# A negative height is a top-down BMP; the magnitude is the pixel count.
return (abs(width), abs(height)) if width and height else None
def read_image(data: bytes, *, name: str, label: str | None = None) -> ExtractedImage:
"""One image, typed by its bytes, or a coded refusal.
Raises :class:`ExtractionError` with `asset_type_unknown` when the bytes
are not an image this package recognises. The caller records that as an
:class:`AssetRejection` and keeps going -- a document is not lost over one
picture.
"""
kind = sniff(data)
if kind is None:
raise ExtractionError(
f"the bytes behind {name!r} are not an image format this package "
f"recognises (first bytes {data[:8]!r})",
code="asset_type_unknown",
)
media_type, suffix = kind
size = dimensions(data)
return ExtractedImage(
data=data,
name=name,
media_type=media_type,
suffix=suffix,
width=size[0] if size else None,
height=size[1] if size else None,
label=label,
)
def digest(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def _reduce(text: str) -> str:
return _SEPARATOR_RUN.sub("-", unicodedata.normalize("NFC", text).lower()).strip("-")
def asset_name(image: ExtractedImage) -> str:
"""`<sha256[:12]>-<reduced original BASENAME><sniffed suffix>`.
DEDUP IS ON CONTENT, and the digest is what makes it so: the same bytes
arriving twice are one file, whichever document dropped them. The residual
is stated rather than claimed away -- identical bytes shipped under two
different base names are two files in the bundle, both correct and both
holding the same picture. Measured on the fixture inbox and on the R761
delivery, that case occurs 0 times; a rule that could not produce it at all
would have to drop the readable tail entirely, and the tail is what makes
`assets/` legible to the person checking a bundle by hand.
The suffix comes from the bytes and the stem from the name, reduced to the
same grammar every generated filename in this package uses. A link target
sits inside `](...)`, which a space or a closing parenthesis terminates --
and the R761 delivery's own hrefs carry both (`25-0143 - Tabeller -
Prosesskoden (R761-R762).jpg`), so reducing is what makes the pointer
followable rather than merely tidy.
"""
# The BASENAME, never the path the document reached it through. Measured
# on the fixture inbox: one image pointed at as `graphics/figur-84-1.png`
# from an HTML document and as `figur-84-1.png` from an STS one was written
# twice, under two names, in one run -- with the digest in both announcing
# that the bytes were identical. The path is a property of the pointer, not
# of the picture, and the full original survives on the pointer's own line.
base = image.name.rsplit("/", 1)[-1]
stem = _reduce(base.rsplit(".", 1)[0] if "." in base else base)
head = digest(image.data)[:DIGEST_PREFIX]
if not stem:
return f"{head}{image.suffix}"
return f"{head}-{stem[:NAME_TAIL_MAX].rstrip('-')}{image.suffix}"
def asset_href(image: ExtractedImage) -> str:
"""The bundle-absolute path SPEC SS 6.2 allows.
Absolute rather than relative because a segmented bundle puts concepts at
different depths: `assets/x.png` resolves to two different places from two
concepts of one document, and `/assets/x.png` to one place from every
concept in the bundle.
"""
return f"/{ASSETS_DIR}/{asset_name(image)}"
def _inline(value: str) -> str:
"""A label, made safe for the one line it is written on.
`[` and `]` are the link grammar's own delimiters and a newline would open
a third line in a two-line block, so both are replaced rather than escaped:
Door B refuses a title containing a bracket outright (`inbox_title_invalid`)
and this text reaches a title through no route, but the pointer is body text
a proposer reads, and a half-open link there is a pointer that resolves
nowhere.
"""
collapsed = " ".join(value.split())
return collapsed.replace("[", "(").replace("]", ")")
def render_block(image: ExtractedImage) -> str:
"""The two lines that stand where the image stands.
Line one is markdown, so a reader that renders the concept sees the picture
and a reader that does not sees the label. Line two states what the first
line cannot: the name the SOURCE gave the file, and the size in pixels --
the two facts a person checking the bundle against the original needs, and
the two a transcription in step 2 has to be judged against.
"""
label = image.label or image.name or asset_name(image)
size = (
f"{image.width}x{image.height} px"
if image.width is not None and image.height is not None
else "dimensions unknown"
)
detail = f"Image: {_inline(image.name or asset_name(image))} ({size})"
if image.label:
detail += f" -- {_inline(image.label)}"
return f"![{_inline(label)}]({asset_href(image)})\n{detail}"
def encode_png(
width: int,
height: int,
samples: bytes,
*,
channels: int,
palette: bytes | None = None,
alpha: bytes | None = None,
) -> bytes:
"""8-bit samples as a PNG, using nothing but `zlib`.
A PDF image is usually not a file: `FlateDecode` hands back raw samples
with the colour model in the dictionary beside them, so carrying one at all
means encoding it. Doing that with the stdlib rather than with a renderer
is what keeps the `pdf` row free of a new dependency AND keeps an asset's
bytes -- and therefore its content-addressed name -- independent of which
version of a rasteriser happened to be installed. `OCR_DPI`'s docstring
states the opposite property for OCR text, and the difference is deliberate.
`channels` is 1 (grey, or indexed when `palette` is given) or 3 (RGB).
`alpha` is one byte per pixel, from a PDF `SMask`; absent means opaque.
"""
if channels not in (1, 3):
raise ExtractionError(
f"PNG encoding supports 1 or 3 channels, not {channels}",
code="asset_samples_invalid",
)
if width <= 0 or height <= 0:
raise ExtractionError(
f"PNG encoding needs positive dimensions, got {width}x{height}",
code="asset_samples_invalid",
)
expected = width * height * channels
if len(samples) < expected:
raise ExtractionError(
f"the sample buffer holds {len(samples)} bytes where {width}x{height} "
f"at {channels} channels needs {expected} -- refusing to pad, because a "
"short buffer means the image dictionary was read wrong",
code="asset_samples_invalid",
)
if alpha is not None and len(alpha) < width * height:
raise ExtractionError(
f"the alpha buffer holds {len(alpha)} bytes where {width}x{height} needs "
f"{width * height}",
code="asset_samples_invalid",
)
if palette is not None:
if channels != 1:
raise ExtractionError(
"a palette applies to single-channel samples only",
code="asset_samples_invalid",
)
if not palette or len(palette) % 3:
raise ExtractionError(
f"a palette must be whole RGB triples, got {len(palette)} bytes",
code="asset_samples_invalid",
)
colour_type = 3
elif alpha is not None:
colour_type = 6 if channels == 3 else 4
else:
colour_type = 2 if channels == 3 else 0
rows = bytearray()
for row in range(height):
# Filter type 0 (None) on every row. A filter would shrink the file and
# would make the bytes depend on a heuristic; this encoder's output has
# to be reproducible from the samples alone, for as long as the bundle
# is quoted by its digest.
rows.append(0)
start = row * width * channels
line = samples[start : start + width * channels]
if alpha is None:
rows += line
else:
for pixel in range(width):
rows += line[pixel * channels : (pixel + 1) * channels]
rows.append(alpha[row * width + pixel])
def chunk(kind: bytes, payload: bytes) -> bytes:
return (
len(payload).to_bytes(4, "big")
+ kind
+ payload
+ zlib.crc32(kind + payload).to_bytes(4, "big")
)
ihdr = struct.pack(">IIBBBBB", width, height, 8, colour_type, 0, 0, 0)
body = b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", ihdr)
if palette is not None:
body += chunk(b"PLTE", palette)
# Level 9 with no filtering: `zlib` is stdlib and its output for a given
# level is stable within a Python build, which is the same guarantee the
# rest of this package's byte-determinism rests on.
body += chunk(b"IDAT", zlib.compress(bytes(rows), 9))
return body + chunk(b"IEND", b"")
def render_missing(
name: str,
*,
reason: str,
label: str | None = None,
href: str | None = None,
) -> str:
"""A pointer to an image the bundle does NOT carry, and why.
An image this package found and could not carry is stated in the concept,
not dropped. The reader of the bundle is the person who has to decide
whether the missing picture mattered, and they cannot decide about an
absence they were never shown -- which is precisely the shape of the defect
this whole capability exists to close.
The href is kept when the source had one and it survives the link grammar,
so a remote figure says WHERE it was. Extraction never opens a socket: the
network gate is an explicit per-run opt-in and extraction is not on that
path, so a remote source is carried as a name and never as bytes.
"""
shown = _inline(name or "image")
detail = f"Image: {shown} (not carried: {_inline(reason)})"
if href and not re.search(r"[\s()\[\]]", href):
return f"![{_inline(label or name or 'image')}]({href})\n{detail}"
return detail