open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(assets): a bundle carries the images its sources declare (0.10.0)

Browse source 
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-17 10:01:31 +02:00
commit bc39e8091f
33 changed files with 3638 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

118
src/llm_ingestion_okf/inbox.py
View file

@ -27,8 +27,23 @@ from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass, replace
from pathlib import Path, PurePosixPath
from .assets import (
ASSETS_DIR,
IMAGE_POINTER,
AssetRejection,
ExtractedImage,
asset_name,
)
from .connectors import safe_resolve
from .errors import IngestError, MaterializationError, SegmentationError, SourceError
from .extract import DeclaredIdentity, SourceUnits, declared_identity, extract_text, source_units
from .extract import (
DeclaredIdentity,
SourceUnits,
declared_identity,
directory_resolver,
extract_document,
source_units,
)
from .materialize import (
_render_root_frontmatter,
check_filename_length,
@ -39,6 +54,7 @@ from .materialize import (
write_bytes,
)
from .profiles import (
ASSET_COUNT_KEY,
DEFAULT,
BundleProfile,
IndexEntry,
@ -210,6 +226,19 @@ def render_inbox_concept(
code="inbox_source_file_invalid",
)
# HOW MANY ASSET POINTERS THIS CONCEPT CARRIES, counted out of the concept's
# own text rather than threaded down from the extraction. Two reasons, and
# the second is the load-bearing one: a segmented document's images belong
# to the segments whose spans hold them, so a document-level total would be
# written onto every segment and be wrong on all but one of them; and a
# count a reader can verify from the file in front of them is a different
# kind of fact from a count only the producer could have known.
#
# Named on this repository's own profiles only, so a bundle written under
# `DEFAULT` or `STRICT_V1` keeps exactly the key set its contract names.
assets_carried = (
len(IMAGE_POINTER.findall(text)) if ASSET_COUNT_KEY in profile.frontmatter.order else 0
)
frontmatter = {
"type": okf_type,
"title": title,
@ -287,6 +316,11 @@ def render_inbox_concept(
title=source_title,
)
)
if assets_carried:
# Conditional, like `req_number`: absent is "this concept carries no
# image", which is what every bundle written before 0.10.0 says, so a
# corpus without pictures is byte-identical across the move.
frontmatter[ASSET_COUNT_KEY] = str(assets_carried)
if concept_frontmatter_values:
# LAST, and the position is the precedence: a value the caller states
# for the run beats what the document declares, which beats the file
@ -592,6 +626,13 @@ class InboxResult:
# consumer's four buckets keep their meaning: a skipped directory holds no
# dropped FILE outcome, it explains a set of files that were never dropped.
skipped: tuple[SkippedPath, ...] = ()
# THE ASSET DENOMINATOR (0.10.0). `assets` is what reached the bundle;
# `assets_rejected` is what was found and could not be. Both, or neither
# number means anything: "51 carried" is a measurement only beside "of 53
# found", and a run whose figures were all refused would otherwise look
# exactly like a run over documents that had none.
assets: tuple[str, ...] = ()
assets_rejected: tuple[AssetRejection, ...] = ()
def relative_source(path: Path, inbox: Path) -> str:
@ -970,6 +1011,7 @@ def process_inbox(
pdf_headings: bool = False,
heading_reserve: Callable[[str], bool] | None = None,
ocr: bool = False,
assets: bool = False,
concept_frontmatter_values: Mapping[str, str] | None = None,
) -> InboxResult:
"""Convert every file dropped in `inbox_dir` into an OKF concept.
@ -1037,6 +1079,14 @@ def process_inbox(
quarantined: list[BlockedFile] = []
rejected: list[BlockedFile] = []
failed: list[FailedFile] = []
# Keyed by asset name, so one image dropped by two documents is one entry
# and the bundle holds one file. The bytes are kept until the write, which
# happens per document AFTER that document's gate decision -- an image
# belonging to a document the guard refused must not be left behind in
# `assets/`, where nothing would ever point at it and nothing would ever
# retire it.
carried_assets: dict[str, bytes] = {}
refused_assets: list[AssetRejection] = []
# Phase 1: name every file BEFORE any gate call or write, so an intra-run
# collision is caught while both files can still be refused together. Under
@ -1184,13 +1234,23 @@ def process_inbox(
continue
outputs: list[tuple[str, str, tuple[str, ...]]] = []
try:
text = extract_text(
# The resolver is rooted at the DOCUMENT's own directory, which is
# the same root `propose.propose_segments` computes from the file it
# reads off disk. One root both sides derive independently is what
# makes the two renderings identical -- and a plan indexes the exact
# string it was proposed against, so a resolver that disagreed would
# turn every document carrying a pointer into a coded rejection.
resolve = directory_resolver(path.parent) if assets else None
document = extract_document(
source_name(path),
source_bytes,
renderer=_resolve_renderer(profile, path.name),
pdf_headings=pdf_headings,
ocr=ocr,
assets=assets,
resolve=resolve,
)
text = document.text
# The heading RESERVE, supplied as a predicate rather than decided
# here: the condition is the proposer's outline grammar, and the
# door does not own that grammar. A callable keeps the dependency
@ -1201,13 +1261,16 @@ def process_inbox(
reading_fonts = pdf_headings
if heading_reserve is not None and not pdf_headings and heading_reserve(text):
reading_fonts = True
text = extract_text(
document = extract_document(
source_name(path),
source_bytes,
renderer=_resolve_renderer(profile, path.name),
pdf_headings=True,
ocr=ocr,
assets=assets,
resolve=resolve,
)
text = document.text
# Computed from the SAME text the plan's offsets index, so the
# locator and the offset can never disagree about which rendering
# they describe. `None` when the profile names no provenance:
@ -1220,6 +1283,7 @@ def process_inbox(
text,
pdf_headings=reading_fonts,
ocr=ocr,
assets=assets,
)
if profile.provenance is not None
else None
@ -1321,6 +1385,12 @@ def process_inbox(
continue
bundle.mkdir(parents=True, exist_ok=True)
if outputs and (document.images or document.rejected):
# AFTER the gate, and only where the document actually produced
# concepts. An asset written for a refused document would be an
# orphan no pointer names and no retirement pass reaches.
_write_assets(bundle, document.images, carried_assets)
refused_assets.extend(document.rejected)
for target_name, content, reasons in outputs:
# `write_bytes` resolves a subpath through `safe_resolve` but never
# creates one. Without this the very first hierarchical write fails.
@ -1377,6 +1447,8 @@ def process_inbox(
failed=tuple(sorted(failed, key=lambda entry: entry.source_file)),
concepts=tuple(concepts),
skipped=skipped,
assets=tuple(sorted(carried_assets)),
assets_rejected=tuple(refused_assets),
)
@ -1397,6 +1469,46 @@ ADJUDICATION_STATES = (ADJUDICATION_PROPOSED, ADJUDICATION_ADJUDICATED)
ADJUDICATION_COMPANION_KEYS = ("adjudicated_by", "adjudicated_at", "adjudication_dwell_s")
def _write_assets(bundle: Path, images: Sequence[ExtractedImage], seen: dict[str, bytes]) -> None:
"""Put one document's images in the bundle's `assets/` directory.
OWNERSHIP IS PROVEN BY CONTENT IDENTITY, which is Door C's rule reused
verbatim: an occupied name is re-used only when the bytes there are already
identical, and never overwritten otherwise. Here the name carries the
digest of those very bytes, so an occupied name with different contents is
a `sha256[:12]` collision -- refused loudly rather than resolved silently,
because silently resolving it would mean one of two pictures is lost and
every pointer to it shows the other.
A binary write, and the only one in this package. `materialize.write_bytes`
takes `content: str` and encodes UTF-8, which is correct for every text
guarantee it holds and cannot carry a JPEG.
"""
directory = bundle / ASSETS_DIR
for image in images:
name = asset_name(image)
known = seen.get(name)
if known is not None:
if known != image.data:
raise MaterializationError(
f"two different images reduce to the asset name {name!r} in one run; "
"refusing to overwrite the first, because every pointer to it would "
"then show the second",
code="asset_collision",
)
continue
directory.mkdir(parents=True, exist_ok=True)
target = safe_resolve(directory, name)
if target.exists() and target.read_bytes() != image.data:
raise MaterializationError(
f"the asset {name!r} already exists in the bundle with different bytes; "
"refusing to overwrite content this run did not write",
code="asset_collision",
)
target.write_bytes(image.data)
seen[name] = image.data
def _validate_facets(structure: DocumentStructure, profile: BundleProfile) -> None:
"""Refuse a document whose values cannot be rendered as index facets.