llm-ingestion-okf/tests/test_asset_import.py
Kjell Tore Guttormsen bc39e8091f feat(assets): a bundle carries the images its sources declare (0.10.0)
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 10:01:31 +02:00

128 lines
4.7 KiB
Python

"""Door C carries the assets its merged concepts point at (0.10.0).
Measured on 2026-09-17, before this was fixed: importing a bundle built with
`--assets` merged 6 of 6 concepts -- the guard cleared every pointer block as
ordinary body text -- and wrote **no `assets/` directory at all**. Every
`![Tabell 84-2](/assets/...)` in the imported bundle pointed at a file that was
not there.
That is the same defect one door over: a bundle that reads as complete and is
not. Door C's two invariants make the repair obvious rather than a new
mechanism -- a merged concept is written VERBATIM, so its pointer cannot be
rewritten to point somewhere else, and ownership is proven by CONTENT IDENTITY,
which is exactly the rule an asset name already encodes.
WHAT IS NOT CARRIED IS AS IMPORTANT: an asset no merged concept points at stays
behind. A concept the guard refused takes its pictures with it, or the import
would persist bytes the gate never cleared a reader for -- and an orphan in
`assets/` is a file nothing names and nothing retires.
"""
from __future__ import annotations
import warnings
from pathlib import Path
import pytest
from llm_ingestion_okf import cli
from llm_ingestion_okf.assets import ASSETS_DIR, IMAGE_POINTER
from llm_ingestion_okf.importer import import_bundle
FIXTURES = Path(__file__).parent / "fixtures" / "image-inbox"
INGESTED_AT = "2026-09-17T00:00:00Z"
def _source_bundle(tmp_path: Path) -> Path:
inbox = tmp_path / "inbox"
(inbox / "graphics").mkdir(parents=True)
for source in sorted(FIXTURES.rglob("*")):
if source.is_file():
(inbox / source.relative_to(FIXTURES)).write_bytes(source.read_bytes())
bundle = tmp_path / "source-bundle"
with warnings.catch_warnings():
warnings.simplefilter("ignore")
assert (
cli.main(
[
"build",
str(inbox),
"--bundle",
str(bundle),
"--bundle-id",
"asset-import-fixture",
"--okf-version",
"0.2",
]
)
== 0
)
return bundle
def _imported(tmp_path: Path) -> Path:
from llm_ingestion_okf.guard_adapter import import_gate
target = tmp_path / "imported"
import_bundle(
_source_bundle(tmp_path),
target,
INGESTED_AT,
origin="external",
channel="manual",
gate=import_gate,
)
return target
def test_every_pointer_a_merged_concept_carries_resolves(tmp_path: Path) -> None:
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
target = _imported(tmp_path)
pointers = 0
for concept in sorted(target.rglob("*.md")):
for match in IMAGE_POINTER.finditer(concept.read_text("utf-8")):
pointers += 1
asset = target / ASSETS_DIR / match.group("asset")
assert asset.is_file(), f"{concept.name} points at a missing {asset.name}"
assert pointers, "the fixture bundle carried no pointers at all"
def test_the_imported_bytes_are_the_senders_own(tmp_path: Path) -> None:
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
source = _source_bundle(tmp_path)
from llm_ingestion_okf.guard_adapter import import_gate
target = tmp_path / "imported"
import_bundle(
source, target, INGESTED_AT, origin="external", channel="manual", gate=import_gate
)
carried = sorted((target / ASSETS_DIR).glob("*"))
# The known-positive in the same test: a loop over an empty directory is
# green and proves nothing, which is exactly how a repair can empty the set
# a test iterates over and stay passing.
assert carried, "nothing was carried, so the comparison below ran over nothing"
for asset in carried:
assert asset.read_bytes() == (source / ASSETS_DIR / asset.name).read_bytes()
def test_an_asset_nothing_points_at_stays_behind(tmp_path: Path) -> None:
"""The negative control, and it is the security half of the rule.
An asset belonging to a concept the gate refused must not ride in on the
back of one it cleared.
"""
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
source = _source_bundle(tmp_path)
(source / ASSETS_DIR / "deadbeefdead-ingen-peker.png").write_bytes(
(source / ASSETS_DIR).glob("*.png").__next__().read_bytes() + b"\x00"
)
from llm_ingestion_okf.guard_adapter import import_gate
target = tmp_path / "imported"
import_bundle(
source, target, INGESTED_AT, origin="external", channel="manual", gate=import_gate
)
assert not (target / ASSETS_DIR / "deadbeefdead-ingen-peker.png").exists()