open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(assets): a bundle carries the images its sources declare (0.10.0)

Browse source 
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-17 10:01:31 +02:00
commit bc39e8091f
33 changed files with 3638 additions and 64 deletions
Download patch file Download diff file Expand all files Collapse all files

BIN
tests/fixtures/image-inbox/graphics/figur-84-1.png vendored Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 94 B

BIN
tests/fixtures/image-inbox/graphics/tabell-84-2.png vendored Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 122 B

BIN
tests/fixtures/image-inbox/prosess-84-notat.docx vendored Normal file
View file

Binary file not shown.

BIN
tests/fixtures/image-inbox/prosess-84-presentasjon.pptx vendored Normal file
View file

Binary file not shown.

18
tests/fixtures/image-inbox/prosess-84-sts.xml vendored Normal file
View file

@ -0,0 +1,18 @@
<?xml version="1.0" encoding="UTF-8"?>
<standard xmlns:xlink="http://www.w3.org/1999/xlink">
<front><std-ident><doc-number>R761</doc-number></std-ident></front>
<body>
<sec>
<label>84</label>
<title>Konstruksjoner av betong</title>
<sec>
<label>84.1</label>
<title>Toleranser</title>
<p>Toleranseklasse er gitt i tabell 84-2.</p>
<graphic xlink:href="graphics/tabell-84-2.png"/>
<p>Figur 84-1 viser prinsippet.</p>
<graphic xlink:href="figur-84-1.png"/>
</sec>
</sec>
</body>
</standard>

BIN
tests/fixtures/image-inbox/prosess-84-tabell.pdf vendored Normal file
View file

Binary file not shown.

14
tests/fixtures/image-inbox/prosess-84-web.html vendored Normal file
View file

@ -0,0 +1,14 @@
<!doctype html>
<html><head><title>Prosess 84</title></head>
<body>
<h1>84 Konstruksjoner av betong</h1>
<p>Toleranseklasse for de enkelte konstruksjonsdeler er gitt i tabell 84-2.</p>
<img src="graphics/tabell-84-2.png" alt="Tabell 84-2 Toleranseklasser">
<p>Figuren under viser prinsippet.</p>
<figure>
<img src="graphics/figur-84-1.png" alt="Figur 84-1 Prinsippskisse">
<figcaption>Figur 84-1 Prinsippskisse</figcaption>
</figure>
<p>Og en som ligger et annet sted:</p>
<img src="https://example.invalid/ekstern.png" alt="Ekstern figur">
</body></html>

327
tests/fixtures/make_image_fixtures.py vendored Normal file
View file

@ -0,0 +1,327 @@
"""Regenerate the image-bearing fixtures for the asset path (0.10.0).
Five documents, one per reader, each carrying a KNOWN number of images so a
gate can state "carried N of M" with M read out of the source rather than out
of this package. Written in a second file rather than appended to
`make_fixtures.py` for one reason: every fixture that file emits is byte-pinned
by a test, and the object numbering of the PDF builders is part of those bytes.
Adding an XObject to a shared builder would regenerate files whose whole value
is that they have not moved.
The same policy holds here as there: no generator library. The PNG is written
out with `zlib` from the stdlib, the JPEG as a header sequence (the readers
copy JPEG bytes through and read nothing but the frame marker, so a decodable
photograph would test nothing extra and could not be hand-audited), and the
containers are assembled part by part.
Run from the repository root: python3 tests/fixtures/make_image_fixtures.py
"""
from __future__ import annotations
import io
import struct
import zipfile
import zlib
from pathlib import Path
HERE = Path(__file__).parent
IMAGES = HERE / "image-inbox"
_XML = '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
_ZIP_DATE = (2020, 1, 1, 0, 0, 0)
def png(width: int, height: int, value: int = 0x40) -> bytes:
"""A real, single-channel PNG of a flat grey."""
def chunk(kind: bytes, payload: bytes) -> bytes:
return (
len(payload).to_bytes(4, "big")
+ kind
+ payload
+ zlib.crc32(kind + payload).to_bytes(4, "big")
)
ihdr = struct.pack(">IIBBBBB", width, height, 8, 0, 0, 0, 0)
raw = b"".join(b"\x00" + bytes([value] * width) for _ in range(height))
return (
b"\x89PNG\r\n\x1a\n"
+ chunk(b"IHDR", ihdr)
+ chunk(b"IDAT", zlib.compress(raw, 9))
+ chunk(b"IEND", b"")
)
def jpeg_header(width: int, height: int) -> bytes:
"""A JPEG's marker sequence: SOI, JFIF, a baseline frame header, EOI.
Not a decodable photograph, and that is the point of it. The `pdf` reader
passes `DCTDecode` bytes through untouched and reads only the frame header
for the size, which is precisely what R761's own table images need -- 29 of
its 50 image objects are `DCTDecode`. A fixture that also carried entropy
data would exercise no additional line of this package and could not be read
byte by byte by a person.
"""
frame = bytes([8, height >> 8, height & 0xFF, width >> 8, width & 0xFF, 1, 1, 0x11, 0])
return (
b"\xff\xd8"
b"\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00"
+ b"\xff\xc0"
+ (len(frame) + 2).to_bytes(2, "big")
+ frame
+ b"\xff\xd9"
)
# --- pdf ---------------------------------------------------------------------
#
# TWO images on one page, and they are deliberately of the two kinds the
# measurement on R761 found: 29 `DCTDecode` objects, which arrive as a finished
# JPEG file, and 21 `FlateDecode` ones, which arrive as raw samples with the
# colour model in the dictionary beside them and have to be encoded to be
# carried at all. A fixture with only one kind would leave half the reader
# unexercised, and it is the encoded half that can be silently wrong.
PDF_GRAY_WIDTH, PDF_GRAY_HEIGHT = 4, 3
PDF_GRAY_SAMPLES = bytes([0, 60, 120, 180, 20, 80, 140, 200, 40, 100, 160, 255])
PDF_JPEG_WIDTH, PDF_JPEG_HEIGHT = 360, 269
PDF_CONTENT = (
b"BT /F1 12 Tf 20 170 Td (Toleranseklasse er gitt i tabell 84-2) Tj ET\n"
b"q 80 0 0 60 20 90 cm /ImFlate Do Q\n"
b"q 80 0 0 60 20 20 cm /ImJpeg Do Q\n"
)
def build_image_pdf() -> bytes:
"""A one-page PDF with a Flate image and a DCT image in its resources."""
flate = zlib.compress(PDF_GRAY_SAMPLES, 9)
jpeg = jpeg_header(PDF_JPEG_WIDTH, PDF_JPEG_HEIGHT)
objects = [
b"<< /Type /Catalog /Pages 2 0 R >>",
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 4 0 R "
b"/Resources << /Font << /F1 5 0 R >> "
b"/XObject << /ImFlate 6 0 R /ImJpeg 7 0 R >> >> >>",
b"<< /Length "
+ str(len(PDF_CONTENT)).encode()
+ b" >>\nstream\n"
+ PDF_CONTENT
+ b"endstream",
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>",
b"<< /Type /XObject /Subtype /Image /Width "
+ str(PDF_GRAY_WIDTH).encode()
+ b" /Height "
+ str(PDF_GRAY_HEIGHT).encode()
+ b" /ColorSpace /DeviceGray /BitsPerComponent 8 /Filter /FlateDecode /Length "
+ str(len(flate)).encode()
+ b" >>\nstream\n"
+ flate
+ b"\nendstream",
b"<< /Type /XObject /Subtype /Image /Width "
+ str(PDF_JPEG_WIDTH).encode()
+ b" /Height "
+ str(PDF_JPEG_HEIGHT).encode()
+ b" /ColorSpace /DeviceGray /BitsPerComponent 8 /Filter /DCTDecode /Length "
+ str(len(jpeg)).encode()
+ b" >>\nstream\n"
+ jpeg
+ b"\nendstream",
]
out = bytearray(b"%PDF-1.4\n")
offsets = []
for number, body in enumerate(objects, start=1):
offsets.append(len(out))
out += str(number).encode() + b" 0 obj\n" + body + b"\nendobj\n"
xref_at = len(out)
size = str(len(objects) + 1).encode()
out += b"xref\n0 " + size + b"\n0000000000 65535 f \n"
for offset in offsets:
out += ("%010d 00000 n \n" % offset).encode()
out += b"trailer\n<< /Size " + size + b" /Root 1 0 R >>\n"
out += b"startxref\n" + str(xref_at).encode() + b"\n%%EOF\n"
return bytes(out)
# --- html --------------------------------------------------------------------
#
# THREE `<img>` and only two of them can be carried. The remote one is the
# boundary written as a fixture: this package never opens a socket during
# extraction (the network gate is an explicit per-run opt-in and extraction is
# not on that path), so a remote source becomes a pointer WITHOUT a file, and
# the gate counts it as found-and-not-carried rather than as absent.
HTML_DOCUMENT = """<!doctype html>
<html><head><title>Prosess 84</title></head>
<body>
<h1>84 Konstruksjoner av betong</h1>
<p>Toleranseklasse for de enkelte konstruksjonsdeler er gitt i tabell 84-2.</p>
<img src="graphics/tabell-84-2.png" alt="Tabell 84-2 Toleranseklasser">
<p>Figuren under viser prinsippet.</p>
<figure>
<img src="graphics/figur-84-1.png" alt="Figur 84-1 Prinsippskisse">
<figcaption>Figur 84-1 Prinsippskisse</figcaption>
</figure>
<p>Og en som ligger et annet sted:</p>
<img src="https://example.invalid/ekstern.png" alt="Ekstern figur">
</body></html>
"""
# --- niso-sts ----------------------------------------------------------------
#
# The shape the R761 delivery actually has, measured 2026-09-16: 50 `<graphic>`
# elements, every one a direct child of a `<sec>`, none inside a `<table-wrap>`,
# each carrying a bare file name in `xlink:href` that resolves against a sibling
# `graphics/` directory. No `<caption>` anywhere near them -- the caption a
# human reads is a `<p>` the extractor already emits on its own line.
STS_DOCUMENT = """<?xml version="1.0" encoding="UTF-8"?>
<standard xmlns:xlink="http://www.w3.org/1999/xlink">
<front><std-ident><doc-number>R761</doc-number></std-ident></front>
<body>
<sec>
<label>84</label>
<title>Konstruksjoner av betong</title>
<sec>
<label>84.1</label>
<title>Toleranser</title>
<p>Toleranseklasse er gitt i tabell 84-2.</p>
<graphic xlink:href="graphics/tabell-84-2.png"/>
<p>Figur 84-1 viser prinsippet.</p>
<graphic xlink:href="figur-84-1.png"/>
</sec>
</sec>
</body>
</standard>
"""
def build_docx() -> bytes:
"""A `.docx` with one embedded image, its alt text on the drawing."""
image = png(40, 30, value=0x30)
parts: dict[str, str | bytes] = {
"[Content_Types].xml": _XML
+ '<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
+ '<Default Extension="xml" ContentType="application/xml"/>'
+ '<Default Extension="rels" ContentType="application/vnd.openxmlformats-package'
+ '.relationships+xml"/>'
+ '<Default Extension="png" ContentType="image/png"/>'
+ '<Override PartName="/word/document.xml" ContentType="application/vnd'
+ '.openxmlformats-officedocument.wordprocessingml.document.main+xml"/>'
+ "</Types>",
"_rels/.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006'
+ '/relationships/officeDocument" Target="word/document.xml"/></Relationships>',
"word/_rels/document.xml.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rIdImg" Type="http://schemas.openxmlformats.org/officeDocument'
+ '/2006/relationships/image" Target="media/tabell-84-2.png"/></Relationships>',
"word/media/tabell-84-2.png": image,
"word/document.xml": _XML
+ '<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"'
+ ' xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships"'
+ ' xmlns:wp="http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing"'
+ ' xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main"'
+ ' xmlns:pic="http://schemas.openxmlformats.org/drawingml/2006/picture"><w:body>'
+ "<w:p><w:r><w:t>Toleranseklasse er gitt i tabell 84-2.</w:t></w:r></w:p>"
+ '<w:p><w:r><w:drawing><wp:inline><wp:extent cx="381000" cy="285750"/>'
+ '<wp:docPr id="1" name="Bilde 1" descr="Tabell 84-2 Toleranseklasser"/>'
+ '<a:graphic><a:graphicData uri="http://schemas.openxmlformats.org/drawingml/2006'
+ '/picture"><pic:pic><pic:nvPicPr><pic:cNvPr id="1" name="tabell-84-2.png"'
+ ' descr="Tabell 84-2 Toleranseklasser"/><pic:cNvPicPr/></pic:nvPicPr>'
+ '<pic:blipFill><a:blip r:embed="rIdImg"/><a:stretch><a:fillRect/></a:stretch>'
+ '</pic:blipFill><pic:spPr><a:xfrm><a:off x="0" y="0"/>'
+ '<a:ext cx="381000" cy="285750"/></a:xfrm>'
+ '<a:prstGeom prst="rect"><a:avLst/></a:prstGeom></pic:spPr></pic:pic>'
+ "</a:graphicData></a:graphic></wp:inline></w:drawing></w:r></w:p>"
+ "<w:p><w:r><w:t>Etter tabellen gjelder NS-EN 13670.</w:t></w:r></w:p>"
+ "</w:body></w:document>",
}
return build_container(parts)
def build_pptx() -> bytes:
"""A `.pptx` with one titled slide and one embedded image on it."""
image = png(48, 36, value=0x70)
parts: dict[str, str | bytes] = {
"[Content_Types].xml": _XML
+ '<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
+ '<Default Extension="xml" ContentType="application/xml"/>'
+ '<Default Extension="rels" ContentType="application/vnd.openxmlformats-package'
+ '.relationships+xml"/>'
+ '<Default Extension="png" ContentType="image/png"/>'
+ '<Override PartName="/ppt/presentation.xml" ContentType="application/vnd'
+ '.openxmlformats-officedocument.presentationml.presentation.main+xml"/>'
+ '<Override PartName="/ppt/slides/slide1.xml" ContentType="application/vnd'
+ '.openxmlformats-officedocument.presentationml.slide+xml"/>'
+ "</Types>",
"_rels/.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006'
+ '/relationships/officeDocument" Target="ppt/presentation.xml"/></Relationships>',
"ppt/_rels/presentation.xml.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rId2" Type="http://schemas.openxmlformats.org/officeDocument/2006'
+ '/relationships/slide" Target="slides/slide1.xml"/></Relationships>',
"ppt/presentation.xml": _XML
+ '<p:presentation xmlns:p="http://schemas.openxmlformats.org/presentationml/2006/main"'
+ ' xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">'
+ '<p:sldIdLst><p:sldId id="256" r:id="rId2"/></p:sldIdLst></p:presentation>',
"ppt/slides/_rels/slide1.xml.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rIdImg" Type="http://schemas.openxmlformats.org/officeDocument'
+ '/2006/relationships/image" Target="../media/skisse.png"/></Relationships>',
"ppt/media/skisse.png": image,
"ppt/slides/slide1.xml": _XML
+ '<p:sld xmlns:p="http://schemas.openxmlformats.org/presentationml/2006/main"'
+ ' xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main"'
+ ' xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">'
+ "<p:cSld><p:spTree>"
+ '<p:nvGrpSpPr><p:cNvPr id="1" name=""/><p:cNvGrpSpPr/><p:nvPr/></p:nvGrpSpPr>'
+ "<p:grpSpPr/>"
+ '<p:sp><p:nvSpPr><p:cNvPr id="2" name="Tittel 1"/><p:cNvSpPr/>'
+ '<p:nvPr><p:ph type="title"/></p:nvPr></p:nvSpPr><p:spPr/>'
+ "<p:txBody><a:bodyPr/><a:p><a:r><a:t>Toleranser</a:t></a:r></a:p></p:txBody></p:sp>"
+ '<p:pic><p:nvPicPr><p:cNvPr id="3" name="skisse.png" descr="Prinsippskisse"/>'
+ "<p:cNvPicPr/><p:nvPr/></p:nvPicPr>"
+ '<p:blipFill><a:blip r:embed="rIdImg"/><a:stretch><a:fillRect/></a:stretch>'
+ '</p:blipFill><p:spPr><a:xfrm><a:off x="0" y="0"/>'
+ '<a:ext cx="457200" cy="342900"/></a:xfrm>'
+ '<a:prstGeom prst="rect"><a:avLst/></a:prstGeom></p:spPr></p:pic>'
+ "</p:spTree></p:cSld></p:sld>",
}
return build_container(parts)
def build_container(parts: dict[str, str | bytes]) -> bytes:
out = io.BytesIO()
with zipfile.ZipFile(out, "w", compression=zipfile.ZIP_DEFLATED) as archive:
for name, payload in parts.items():
info = zipfile.ZipInfo(name, date_time=_ZIP_DATE)
info.compress_type = zipfile.ZIP_DEFLATED
archive.writestr(info, payload)
return out.getvalue()
if __name__ == "__main__":
(IMAGES / "graphics").mkdir(parents=True, exist_ok=True)
# FIVE DISTINCT STEMS. One stem across five extensions is refused by the
# door's own SS 3 collision rule -- measured while building this fixture:
# `prosess-84.{pdf,docx,pptx}` gave `inbox_slug_collision: 2/7` and two of
# the five readers were never exercised at all, with the gate reporting a
# carrying defect that was really a fixture defect.
written: list[tuple[str, bytes]] = [
("graphics/tabell-84-2.png", png(120, 90, value=0x20)),
("graphics/figur-84-1.png", png(64, 48, value=0x80)),
("prosess-84-web.html", HTML_DOCUMENT.encode("utf-8")),
("prosess-84-sts.xml", STS_DOCUMENT.encode("utf-8")),
("prosess-84-tabell.pdf", build_image_pdf()),
("prosess-84-notat.docx", build_docx()),
("prosess-84-presentasjon.pptx", build_pptx()),
]
for name, payload in written:
(IMAGES / name).write_bytes(payload)
print(f"wrote image-inbox/{name}")

181
tests/test_asset_bundle.py Normal file
View file

@ -0,0 +1,181 @@
"""Door B writes the images beside the text, and says how many (0.10.0).
The gate (`test_asset_gate.py`) asks whether the bytes arrive. This file pins
the four properties that decide whether their arrival is trustworthy.
- **A concept says how many images it carries.** `images: N`, conditional, so a
document without figures pays nothing and every bundle built before this
existed is byte-identical. Without the count a consumer cannot tell "this
document had no figures" from "this build dropped them", which is the
distinction the whole capability exists to restore.
- **The key belongs to the profiles this repository owns.** `DEFAULT` states
commons' ingest-spec SS 5 layer and `STRICT_V1` the wiki's ratified contract;
naming a key in either from here is this repository editing someone else's
contract (O2), the same reason `sources` sits on the segmented v0.2 profile
alone. Under those profiles the images are still CARRIED and still POINTED
at -- only the count is absent.
- **`--no-assets` reproduces the pre-move bytes.** Every default this package
has ever moved carries an opt-out that does, and the claim is measured on a
whole bundle rather than asserted.
- **The run log states the denominator.** SS 9's `log.md` already carries the
gate's name and the file counts because they are the facts about a run that
the bundle cannot otherwise recover. "51 images carried of 53 found" is the
same class of fact, and without it a bundle whose figures were all refused
looks exactly like a bundle of documents that had none.
"""
from __future__ import annotations
import filecmp
import warnings
from pathlib import Path
import pytest
from llm_ingestion_okf import cli, corpus
from llm_ingestion_okf.assets import ASSETS_DIR
from llm_ingestion_okf.profiles import DEFAULT, SEGMENTED_OKF_V0_2, STRICT_V1, STRUCTURED_V1
FIXTURES = Path(__file__).parent / "fixtures" / "image-inbox"
BUNDLE_ID = "asset-bundle-fixture"
OKF_VERSION = "0.2"
def _inbox(root: Path) -> Path:
inbox = root / "inbox"
(inbox / "graphics").mkdir(parents=True)
for source in sorted(FIXTURES.rglob("*")):
if source.is_file():
(inbox / source.relative_to(FIXTURES)).write_bytes(source.read_bytes())
return inbox
def _build(inbox: Path, bundle: Path, *extra: str) -> int:
with warnings.catch_warnings():
warnings.simplefilter("ignore")
return cli.main(
[
"build",
str(inbox),
"--bundle",
str(bundle),
"--bundle-id",
BUNDLE_ID,
"--okf-version",
OKF_VERSION,
*extra,
]
)
def _concepts(bundle: Path) -> list[Path]:
return [
path
for path in sorted(bundle.rglob("*.md"))
if path.name not in {"index.md", corpus.LOG_NAME}
]
# --- the count -------------------------------------------------------------
def test_a_concept_that_carries_images_counts_them(tmp_path: Path) -> None:
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
bundle = tmp_path / "bundle"
assert _build(_inbox(tmp_path), bundle) == 0
counted = [path for path in _concepts(bundle) if "\nimages: " in path.read_text("utf-8")]
assert counted, "no concept declared an image count"
for path in counted:
text = path.read_text("utf-8")
line = next(row for row in text.splitlines() if row.startswith("images: "))
assert int(line.split(":", 1)[1]) == text.count("](/assets/")
def test_a_concept_without_images_declares_no_count(tmp_path: Path) -> None:
"""Conditional, like `req_number`: absent is the document saying nothing."""
pytest.importorskip("llm_ingestion_guard")
inbox = tmp_path / "inbox"
inbox.mkdir()
(inbox / "ren.md").write_text("# Kostnader\n\nIngen figurer her.\n", encoding="utf-8")
bundle = tmp_path / "bundle"
assert _build(inbox, bundle) == 0
for path in _concepts(bundle):
assert "images:" not in path.read_text("utf-8")
def test_the_key_is_named_by_the_profiles_this_repository_owns() -> None:
assert "images" in STRUCTURED_V1.frontmatter.order
assert "images" in SEGMENTED_OKF_V0_2.frontmatter.order
# O2: naming a key in either of these is editing a contract owned elsewhere.
assert "images" not in DEFAULT.frontmatter.order
assert "images" not in STRICT_V1.frontmatter.order
# --- the opt-out reproduces the pre-move bytes -----------------------------
def test_no_assets_writes_no_assets_directory(tmp_path: Path) -> None:
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
bundle = tmp_path / "bundle"
assert _build(_inbox(tmp_path), bundle, "--no-assets") == 0
assert not (bundle / ASSETS_DIR).exists()
for path in _concepts(bundle):
assert "](/assets/" not in path.read_text("utf-8")
def test_the_two_runs_differ_in_the_asset_layer_and_nowhere_else(tmp_path: Path) -> None:
"""A document with NO images is byte-identical under both settings.
The measurement that matters for every consumer who has already built a
bundle: turning the capability on must move nothing in a corpus that has no
pictures in it. Asserted on files rather than on a count, because a count is
insensitive to exactly the change this would hide.
"""
pytest.importorskip("llm_ingestion_guard")
inbox = tmp_path / "inbox"
inbox.mkdir()
(inbox / "ren.md").write_text(
"# Kostnader\n\nIngen figurer.\n\n## Pris\n\nEn tabell uten bilde.\n", encoding="utf-8"
)
on, off = tmp_path / "on", tmp_path / "off"
assert _build(inbox, on) == 0
assert _build(inbox, off, "--no-assets") == 0
names = sorted(path.relative_to(on) for path in on.rglob("*") if path.is_file())
assert names == sorted(path.relative_to(off) for path in off.rglob("*") if path.is_file())
for name in names:
if name.name == corpus.LOG_NAME:
continue # the log names the setting on purpose
assert filecmp.cmp(on / name, off / name, shallow=False), name
# --- the log states the denominator ----------------------------------------
def test_the_log_carries_images_found_and_carried(tmp_path: Path) -> None:
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
bundle = tmp_path / "bundle"
assert _build(_inbox(tmp_path), bundle) == 0
log = (bundle / corpus.LOG_NAME).read_text("utf-8")
assert "**Images**:" in log
assert " of " in log.split("**Images**:", 1)[1].splitlines()[0]
def test_the_log_names_the_capability_as_off_when_it_is(tmp_path: Path) -> None:
"""`--no-assets` is stated, never implied by a missing line.
The same rule `--gate none` follows: a bundle built without screening says
`NOTHING WAS SCREENED` rather than leaving a reader to infer it from an
absent bullet. A bundle whose figures were never looked for must not read
as a bundle of documents that had none.
"""
pytest.importorskip("llm_ingestion_guard")
inbox = tmp_path / "inbox"
inbox.mkdir()
(inbox / "ren.md").write_text("# Kostnader\n\nIngen figurer.\n", encoding="utf-8")
bundle = tmp_path / "bundle"
assert _build(inbox, bundle, "--no-assets") == 0
log = (bundle / corpus.LOG_NAME).read_text("utf-8")
assert "NOT CARRIED" in log

244
tests/test_asset_gate.py Normal file
View file

@ -0,0 +1,244 @@
"""THE GATE: does a bundle carry the images its sources declare? (0.10.0)
Written BEFORE the capability, and red on purpose. Its whole job is to state
one number per reader with the DENOMINATOR read out of the source document
rather than out of this package -- "carried 0 of 9" is a measurement; "no image
support" is a sentence. The denominators here are computed by opening the
fixture bytes and counting what the FORMAT says is there (`page.images`,
`word/media/`, `ppt/media/`, `<img`, `<graphic`), so the gate keeps working
when the fixtures change and cannot drift into asserting our own output back at
us.
**Measured at `332961a`**, built from `git archive` rather than from the
editable tree -- an editable install reads `src/` live, so a "before" run taken
in this working tree would have been measuring the change it was supposed to
predate:
carried 0 of 2 local (2 declared) prosess-84-tabell.pdf
carried 0 of 1 local (1 declared) prosess-84-notat.docx
carried 0 of 1 local (1 declared) prosess-84-presentasjon.pptx
carried 0 of 2 local (3 declared) prosess-84-web.html
carried 0 of 2 local (2 declared) prosess-84-sts.xml
---------------------------------------------------------------
carried 0 of 8 local images across 5 documents (9 declared),
and the bundle held no `assets/` directory at all.
Two of the seven files in the fixture inbox are the PNGs the HTML and STS
documents point at, and the run reports them as `extractor_unknown: 2/7` at
both commits. That is deliberate and unchanged: `.png` as a DROPPED FILE is a
separate question with its own order, and an image reached through a document
is this one.
WHY THE GATE IS A BUNDLE-LEVEL COUNT and not a per-function assertion: the
failure this exists to stop is not "the reader returned nothing". It is a
bundle that looks complete and is not -- R761's process 84 saying "toleranse-
klasse ... er gitt i tabell 84-2" over an empty space, with `okf check` green
and `okf quality` unable to see it. So the gate runs the real command, walks
the written bundle, and asks whether the bytes are there.
THE REMOTE IMAGE IS PART OF THE DENOMINATOR AND NOT PART OF THE TARGET. The
HTML fixture carries three `<img>`; one points at `https://example.invalid/`.
Extraction opens no socket -- network access here is an explicit per-run opt-in
and extraction is not on that path -- so that image is FOUND, counted, and
carried as a pointer without a file. A gate that quietly dropped it from the
denominator would report 2 of 2 and hide the one case a reader most needs told.
"""
from __future__ import annotations
import re
import warnings
import zipfile
from pathlib import Path
import pytest
from llm_ingestion_okf import cli, corpus
from llm_ingestion_okf.assets import ASSETS_DIR, IMAGE_POINTER
FIXTURES = Path(__file__).parent / "fixtures" / "image-inbox"
BUNDLE_ID = "asset-gate-fixture"
OKF_VERSION = "0.2"
#: Every document in the fixture inbox, with the reader it exercises.
DOCUMENTS = (
"prosess-84-tabell.pdf",
"prosess-84-notat.docx",
"prosess-84-presentasjon.pptx",
"prosess-84-web.html",
"prosess-84-sts.xml",
)
# --- denominators, read out of the source ----------------------------------
def declared_images(path: Path) -> int:
"""How many images the SOURCE says it holds, by the format's own rule.
Never a constant: a constant is this package asserting its own expectation,
and it goes stale the moment a fixture is regenerated. Each branch reads the
container the way the format defines it, which is also the number a person
checking the bundle by hand would arrive at.
"""
suffix = path.suffix.lower()
data = path.read_bytes()
if suffix == ".pdf":
pdfplumber = pytest.importorskip("pdfplumber")
with warnings.catch_warnings():
warnings.simplefilter("ignore")
with pdfplumber.open(path) as pdf:
return sum(len(page.images) for page in pdf.pages)
if suffix in (".docx", ".pptx", ".xlsx"):
with zipfile.ZipFile(path) as archive:
return sum(1 for name in archive.namelist() if "/media/" in name)
text = data.decode("utf-8")
if suffix in (".html", ".htm"):
return len(re.findall(r"<img\b", text))
if suffix == ".xml":
return len(re.findall(r"<(?:inline-)?graphic\b", text))
raise AssertionError(f"no declared-image rule for {path.name}")
def local_images(path: Path) -> int:
"""The declared images MINUS the ones whose source is off this machine.
Separate from :func:`declared_images` and both are reported: the difference
between them is exactly the network boundary, and collapsing the two would
turn a boundary into an absence.
"""
declared = declared_images(path)
if path.suffix.lower() in (".html", ".htm"):
remote = len(re.findall(r'<img\b[^>]*src="(?:https?:)?//', path.read_text("utf-8")))
return declared - remote
return declared
# --- the run ---------------------------------------------------------------
def _inbox(tmp_path: Path) -> Path:
inbox = tmp_path / "inbox"
(inbox / "graphics").mkdir(parents=True)
for source in sorted(FIXTURES.rglob("*")):
if source.is_file():
target = inbox / source.relative_to(FIXTURES)
target.write_bytes(source.read_bytes())
return inbox
def _build(inbox: Path, bundle: Path, *extra: str) -> int:
return cli.main(
[
"build",
str(inbox),
"--bundle",
str(bundle),
"--bundle-id",
BUNDLE_ID,
"--okf-version",
OKF_VERSION,
*extra,
]
)
def _concepts(bundle: Path) -> list[Path]:
return [
path
for path in sorted(bundle.rglob("*.md"))
if path.name not in {"index.md", corpus.LOG_NAME}
]
def carried_per_source(bundle: Path) -> dict[str, int]:
"""Pointers in the bundle whose asset file is actually present, per source.
A pointer whose bytes are missing is NOT carried. Counting pointers alone
would let an empty `assets/` pass the gate, which is the same defect one
level down -- a bundle that looks complete and is not.
"""
counts: dict[str, int] = {}
for concept in _concepts(bundle):
text = concept.read_text("utf-8")
source = ""
for line in text.splitlines():
if line.startswith("source_file:"):
source = line.split(":", 1)[1].strip().strip("'\"")
break
for match in IMAGE_POINTER.finditer(text):
if (bundle / ASSETS_DIR / match.group("asset")).is_file():
counts[source] = counts.get(source, 0) + 1
return counts
@pytest.fixture(scope="module")
def built(tmp_path_factory: pytest.TempPathFactory) -> tuple[Path, dict[str, int]]:
pytest.importorskip("pdfplumber")
pytest.importorskip("pypandoc")
pytest.importorskip("llm_ingestion_guard")
root = tmp_path_factory.mktemp("asset-gate")
inbox = _inbox(root)
bundle = root / "bundle"
with warnings.catch_warnings():
warnings.simplefilter("ignore")
assert _build(inbox, bundle) == 0
return bundle, carried_per_source(bundle)
# --- the gate --------------------------------------------------------------
@pytest.mark.parametrize("document", DOCUMENTS)
def test_every_local_image_reaches_the_bundle(
document: str, built: tuple[Path, dict[str, int]]
) -> None:
"""carried == local, per source document, with both numbers printed."""
bundle, carried = built
want = local_images(FIXTURES / document)
got = carried.get(document, 0)
assert got == want, (
f"{document}: carried {got} of {want} local images "
f"({declared_images(FIXTURES / document)} declared by the source)"
)
def test_the_bundle_carries_every_local_image_of_every_document(
built: tuple[Path, dict[str, int]],
) -> None:
"""The whole-bundle row, so the gate reports one number a person can quote."""
bundle, carried = built
want = sum(local_images(FIXTURES / document) for document in DOCUMENTS)
got = sum(carried.values())
assert got == want, f"carried {got} of {want} local images across {len(DOCUMENTS)} documents"
def test_a_remote_image_is_a_pointer_without_a_file_never_a_silent_drop(
built: tuple[Path, dict[str, int]],
) -> None:
"""The network boundary, stated in the artifact rather than implied by absence."""
bundle, _ = built
html = [path for path in _concepts(bundle) if "prosess-84-web.html" in path.read_text("utf-8")]
assert html, "the html document produced no concept at all"
body = "\n".join(path.read_text("utf-8") for path in html)
assert "https://example.invalid/ekstern.png" in body, (
"a remote image must leave a pointer naming where it was, so a reader "
"learns the document had a figure this bundle does not hold"
)
assert "not carried" in body
def test_the_asset_bytes_are_the_sources_own_bytes(
built: tuple[Path, dict[str, int]],
) -> None:
"""A carried image is byte-identical to the file the source shipped.
Only the two files the fixture carries as real files can be checked this
way; the PDF and office images arrive inside a container. That is enough to
pin the property that matters -- nothing re-encodes an image that already
is one.
"""
bundle, _ = built
written = {path.read_bytes() for path in (bundle / ASSETS_DIR).glob("*")}
for original in sorted((FIXTURES / "graphics").glob("*.png")):
assert original.read_bytes() in written, original.name

128
tests/test_asset_import.py Normal file
View file

@ -0,0 +1,128 @@
"""Door C carries the assets its merged concepts point at (0.10.0).
Measured on 2026-09-17, before this was fixed: importing a bundle built with
`--assets` merged 6 of 6 concepts -- the guard cleared every pointer block as
ordinary body text -- and wrote **no `assets/` directory at all**. Every
`![Tabell 84-2](/assets/...)` in the imported bundle pointed at a file that was
not there.
That is the same defect one door over: a bundle that reads as complete and is
not. Door C's two invariants make the repair obvious rather than a new
mechanism -- a merged concept is written VERBATIM, so its pointer cannot be
rewritten to point somewhere else, and ownership is proven by CONTENT IDENTITY,
which is exactly the rule an asset name already encodes.
WHAT IS NOT CARRIED IS AS IMPORTANT: an asset no merged concept points at stays
behind. A concept the guard refused takes its pictures with it, or the import
would persist bytes the gate never cleared a reader for -- and an orphan in
`assets/` is a file nothing names and nothing retires.
"""
from __future__ import annotations
import warnings
from pathlib import Path
import pytest
from llm_ingestion_okf import cli
from llm_ingestion_okf.assets import ASSETS_DIR, IMAGE_POINTER
from llm_ingestion_okf.importer import import_bundle
FIXTURES = Path(__file__).parent / "fixtures" / "image-inbox"
INGESTED_AT = "2026-09-17T00:00:00Z"
def _source_bundle(tmp_path: Path) -> Path:
inbox = tmp_path / "inbox"
(inbox / "graphics").mkdir(parents=True)
for source in sorted(FIXTURES.rglob("*")):
if source.is_file():
(inbox / source.relative_to(FIXTURES)).write_bytes(source.read_bytes())
bundle = tmp_path / "source-bundle"
with warnings.catch_warnings():
warnings.simplefilter("ignore")
assert (
cli.main(
[
"build",
str(inbox),
"--bundle",
str(bundle),
"--bundle-id",
"asset-import-fixture",
"--okf-version",
"0.2",
]
)
== 0
)
return bundle
def _imported(tmp_path: Path) -> Path:
from llm_ingestion_okf.guard_adapter import import_gate
target = tmp_path / "imported"
import_bundle(
_source_bundle(tmp_path),
target,
INGESTED_AT,
origin="external",
channel="manual",
gate=import_gate,
)
return target
def test_every_pointer_a_merged_concept_carries_resolves(tmp_path: Path) -> None:
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
target = _imported(tmp_path)
pointers = 0
for concept in sorted(target.rglob("*.md")):
for match in IMAGE_POINTER.finditer(concept.read_text("utf-8")):
pointers += 1
asset = target / ASSETS_DIR / match.group("asset")
assert asset.is_file(), f"{concept.name} points at a missing {asset.name}"
assert pointers, "the fixture bundle carried no pointers at all"
def test_the_imported_bytes_are_the_senders_own(tmp_path: Path) -> None:
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
source = _source_bundle(tmp_path)
from llm_ingestion_okf.guard_adapter import import_gate
target = tmp_path / "imported"
import_bundle(
source, target, INGESTED_AT, origin="external", channel="manual", gate=import_gate
)
carried = sorted((target / ASSETS_DIR).glob("*"))
# The known-positive in the same test: a loop over an empty directory is
# green and proves nothing, which is exactly how a repair can empty the set
# a test iterates over and stay passing.
assert carried, "nothing was carried, so the comparison below ran over nothing"
for asset in carried:
assert asset.read_bytes() == (source / ASSETS_DIR / asset.name).read_bytes()
def test_an_asset_nothing_points_at_stays_behind(tmp_path: Path) -> None:
"""The negative control, and it is the security half of the rule.
An asset belonging to a concept the gate refused must not ride in on the
back of one it cleared.
"""
pytest.importorskip("pdfplumber")
pytest.importorskip("llm_ingestion_guard")
source = _source_bundle(tmp_path)
(source / ASSETS_DIR / "deadbeefdead-ingen-peker.png").write_bytes(
(source / ASSETS_DIR).glob("*.png").__next__().read_bytes() + b"\x00"
)
from llm_ingestion_okf.guard_adapter import import_gate
target = tmp_path / "imported"
import_bundle(
source, target, INGESTED_AT, origin="external", channel="manual", gate=import_gate
)
assert not (target / ASSETS_DIR / "deadbeefdead-ingen-peker.png").exists()

55
tests/test_asset_structure.py Normal file
View file

@ -0,0 +1,55 @@
"""An image is not a cross-reference, and a carried image is counted (0.10.0).
Two door-level properties that the asset layer would otherwise get wrong in
opposite directions.
**A markdown image is not a pointer to another concept.** `structure._LINK`
reads `[...](target)` and does not care what stands in front of the bracket, so
every pointer this capability writes would arrive in the index as a `references`
edge to a concept that cannot exist. That is the same defect K3-21 hit when the
`Enclosing section:` line was read as body text and became a second, unresolved
edge; the repair there was to append after derivation, and the repair here is
that an image link is a different construct from a link.
**A concept says how many images it carries.** A bundle that holds an image and
says nothing about it is only half an improvement: the count is what lets a
consumer, or `okf quality`, tell "this document had no figures" from "this
build dropped them". The key is CONDITIONAL -- written only where the count is
non-zero -- so a bundle of documents without images is byte-identical to one
built before this existed.
"""
from __future__ import annotations
from llm_ingestion_okf.structure import derive_document_structure
POINTER = (
"# 84 Konstruksjoner\n\n"
"Toleranseklasse er gitt i tabell 84-2, jf. N500.\n\n"
"![Tabell 84-2](/assets/e54e5f5da0e8-tabell-84-2.png)\n"
"Image: graphics/tabell-84-2.png (120x90 px) -- Tabell 84-2\n\n"
"Se ellers [N100](n100.md).\n"
)
def test_an_image_pointer_is_not_a_reference() -> None:
structure = derive_document_structure(POINTER, source_file="r761.xml")
assert "/assets/e54e5f5da0e8-tabell-84-2.png" not in structure.references
def test_the_links_beside_it_still_are() -> None:
"""The known-positive in the same text: masking must not eat real edges."""
structure = derive_document_structure(POINTER, source_file="r761.xml")
assert "n100.md" in structure.references
def test_the_asset_name_does_not_leak_a_number_into_references() -> None:
"""A masked span is still masked: `84-2` inside the file name is not a subject.
The span is blanked rather than deleted for exactly this reason -- deleting
would move every later offset and break first-appearance order, and leaving
it would let the number scan read an identifier out of a file name.
"""
structure = derive_document_structure(POINTER, source_file="r761.xml")
assert all("ASSETS" not in subject.upper() for subject in structure.references)
assert "N500" in structure.references

336
tests/test_assets.py Normal file
View file

@ -0,0 +1,336 @@
"""Binary assets: the layer that lets a bundle carry an image at all.
Until 0.10.0 no reader in this package fetched, named, described or copied a
single image, and the only writer into a bundle was
`materialize.write_bytes(bundle_dir, name, content: str)` -- UTF-8, text, no
binary path anywhere. A document whose table is a raster picture therefore
reached a concept as an absence with no denominator: measured on R761
Prosesskoden:2025, the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, and process 84 says "toleranse-
klasse ... er gitt i tabell 84-2" above an empty space.
THIS MODULE IS THE ONE PLACE THAT DECIDES WHAT AN IMAGE IS. Every reader hands
it bytes and gets back either a carried image or a coded rejection, so a
format's own quirks never reach the bundle layout, and "N images carried of M
found" has one definition for `pdf`, `docx`, `html` and `xml` alike.
Three properties are pinned here because a bundle is downstream of all of them:
- **the type is SNIFFED, never claimed.** A `.jpg` that is really a PNG would
otherwise be written under a name whose extension lies, and a consumer that
dispatches on the extension reads it wrong with full confidence.
- **the name is CONTENT-ADDRESSED.** `<sha256[:12]>-<reduced original name>`
makes two drops of one image one file, and makes a rebuild of the same
corpus produce the same bundle -- the byte-determinism rule this package
already holds for text.
- **the pointer is one GRAMMAR.** Step 2 (`okf describe`) has to find every
pointer mechanically in order to write a transcription under it, so the
block is a regex this module owns rather than a shape each reader invents.
"""
from __future__ import annotations
import hashlib
import re
import zlib
import pytest
from llm_ingestion_okf import assets
from llm_ingestion_okf.errors import ExtractionError
# --- hand-laid image bytes -------------------------------------------------
#
# Written out here rather than committed as files: every byte is visible in the
# test that depends on it, and the readers below are header readers, so a
# header is the whole input they have. The PNG is a REAL image (zlib from the
# stdlib); the JPEG is a structurally valid header sequence and not a decodable
# photograph, which is exactly what the JPEG path needs -- it copies the bytes
# through and reads nothing but the SOF marker.
def _png(width: int, height: int) -> bytes:
def chunk(kind: bytes, payload: bytes) -> bytes:
return (
len(payload).to_bytes(4, "big")
+ kind
+ payload
+ zlib.crc32(kind + payload).to_bytes(4, "big")
)
ihdr = width.to_bytes(4, "big") + height.to_bytes(4, "big") + bytes([8, 0, 0, 0, 0])
raw = b"".join(b"\x00" + bytes([0x40] * width) for _ in range(height))
return (
b"\x89PNG\r\n\x1a\n"
+ chunk(b"IHDR", ihdr)
+ chunk(b"IDAT", zlib.compress(raw, 9))
+ chunk(b"IEND", b"")
)
def _jpeg(width: int, height: int) -> bytes:
sof = bytes([8, height >> 8, height & 0xFF, width >> 8, width & 0xFF, 1, 1, 0x11, 0])
return (
b"\xff\xd8"
b"\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00"
+ b"\xff\xc0"
+ (len(sof) + 2).to_bytes(2, "big")
+ sof
+ b"\xff\xd9"
)
def _gif(width: int, height: int) -> bytes:
return (
b"GIF89a"
+ width.to_bytes(2, "little")
+ height.to_bytes(2, "little")
+ b"\x00\x00\x00"
+ b";"
)
def _bmp(width: int, height: int) -> bytes:
header = b"BM" + (54).to_bytes(4, "little") + b"\x00" * 4 + (54).to_bytes(4, "little")
dib = (
(40).to_bytes(4, "little")
+ width.to_bytes(4, "little", signed=True)
+ height.to_bytes(4, "little", signed=True)
+ b"\x01\x00\x18\x00"
+ b"\x00" * 24
)
return header + dib
# --- the type is sniffed, never claimed ------------------------------------
@pytest.mark.parametrize(
("data", "media_type", "suffix"),
[
(_png(4, 3), "image/png", ".png"),
(_jpeg(360, 269), "image/jpeg", ".jpg"),
(_gif(7, 5), "image/gif", ".gif"),
(_bmp(9, 2), "image/bmp", ".bmp"),
],
)
def test_sniff_reads_the_bytes(data: bytes, media_type: str, suffix: str) -> None:
assert assets.sniff(data) == (media_type, suffix)
def test_sniff_refuses_what_is_not_an_image() -> None:
assert assets.sniff(b"%PDF-1.7\n") is None
assert assets.sniff(b"") is None
def test_the_claimed_extension_never_decides() -> None:
"""A PNG named `.jpg` is carried as a PNG, under a `.png` name."""
image = assets.read_image(_png(4, 3), name="tabell-84-2.jpg")
assert image.media_type == "image/png"
assert assets.asset_name(image).endswith(".png")
def test_unknown_bytes_are_a_coded_rejection_not_a_crash() -> None:
with pytest.raises(ExtractionError) as excinfo:
assets.read_image(b"not an image at all", name="x.png")
assert excinfo.value.code == "asset_type_unknown"
# --- dimensions, from the header of each format ----------------------------
@pytest.mark.parametrize(
("data", "size"),
[
(_png(4, 3), (4, 3)),
(_jpeg(360, 269), (360, 269)),
(_gif(7, 5), (7, 5)),
(_bmp(9, 2), (9, 2)),
],
)
def test_dimensions_from_the_header(data: bytes, size: tuple[int, int]) -> None:
assert assets.dimensions(data) == size
def test_dimensions_absent_is_absent_never_zero() -> None:
"""A format whose size this module does not read says so, rather than 0x0.
`0x0 px` in a concept is a measurement nobody took, printed as a fact.
"""
tiff = b"II\x2a\x00" + b"\x00" * 16
assert assets.sniff(tiff) == ("image/tiff", ".tiff")
assert assets.dimensions(tiff) is None
image = assets.read_image(tiff, name="scan.tiff")
assert image.width is None and image.height is None
assert "dimensions unknown" in assets.render_block(image)
# --- the name is content-addressed -----------------------------------------
def test_asset_name_is_digest_plus_a_reduced_original() -> None:
data = _jpeg(360, 269)
image = assets.read_image(data, name="25-0143 - Tabeller - Prosesskoden (R761-R762).jpg")
name = assets.asset_name(image)
assert name.startswith(hashlib.sha256(data).hexdigest()[: assets.DIGEST_PREFIX] + "-")
assert name.endswith(".jpg")
# The link target sits inside `](...)`, which a space or a parenthesis
# terminates -- and `structure._LINK` reads exactly that grammar. A name
# carrying either would produce a pointer no reader can follow.
assert re.fullmatch(r"[a-z0-9][a-z0-9.-]*", name), name
def test_two_drops_of_one_image_are_one_asset() -> None:
data = _png(4, 3)
first = assets.read_image(data, name="figur.png")
second = assets.read_image(data, name="figur.png")
assert assets.asset_name(first) == assets.asset_name(second)
def test_one_name_over_two_contents_stays_two_assets() -> None:
a = assets.read_image(_png(4, 3), name="figur.png")
b = assets.read_image(_png(5, 3), name="figur.png")
assert assets.asset_name(a) != assets.asset_name(b)
def test_a_nameless_image_still_gets_a_name() -> None:
image = assets.read_image(_png(4, 3), name="")
assert re.fullmatch(r"[0-9a-f]{12}\.png", assets.asset_name(image))
# --- the pointer is one grammar --------------------------------------------
def test_block_names_the_original_the_size_and_the_label() -> None:
image = assets.read_image(
_jpeg(360, 269),
name="25-0143 - Tabeller - Prosesskoden (R761-R762).jpg",
label="Tabell 84-2",
)
block = assets.render_block(image)
first, second = block.split("\n")
assert first == f"![Tabell 84-2]({assets.asset_href(image)})"
assert second.startswith("Image: 25-0143 - Tabeller - Prosesskoden (R761-R762).jpg")
assert "360x269 px" in second
assert second.endswith("Tabell 84-2")
def test_the_label_falls_back_to_the_original_name() -> None:
"""Two of the four readers have no caption element at all.
A PDF image object and an STS `<graphic>` carry no caption: the caption on
the page is running text the extractor already emitted. Inventing one from
the nearest line would be an unmarked heuristic, so the alt slot carries
the name the source gave the file.
"""
image = assets.read_image(_png(4, 3), name="graphic_0003.jpg")
assert assets.render_block(image).startswith("![graphic_0003.jpg](")
def test_a_bracket_in_a_label_cannot_break_the_link() -> None:
image = assets.read_image(_png(4, 3), name="f.png", label="Tabell [84-2] jf. pkt (3)")
first = assets.render_block(image).split("\n")[0]
assert first == f"![Tabell (84-2) jf. pkt (3)]({assets.asset_href(image)})"
def test_a_newline_in_a_label_cannot_open_a_third_line() -> None:
image = assets.read_image(_png(4, 3), name="f.png", label="Tabell\n84-2")
assert len(assets.render_block(image).split("\n")) == 2
def test_the_pointer_regex_finds_every_shipped_block() -> None:
"""Step 2 has to find these mechanically; the finder ships with the writer."""
images = [
assets.read_image(_png(4, 3), name="a.png", label="Figur 1"),
assets.read_image(_jpeg(9, 9), name="b.jpg"),
]
text = "Prosess 84\n\n" + "\n\n".join(assets.render_block(i) for i in images) + "\n\nSlutt\n"
found = assets.IMAGE_POINTER.findall(text)
assert [match[1] for match in found] == [assets.asset_name(i) for i in images]
def test_href_is_bundle_absolute() -> None:
"""SPEC SS 6.2 allows a bundle-relative path with a leading `/`.
A concept can sit at any depth under a segmented bundle, so a relative
`assets/...` would resolve differently from two concepts of one document.
"""
image = assets.read_image(_png(4, 3), name="f.png")
assert assets.asset_href(image) == f"/{assets.ASSETS_DIR}/{assets.asset_name(image)}"
# --- PNG encoding, for samples that arrive without a container -------------
#
# A PDF image is usually not a file: `FlateDecode` hands back raw samples with
# the colour model in the dictionary beside them. Encoding those is the only
# way to carry them at all, and it is stdlib (`zlib`), so the `pdf` row does
# not gain a dependency and the output is not bound to a renderer's version.
def test_png_from_gray_samples_round_trips_the_header() -> None:
encoded = assets.encode_png(3, 2, bytes([0, 64, 128, 192, 255, 32]), channels=1)
assert assets.sniff(encoded) == ("image/png", ".png")
assert assets.dimensions(encoded) == (3, 2)
def test_png_from_rgb_samples_round_trips_the_header() -> None:
encoded = assets.encode_png(2, 1, bytes(range(6)), channels=3)
assert assets.dimensions(encoded) == (2, 1)
def test_png_from_indexed_samples_carries_the_palette() -> None:
palette = bytes([255, 0, 0, 0, 255, 0])
encoded = assets.encode_png(2, 1, bytes([0, 1]), channels=1, palette=palette)
assert assets.dimensions(encoded) == (2, 1)
assert b"PLTE" in encoded
def test_png_refuses_a_sample_count_that_does_not_fit() -> None:
"""Refused rather than padded: a short buffer is a misread dictionary."""
with pytest.raises(ExtractionError) as excinfo:
assets.encode_png(4, 4, b"\x00\x01", channels=1)
assert excinfo.value.code == "asset_samples_invalid"
def test_png_encoding_is_byte_stable() -> None:
first = assets.encode_png(3, 2, bytes([0, 64, 128, 192, 255, 32]), channels=1)
second = assets.encode_png(3, 2, bytes([0, 64, 128, 192, 255, 32]), channels=1)
assert first == second
def test_the_restated_reduction_is_materializes_own() -> None:
"""`assets` cannot import `materialize`, so the rule is held equal instead.
`extract.py`'s registry must not import the contract layer (its own
docstring says so, and `materialize` reaches `manifest` and `profiles`), so
the id-grammar reduction is written out a second time. This is what stops
the second copy becoming a second grammar.
"""
from llm_ingestion_okf.materialize import reduce_to_id_grammar
for value in (
"25-0143 - Tabeller - Prosesskoden (R761-R762)",
"Figur 11.1 Toleransekrav",
"grafikk_med_æøå",
"---",
"",
):
assert assets._reduce(value) == reduce_to_id_grammar(value), value
def test_one_image_reached_by_two_paths_is_one_asset() -> None:
"""The asset name reads the BASENAME, never the path the document used.
Measured on the fixture inbox before this rule existed: the HTML document
points at `graphics/figur-84-1.png` and the STS document at
`figur-84-1.png` (resolved through the `graphics/` sibling convention), so
one image was written twice, under two names, from one run -- with the
digest in both of them announcing that the bytes were identical. The path a
document happened to use is not a property of the picture.
"""
data = _png(6, 4)
through_directory = assets.read_image(data, name="graphics/figur-84-1.png")
bare = assets.read_image(data, name="figur-84-1.png")
assert assets.asset_name(through_directory) == assets.asset_name(bare)
# The full original is not lost -- it moves to the line a person reads.
assert "graphics/figur-84-1.png" in assets.render_block(through_directory)

38
tests/test_docs_promises.py
View file

@ -317,3 +317,41 @@ def test_the_readme_names_every_gate_the_command_accepts() -> None:
text = README.read_text(encoding="utf-8")
for name in GATE_NAMES:
assert f"`{name}`" in text, f"README does not name the gate {name}"
# --- the asset default is published and pinned (0.10.0) --------------------
#
# Same obligation as the gate marker above it, for the same reason: 0.10.0
# changes what `okf build` writes for every consumer whose sources carry
# pictures, and a documented default that can drift from the applied one is how
# F1 survived for months.
_ASSETS_LINE = re.compile(r"^<!-- cli-default-assets: (on|off) -->$", re.MULTILINE)
def test_the_readme_names_the_asset_default_the_build_command_applies() -> None:
from llm_ingestion_okf import cli
match = _ASSETS_LINE.search(README.read_text(encoding="utf-8"))
assert match is not None, (
"README.md carries no `<!-- cli-default-assets: ... -->` marker; without it "
"the documented default can drift from the one the command applies"
)
assert (match.group(1) == "on") is cli.DEFAULT_ASSETS
def test_the_readme_names_the_opt_out_that_reproduces_the_old_bytes() -> None:
text = README.read_text(encoding="utf-8")
assert "`--no-assets`" in text
assert "NOT CARRIED" in text
def test_the_readme_states_that_image_bytes_are_not_screened() -> None:
"""The boundary, published rather than left to be discovered.
The guard is text-only. A consumer weighing an untrusted drop has to be
able to learn which half of a concept was looked at without reading this
package's source.
"""
text = README.read_text(encoding="utf-8")
assert "image bytes are not screened" in text.lower()

91
tests/test_error_codes.py
View file

@ -13,7 +13,9 @@ import importlib.util
import json
import sqlite3
import sys
import tempfile
import urllib.error
import warnings
from pathlib import Path
from typing import Any
@ -700,3 +702,92 @@ def test_segmentation_plan_unmatched(tmp_path: Path) -> None:
root_frontmatter_values={"bundle_id": "b-1"},
)
assert code_of(excinfo) == "segmentation_plan_unmatched"
# --- asset codes (0.10.0) --------------------------------------------------
#
# One test per code, like every code above it. These five are the only codes in
# the registry that a caller is expected to COUNT rather than to act on: an
# image a reader could not carry becomes a row in the run log and a line in the
# concept, never a failed document.
def test_asset_type_unknown() -> None:
from llm_ingestion_okf import assets
with pytest.raises(ExtractionError) as excinfo:
assets.read_image(b"%PDF-1.7\n", name="figur.png")
assert excinfo.value.code == "asset_type_unknown"
def test_asset_samples_invalid() -> None:
from llm_ingestion_okf import assets
with pytest.raises(ExtractionError) as excinfo:
assets.encode_png(8, 8, b"\x00", channels=1)
assert excinfo.value.code == "asset_samples_invalid"
def test_asset_remote() -> None:
from llm_ingestion_okf.extract import extract_document
document = extract_document(
"side.html",
b'<html><body><img src="https://example.invalid/x.png" alt="x"></body></html>',
assets=True,
)
assert [item.code for item in document.rejected] == ["asset_remote"]
def test_asset_unresolved() -> None:
from llm_ingestion_okf.extract import extract_document
document = extract_document(
"side.html",
b'<html><body><img src="mangler.png" alt="x"></body></html>',
assets=True,
)
assert [item.code for item in document.rejected] == ["asset_unresolved"]
def test_asset_pdf_unsupported() -> None:
"""A PDF image this encoder refuses rather than approximates.
A 1-bit stencil: carrying it at 8 bits would be a decision about what black
means, and a wrong one is indistinguishable from a right one in the output.
"""
pytest.importorskip("pdfplumber")
from llm_ingestion_okf.extract import extract_document
data = (
Path(__file__).parent / "fixtures" / "image-inbox" / "prosess-84-tabell.pdf"
).read_bytes()
stencil = data.replace(
b"/BitsPerComponent 8 /Filter /FlateDecode", b"/BitsPerComponent 1 /Filter /FlateDecode", 1
)
assert stencil != data
with warnings.catch_warnings():
warnings.simplefilter("ignore")
document = extract_document("krav.pdf", stencil, assets=True)
assert "asset_pdf_unsupported" in [item.code for item in document.rejected]
def test_asset_collision() -> None:
"""Two different pictures reducing to one asset name, refused in the run.
Constructed rather than found: the name carries 12 hex of the digest of its
own bytes, so reaching this by accident is a 48-bit collision. The code
exists because resolving it silently would lose one of the two pictures
while every pointer to it kept showing the other.
"""
from llm_ingestion_okf.assets import ExtractedImage, asset_name
from llm_ingestion_okf.inbox import _write_assets
image = ExtractedImage(b"AAAA", "f.png", "image/png", ".png", 1, 1)
# One name already holding DIFFERENT bytes, which is what a 48-bit digest
# collision would look like from inside the run.
seen = {asset_name(image): b"BBBB"}
with tempfile.TemporaryDirectory() as root:
with pytest.raises(MaterializationError) as excinfo:
_write_assets(Path(root) / "bundle", [image], seen)
assert excinfo.value.code == "asset_collision"