open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity
llm-ingestion-okf/tests/fixtures/make_image_fixtures.py
Kjell Tore Guttormsen bc39e8091f feat(assets): a bundle carries the images its sources declare (0.10.0) 
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 10:01:31 +02:00

327 lines
15 KiB
Python
Raw Blame History

"""Regenerate the image-bearing fixtures for the asset path (0.10.0).
Five documents, one per reader, each carrying a KNOWN number of images so a
gate can state "carried N of M" with M read out of the source rather than out
of this package. Written in a second file rather than appended to
`make_fixtures.py` for one reason: every fixture that file emits is byte-pinned
by a test, and the object numbering of the PDF builders is part of those bytes.
Adding an XObject to a shared builder would regenerate files whose whole value
is that they have not moved.
The same policy holds here as there: no generator library. The PNG is written
out with `zlib` from the stdlib, the JPEG as a header sequence (the readers
copy JPEG bytes through and read nothing but the frame marker, so a decodable
photograph would test nothing extra and could not be hand-audited), and the
containers are assembled part by part.
Run from the repository root: python3 tests/fixtures/make_image_fixtures.py
"""
from __future__ import annotations
import io
import struct
import zipfile
import zlib
from pathlib import Path
HERE = Path(__file__).parent
IMAGES = HERE / "image-inbox"
_XML = '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
_ZIP_DATE = (2020, 1, 1, 0, 0, 0)
def png(width: int, height: int, value: int = 0x40) -> bytes:
"""A real, single-channel PNG of a flat grey."""
def chunk(kind: bytes, payload: bytes) -> bytes:
return (
len(payload).to_bytes(4, "big")
+ kind
+ payload
+ zlib.crc32(kind + payload).to_bytes(4, "big")
)
ihdr = struct.pack(">IIBBBBB", width, height, 8, 0, 0, 0, 0)
raw = b"".join(b"\x00" + bytes([value] * width) for _ in range(height))
return (
b"\x89PNG\r\n\x1a\n"
+ chunk(b"IHDR", ihdr)
+ chunk(b"IDAT", zlib.compress(raw, 9))
+ chunk(b"IEND", b"")
)
def jpeg_header(width: int, height: int) -> bytes:
"""A JPEG's marker sequence: SOI, JFIF, a baseline frame header, EOI.
Not a decodable photograph, and that is the point of it. The `pdf` reader
passes `DCTDecode` bytes through untouched and reads only the frame header
for the size, which is precisely what R761's own table images need -- 29 of
its 50 image objects are `DCTDecode`. A fixture that also carried entropy
data would exercise no additional line of this package and could not be read
byte by byte by a person.
"""
frame = bytes([8, height >> 8, height & 0xFF, width >> 8, width & 0xFF, 1, 1, 0x11, 0])
return (
b"\xff\xd8"
b"\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00"
+ b"\xff\xc0"
+ (len(frame) + 2).to_bytes(2, "big")
+ frame
+ b"\xff\xd9"
)
# --- pdf ---------------------------------------------------------------------
#
# TWO images on one page, and they are deliberately of the two kinds the
# measurement on R761 found: 29 `DCTDecode` objects, which arrive as a finished
# JPEG file, and 21 `FlateDecode` ones, which arrive as raw samples with the
# colour model in the dictionary beside them and have to be encoded to be
# carried at all. A fixture with only one kind would leave half the reader
# unexercised, and it is the encoded half that can be silently wrong.
PDF_GRAY_WIDTH, PDF_GRAY_HEIGHT = 4, 3
PDF_GRAY_SAMPLES = bytes([0, 60, 120, 180, 20, 80, 140, 200, 40, 100, 160, 255])
PDF_JPEG_WIDTH, PDF_JPEG_HEIGHT = 360, 269
PDF_CONTENT = (
b"BT /F1 12 Tf 20 170 Td (Toleranseklasse er gitt i tabell 84-2) Tj ET\n"
b"q 80 0 0 60 20 90 cm /ImFlate Do Q\n"
b"q 80 0 0 60 20 20 cm /ImJpeg Do Q\n"
)
def build_image_pdf() -> bytes:
"""A one-page PDF with a Flate image and a DCT image in its resources."""
flate = zlib.compress(PDF_GRAY_SAMPLES, 9)
jpeg = jpeg_header(PDF_JPEG_WIDTH, PDF_JPEG_HEIGHT)
objects = [
b"<< /Type /Catalog /Pages 2 0 R >>",
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 4 0 R "
b"/Resources << /Font << /F1 5 0 R >> "
b"/XObject << /ImFlate 6 0 R /ImJpeg 7 0 R >> >> >>",
b"<< /Length "
+ str(len(PDF_CONTENT)).encode()
+ b" >>\nstream\n"
+ PDF_CONTENT
+ b"endstream",
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>",
b"<< /Type /XObject /Subtype /Image /Width "
+ str(PDF_GRAY_WIDTH).encode()
+ b" /Height "
+ str(PDF_GRAY_HEIGHT).encode()
+ b" /ColorSpace /DeviceGray /BitsPerComponent 8 /Filter /FlateDecode /Length "
+ str(len(flate)).encode()
+ b" >>\nstream\n"
+ flate
+ b"\nendstream",
b"<< /Type /XObject /Subtype /Image /Width "
+ str(PDF_JPEG_WIDTH).encode()
+ b" /Height "
+ str(PDF_JPEG_HEIGHT).encode()
+ b" /ColorSpace /DeviceGray /BitsPerComponent 8 /Filter /DCTDecode /Length "
+ str(len(jpeg)).encode()
+ b" >>\nstream\n"
+ jpeg
+ b"\nendstream",
]
out = bytearray(b"%PDF-1.4\n")
offsets = []
for number, body in enumerate(objects, start=1):
offsets.append(len(out))
out += str(number).encode() + b" 0 obj\n" + body + b"\nendobj\n"
xref_at = len(out)
size = str(len(objects) + 1).encode()
out += b"xref\n0 " + size + b"\n0000000000 65535 f \n"
for offset in offsets:
out += ("%010d 00000 n \n" % offset).encode()
out += b"trailer\n<< /Size " + size + b" /Root 1 0 R >>\n"
out += b"startxref\n" + str(xref_at).encode() + b"\n%%EOF\n"
return bytes(out)
# --- html --------------------------------------------------------------------
#
# THREE `<img>` and only two of them can be carried. The remote one is the
# boundary written as a fixture: this package never opens a socket during
# extraction (the network gate is an explicit per-run opt-in and extraction is
# not on that path), so a remote source becomes a pointer WITHOUT a file, and
# the gate counts it as found-and-not-carried rather than as absent.
HTML_DOCUMENT = """<!doctype html>
<html><head><title>Prosess 84</title></head>
<body>
<h1>84 Konstruksjoner av betong</h1>
<p>Toleranseklasse for de enkelte konstruksjonsdeler er gitt i tabell 84-2.</p>
<img src="graphics/tabell-84-2.png" alt="Tabell 84-2 Toleranseklasser">
<p>Figuren under viser prinsippet.</p>
<figure>
<img src="graphics/figur-84-1.png" alt="Figur 84-1 Prinsippskisse">
<figcaption>Figur 84-1 Prinsippskisse</figcaption>
</figure>
<p>Og en som ligger et annet sted:</p>
<img src="https://example.invalid/ekstern.png" alt="Ekstern figur">
</body></html>
"""
# --- niso-sts ----------------------------------------------------------------
#
# The shape the R761 delivery actually has, measured 2026-09-16: 50 `<graphic>`
# elements, every one a direct child of a `<sec>`, none inside a `<table-wrap>`,
# each carrying a bare file name in `xlink:href` that resolves against a sibling
# `graphics/` directory. No `<caption>` anywhere near them -- the caption a
# human reads is a `<p>` the extractor already emits on its own line.
STS_DOCUMENT = """<?xml version="1.0" encoding="UTF-8"?>
<standard xmlns:xlink="http://www.w3.org/1999/xlink">
<front><std-ident><doc-number>R761</doc-number></std-ident></front>
<body>
<sec>
<label>84</label>
<title>Konstruksjoner av betong</title>
<sec>
<label>84.1</label>
<title>Toleranser</title>
<p>Toleranseklasse er gitt i tabell 84-2.</p>
<graphic xlink:href="graphics/tabell-84-2.png"/>
<p>Figur 84-1 viser prinsippet.</p>
<graphic xlink:href="figur-84-1.png"/>
</sec>
</sec>
</body>
</standard>
"""
def build_docx() -> bytes:
"""A `.docx` with one embedded image, its alt text on the drawing."""
image = png(40, 30, value=0x30)
parts: dict[str, str | bytes] = {
"[Content_Types].xml": _XML
+ '<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
+ '<Default Extension="xml" ContentType="application/xml"/>'
+ '<Default Extension="rels" ContentType="application/vnd.openxmlformats-package'
+ '.relationships+xml"/>'
+ '<Default Extension="png" ContentType="image/png"/>'
+ '<Override PartName="/word/document.xml" ContentType="application/vnd'
+ '.openxmlformats-officedocument.wordprocessingml.document.main+xml"/>'
+ "</Types>",
"_rels/.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006'
+ '/relationships/officeDocument" Target="word/document.xml"/></Relationships>',
"word/_rels/document.xml.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rIdImg" Type="http://schemas.openxmlformats.org/officeDocument'
+ '/2006/relationships/image" Target="media/tabell-84-2.png"/></Relationships>',
"word/media/tabell-84-2.png": image,
"word/document.xml": _XML
+ '<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"'
+ ' xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships"'
+ ' xmlns:wp="http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing"'
+ ' xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main"'
+ ' xmlns:pic="http://schemas.openxmlformats.org/drawingml/2006/picture"><w:body>'
+ "<w:p><w:r><w:t>Toleranseklasse er gitt i tabell 84-2.</w:t></w:r></w:p>"
+ '<w:p><w:r><w:drawing><wp:inline><wp:extent cx="381000" cy="285750"/>'
+ '<wp:docPr id="1" name="Bilde 1" descr="Tabell 84-2 Toleranseklasser"/>'
+ '<a:graphic><a:graphicData uri="http://schemas.openxmlformats.org/drawingml/2006'
+ '/picture"><pic:pic><pic:nvPicPr><pic:cNvPr id="1" name="tabell-84-2.png"'
+ ' descr="Tabell 84-2 Toleranseklasser"/><pic:cNvPicPr/></pic:nvPicPr>'
+ '<pic:blipFill><a:blip r:embed="rIdImg"/><a:stretch><a:fillRect/></a:stretch>'
+ '</pic:blipFill><pic:spPr><a:xfrm><a:off x="0" y="0"/>'
+ '<a:ext cx="381000" cy="285750"/></a:xfrm>'
+ '<a:prstGeom prst="rect"><a:avLst/></a:prstGeom></pic:spPr></pic:pic>'
+ "</a:graphicData></a:graphic></wp:inline></w:drawing></w:r></w:p>"
+ "<w:p><w:r><w:t>Etter tabellen gjelder NS-EN 13670.</w:t></w:r></w:p>"
+ "</w:body></w:document>",
}
return build_container(parts)
def build_pptx() -> bytes:
"""A `.pptx` with one titled slide and one embedded image on it."""
image = png(48, 36, value=0x70)
parts: dict[str, str | bytes] = {
"[Content_Types].xml": _XML
+ '<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
+ '<Default Extension="xml" ContentType="application/xml"/>'
+ '<Default Extension="rels" ContentType="application/vnd.openxmlformats-package'
+ '.relationships+xml"/>'
+ '<Default Extension="png" ContentType="image/png"/>'
+ '<Override PartName="/ppt/presentation.xml" ContentType="application/vnd'
+ '.openxmlformats-officedocument.presentationml.presentation.main+xml"/>'
+ '<Override PartName="/ppt/slides/slide1.xml" ContentType="application/vnd'
+ '.openxmlformats-officedocument.presentationml.slide+xml"/>'
+ "</Types>",
"_rels/.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006'
+ '/relationships/officeDocument" Target="ppt/presentation.xml"/></Relationships>',
"ppt/_rels/presentation.xml.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rId2" Type="http://schemas.openxmlformats.org/officeDocument/2006'
+ '/relationships/slide" Target="slides/slide1.xml"/></Relationships>',
"ppt/presentation.xml": _XML
+ '<p:presentation xmlns:p="http://schemas.openxmlformats.org/presentationml/2006/main"'
+ ' xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">'
+ '<p:sldIdLst><p:sldId id="256" r:id="rId2"/></p:sldIdLst></p:presentation>',
"ppt/slides/_rels/slide1.xml.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rIdImg" Type="http://schemas.openxmlformats.org/officeDocument'
+ '/2006/relationships/image" Target="../media/skisse.png"/></Relationships>',
"ppt/media/skisse.png": image,
"ppt/slides/slide1.xml": _XML
+ '<p:sld xmlns:p="http://schemas.openxmlformats.org/presentationml/2006/main"'
+ ' xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main"'
+ ' xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">'
+ "<p:cSld><p:spTree>"
+ '<p:nvGrpSpPr><p:cNvPr id="1" name=""/><p:cNvGrpSpPr/><p:nvPr/></p:nvGrpSpPr>'
+ "<p:grpSpPr/>"
+ '<p:sp><p:nvSpPr><p:cNvPr id="2" name="Tittel 1"/><p:cNvSpPr/>'
+ '<p:nvPr><p:ph type="title"/></p:nvPr></p:nvSpPr><p:spPr/>'
+ "<p:txBody><a:bodyPr/><a:p><a:r><a:t>Toleranser</a:t></a:r></a:p></p:txBody></p:sp>"
+ '<p:pic><p:nvPicPr><p:cNvPr id="3" name="skisse.png" descr="Prinsippskisse"/>'
+ "<p:cNvPicPr/><p:nvPr/></p:nvPicPr>"
+ '<p:blipFill><a:blip r:embed="rIdImg"/><a:stretch><a:fillRect/></a:stretch>'
+ '</p:blipFill><p:spPr><a:xfrm><a:off x="0" y="0"/>'
+ '<a:ext cx="457200" cy="342900"/></a:xfrm>'
+ '<a:prstGeom prst="rect"><a:avLst/></a:prstGeom></p:spPr></p:pic>'
+ "</p:spTree></p:cSld></p:sld>",
}
return build_container(parts)
def build_container(parts: dict[str, str | bytes]) -> bytes:
out = io.BytesIO()
with zipfile.ZipFile(out, "w", compression=zipfile.ZIP_DEFLATED) as archive:
for name, payload in parts.items():
info = zipfile.ZipInfo(name, date_time=_ZIP_DATE)
info.compress_type = zipfile.ZIP_DEFLATED
archive.writestr(info, payload)
return out.getvalue()
if __name__ == "__main__":
(IMAGES / "graphics").mkdir(parents=True, exist_ok=True)
# FIVE DISTINCT STEMS. One stem across five extensions is refused by the
# door's own SS 3 collision rule -- measured while building this fixture:
# `prosess-84.{pdf,docx,pptx}` gave `inbox_slug_collision: 2/7` and two of
# the five readers were never exercised at all, with the gate reporting a
# carrying defect that was really a fixture defect.
written: list[tuple[str, bytes]] = [
("graphics/tabell-84-2.png", png(120, 90, value=0x20)),
("graphics/figur-84-1.png", png(64, 48, value=0x80)),
("prosess-84-web.html", HTML_DOCUMENT.encode("utf-8")),
("prosess-84-sts.xml", STS_DOCUMENT.encode("utf-8")),
("prosess-84-tabell.pdf", build_image_pdf()),
("prosess-84-notat.docx", build_docx()),
("prosess-84-presentasjon.pptx", build_pptx()),
]
for name, payload in written:
(IMAGES / name).write_bytes(payload)
print(f"wrote image-inbox/{name}")
Reference in a new issue View git blame Copy permalink