Kjell Tore Guttormsen
574cf61272
Green: 23 of 23 in the suite that was 3 red, and the sweep's carried-here- refused-there count goes 703 of 22 620 to 0, the drawn-differently count 1 492 to 32. Two clauses, both refusing with the code the path already uses: * an end-of-line escape at column 0. It closes no row, so the row it moves the cursor over is a row the stream never wrote -- PM's four-EOL frame, carried with 32 of 32 pixels never decoded while Pillow refuses the file. * a delta whose horizontal offset would leave the row. The format puts that offset inside the line; this reader keeps the cursor past the row end and a flat decoder rolls it into the next row, which is two pictures from one stream. CHOSEN OVER PM's RECOMMENDATION, and the recommendation was measured first: refusing only a stream that painted nothing leaves 512 streams carried here and refused there, and 1 171 drawn differently. It would have narrowed this class for the third round running instead of closing it. WHAT IS NOT CLOSED IS STATED, in the docstring and in the test: 32 of 22 620 streams are still drawn differently, every one of them a run or absolute block that overruns its row. Refusing those gives 0 and 0 -- and costs 15 of the 25 real RLE8 files, which would drop 15 figures and move a pinned bundle's bytes. Measured on the corpus first: over 11 441 files scanned across the four raw deliveries and the K2 reference corpus, the 25 BMPs on this machine use an end-of-line at column 0 in 0 of 25 and a delta in 0 of 25, and 25 of 25 still decode to Pillow's pixels exactly (3 117 220 pixels compared, after the change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1232 lines
56 KiB
Python
1232 lines
56 KiB
Python
"""Binary assets: what an image IS, what it is called, and how it is pointed at.
|
|
|
|
Until 0.10.0 this package had no image path at all. Every reader recovered
|
|
text, every warning said so, and the only writer into a bundle was
|
|
:func:`materialize.write_bytes`, whose signature is ``(bundle_dir, name,
|
|
content: str)`` -- UTF-8 in, text out, no binary route anywhere. A document
|
|
whose table is a raster picture therefore reached a concept as an absence with
|
|
no denominator. Measured on R761 Prosesskoden:2025, the publisher's own
|
|
NISO-STS delivery: the process text is carried in full, and 12 ``Tabell N-N``
|
|
and 9 ``Figur N-N`` captions stand over nothing, so process 84's
|
|
"toleranseklasse ... er gitt i tabell 84-2" points at empty space.
|
|
|
|
THIS MODULE IS THE ONE PLACE THAT DECIDES WHAT AN IMAGE IS. A reader hands it
|
|
bytes and a name; it returns a carried image or raises a coded rejection. That
|
|
is what makes "N carried of M found" mean the same thing for ``pdf``, ``docx``,
|
|
``html`` and ``xml``, and it is what keeps each format's quirks out of the
|
|
bundle layout.
|
|
|
|
THREE RULES, and each one exists because the alternative is a silent lie:
|
|
|
|
- **The type is SNIFFED, never claimed.** Measured on the R761 delivery, the
|
|
graphics directory holds ``.bmp``, ``.jpg`` and ``.png`` side by side and the
|
|
document's ``xlink:href`` values are whatever the publisher's tool wrote. A
|
|
name is a claim; the magic bytes are the fact. A consumer dispatching on the
|
|
extension of a name that lies reads the file wrong with full confidence.
|
|
- **The name is CONTENT-ADDRESSED** -- ``<sha256[:12]>-<reduced original>``.
|
|
Two drops of one image are one file, a rebuild of one corpus is one bundle,
|
|
and the digest carries the uniqueness so the readable tail can be shortened
|
|
without any risk of collision. That is the byte-determinism rule this package
|
|
already holds for text, extended to the bytes beside it.
|
|
- **The pointer is ONE GRAMMAR**, owned here. ``okf describe`` (step 2) has to
|
|
find every pointer mechanically in order to write a transcription under it,
|
|
so the block is a regex this module ships beside the writer rather than a
|
|
shape each reader invents and each consumer re-derives.
|
|
|
|
WHAT THIS MODULE DOES NOT DO: it never looks at a picture. Classifying an image
|
|
as a table or a figure, and reading what it says, is a model call, and the
|
|
invariant "no model calls anywhere in the run path" is not negotiated here --
|
|
step 2 is a separate command, outside the build path, and this module is
|
|
importable without it.
|
|
|
|
**The image BYTES are not screened.** The guard is text-only (its own boundary,
|
|
not ours), so what passes a persist gate is the pointer block, as body text,
|
|
like every other line. The bytes of a carried image are written to the bundle
|
|
unscreened. Stated here rather than implied, because a consumer weighing an
|
|
untrusted drop needs to know which half of the concept was looked at.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
import re
|
|
import struct
|
|
import unicodedata
|
|
import zlib
|
|
from collections.abc import Iterator
|
|
from dataclasses import dataclass
|
|
|
|
from .errors import ExtractionError
|
|
|
|
#: The one directory a bundle keeps its binary assets in, at the bundle root.
|
|
#: Fixed rather than configurable: a consumer resolving `/assets/...` out of a
|
|
#: concept has only the bundle, and a per-profile directory would make that
|
|
#: pointer unresolvable without also shipping the profile that wrote it.
|
|
ASSETS_DIR = "assets"
|
|
|
|
#: How much of the digest names the file. 12 hex characters is 48 bits; over
|
|
#: the largest asset population measured here (4 828 image objects in one
|
|
#: 33-document corpus) the birthday probability of a collision is about
|
|
#: 4e-11. A collision would be caught anyway -- an occupied name is re-used
|
|
#: only when the bytes there are already identical, the same content-identity
|
|
#: rule Door C proves ownership with.
|
|
DIGEST_PREFIX = 12
|
|
|
|
#: How much of the original name survives into the asset name. The digest
|
|
#: carries uniqueness, so this is decoration and truncating it is safe -- which
|
|
#: is the opposite of `materialize.check_filename_length`'s situation, where the
|
|
#: name IS the identity and truncation would silently merge two documents.
|
|
NAME_TAIL_MAX = 60
|
|
|
|
#: Magic bytes -> (media type, suffix). Sniffed in this order; the first match
|
|
#: wins, and nothing here overlaps.
|
|
_MAGIC: tuple[tuple[bytes, str, str], ...] = (
|
|
(b"\x89PNG\r\n\x1a\n", "image/png", ".png"),
|
|
(b"\xff\xd8\xff", "image/jpeg", ".jpg"),
|
|
(b"GIF87a", "image/gif", ".gif"),
|
|
(b"GIF89a", "image/gif", ".gif"),
|
|
(b"BM", "image/bmp", ".bmp"),
|
|
(b"II\x2a\x00", "image/tiff", ".tiff"),
|
|
(b"MM\x00\x2a", "image/tiff", ".tiff"),
|
|
)
|
|
|
|
#: The formats a model can be SHOWN. Everything a document ships outside this
|
|
#: set is converted losslessly to PNG, or refused with a code -- never carried
|
|
#: silently, which is what this package did until this round of 0.10.1.
|
|
#:
|
|
#: MEASURED 2026-09-19 over the frozen R761 delivery's own `assets/`
|
|
#: (denominator 50): 29 JPEG, 2 PNG and **19 "PC bitmap, Windows 3.x, 8-bit,
|
|
#: compression 1"**. The 19 are byte-correct files that nothing reads, so 19 of
|
|
#: that document's figures were present and invisible at the same time -- and
|
|
#: the `images: N` count said they had arrived. An absence a reader is shown is
|
|
#: information; a picture that is there and unreadable is worse than either.
|
|
#:
|
|
#: IT IS A PROPERTY, NOT A LIST OF FORMATS WE HAPPENED TO MEET. A carried
|
|
#: asset's type is read off its bytes and tested against this set, so a format
|
|
#: nobody here has seen is refused by the same rule that refuses TIFF.
|
|
#:
|
|
#: WebP is on the list and `sniff` does not recognise it: the set states what a
|
|
#: model can be shown, not what this package can read. A WebP is therefore
|
|
#: refused one step earlier, as `asset_type_unknown`, and never reaches this
|
|
#: test. Stating that is cheaper than a set whose name is wider than its reach.
|
|
VIEWABLE_MEDIA_TYPES = frozenset({"image/png", "image/jpeg", "image/gif", "image/webp"})
|
|
|
|
#: JPEG 2000, in both the forms a PDF `JPXDecode` stream hands back: the JP2
|
|
#: container and a bare codestream.
|
|
_JP2_SIGNATURE = b"\x00\x00\x00\x0cjP \r\n\x87\n"
|
|
_J2K_SIGNATURE = b"\xff\x4f\xff\x51"
|
|
|
|
#: The frame markers that carry a JPEG's dimensions. Every SOF except the four
|
|
#: that are not frame headers at all (`DHT` 0xC4, `JPG` 0xC8, `DAC` 0xCC).
|
|
_JPEG_SOF = frozenset(range(0xC0, 0xD0)) - {0xC4, 0xC8, 0xCC}
|
|
|
|
#: `materialize.reduce_to_id_grammar`'s rule, restated. Not imported: this
|
|
#: module is reached from `extract.py`, whose registry must not import the
|
|
#: contract layer, and `materialize` pulls in `manifest` and `profiles`.
|
|
#: `tests/test_assets.py` holds the two forms equal on the same inputs, so the
|
|
#: restatement cannot drift into a second grammar.
|
|
_SEPARATOR_RUN = re.compile(r"[^a-z0-9]+")
|
|
|
|
#: One pointer block, as this module writes it. Group 1 is the label, group 2
|
|
#: is the asset file name, group 3 is the whole second line. `okf describe`
|
|
#: finds its work with this and writes under the match; a consumer wanting to
|
|
#: strip pointers uses the same expression, so there is one definition of what
|
|
#: a pointer looks like rather than one per reader.
|
|
IMAGE_POINTER = re.compile(
|
|
r"^!\[(?P<label>[^\]\n]*)\]\(/" + ASSETS_DIR + r"/(?P<asset>[^)\s]+)\)\n"
|
|
r"(?P<detail>Image: [^\n]*)$",
|
|
re.MULTILINE,
|
|
)
|
|
|
|
|
|
#: THE SIZE A PICTURE MAY DECLARE. Read off the corpora rather than chosen:
|
|
#: over the 4 828 image objects of the 43-document reference corpus the largest
|
|
#: is 4 515 x 4 128 (18.6 MP, a landscape drawing), and over R761's 109
|
|
#: delivered pictures it is 2 072 x 656 (1.4 MP). This sits an order of
|
|
#: magnitude above both, so nothing anyone has measured is refused -- and a
|
|
#: declared size beyond it is a COUNTED refusal instead of a killed build.
|
|
#:
|
|
#: The bound exists because a few kilobytes can declare an enormous picture: a
|
|
#: 9.6 KB PDF declaring 3 000 x 3 000 grayscale zeros took 83 MB of peak RSS
|
|
#: and a 63 KB one declaring 8 000 x 8 000 took 276 MB, linear in the pixel
|
|
#: count, so 50 000 x 50 000 is several GB and takes the whole batch with it.
|
|
#: Found by an independent review of 0.10.0 before it was pushed.
|
|
MAX_IMAGE_PIXELS = 40_000_000
|
|
|
|
#: The same bound in bytes: for a payload whose pixels are not yet known (an
|
|
#: inline `data:` URI) and for the DECOMPRESSED size of a stream, which is a
|
|
#: different number from anything the container declares. `/Length` in a PDF
|
|
#: image dictionary is the COMPRESSED length and `/Width` and `/Height` are a
|
|
#: claim; nothing in the dictionary states what the decompression will cost.
|
|
MAX_IMAGE_BYTES = 256 * 1024 * 1024
|
|
|
|
#: How much of a stream is inflated at a time while `inflated_size` measures
|
|
#: it. The cap on the OUTPUT is what keeps the measurement cheaper than the
|
|
#: bomb; the input is handed over whole because it is already in memory.
|
|
_INFLATE_CHUNK = 1 << 20
|
|
|
|
|
|
def check_size(
|
|
width: int | None, height: int | None, *, name: str, channels: int = 1, bits: int = 8
|
|
) -> None:
|
|
"""Refuse a declared size beyond the bound, BEFORE anything is decoded.
|
|
|
|
Read off what the container DECLARES. That is the only number available
|
|
before a cost is paid, and it is a CLAIM by an untrusted document rather
|
|
than a measurement: this refuses a picture too large to be one we carry,
|
|
and it says nothing about what decompressing the stream beside it costs.
|
|
`inflated_size` is the other half, and neither substitutes for the other.
|
|
|
|
`None` is UNKNOWN -- a container that declares no size leaves nothing to
|
|
bound, and inventing a number would refuse a legitimate picture. A size
|
|
that is declared and is not positive is neither unknown nor large: it is a
|
|
dictionary that was written wrong or written to be read wrong, and it is
|
|
refused here rather than three steps later by the encoder. Measured on
|
|
`230d1cb`: `/Width -1 /Height 40000000000` multiplies to a NEGATIVE pixel
|
|
count, so every `>` below was false, the function returned silently, 400 MB
|
|
was decompressed, and the refusal arrived from `encode_png` under
|
|
`asset_samples_invalid` -- a code about a sample buffer for a defect in the
|
|
declaration.
|
|
"""
|
|
if width is None or height is None:
|
|
return
|
|
if width <= 0 or height <= 0:
|
|
raise ExtractionError(
|
|
f"the image {name!r} declares {width}x{height}, which is not a size; "
|
|
"refused unread rather than multiplied out, because a non-positive "
|
|
"dimension makes every bound below it read as satisfied",
|
|
code="asset_size_invalid",
|
|
)
|
|
pixels = width * height
|
|
expected = pixels * channels * (bits // 8 or 1)
|
|
if pixels > MAX_IMAGE_PIXELS or expected > MAX_IMAGE_BYTES:
|
|
raise ExtractionError(
|
|
f"the image {name!r} declares {width}x{height} = {pixels} pixels "
|
|
f"({expected} bytes of samples), over this package's bound of "
|
|
f"{MAX_IMAGE_PIXELS} pixels and {MAX_IMAGE_BYTES} bytes; refused "
|
|
"unread so one picture cannot take the run with it",
|
|
code="asset_too_large",
|
|
)
|
|
|
|
|
|
def check_payload(size: int, *, name: str) -> None:
|
|
"""The same bound for an encoded payload of `size` bytes."""
|
|
if size > MAX_IMAGE_BYTES:
|
|
raise ExtractionError(
|
|
f"the image {name!r} carries {size} encoded bytes, over this package's "
|
|
f"bound of {MAX_IMAGE_BYTES}; refused unread",
|
|
code="asset_too_large",
|
|
)
|
|
|
|
|
|
#: What decoding ONE link of a PDF filter chain may cost this package, in bytes
|
|
#: of memory. A SEPARATE number from `MAX_IMAGE_BYTES`, and the distinction is
|
|
#: the whole of round 3: that one bounds the picture this package will carry,
|
|
#: this one bounds what producing it costs on the way. Three rounds of this
|
|
#: review each bound an output and the bomb moved one link along, because a
|
|
#: decoder's working set is not its output. Twice `MAX_IMAGE_BYTES`, so a run
|
|
#: may hold the stream it was given and one stage of decoding at once and no
|
|
#: more.
|
|
MAX_FILTER_DECODE_BYTES = 512 * 1024 * 1024
|
|
|
|
#: MEASURED peak memory per byte of INPUT, for each filter this package lets an
|
|
#: image be reached through. `None` means the decoder is driven a chunk at a
|
|
#: time here, so the cost is measured as it is paid and no ratio is needed --
|
|
#: today that is `FlateDecode` alone (`_inflate`).
|
|
#:
|
|
#: The numbers are read off CPython 3.14 on 2026-09-18, worst case per filter:
|
|
#:
|
|
#: * `ASCII85Decode` 101.4x at 1 MiB of input, 96.1x at 4 MiB, 94.5x at 16 MiB.
|
|
#: `z` is the shorthand for four zero bytes, so `base64.a85decode` appends one
|
|
#: 4-byte object per INPUT byte to a list -- the output ratio is 4, the cost
|
|
#: ratio is a hundred, and 0.10.1 documented this filter as "bounded by its
|
|
#: own input because it shrinks". The constant sits above the worst of the
|
|
#: three, and `test_the_ascii85_cost_ratio_is_not_below_the_one_this_package
|
|
#: _measured` re-measures it so it cannot rot when CPython changes.
|
|
#: * `ASCIIHexDecode` 1.5x at 16 MiB: it strips whitespace into a copy and
|
|
#: `unhexlify`s that, and its output is half its input.
|
|
#: * `DCTDecode`, `JPXDecode` and `JBIG2Decode` are pass-through in pdfminer --
|
|
#: the bytes are handed to the image reader unchanged -- so the ratio is 1.
|
|
#:
|
|
#: A filter that is not in this table has no measured ratio and is refused
|
|
#: unread (`asset_pdf_unbounded`). That is the same decision `corpus.resolve
|
|
#: _gate` takes for an unknown gate name: a fallback reproduces the defect with
|
|
#: an extra step.
|
|
PDF_FILTER_COST_RATIO: dict[str, float | None] = {
|
|
"FlateDecode": None,
|
|
"ASCII85Decode": 104.0,
|
|
"ASCIIHexDecode": 2.0,
|
|
"DCTDecode": 1.0,
|
|
"JPXDecode": 1.0,
|
|
"JBIG2Decode": 1.0,
|
|
}
|
|
|
|
#: The largest OUTPUT each of those filters can produce per byte of input, used
|
|
#: to carry a bound forward when the bytes themselves have been discarded.
|
|
#: `ASCII85Decode` is 4 (one `z`), `ASCIIHexDecode` 0.5 (two digits to a byte),
|
|
#: pass-through 1. `FlateDecode` has no such number, which is why it is the one
|
|
#: filter measured a chunk at a time.
|
|
PDF_FILTER_OUTPUT_RATIO: dict[str, float | None] = {
|
|
"FlateDecode": None,
|
|
"ASCII85Decode": 4.0,
|
|
"ASCIIHexDecode": 0.5,
|
|
"DCTDecode": 1.0,
|
|
"JPXDecode": 1.0,
|
|
"JBIG2Decode": 1.0,
|
|
}
|
|
|
|
|
|
def filter_input_limit(canonical: str) -> int | None:
|
|
"""The largest input this package will hand to `canonical`'s decoder.
|
|
|
|
`None` for a filter decoded a chunk at a time, which needs no input limit
|
|
because its cost is measured while it is paid.
|
|
|
|
The number this produces for `ASCII85Decode` -- about 5.0 MB -- is READ OFF
|
|
the corpora the way `MAX_IMAGE_PIXELS` is: over the 9 668 image objects of
|
|
the 77 PDFs on this machine (2026-09-18), 16 decode through an
|
|
`ASCII85Decode` link and the largest input to one is 450 739 bytes, so the
|
|
limit stands more than ten times above anything measured.
|
|
"""
|
|
ratio = PDF_FILTER_COST_RATIO.get(canonical)
|
|
if ratio is None:
|
|
return None
|
|
return int(MAX_FILTER_DECODE_BYTES // ratio)
|
|
|
|
|
|
def check_filter_cost(size: int, *, canonical: str, name: str) -> None:
|
|
"""Refuse a link whose decoder would cost more than the budget, BEFORE it
|
|
decodes anything.
|
|
|
|
This is the half `inflated_size` cannot cover. That one drives zlib a chunk
|
|
at a time and stops the moment the running total crosses the bound, which
|
|
is only possible because zlib hands its output over incrementally. Nothing
|
|
else in a PDF filter chain does: `base64.a85decode` is asked for a whole
|
|
string and gives back a whole string, so by the time its output could be
|
|
measured the memory has been spent. For those the cost is PREDICTED from a
|
|
measured ratio and the input size, and predicted before the call.
|
|
"""
|
|
limit = filter_input_limit(canonical)
|
|
if limit is None or size <= limit:
|
|
return
|
|
ratio = PDF_FILTER_COST_RATIO[canonical]
|
|
raise ExtractionError(
|
|
f"the image {name!r} hands {size} bytes to {canonical}, whose decoder costs "
|
|
f"about {ratio} bytes of memory per byte of input -- over this package's "
|
|
f"budget of {MAX_FILTER_DECODE_BYTES} bytes for one link; refused before "
|
|
"the decode, because a bound on what a link OUTPUTS is not a bound on "
|
|
"what producing it costs",
|
|
code="asset_too_large",
|
|
)
|
|
|
|
|
|
def inflate_limit_for(canonical: str | None) -> int:
|
|
"""How much a `FlateDecode` link may produce, given what comes AFTER it.
|
|
|
|
The picture's own bound is `MAX_IMAGE_BYTES`, but a link's output is the
|
|
next link's input, and a decoder with a cost ratio cannot be handed more
|
|
than `filter_input_limit` allows. Carrying the budget down the chain this
|
|
way is what stops `[/FlateDecode /ASCII85Decode]` from inflating 256 MiB of
|
|
`z` before the link behind it is asked anything.
|
|
"""
|
|
limit = MAX_IMAGE_BYTES
|
|
if canonical is not None:
|
|
behind = filter_input_limit(canonical)
|
|
if behind is not None:
|
|
limit = min(limit, behind)
|
|
return limit
|
|
|
|
|
|
def inflated_size(raw: bytes, *, name: str, limit: int | None = None) -> int:
|
|
"""What a deflate stream costs to decompress, measured without paying it.
|
|
|
|
THE DECLARED SIZE AND THE COST ARE TWO INDEPENDENT NUMBERS, and binding
|
|
only the first is what an independent review of 0.10.1 measured on
|
|
`230d1cb`: a 408 516-byte PDF declaring a 1x1 picture and carrying 400 MB
|
|
of deflated zeros was CARRIED, with no rejection, at 892 MB of peak RSS --
|
|
about 2 100x the file size, linear, so a 10 MB document is ~21 GB and takes
|
|
the whole batch build with it. `check_size` was reading a claim as though
|
|
it were a cost.
|
|
|
|
The output is inflated a chunk at a time and DISCARDED: only the running
|
|
total is kept, so the measurement stays bounded whatever the stream holds,
|
|
and it stops at the first chunk that crosses `limit`. A legitimate image is
|
|
therefore inflated twice -- once here and once by the reader that carries
|
|
it -- which is the price of not holding an unbounded buffer to find out how
|
|
big it is. Measured on R761 Prosesskoden:2025 (50 image objects): the
|
|
second pass costs under a second of a 200-second extraction.
|
|
|
|
A stream that is not valid deflate data is not this function's problem: the
|
|
reader behind it reports that in its own vocabulary, so a `zlib.error` ends
|
|
the measurement at whatever was produced up to it.
|
|
"""
|
|
return sum(len(chunk) for chunk in _inflate(raw, name=name, limit=limit))
|
|
|
|
|
|
def inflate_bounded(raw: bytes, *, name: str, limit: int | None = None) -> bytes:
|
|
"""The same measurement, KEEPING the output rather than discarding it.
|
|
|
|
One chain link is not the end of a chain: a PDF may decode a stream through
|
|
`/Filter [/FlateDecode /FlateDecode]`, and measuring the first link says
|
|
nothing about the second, which is where the cost is. To bound the second
|
|
link the first one's bytes have to exist, so this inflates under the same
|
|
running bound and hands the result on. It refuses at the same point
|
|
`inflated_size` does, so what is held is never more than the bound -- which
|
|
is what separates carrying an intermediate stage from paying for a bomb.
|
|
|
|
Measured 2026-09-18: 400 MB of zeros deflated twice is 795 bytes of stream,
|
|
and the first link of that chain inflates to 407 685 bytes. The whole
|
|
expansion lives in the LAST link, which is the one nothing measured.
|
|
"""
|
|
return b"".join(_inflate(raw, name=name, limit=limit))
|
|
|
|
|
|
def _inflate(raw: bytes, *, name: str, limit: int | None) -> Iterator[bytes]:
|
|
"""Inflate `raw` a chunk at a time, refusing the moment the running total
|
|
crosses the bound. The two callers differ only in whether they keep what
|
|
comes out."""
|
|
bound = MAX_IMAGE_BYTES if limit is None else limit
|
|
decompressor = zlib.decompressobj()
|
|
total = 0
|
|
pending = raw
|
|
try:
|
|
while True:
|
|
produced = decompressor.decompress(pending, _INFLATE_CHUNK)
|
|
total += len(produced)
|
|
if total > bound:
|
|
raise ExtractionError(
|
|
f"the stream behind {name!r} decompresses to more than {bound} "
|
|
f"bytes from {len(raw)} bytes of input, over this package's "
|
|
"bound; refused without being held, because the size a "
|
|
"container declares is a claim and this is the cost",
|
|
code="asset_too_large",
|
|
)
|
|
yield produced
|
|
pending = decompressor.unconsumed_tail
|
|
if decompressor.eof or not pending:
|
|
break
|
|
except zlib.error:
|
|
return
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class ExtractedImage:
|
|
"""One image a document carries, as this package will write it.
|
|
|
|
`name` is what the SOURCE calls the file -- a `xlink:href`, an `<img src>`,
|
|
a media entry inside an OOXML container, or a synthesised name for a PDF
|
|
image object, which has none. It is carried verbatim into the pointer's
|
|
second line and reduced (never trusted) into the asset's own name.
|
|
|
|
`label` is a caption or alt text where the format HAS one, and `None`
|
|
where it does not. Two of the four readers have none: a PDF image object
|
|
and an STS `<graphic>` carry no caption element, and the caption a human
|
|
sees is running text the extractor already emitted on its own line.
|
|
Deriving a label from the nearest line would be an unmarked heuristic,
|
|
which this package treats as worse than no heuristic at all.
|
|
|
|
`converted_from` and `source_sha256` are set when the source was NOT a
|
|
format a model can be shown and this package rewrote it (today: a BMP, as
|
|
a PNG). They are the whole of the traceability: `converted_from` is the
|
|
media type the file had, `source_sha256` is the digest of the bytes the
|
|
document actually shipped, and the digest of `data` is what the bundle
|
|
holds. With the three of them a reader can take the original delivery,
|
|
run `shasum -a 256`, and find the row. Both are `None` for an image
|
|
carried verbatim, which is every image this package has ever carried
|
|
until now -- so a bundle of JPEGs is byte-identical across the move.
|
|
"""
|
|
|
|
data: bytes
|
|
name: str
|
|
media_type: str
|
|
suffix: str
|
|
width: int | None
|
|
height: int | None
|
|
label: str | None = None
|
|
converted_from: str | None = None
|
|
source_sha256: str | None = None
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class AssetRejection:
|
|
"""An image that was FOUND and not carried, with the reason as a code.
|
|
|
|
A rejection is a row in the run log, never a failed document: one
|
|
unreadable picture must not cost the 3 000 concepts of text around it. The
|
|
denominator is what makes the log worth reading -- "51 carried" says
|
|
nothing without "of 53 found".
|
|
"""
|
|
|
|
name: str
|
|
code: str
|
|
reason: str
|
|
|
|
|
|
def sniff(data: bytes) -> tuple[str, str] | None:
|
|
"""`(media type, suffix)` read from the bytes, or `None` if not an image.
|
|
|
|
The claimed extension never participates. A `.jpg` that is really a PNG is
|
|
carried as a PNG under a `.png` name, because the alternative is a bundle
|
|
whose file names disagree with their contents.
|
|
"""
|
|
if data.startswith(_JP2_SIGNATURE) or data.startswith(_J2K_SIGNATURE):
|
|
return "image/jp2", ".jp2"
|
|
for magic, media_type, suffix in _MAGIC:
|
|
if data.startswith(magic):
|
|
return media_type, suffix
|
|
return None
|
|
|
|
|
|
def dimensions(data: bytes) -> tuple[int, int] | None:
|
|
"""`(width, height)` in pixels from the file's own header, or `None`.
|
|
|
|
`None` is a MEASUREMENT: this reader does not read every format's geometry
|
|
(JPEG 2000 and TIFF are absent), and a concept that printed `0x0 px` would
|
|
be stating a number nobody took. The pointer says "dimensions unknown"
|
|
instead.
|
|
"""
|
|
kind = sniff(data)
|
|
if kind is None:
|
|
return None
|
|
suffix = kind[1]
|
|
try:
|
|
if suffix == ".png":
|
|
if len(data) < 24 or data[12:16] != b"IHDR":
|
|
return None
|
|
width, height = struct.unpack(">II", data[16:24])
|
|
return (width, height) if width and height else None
|
|
if suffix == ".jpg":
|
|
return _jpeg_dimensions(data)
|
|
if suffix == ".gif":
|
|
if len(data) < 10:
|
|
return None
|
|
width, height = struct.unpack("<HH", data[6:10])
|
|
return (width, height) if width and height else None
|
|
if suffix == ".bmp":
|
|
return _bmp_dimensions(data)
|
|
except (struct.error, IndexError):
|
|
return None
|
|
return None
|
|
|
|
|
|
def _jpeg_dimensions(data: bytes) -> tuple[int, int] | None:
|
|
position = 2
|
|
end = len(data)
|
|
while position + 3 < end:
|
|
if data[position] != 0xFF:
|
|
position += 1
|
|
continue
|
|
marker = data[position + 1]
|
|
if marker in (0xD8, 0x01) or 0xD0 <= marker <= 0xD7:
|
|
position += 2
|
|
continue
|
|
if marker == 0xFF:
|
|
position += 1
|
|
continue
|
|
length = struct.unpack(">H", data[position + 2 : position + 4])[0]
|
|
if marker in _JPEG_SOF:
|
|
if position + 9 > end:
|
|
return None
|
|
height, width = struct.unpack(">HH", data[position + 5 : position + 9])
|
|
return (width, height) if width and height else None
|
|
position += 2 + length
|
|
return None
|
|
|
|
|
|
def _bmp_dimensions(data: bytes) -> tuple[int, int] | None:
|
|
if len(data) < 26:
|
|
return None
|
|
header_size = struct.unpack("<I", data[14:18])[0]
|
|
if header_size == 12:
|
|
width, height = struct.unpack("<hh", data[18:22])
|
|
else:
|
|
width, height = struct.unpack("<ii", data[18:26])
|
|
# A negative height is a top-down BMP; the magnitude is the pixel count.
|
|
return (abs(width), abs(height)) if width and height else None
|
|
|
|
|
|
#: `biCompression`: uncompressed, and the 8-bit run-length encoding 19 of
|
|
#: R761's 50 assets use. Every other value -- RLE4, BITFIELDS, embedded JPEG or
|
|
#: PNG -- is refused by name rather than guessed at.
|
|
_BMP_RGB = 0
|
|
_BMP_RLE8 = 1
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class _BmpHeader:
|
|
"""What the DIB header declares, before anything is decoded."""
|
|
|
|
header_size: int
|
|
width: int
|
|
height: int
|
|
top_down: bool
|
|
bits: int
|
|
compression: int
|
|
entries: int
|
|
offbits: int
|
|
|
|
|
|
def _bmp_header(data: bytes, *, name: str) -> _BmpHeader:
|
|
"""The declaration, validated -- never the pixels.
|
|
|
|
Everything this returns is a CLAIM by an untrusted file. It is read first
|
|
precisely so the claim can be bounded before the cost of believing it is
|
|
paid: the row buffers below are `width * height` bytes, and that product
|
|
comes from these two fields.
|
|
"""
|
|
if len(data) < 54:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} stops inside its own header ({len(data)} bytes)",
|
|
code="asset_samples_invalid",
|
|
)
|
|
offbits = struct.unpack("<I", data[10:14])[0]
|
|
header_size = struct.unpack("<I", data[14:18])[0]
|
|
if header_size < 40:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} carries a {header_size}-byte DIB header; this reader "
|
|
"expresses BITMAPINFOHEADER and its extensions, and a 12-byte "
|
|
"BITMAPCOREHEADER stores its palette in 3-byte entries, which is a "
|
|
"different format wearing the same magic",
|
|
code="asset_bmp_unsupported",
|
|
)
|
|
if len(data) < 14 + header_size:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} declares a {header_size}-byte DIB header and holds "
|
|
f"{len(data) - 14} bytes after the file header",
|
|
code="asset_samples_invalid",
|
|
)
|
|
width, height = struct.unpack("<ii", data[18:26])
|
|
bits = struct.unpack("<H", data[28:30])[0]
|
|
compression = struct.unpack("<I", data[30:34])[0]
|
|
clr_used = struct.unpack("<I", data[46:50])[0]
|
|
if width <= 0 or height == 0:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} declares {width}x{height}, which is not a size",
|
|
code="asset_size_invalid",
|
|
)
|
|
top_down = height < 0
|
|
entries = clr_used or (1 << bits if bits <= 8 else 0)
|
|
if entries > 256:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} declares {entries} palette entries; a PNG palette holds at most 256",
|
|
code="asset_bmp_unsupported",
|
|
)
|
|
return _BmpHeader(
|
|
header_size=header_size,
|
|
width=width,
|
|
height=-height if top_down else height,
|
|
top_down=top_down,
|
|
bits=bits,
|
|
compression=compression,
|
|
entries=entries,
|
|
offbits=offbits,
|
|
)
|
|
|
|
|
|
def _bmp_palette(data: bytes, head: _BmpHeader, *, name: str) -> bytes:
|
|
"""The colour table as PNG wants it: RGB triples, from the file's BGRA.
|
|
|
|
The order matters and getting it wrong is invisible in every structural
|
|
check: a palette read as RGB gives a picture with red and blue swapped,
|
|
the right size, the right number of colours, and the wrong answer.
|
|
"""
|
|
entries = head.entries or 256
|
|
start = 14 + head.header_size
|
|
table = data[start : start + entries * 4]
|
|
if len(table) < entries * 4:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} declares {entries} palette entries and holds {len(table) // 4}",
|
|
code="asset_samples_invalid",
|
|
)
|
|
palette = bytearray()
|
|
for index in range(entries):
|
|
blue, green, red = table[index * 4], table[index * 4 + 1], table[index * 4 + 2]
|
|
palette += bytes((red, green, blue))
|
|
return bytes(palette)
|
|
|
|
|
|
def _bmp_flat_rows(data: bytes, head: _BmpHeader, *, per_pixel: int, name: str) -> bytes:
|
|
"""Uncompressed rows, unpadded and in top-down order.
|
|
|
|
A BMP row is padded to a 4-byte boundary and stored bottom-up unless the
|
|
declared height is negative. Both are read off the header rather than
|
|
assumed, because either mistake produces a picture that is the right size
|
|
and is sheared or upside down.
|
|
"""
|
|
stride = ((head.width * per_pixel + 3) // 4) * 4
|
|
needed = stride * head.height
|
|
available = len(data) - head.offbits
|
|
if available < needed:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} needs {needed} bytes of pixel data and holds {available} "
|
|
"-- refusing to pad, because a short buffer means the header was read wrong",
|
|
code="asset_samples_invalid",
|
|
)
|
|
rows = [
|
|
data[head.offbits + index * stride : head.offbits + index * stride + head.width * per_pixel]
|
|
for index in range(head.height)
|
|
]
|
|
if not head.top_down:
|
|
rows.reverse()
|
|
return b"".join(rows)
|
|
|
|
|
|
def _bmp_rle8_rows(data: bytes, head: _BmpHeader, *, name: str) -> bytes:
|
|
"""The RLE8 opcode stream painted into a frame of the DECLARED size.
|
|
|
|
Five opcodes, and a decoder implementing only the first is wrong on real
|
|
files: an encoded run, an absolute run (whose byte count is padded to an
|
|
even length), a delta that SKIPS pixels, end-of-line and end-of-bitmap.
|
|
Skipped pixels keep index 0, which is what the format says and what an
|
|
independent decoder produces.
|
|
|
|
THE COST IS THE FRAME, NOT THE STREAM. The rows are allocated once from the
|
|
declared size -- already bounded by `check_size` before this is called --
|
|
and each run is written as one CLIPPED slice. Painting pixel by pixel would
|
|
leave the memory bounded and the CPU unbounded: a megabyte of `FF` runs is
|
|
a hundred million paint steps against a frame of 32 pixels.
|
|
|
|
A STREAM THAT NEVER SAYS IT IS FINISHED IS REFUSED, and that is the
|
|
difference between a skipped pixel and a missing one. Running out of bytes
|
|
leaves the rest of the frame at index 0 -- indistinguishable, in the
|
|
output, from a delta that skipped it, which is why every decoder agrees on
|
|
the wrong picture: measured 2026-09-19 on a real 352x548 R761 asset, a
|
|
stream cut to 90 % was carried with 13 923 pixels wrong, to 50 % with
|
|
95 890, to 10 % with 166 525, with no code and no row. So the loop may end
|
|
HERE and nowhere else, and `_bmp_flat_rows` refuses the same shape with the
|
|
same code.
|
|
|
|
The terminator is checked rather than `biSizeImage`, which is a claim by
|
|
the same untrusted header. Read off the corpus: over the 25 RLE8 BMPs
|
|
the frozen R761 delivery ships (24 distinct; the bundle carries 19 of
|
|
them, the rest being an unpointed duplicate and four no concept names),
|
|
25 of 25 end at an explicit
|
|
end-of-bitmap, on 25 of 25 it is the stream's LAST two bytes, and on 25 of
|
|
25 `biSizeImage` equals the available bytes -- so requiring it costs
|
|
nothing measured here, and a whole stream that omits it is refused
|
|
alongside a cut one because from the reader's side they are the same bytes.
|
|
|
|
AND THE TERMINATOR ALONE IS NOT A COVERAGE PROOF, because a stream may say
|
|
it is finished anywhere: measured 2026-09-19, one whose FIRST two bytes are
|
|
the end-of-bitmap escape was carried with every pixel of the frame never
|
|
decoded. So the cursor must also stand at or past the end of the last row.
|
|
|
|
THE LINE IS THE CURSOR AND NOT THE PIXELS, and that is a format argument
|
|
rather than a corpus one -- the corpus cannot choose between the two, since
|
|
25 of 25 of those files paint every pixel, 25 of 25 reach the end of the
|
|
frame and 0 of 25 use a delta. A pixel-coverage count would refuse both
|
|
constructions the format defines.
|
|
|
|
BUT A CURSOR THAT MOVED IS NOT A SKIP THAT WAS STATED, and the round that
|
|
wrote the cursor rule claimed more than it had measured: it said a delta
|
|
escape and an end-of-line escape both leave pixels every decoder agrees
|
|
on. Only the delta does. A delta states a DISTANCE, so an independent
|
|
decoder passes over exactly the same pixels; an end-of-line states "the
|
|
rest of THIS row", so an end-of-line at column 0 closes nothing and claims
|
|
a row the stream never wrote. Measured 2026-09-19: four end-of-line
|
|
escapes and an end-of-bitmap carried an 8x4 frame with 32 of 32 pixels
|
|
never decoded, and Pillow refuses those same bytes. So an end-of-line at
|
|
column 0 is REFUSED, and so is a delta whose horizontal offset would leave
|
|
the row -- the format puts that offset inside the line, and a reader that
|
|
keeps the cursor past the row end and a reader that rolls it into the next
|
|
row draw two different pictures from one stream.
|
|
|
|
THE DIRECTION THAT IS CLOSED, AND THE ONE THAT IS NOT. Over every opcode
|
|
sequence of length 1 to 4 on a 4x3 frame -- 22 620 streams, swept in
|
|
`tests/test_asset_viewable.py` rather than curated -- this reader carried
|
|
703 streams the independent decoder refuses and drew 1 492 more
|
|
differently. With these two clauses: 0 and 32. All 32 remaining are a run
|
|
or absolute block that OVERRUNS its row, which this reader clips at the
|
|
row end and Pillow spills into the next one. That last class is STATED
|
|
rather than closed, because closing it costs pictures: 15 of the 25 real
|
|
files overrun a row, and refusing them would drop 15 real figures and move
|
|
a pinned bundle's bytes. The two clauses that shipped cost nothing
|
|
measured -- over 11 441 files scanned across four raw standard deliveries
|
|
and the K2 reference corpus, the only 25 BMPs on this machine use an
|
|
end-of-line at column 0 in 0 of 25 and a delta in 0 of 25, and 25 of 25
|
|
still decode to Pillow's pixels exactly (3 117 220 compared).
|
|
"""
|
|
width, height = head.width, head.height
|
|
rows = [bytearray(width) for _ in range(height)]
|
|
position = head.offbits
|
|
end = len(data)
|
|
x = 0
|
|
y = 0
|
|
finished = False
|
|
while position + 1 < end:
|
|
count = data[position]
|
|
value = data[position + 1]
|
|
position += 2
|
|
if count:
|
|
if 0 <= y < height and x < width:
|
|
stop = min(x + count, width)
|
|
rows[y][x:stop] = bytes((value,)) * (stop - x)
|
|
x += count
|
|
continue
|
|
if value == 0:
|
|
if x == 0:
|
|
raise ExtractionError(
|
|
f"the RLE8 stream in {name!r} ends a row at column 0 of a "
|
|
f"{width}x{height} frame -- an end-of-line escape states the rest of the "
|
|
"row it started, so one that started no row claims pixels the stream "
|
|
"never wrote",
|
|
code="asset_samples_invalid",
|
|
)
|
|
x = 0
|
|
y += 1
|
|
elif value == 1:
|
|
finished = True
|
|
break
|
|
elif value == 2:
|
|
if position + 2 > end:
|
|
break
|
|
if x + data[position] > width:
|
|
raise ExtractionError(
|
|
f"the RLE8 stream in {name!r} deltas from column {x} by "
|
|
f"{data[position]} in a {width}-wide frame -- the format puts that offset "
|
|
"inside the line, and readers disagree about where it lands",
|
|
code="asset_samples_invalid",
|
|
)
|
|
x += data[position]
|
|
y += data[position + 1]
|
|
position += 2
|
|
else:
|
|
run = data[position : position + value]
|
|
position += value + (value & 1)
|
|
if 0 <= y < height and x < width:
|
|
stop = min(x + len(run), width)
|
|
rows[y][x:stop] = run[: stop - x]
|
|
x += value
|
|
if not finished:
|
|
raise ExtractionError(
|
|
f"the RLE8 stream in {name!r} ends after {position - head.offbits} of "
|
|
f"{len(data) - head.offbits} bytes without an end-of-bitmap escape "
|
|
"-- refusing to carry a frame whose remaining pixels were never decoded",
|
|
code="asset_samples_invalid",
|
|
)
|
|
if y < height - 1 or (y == height - 1 and x < width):
|
|
raise ExtractionError(
|
|
f"the RLE8 stream in {name!r} ends at row {y} column {x} of a {width}x{height} "
|
|
"frame -- refusing to carry a picture whose last rows the stream never reached",
|
|
code="asset_samples_invalid",
|
|
)
|
|
if not head.top_down:
|
|
rows.reverse()
|
|
return b"".join(bytes(row) for row in rows)
|
|
|
|
|
|
def bmp_to_png(data: bytes, *, name: str) -> bytes:
|
|
"""A BMP as a PNG with the same pixels, using nothing but the stdlib.
|
|
|
|
WHY A READER HERE AND NOT PILLOW, which this tree already carries
|
|
transitively under `pdfplumber`. Two reasons, measured rather than
|
|
preferred. First, `read_image` is on the CORE path: `.html` and `.xml` are
|
|
stdlib file types that carry images with no `[extract]` extra installed, so
|
|
a Pillow-based converter would either make a core path depend on an
|
|
optional binary wheel or buy this package its second runtime dependency.
|
|
Second, and decisive: an asset's name is its content digest, so the bytes
|
|
this function emits are part of the bundle's identity. Encoding through an
|
|
installed library would make that identity move with the library's version
|
|
-- the exact property 0.10.0 felled page rasterisation over. `encode_png`
|
|
already writes a PNG from samples with `zlib` alone; this adds the reader in
|
|
front of it.
|
|
|
|
Pillow is still the INDEPENDENT decoder in the tests, which is the job it
|
|
is good for here: 19 of 19 of R761's real RLE8 assets decode to identical
|
|
RGB through both paths (measured 2026-09-19, before this was written).
|
|
"""
|
|
head = _bmp_header(data, name=name)
|
|
channels = 3 if head.bits == 24 else 1
|
|
# THE CEILING FIRST, on the DECLARATION, before one row is allocated. The
|
|
# rows below are `width * height` bytes of an untrusted document's claim.
|
|
check_size(head.width, head.height, name=name, channels=channels, bits=8)
|
|
if head.bits == 8 and head.compression in (_BMP_RGB, _BMP_RLE8):
|
|
palette = _bmp_palette(data, head, name=name)
|
|
if head.compression == _BMP_RLE8:
|
|
if head.top_down:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} declares a top-down RLE8 image, which the format "
|
|
"does not define",
|
|
code="asset_bmp_unsupported",
|
|
)
|
|
samples = _bmp_rle8_rows(data, head, name=name)
|
|
else:
|
|
samples = _bmp_flat_rows(data, head, per_pixel=1, name=name)
|
|
limit = len(palette) // 3
|
|
if samples and max(samples) >= limit:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} uses palette index {max(samples)} and declares {limit} "
|
|
"entries; carrying it would invent a colour",
|
|
code="asset_samples_invalid",
|
|
)
|
|
return encode_png(head.width, head.height, samples, channels=1, palette=palette)
|
|
if head.bits == 24 and head.compression == _BMP_RGB:
|
|
raw = _bmp_flat_rows(data, head, per_pixel=3, name=name)
|
|
swapped = bytearray(raw)
|
|
# BGR on disk, RGB in a PNG. Two slices rather than a loop, and read
|
|
# from `raw` both times so the first assignment cannot feed the second.
|
|
swapped[0::3] = raw[2::3]
|
|
swapped[2::3] = raw[0::3]
|
|
return encode_png(head.width, head.height, bytes(swapped), channels=3)
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} stores {head.bits}-bit samples under compression "
|
|
f"{head.compression}; this reader expresses 8-bit (uncompressed and RLE8) and "
|
|
"24-bit uncompressed, and will not guess at the rest",
|
|
code="asset_bmp_unsupported",
|
|
)
|
|
|
|
|
|
#: Source media type -> the function that makes a viewable file of it. A map
|
|
#: rather than a branch, so what this package can convert is one readable line
|
|
#: and adding a format is adding a row.
|
|
_CONVERTERS = {"image/bmp": bmp_to_png}
|
|
|
|
|
|
def to_viewable(data: bytes, *, media_type: str, name: str) -> bytes:
|
|
"""Bytes a model can be shown, or a coded refusal. Never a silent carry."""
|
|
convert = _CONVERTERS.get(media_type)
|
|
if convert is None:
|
|
raise ExtractionError(
|
|
f"the image {name!r} is {media_type}, which no model can be shown, and this "
|
|
"package has no lossless conversion for it; refused rather than carried in a "
|
|
"format nothing reads",
|
|
code="asset_not_viewable",
|
|
)
|
|
return convert(data, name=name)
|
|
|
|
|
|
def read_image(data: bytes, *, name: str, label: str | None = None) -> ExtractedImage:
|
|
"""One image, typed by its bytes, or a coded refusal.
|
|
|
|
Raises :class:`ExtractionError` with `asset_type_unknown` when the bytes
|
|
are not an image this package recognises. The caller records that as an
|
|
:class:`AssetRejection` and keeps going -- a document is not lost over one
|
|
picture.
|
|
"""
|
|
kind = sniff(data)
|
|
if kind is None:
|
|
raise ExtractionError(
|
|
f"the bytes behind {name!r} are not an image format this package "
|
|
f"recognises (first bytes {data[:8]!r})",
|
|
code="asset_type_unknown",
|
|
)
|
|
media_type, suffix = kind
|
|
# THE VIEWABILITY GATE, and it stands before the size is read because the
|
|
# size that matters is the one the CARRIED file has. A BMP that becomes a
|
|
# PNG is measured as the PNG a consumer will open.
|
|
converted_from: str | None = None
|
|
source_sha256: str | None = None
|
|
if media_type not in VIEWABLE_MEDIA_TYPES:
|
|
converted_from = media_type
|
|
source_sha256 = digest(data)
|
|
data = to_viewable(data, media_type=media_type, name=name)
|
|
media_type, suffix = "image/png", ".png"
|
|
size = dimensions(data)
|
|
# THE BOUND HOLDS FOR A FILE CARRIED VERBATIM TOO. This package does not
|
|
# decode one, so it pays nothing for it -- but writing a 7 000 x 7 000 PNG
|
|
# of 47 705 bytes into a bundle hands the consumer the same bomb with
|
|
# `7000x7000 px` printed beside it, and the README's first sentence about
|
|
# this bound says such an image is refused. Sniffed from the header rather
|
|
# than claimed, like the type beside it. Measured: the largest of the
|
|
# 4 828 objects in the reference corpus is 18.6 MP, so nothing anyone has
|
|
# measured is refused here.
|
|
if size is not None:
|
|
check_size(size[0], size[1], name=name)
|
|
return ExtractedImage(
|
|
data=data,
|
|
name=name,
|
|
media_type=media_type,
|
|
suffix=suffix,
|
|
width=size[0] if size else None,
|
|
height=size[1] if size else None,
|
|
label=label,
|
|
converted_from=converted_from,
|
|
source_sha256=source_sha256,
|
|
)
|
|
|
|
|
|
def digest(data: bytes) -> str:
|
|
return hashlib.sha256(data).hexdigest()
|
|
|
|
|
|
def conversion(image: ExtractedImage) -> tuple[str, str] | None:
|
|
"""`(the source's digest, the carried asset's digest)`, or `None`.
|
|
|
|
THE RUN'S OWN RECORD OF WHAT IT REWROTE, for a reader that must not have
|
|
to take the bundle's word for it. `render_block` states the same pair on
|
|
the pointer's second line, which is where a person reads it -- but that
|
|
line is markdown in a concept body, and measured by PM 2026-09-19 an
|
|
ordinary HTML document with two `<p>` elements produces the same two
|
|
lines. A judge reading the claim off the bundle text is therefore reading
|
|
an untrusted document; a judge reading it off the accounting is reading
|
|
this function's output, which no document can reach.
|
|
"""
|
|
if image.converted_from is None or image.source_sha256 is None:
|
|
return None
|
|
return (image.source_sha256, digest(image.data))
|
|
|
|
|
|
def _reduce(text: str) -> str:
|
|
return _SEPARATOR_RUN.sub("-", unicodedata.normalize("NFC", text).lower()).strip("-")
|
|
|
|
|
|
def asset_name(image: ExtractedImage) -> str:
|
|
"""`<sha256[:12]>-<reduced original BASENAME><sniffed suffix>`.
|
|
|
|
DEDUP IS ON CONTENT, and the digest is what makes it so: the same bytes
|
|
arriving twice are one file, whichever document dropped them. The residual
|
|
is stated rather than claimed away -- identical bytes shipped under two
|
|
different base names are two files in the bundle, both correct and both
|
|
holding the same picture. Measured on the fixture inbox and on the R761
|
|
delivery, that case occurs 0 times; a rule that could not produce it at all
|
|
would have to drop the readable tail entirely, and the tail is what makes
|
|
`assets/` legible to the person checking a bundle by hand.
|
|
|
|
|
|
The suffix comes from the bytes and the stem from the name, reduced to the
|
|
same grammar every generated filename in this package uses. A link target
|
|
sits inside `](...)`, which a space or a closing parenthesis terminates --
|
|
and the R761 delivery's own hrefs carry both (`25-0143 - Tabeller -
|
|
Prosesskoden (R761-R762).jpg`), so reducing is what makes the pointer
|
|
followable rather than merely tidy.
|
|
"""
|
|
# The BASENAME, never the path the document reached it through. Measured
|
|
# on the fixture inbox: one image pointed at as `graphics/figur-84-1.png`
|
|
# from an HTML document and as `figur-84-1.png` from an STS one was written
|
|
# twice, under two names, in one run -- with the digest in both announcing
|
|
# that the bytes were identical. The path is a property of the pointer, not
|
|
# of the picture, and the full original survives on the pointer's own line.
|
|
base = image.name.rsplit("/", 1)[-1]
|
|
stem = _reduce(base.rsplit(".", 1)[0] if "." in base else base)
|
|
head = digest(image.data)[:DIGEST_PREFIX]
|
|
if not stem:
|
|
return f"{head}{image.suffix}"
|
|
return f"{head}-{stem[:NAME_TAIL_MAX].rstrip('-')}{image.suffix}"
|
|
|
|
|
|
def asset_href(image: ExtractedImage) -> str:
|
|
"""The bundle-absolute path SPEC SS 6.2 allows.
|
|
|
|
Absolute rather than relative because a segmented bundle puts concepts at
|
|
different depths: `assets/x.png` resolves to two different places from two
|
|
concepts of one document, and `/assets/x.png` to one place from every
|
|
concept in the bundle.
|
|
"""
|
|
return f"/{ASSETS_DIR}/{asset_name(image)}"
|
|
|
|
|
|
#: `sha256:` immediately in front of 64 hex digits -- the CHECKSUM FIELD this
|
|
#: module writes on a pointer's second line, and the grammar the content
|
|
#: accounting gate reads a conversion claim with.
|
|
_CHECKSUM_FIELD = re.compile(r"sha256:(?=[0-9a-fA-F]{64})")
|
|
|
|
|
|
def _inline(value: str) -> str:
|
|
"""A label, made safe for the one line it is written on.
|
|
|
|
`[` and `]` are the link grammar's own delimiters and a newline would open
|
|
a third line in a two-line block, so both are replaced rather than escaped:
|
|
Door B refuses a title containing a bracket outright (`inbox_title_invalid`)
|
|
and this text reaches a title through no route, but the pointer is body text
|
|
a proposer reads, and a half-open link there is a pointer that resolves
|
|
nowhere.
|
|
|
|
WHERE THE BOUNDARY RUNS. Everything this function returns came from the
|
|
DOCUMENT -- an `alt` attribute, an STS `<caption>`, a file name a publisher
|
|
chose. Everything `render_block` appends after it came from the run: the
|
|
size it measured, the type it sniffed, the digests it computed. The second
|
|
line carries both, so document text must not be able to emit the metadata
|
|
grammar the run writes there. Measured by PM 2026-09-19: an `alt` attribute
|
|
stating `converted from ... sha256:<a> to ... sha256:<b>` made the content
|
|
accounting gate report a picture as carried that was refused
|
|
`asset_too_large` and is not in `assets/` at all. A checksum field is
|
|
therefore disarmed here -- the digits are kept, because a reader is owed
|
|
what the document said, and the colon that makes them a FIELD is not.
|
|
"""
|
|
collapsed = " ".join(value.split())
|
|
return _CHECKSUM_FIELD.sub("sha256 ", collapsed.replace("[", "(").replace("]", ")"))
|
|
|
|
|
|
def render_block(image: ExtractedImage) -> str:
|
|
"""The two lines that stand where the image stands.
|
|
|
|
Line one is markdown, so a reader that renders the concept sees the picture
|
|
and a reader that does not sees the label. Line two states what the first
|
|
line cannot: the name the SOURCE gave the file, and the size in pixels --
|
|
the two facts a person checking the bundle against the original needs, and
|
|
the two a transcription in step 2 has to be judged against.
|
|
"""
|
|
label = image.label or image.name or asset_name(image)
|
|
size = (
|
|
f"{image.width}x{image.height} px"
|
|
if image.width is not None and image.height is not None
|
|
else "dimensions unknown"
|
|
)
|
|
detail = f"Image: {_inline(image.name or asset_name(image))} ({size})"
|
|
if image.label:
|
|
detail += f" -- {_inline(image.label)}"
|
|
# WHAT THE CONVERSION DID, on the line the rest of the asset metadata is
|
|
# already on. Both digests in full: the asset's file name carries only the
|
|
# first 12 hex characters of the new one, and a checksum a reader cannot
|
|
# paste into `shasum -a 256` is decoration. Written LAST so a labelled
|
|
# image that was not converted keeps the bytes it has today.
|
|
if image.converted_from and image.source_sha256:
|
|
detail += (
|
|
f" -- converted from {image.converted_from} sha256:{image.source_sha256}"
|
|
f" to {image.media_type} sha256:{digest(image.data)}"
|
|
)
|
|
return f"})\n{detail}"
|
|
|
|
|
|
def encode_png(
|
|
width: int,
|
|
height: int,
|
|
samples: bytes,
|
|
*,
|
|
channels: int,
|
|
palette: bytes | None = None,
|
|
alpha: bytes | None = None,
|
|
) -> bytes:
|
|
"""8-bit samples as a PNG, using nothing but `zlib`.
|
|
|
|
Refuses a size over :data:`MAX_IMAGE_PIXELS` on its own rather than
|
|
trusting the caller to have checked: this function is what allocates
|
|
`width * height * channels` bytes twice over.
|
|
|
|
A PDF image is usually not a file: `FlateDecode` hands back raw samples
|
|
with the colour model in the dictionary beside them, so carrying one at all
|
|
means encoding it. Doing that with the stdlib rather than with a renderer
|
|
is what keeps the `pdf` row free of a new dependency AND keeps an asset's
|
|
bytes -- and therefore its content-addressed name -- independent of which
|
|
version of a rasteriser happened to be installed. `OCR_DPI`'s docstring
|
|
states the opposite property for OCR text, and the difference is deliberate.
|
|
|
|
`channels` is 1 (grey, or indexed when `palette` is given) or 3 (RGB).
|
|
`alpha` is one byte per pixel, from a PDF `SMask`; absent means opaque.
|
|
"""
|
|
check_size(width, height, name=f"{width}x{height}", channels=channels)
|
|
if channels not in (1, 3):
|
|
raise ExtractionError(
|
|
f"PNG encoding supports 1 or 3 channels, not {channels}",
|
|
code="asset_samples_invalid",
|
|
)
|
|
if width <= 0 or height <= 0:
|
|
raise ExtractionError(
|
|
f"PNG encoding needs positive dimensions, got {width}x{height}",
|
|
code="asset_samples_invalid",
|
|
)
|
|
expected = width * height * channels
|
|
if len(samples) < expected:
|
|
raise ExtractionError(
|
|
f"the sample buffer holds {len(samples)} bytes where {width}x{height} "
|
|
f"at {channels} channels needs {expected} -- refusing to pad, because a "
|
|
"short buffer means the image dictionary was read wrong",
|
|
code="asset_samples_invalid",
|
|
)
|
|
if alpha is not None and len(alpha) < width * height:
|
|
raise ExtractionError(
|
|
f"the alpha buffer holds {len(alpha)} bytes where {width}x{height} needs "
|
|
f"{width * height}",
|
|
code="asset_samples_invalid",
|
|
)
|
|
if palette is not None:
|
|
if channels != 1:
|
|
raise ExtractionError(
|
|
"a palette applies to single-channel samples only",
|
|
code="asset_samples_invalid",
|
|
)
|
|
if not palette or len(palette) % 3:
|
|
raise ExtractionError(
|
|
f"a palette must be whole RGB triples, got {len(palette)} bytes",
|
|
code="asset_samples_invalid",
|
|
)
|
|
colour_type = 3
|
|
elif alpha is not None:
|
|
colour_type = 6 if channels == 3 else 4
|
|
else:
|
|
colour_type = 2 if channels == 3 else 0
|
|
|
|
rows = bytearray()
|
|
for row in range(height):
|
|
# Filter type 0 (None) on every row. A filter would shrink the file and
|
|
# would make the bytes depend on a heuristic; this encoder's output has
|
|
# to be reproducible from the samples alone, for as long as the bundle
|
|
# is quoted by its digest.
|
|
rows.append(0)
|
|
start = row * width * channels
|
|
line = samples[start : start + width * channels]
|
|
if alpha is None:
|
|
rows += line
|
|
else:
|
|
for pixel in range(width):
|
|
rows += line[pixel * channels : (pixel + 1) * channels]
|
|
rows.append(alpha[row * width + pixel])
|
|
|
|
def chunk(kind: bytes, payload: bytes) -> bytes:
|
|
return (
|
|
len(payload).to_bytes(4, "big")
|
|
+ kind
|
|
+ payload
|
|
+ zlib.crc32(kind + payload).to_bytes(4, "big")
|
|
)
|
|
|
|
ihdr = struct.pack(">IIBBBBB", width, height, 8, colour_type, 0, 0, 0)
|
|
body = b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", ihdr)
|
|
if palette is not None:
|
|
body += chunk(b"PLTE", palette)
|
|
# Level 9 with no filtering: `zlib` is stdlib and its output for a given
|
|
# level is stable within a Python build, which is the same guarantee the
|
|
# rest of this package's byte-determinism rests on.
|
|
body += chunk(b"IDAT", zlib.compress(bytes(rows), 9))
|
|
return body + chunk(b"IEND", b"")
|
|
|
|
|
|
def render_missing(
|
|
name: str,
|
|
*,
|
|
reason: str,
|
|
label: str | None = None,
|
|
href: str | None = None,
|
|
) -> str:
|
|
"""A pointer to an image the bundle does NOT carry, and why.
|
|
|
|
An image this package found and could not carry is stated in the concept,
|
|
not dropped. The reader of the bundle is the person who has to decide
|
|
whether the missing picture mattered, and they cannot decide about an
|
|
absence they were never shown -- which is precisely the shape of the defect
|
|
this whole capability exists to close.
|
|
|
|
The href is kept when the source had one, so a remote figure says WHERE it
|
|
was. Extraction never opens a socket: the network gate is an explicit
|
|
per-run opt-in and extraction is not on that path, so a remote source is
|
|
carried as a name and never as bytes.
|
|
|
|
`label` is the alt text or the figure caption, and it is written for the
|
|
same reason the line exists at all: the reader deciding whether the missing
|
|
picture mattered is much better served by "Figur 84-1 Tverrprofil" than by
|
|
a file name. 0.10.1 dropped it while closing the live-link defect -- the
|
|
parameter stayed in the signature and no branch read it -- which an
|
|
independent review measured as a regression against 0.10.0.
|
|
"""
|
|
# INERT, never ``. A remote reference is an address the
|
|
# document's author chose, and 0.10.0 wrote it as a live markdown image: a
|
|
# consumer that renders the bundle, or an agent that fetches what it
|
|
# renders, turns "this bundle was opened" into a beacon to them -- and a
|
|
# server-side consumer into an SSRF. This package opens no socket, which is
|
|
# not the same as the pointer being harmless.
|
|
#
|
|
# The address is STATED, because a reader has to know what stood there --
|
|
# and stated ONCE, inside a code span. 0.10.1 wrote it twice: once in the
|
|
# span and once as the bare `name`, and a GFM/linkify renderer autolinks a
|
|
# bare URL into `<a href="...">`. It takes a click rather than a render, so
|
|
# it is weaker than `![..]()` -- but "inert" was half the truth, and half
|
|
# is what this line exists not to be.
|
|
if href:
|
|
address = _inline(href).replace("`", "'")
|
|
shown = f"`{address}`"
|
|
else:
|
|
shown = _inline(name or "image")
|
|
detail = f"Image: {shown} (not carried: {_inline(reason)})"
|
|
if label:
|
|
detail += f" -- {_inline(label)}"
|
|
return detail
|