Kjell Tore Guttormsen
561b609a53
Chose the CURSOR over a pixel-coverage count because the corpus cannot choose -- 25 of 25 of the R761 delivery's RLE8 BMPs paint every pixel, 25 of 25 reach the end of the frame, 0 of 25 use a delta -- and an independent decoder can: a delta and an end-of-line escape state their skip, so every decoder agrees on the index-0 pixels they pass over, while a pixel count would refuse both constructions the format defines. `_bmp_rle8_rows` now refuses (`asset_samples_invalid`) when the terminator arrives with the cursor short of the last row. Pillow reads 5 of the 8 streams in the table and refuses the same 3, one of them short by a single pixel. Both docstrings the round was sent to correct are rewritten: the test no longer claims every pixel is decoded (it is not -- a stated skip keeps index 0), and `_bmp_rle8_rows` no longer frames the delta argument as read off the corpus, which it never was. R761 rebuilt: bundle `diff -r`-identical to the build before this commit, 50 assets (29 JPEG + 21 PNG), 19 of 19 conversions, SHY 71, u = 0, d = 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1194 lines
54 KiB
Python
1194 lines
54 KiB
Python
"""Binary assets: what an image IS, what it is called, and how it is pointed at.
|
|
|
|
Until 0.10.0 this package had no image path at all. Every reader recovered
|
|
text, every warning said so, and the only writer into a bundle was
|
|
:func:`materialize.write_bytes`, whose signature is ``(bundle_dir, name,
|
|
content: str)`` -- UTF-8 in, text out, no binary route anywhere. A document
|
|
whose table is a raster picture therefore reached a concept as an absence with
|
|
no denominator. Measured on R761 Prosesskoden:2025, the publisher's own
|
|
NISO-STS delivery: the process text is carried in full, and 12 ``Tabell N-N``
|
|
and 9 ``Figur N-N`` captions stand over nothing, so process 84's
|
|
"toleranseklasse ... er gitt i tabell 84-2" points at empty space.
|
|
|
|
THIS MODULE IS THE ONE PLACE THAT DECIDES WHAT AN IMAGE IS. A reader hands it
|
|
bytes and a name; it returns a carried image or raises a coded rejection. That
|
|
is what makes "N carried of M found" mean the same thing for ``pdf``, ``docx``,
|
|
``html`` and ``xml``, and it is what keeps each format's quirks out of the
|
|
bundle layout.
|
|
|
|
THREE RULES, and each one exists because the alternative is a silent lie:
|
|
|
|
- **The type is SNIFFED, never claimed.** Measured on the R761 delivery, the
|
|
graphics directory holds ``.bmp``, ``.jpg`` and ``.png`` side by side and the
|
|
document's ``xlink:href`` values are whatever the publisher's tool wrote. A
|
|
name is a claim; the magic bytes are the fact. A consumer dispatching on the
|
|
extension of a name that lies reads the file wrong with full confidence.
|
|
- **The name is CONTENT-ADDRESSED** -- ``<sha256[:12]>-<reduced original>``.
|
|
Two drops of one image are one file, a rebuild of one corpus is one bundle,
|
|
and the digest carries the uniqueness so the readable tail can be shortened
|
|
without any risk of collision. That is the byte-determinism rule this package
|
|
already holds for text, extended to the bytes beside it.
|
|
- **The pointer is ONE GRAMMAR**, owned here. ``okf describe`` (step 2) has to
|
|
find every pointer mechanically in order to write a transcription under it,
|
|
so the block is a regex this module ships beside the writer rather than a
|
|
shape each reader invents and each consumer re-derives.
|
|
|
|
WHAT THIS MODULE DOES NOT DO: it never looks at a picture. Classifying an image
|
|
as a table or a figure, and reading what it says, is a model call, and the
|
|
invariant "no model calls anywhere in the run path" is not negotiated here --
|
|
step 2 is a separate command, outside the build path, and this module is
|
|
importable without it.
|
|
|
|
**The image BYTES are not screened.** The guard is text-only (its own boundary,
|
|
not ours), so what passes a persist gate is the pointer block, as body text,
|
|
like every other line. The bytes of a carried image are written to the bundle
|
|
unscreened. Stated here rather than implied, because a consumer weighing an
|
|
untrusted drop needs to know which half of the concept was looked at.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
import re
|
|
import struct
|
|
import unicodedata
|
|
import zlib
|
|
from collections.abc import Iterator
|
|
from dataclasses import dataclass
|
|
|
|
from .errors import ExtractionError
|
|
|
|
#: The one directory a bundle keeps its binary assets in, at the bundle root.
|
|
#: Fixed rather than configurable: a consumer resolving `/assets/...` out of a
|
|
#: concept has only the bundle, and a per-profile directory would make that
|
|
#: pointer unresolvable without also shipping the profile that wrote it.
|
|
ASSETS_DIR = "assets"
|
|
|
|
#: How much of the digest names the file. 12 hex characters is 48 bits; over
|
|
#: the largest asset population measured here (4 828 image objects in one
|
|
#: 33-document corpus) the birthday probability of a collision is about
|
|
#: 4e-11. A collision would be caught anyway -- an occupied name is re-used
|
|
#: only when the bytes there are already identical, the same content-identity
|
|
#: rule Door C proves ownership with.
|
|
DIGEST_PREFIX = 12
|
|
|
|
#: How much of the original name survives into the asset name. The digest
|
|
#: carries uniqueness, so this is decoration and truncating it is safe -- which
|
|
#: is the opposite of `materialize.check_filename_length`'s situation, where the
|
|
#: name IS the identity and truncation would silently merge two documents.
|
|
NAME_TAIL_MAX = 60
|
|
|
|
#: Magic bytes -> (media type, suffix). Sniffed in this order; the first match
|
|
#: wins, and nothing here overlaps.
|
|
_MAGIC: tuple[tuple[bytes, str, str], ...] = (
|
|
(b"\x89PNG\r\n\x1a\n", "image/png", ".png"),
|
|
(b"\xff\xd8\xff", "image/jpeg", ".jpg"),
|
|
(b"GIF87a", "image/gif", ".gif"),
|
|
(b"GIF89a", "image/gif", ".gif"),
|
|
(b"BM", "image/bmp", ".bmp"),
|
|
(b"II\x2a\x00", "image/tiff", ".tiff"),
|
|
(b"MM\x00\x2a", "image/tiff", ".tiff"),
|
|
)
|
|
|
|
#: The formats a model can be SHOWN. Everything a document ships outside this
|
|
#: set is converted losslessly to PNG, or refused with a code -- never carried
|
|
#: silently, which is what this package did until this round of 0.10.1.
|
|
#:
|
|
#: MEASURED 2026-09-19 over the frozen R761 delivery's own `assets/`
|
|
#: (denominator 50): 29 JPEG, 2 PNG and **19 "PC bitmap, Windows 3.x, 8-bit,
|
|
#: compression 1"**. The 19 are byte-correct files that nothing reads, so 19 of
|
|
#: that document's figures were present and invisible at the same time -- and
|
|
#: the `images: N` count said they had arrived. An absence a reader is shown is
|
|
#: information; a picture that is there and unreadable is worse than either.
|
|
#:
|
|
#: IT IS A PROPERTY, NOT A LIST OF FORMATS WE HAPPENED TO MEET. A carried
|
|
#: asset's type is read off its bytes and tested against this set, so a format
|
|
#: nobody here has seen is refused by the same rule that refuses TIFF.
|
|
#:
|
|
#: WebP is on the list and `sniff` does not recognise it: the set states what a
|
|
#: model can be shown, not what this package can read. A WebP is therefore
|
|
#: refused one step earlier, as `asset_type_unknown`, and never reaches this
|
|
#: test. Stating that is cheaper than a set whose name is wider than its reach.
|
|
VIEWABLE_MEDIA_TYPES = frozenset({"image/png", "image/jpeg", "image/gif", "image/webp"})
|
|
|
|
#: JPEG 2000, in both the forms a PDF `JPXDecode` stream hands back: the JP2
|
|
#: container and a bare codestream.
|
|
_JP2_SIGNATURE = b"\x00\x00\x00\x0cjP \r\n\x87\n"
|
|
_J2K_SIGNATURE = b"\xff\x4f\xff\x51"
|
|
|
|
#: The frame markers that carry a JPEG's dimensions. Every SOF except the four
|
|
#: that are not frame headers at all (`DHT` 0xC4, `JPG` 0xC8, `DAC` 0xCC).
|
|
_JPEG_SOF = frozenset(range(0xC0, 0xD0)) - {0xC4, 0xC8, 0xCC}
|
|
|
|
#: `materialize.reduce_to_id_grammar`'s rule, restated. Not imported: this
|
|
#: module is reached from `extract.py`, whose registry must not import the
|
|
#: contract layer, and `materialize` pulls in `manifest` and `profiles`.
|
|
#: `tests/test_assets.py` holds the two forms equal on the same inputs, so the
|
|
#: restatement cannot drift into a second grammar.
|
|
_SEPARATOR_RUN = re.compile(r"[^a-z0-9]+")
|
|
|
|
#: One pointer block, as this module writes it. Group 1 is the label, group 2
|
|
#: is the asset file name, group 3 is the whole second line. `okf describe`
|
|
#: finds its work with this and writes under the match; a consumer wanting to
|
|
#: strip pointers uses the same expression, so there is one definition of what
|
|
#: a pointer looks like rather than one per reader.
|
|
IMAGE_POINTER = re.compile(
|
|
r"^!\[(?P<label>[^\]\n]*)\]\(/" + ASSETS_DIR + r"/(?P<asset>[^)\s]+)\)\n"
|
|
r"(?P<detail>Image: [^\n]*)$",
|
|
re.MULTILINE,
|
|
)
|
|
|
|
|
|
#: THE SIZE A PICTURE MAY DECLARE. Read off the corpora rather than chosen:
|
|
#: over the 4 828 image objects of the 43-document reference corpus the largest
|
|
#: is 4 515 x 4 128 (18.6 MP, a landscape drawing), and over R761's 109
|
|
#: delivered pictures it is 2 072 x 656 (1.4 MP). This sits an order of
|
|
#: magnitude above both, so nothing anyone has measured is refused -- and a
|
|
#: declared size beyond it is a COUNTED refusal instead of a killed build.
|
|
#:
|
|
#: The bound exists because a few kilobytes can declare an enormous picture: a
|
|
#: 9.6 KB PDF declaring 3 000 x 3 000 grayscale zeros took 83 MB of peak RSS
|
|
#: and a 63 KB one declaring 8 000 x 8 000 took 276 MB, linear in the pixel
|
|
#: count, so 50 000 x 50 000 is several GB and takes the whole batch with it.
|
|
#: Found by an independent review of 0.10.0 before it was pushed.
|
|
MAX_IMAGE_PIXELS = 40_000_000
|
|
|
|
#: The same bound in bytes: for a payload whose pixels are not yet known (an
|
|
#: inline `data:` URI) and for the DECOMPRESSED size of a stream, which is a
|
|
#: different number from anything the container declares. `/Length` in a PDF
|
|
#: image dictionary is the COMPRESSED length and `/Width` and `/Height` are a
|
|
#: claim; nothing in the dictionary states what the decompression will cost.
|
|
MAX_IMAGE_BYTES = 256 * 1024 * 1024
|
|
|
|
#: How much of a stream is inflated at a time while `inflated_size` measures
|
|
#: it. The cap on the OUTPUT is what keeps the measurement cheaper than the
|
|
#: bomb; the input is handed over whole because it is already in memory.
|
|
_INFLATE_CHUNK = 1 << 20
|
|
|
|
|
|
def check_size(
|
|
width: int | None, height: int | None, *, name: str, channels: int = 1, bits: int = 8
|
|
) -> None:
|
|
"""Refuse a declared size beyond the bound, BEFORE anything is decoded.
|
|
|
|
Read off what the container DECLARES. That is the only number available
|
|
before a cost is paid, and it is a CLAIM by an untrusted document rather
|
|
than a measurement: this refuses a picture too large to be one we carry,
|
|
and it says nothing about what decompressing the stream beside it costs.
|
|
`inflated_size` is the other half, and neither substitutes for the other.
|
|
|
|
`None` is UNKNOWN -- a container that declares no size leaves nothing to
|
|
bound, and inventing a number would refuse a legitimate picture. A size
|
|
that is declared and is not positive is neither unknown nor large: it is a
|
|
dictionary that was written wrong or written to be read wrong, and it is
|
|
refused here rather than three steps later by the encoder. Measured on
|
|
`230d1cb`: `/Width -1 /Height 40000000000` multiplies to a NEGATIVE pixel
|
|
count, so every `>` below was false, the function returned silently, 400 MB
|
|
was decompressed, and the refusal arrived from `encode_png` under
|
|
`asset_samples_invalid` -- a code about a sample buffer for a defect in the
|
|
declaration.
|
|
"""
|
|
if width is None or height is None:
|
|
return
|
|
if width <= 0 or height <= 0:
|
|
raise ExtractionError(
|
|
f"the image {name!r} declares {width}x{height}, which is not a size; "
|
|
"refused unread rather than multiplied out, because a non-positive "
|
|
"dimension makes every bound below it read as satisfied",
|
|
code="asset_size_invalid",
|
|
)
|
|
pixels = width * height
|
|
expected = pixels * channels * (bits // 8 or 1)
|
|
if pixels > MAX_IMAGE_PIXELS or expected > MAX_IMAGE_BYTES:
|
|
raise ExtractionError(
|
|
f"the image {name!r} declares {width}x{height} = {pixels} pixels "
|
|
f"({expected} bytes of samples), over this package's bound of "
|
|
f"{MAX_IMAGE_PIXELS} pixels and {MAX_IMAGE_BYTES} bytes; refused "
|
|
"unread so one picture cannot take the run with it",
|
|
code="asset_too_large",
|
|
)
|
|
|
|
|
|
def check_payload(size: int, *, name: str) -> None:
|
|
"""The same bound for an encoded payload of `size` bytes."""
|
|
if size > MAX_IMAGE_BYTES:
|
|
raise ExtractionError(
|
|
f"the image {name!r} carries {size} encoded bytes, over this package's "
|
|
f"bound of {MAX_IMAGE_BYTES}; refused unread",
|
|
code="asset_too_large",
|
|
)
|
|
|
|
|
|
#: What decoding ONE link of a PDF filter chain may cost this package, in bytes
|
|
#: of memory. A SEPARATE number from `MAX_IMAGE_BYTES`, and the distinction is
|
|
#: the whole of round 3: that one bounds the picture this package will carry,
|
|
#: this one bounds what producing it costs on the way. Three rounds of this
|
|
#: review each bound an output and the bomb moved one link along, because a
|
|
#: decoder's working set is not its output. Twice `MAX_IMAGE_BYTES`, so a run
|
|
#: may hold the stream it was given and one stage of decoding at once and no
|
|
#: more.
|
|
MAX_FILTER_DECODE_BYTES = 512 * 1024 * 1024
|
|
|
|
#: MEASURED peak memory per byte of INPUT, for each filter this package lets an
|
|
#: image be reached through. `None` means the decoder is driven a chunk at a
|
|
#: time here, so the cost is measured as it is paid and no ratio is needed --
|
|
#: today that is `FlateDecode` alone (`_inflate`).
|
|
#:
|
|
#: The numbers are read off CPython 3.14 on 2026-09-18, worst case per filter:
|
|
#:
|
|
#: * `ASCII85Decode` 101.4x at 1 MiB of input, 96.1x at 4 MiB, 94.5x at 16 MiB.
|
|
#: `z` is the shorthand for four zero bytes, so `base64.a85decode` appends one
|
|
#: 4-byte object per INPUT byte to a list -- the output ratio is 4, the cost
|
|
#: ratio is a hundred, and 0.10.1 documented this filter as "bounded by its
|
|
#: own input because it shrinks". The constant sits above the worst of the
|
|
#: three, and `test_the_ascii85_cost_ratio_is_not_below_the_one_this_package
|
|
#: _measured` re-measures it so it cannot rot when CPython changes.
|
|
#: * `ASCIIHexDecode` 1.5x at 16 MiB: it strips whitespace into a copy and
|
|
#: `unhexlify`s that, and its output is half its input.
|
|
#: * `DCTDecode`, `JPXDecode` and `JBIG2Decode` are pass-through in pdfminer --
|
|
#: the bytes are handed to the image reader unchanged -- so the ratio is 1.
|
|
#:
|
|
#: A filter that is not in this table has no measured ratio and is refused
|
|
#: unread (`asset_pdf_unbounded`). That is the same decision `corpus.resolve
|
|
#: _gate` takes for an unknown gate name: a fallback reproduces the defect with
|
|
#: an extra step.
|
|
PDF_FILTER_COST_RATIO: dict[str, float | None] = {
|
|
"FlateDecode": None,
|
|
"ASCII85Decode": 104.0,
|
|
"ASCIIHexDecode": 2.0,
|
|
"DCTDecode": 1.0,
|
|
"JPXDecode": 1.0,
|
|
"JBIG2Decode": 1.0,
|
|
}
|
|
|
|
#: The largest OUTPUT each of those filters can produce per byte of input, used
|
|
#: to carry a bound forward when the bytes themselves have been discarded.
|
|
#: `ASCII85Decode` is 4 (one `z`), `ASCIIHexDecode` 0.5 (two digits to a byte),
|
|
#: pass-through 1. `FlateDecode` has no such number, which is why it is the one
|
|
#: filter measured a chunk at a time.
|
|
PDF_FILTER_OUTPUT_RATIO: dict[str, float | None] = {
|
|
"FlateDecode": None,
|
|
"ASCII85Decode": 4.0,
|
|
"ASCIIHexDecode": 0.5,
|
|
"DCTDecode": 1.0,
|
|
"JPXDecode": 1.0,
|
|
"JBIG2Decode": 1.0,
|
|
}
|
|
|
|
|
|
def filter_input_limit(canonical: str) -> int | None:
|
|
"""The largest input this package will hand to `canonical`'s decoder.
|
|
|
|
`None` for a filter decoded a chunk at a time, which needs no input limit
|
|
because its cost is measured while it is paid.
|
|
|
|
The number this produces for `ASCII85Decode` -- about 5.0 MB -- is READ OFF
|
|
the corpora the way `MAX_IMAGE_PIXELS` is: over the 9 668 image objects of
|
|
the 77 PDFs on this machine (2026-09-18), 16 decode through an
|
|
`ASCII85Decode` link and the largest input to one is 450 739 bytes, so the
|
|
limit stands more than ten times above anything measured.
|
|
"""
|
|
ratio = PDF_FILTER_COST_RATIO.get(canonical)
|
|
if ratio is None:
|
|
return None
|
|
return int(MAX_FILTER_DECODE_BYTES // ratio)
|
|
|
|
|
|
def check_filter_cost(size: int, *, canonical: str, name: str) -> None:
|
|
"""Refuse a link whose decoder would cost more than the budget, BEFORE it
|
|
decodes anything.
|
|
|
|
This is the half `inflated_size` cannot cover. That one drives zlib a chunk
|
|
at a time and stops the moment the running total crosses the bound, which
|
|
is only possible because zlib hands its output over incrementally. Nothing
|
|
else in a PDF filter chain does: `base64.a85decode` is asked for a whole
|
|
string and gives back a whole string, so by the time its output could be
|
|
measured the memory has been spent. For those the cost is PREDICTED from a
|
|
measured ratio and the input size, and predicted before the call.
|
|
"""
|
|
limit = filter_input_limit(canonical)
|
|
if limit is None or size <= limit:
|
|
return
|
|
ratio = PDF_FILTER_COST_RATIO[canonical]
|
|
raise ExtractionError(
|
|
f"the image {name!r} hands {size} bytes to {canonical}, whose decoder costs "
|
|
f"about {ratio} bytes of memory per byte of input -- over this package's "
|
|
f"budget of {MAX_FILTER_DECODE_BYTES} bytes for one link; refused before "
|
|
"the decode, because a bound on what a link OUTPUTS is not a bound on "
|
|
"what producing it costs",
|
|
code="asset_too_large",
|
|
)
|
|
|
|
|
|
def inflate_limit_for(canonical: str | None) -> int:
|
|
"""How much a `FlateDecode` link may produce, given what comes AFTER it.
|
|
|
|
The picture's own bound is `MAX_IMAGE_BYTES`, but a link's output is the
|
|
next link's input, and a decoder with a cost ratio cannot be handed more
|
|
than `filter_input_limit` allows. Carrying the budget down the chain this
|
|
way is what stops `[/FlateDecode /ASCII85Decode]` from inflating 256 MiB of
|
|
`z` before the link behind it is asked anything.
|
|
"""
|
|
limit = MAX_IMAGE_BYTES
|
|
if canonical is not None:
|
|
behind = filter_input_limit(canonical)
|
|
if behind is not None:
|
|
limit = min(limit, behind)
|
|
return limit
|
|
|
|
|
|
def inflated_size(raw: bytes, *, name: str, limit: int | None = None) -> int:
|
|
"""What a deflate stream costs to decompress, measured without paying it.
|
|
|
|
THE DECLARED SIZE AND THE COST ARE TWO INDEPENDENT NUMBERS, and binding
|
|
only the first is what an independent review of 0.10.1 measured on
|
|
`230d1cb`: a 408 516-byte PDF declaring a 1x1 picture and carrying 400 MB
|
|
of deflated zeros was CARRIED, with no rejection, at 892 MB of peak RSS --
|
|
about 2 100x the file size, linear, so a 10 MB document is ~21 GB and takes
|
|
the whole batch build with it. `check_size` was reading a claim as though
|
|
it were a cost.
|
|
|
|
The output is inflated a chunk at a time and DISCARDED: only the running
|
|
total is kept, so the measurement stays bounded whatever the stream holds,
|
|
and it stops at the first chunk that crosses `limit`. A legitimate image is
|
|
therefore inflated twice -- once here and once by the reader that carries
|
|
it -- which is the price of not holding an unbounded buffer to find out how
|
|
big it is. Measured on R761 Prosesskoden:2025 (50 image objects): the
|
|
second pass costs under a second of a 200-second extraction.
|
|
|
|
A stream that is not valid deflate data is not this function's problem: the
|
|
reader behind it reports that in its own vocabulary, so a `zlib.error` ends
|
|
the measurement at whatever was produced up to it.
|
|
"""
|
|
return sum(len(chunk) for chunk in _inflate(raw, name=name, limit=limit))
|
|
|
|
|
|
def inflate_bounded(raw: bytes, *, name: str, limit: int | None = None) -> bytes:
|
|
"""The same measurement, KEEPING the output rather than discarding it.
|
|
|
|
One chain link is not the end of a chain: a PDF may decode a stream through
|
|
`/Filter [/FlateDecode /FlateDecode]`, and measuring the first link says
|
|
nothing about the second, which is where the cost is. To bound the second
|
|
link the first one's bytes have to exist, so this inflates under the same
|
|
running bound and hands the result on. It refuses at the same point
|
|
`inflated_size` does, so what is held is never more than the bound -- which
|
|
is what separates carrying an intermediate stage from paying for a bomb.
|
|
|
|
Measured 2026-09-18: 400 MB of zeros deflated twice is 795 bytes of stream,
|
|
and the first link of that chain inflates to 407 685 bytes. The whole
|
|
expansion lives in the LAST link, which is the one nothing measured.
|
|
"""
|
|
return b"".join(_inflate(raw, name=name, limit=limit))
|
|
|
|
|
|
def _inflate(raw: bytes, *, name: str, limit: int | None) -> Iterator[bytes]:
|
|
"""Inflate `raw` a chunk at a time, refusing the moment the running total
|
|
crosses the bound. The two callers differ only in whether they keep what
|
|
comes out."""
|
|
bound = MAX_IMAGE_BYTES if limit is None else limit
|
|
decompressor = zlib.decompressobj()
|
|
total = 0
|
|
pending = raw
|
|
try:
|
|
while True:
|
|
produced = decompressor.decompress(pending, _INFLATE_CHUNK)
|
|
total += len(produced)
|
|
if total > bound:
|
|
raise ExtractionError(
|
|
f"the stream behind {name!r} decompresses to more than {bound} "
|
|
f"bytes from {len(raw)} bytes of input, over this package's "
|
|
"bound; refused without being held, because the size a "
|
|
"container declares is a claim and this is the cost",
|
|
code="asset_too_large",
|
|
)
|
|
yield produced
|
|
pending = decompressor.unconsumed_tail
|
|
if decompressor.eof or not pending:
|
|
break
|
|
except zlib.error:
|
|
return
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class ExtractedImage:
|
|
"""One image a document carries, as this package will write it.
|
|
|
|
`name` is what the SOURCE calls the file -- a `xlink:href`, an `<img src>`,
|
|
a media entry inside an OOXML container, or a synthesised name for a PDF
|
|
image object, which has none. It is carried verbatim into the pointer's
|
|
second line and reduced (never trusted) into the asset's own name.
|
|
|
|
`label` is a caption or alt text where the format HAS one, and `None`
|
|
where it does not. Two of the four readers have none: a PDF image object
|
|
and an STS `<graphic>` carry no caption element, and the caption a human
|
|
sees is running text the extractor already emitted on its own line.
|
|
Deriving a label from the nearest line would be an unmarked heuristic,
|
|
which this package treats as worse than no heuristic at all.
|
|
|
|
`converted_from` and `source_sha256` are set when the source was NOT a
|
|
format a model can be shown and this package rewrote it (today: a BMP, as
|
|
a PNG). They are the whole of the traceability: `converted_from` is the
|
|
media type the file had, `source_sha256` is the digest of the bytes the
|
|
document actually shipped, and the digest of `data` is what the bundle
|
|
holds. With the three of them a reader can take the original delivery,
|
|
run `shasum -a 256`, and find the row. Both are `None` for an image
|
|
carried verbatim, which is every image this package has ever carried
|
|
until now -- so a bundle of JPEGs is byte-identical across the move.
|
|
"""
|
|
|
|
data: bytes
|
|
name: str
|
|
media_type: str
|
|
suffix: str
|
|
width: int | None
|
|
height: int | None
|
|
label: str | None = None
|
|
converted_from: str | None = None
|
|
source_sha256: str | None = None
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class AssetRejection:
|
|
"""An image that was FOUND and not carried, with the reason as a code.
|
|
|
|
A rejection is a row in the run log, never a failed document: one
|
|
unreadable picture must not cost the 3 000 concepts of text around it. The
|
|
denominator is what makes the log worth reading -- "51 carried" says
|
|
nothing without "of 53 found".
|
|
"""
|
|
|
|
name: str
|
|
code: str
|
|
reason: str
|
|
|
|
|
|
def sniff(data: bytes) -> tuple[str, str] | None:
|
|
"""`(media type, suffix)` read from the bytes, or `None` if not an image.
|
|
|
|
The claimed extension never participates. A `.jpg` that is really a PNG is
|
|
carried as a PNG under a `.png` name, because the alternative is a bundle
|
|
whose file names disagree with their contents.
|
|
"""
|
|
if data.startswith(_JP2_SIGNATURE) or data.startswith(_J2K_SIGNATURE):
|
|
return "image/jp2", ".jp2"
|
|
for magic, media_type, suffix in _MAGIC:
|
|
if data.startswith(magic):
|
|
return media_type, suffix
|
|
return None
|
|
|
|
|
|
def dimensions(data: bytes) -> tuple[int, int] | None:
|
|
"""`(width, height)` in pixels from the file's own header, or `None`.
|
|
|
|
`None` is a MEASUREMENT: this reader does not read every format's geometry
|
|
(JPEG 2000 and TIFF are absent), and a concept that printed `0x0 px` would
|
|
be stating a number nobody took. The pointer says "dimensions unknown"
|
|
instead.
|
|
"""
|
|
kind = sniff(data)
|
|
if kind is None:
|
|
return None
|
|
suffix = kind[1]
|
|
try:
|
|
if suffix == ".png":
|
|
if len(data) < 24 or data[12:16] != b"IHDR":
|
|
return None
|
|
width, height = struct.unpack(">II", data[16:24])
|
|
return (width, height) if width and height else None
|
|
if suffix == ".jpg":
|
|
return _jpeg_dimensions(data)
|
|
if suffix == ".gif":
|
|
if len(data) < 10:
|
|
return None
|
|
width, height = struct.unpack("<HH", data[6:10])
|
|
return (width, height) if width and height else None
|
|
if suffix == ".bmp":
|
|
return _bmp_dimensions(data)
|
|
except (struct.error, IndexError):
|
|
return None
|
|
return None
|
|
|
|
|
|
def _jpeg_dimensions(data: bytes) -> tuple[int, int] | None:
|
|
position = 2
|
|
end = len(data)
|
|
while position + 3 < end:
|
|
if data[position] != 0xFF:
|
|
position += 1
|
|
continue
|
|
marker = data[position + 1]
|
|
if marker in (0xD8, 0x01) or 0xD0 <= marker <= 0xD7:
|
|
position += 2
|
|
continue
|
|
if marker == 0xFF:
|
|
position += 1
|
|
continue
|
|
length = struct.unpack(">H", data[position + 2 : position + 4])[0]
|
|
if marker in _JPEG_SOF:
|
|
if position + 9 > end:
|
|
return None
|
|
height, width = struct.unpack(">HH", data[position + 5 : position + 9])
|
|
return (width, height) if width and height else None
|
|
position += 2 + length
|
|
return None
|
|
|
|
|
|
def _bmp_dimensions(data: bytes) -> tuple[int, int] | None:
|
|
if len(data) < 26:
|
|
return None
|
|
header_size = struct.unpack("<I", data[14:18])[0]
|
|
if header_size == 12:
|
|
width, height = struct.unpack("<hh", data[18:22])
|
|
else:
|
|
width, height = struct.unpack("<ii", data[18:26])
|
|
# A negative height is a top-down BMP; the magnitude is the pixel count.
|
|
return (abs(width), abs(height)) if width and height else None
|
|
|
|
|
|
#: `biCompression`: uncompressed, and the 8-bit run-length encoding 19 of
|
|
#: R761's 50 assets use. Every other value -- RLE4, BITFIELDS, embedded JPEG or
|
|
#: PNG -- is refused by name rather than guessed at.
|
|
_BMP_RGB = 0
|
|
_BMP_RLE8 = 1
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class _BmpHeader:
|
|
"""What the DIB header declares, before anything is decoded."""
|
|
|
|
header_size: int
|
|
width: int
|
|
height: int
|
|
top_down: bool
|
|
bits: int
|
|
compression: int
|
|
entries: int
|
|
offbits: int
|
|
|
|
|
|
def _bmp_header(data: bytes, *, name: str) -> _BmpHeader:
|
|
"""The declaration, validated -- never the pixels.
|
|
|
|
Everything this returns is a CLAIM by an untrusted file. It is read first
|
|
precisely so the claim can be bounded before the cost of believing it is
|
|
paid: the row buffers below are `width * height` bytes, and that product
|
|
comes from these two fields.
|
|
"""
|
|
if len(data) < 54:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} stops inside its own header ({len(data)} bytes)",
|
|
code="asset_samples_invalid",
|
|
)
|
|
offbits = struct.unpack("<I", data[10:14])[0]
|
|
header_size = struct.unpack("<I", data[14:18])[0]
|
|
if header_size < 40:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} carries a {header_size}-byte DIB header; this reader "
|
|
"expresses BITMAPINFOHEADER and its extensions, and a 12-byte "
|
|
"BITMAPCOREHEADER stores its palette in 3-byte entries, which is a "
|
|
"different format wearing the same magic",
|
|
code="asset_bmp_unsupported",
|
|
)
|
|
if len(data) < 14 + header_size:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} declares a {header_size}-byte DIB header and holds "
|
|
f"{len(data) - 14} bytes after the file header",
|
|
code="asset_samples_invalid",
|
|
)
|
|
width, height = struct.unpack("<ii", data[18:26])
|
|
bits = struct.unpack("<H", data[28:30])[0]
|
|
compression = struct.unpack("<I", data[30:34])[0]
|
|
clr_used = struct.unpack("<I", data[46:50])[0]
|
|
if width <= 0 or height == 0:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} declares {width}x{height}, which is not a size",
|
|
code="asset_size_invalid",
|
|
)
|
|
top_down = height < 0
|
|
entries = clr_used or (1 << bits if bits <= 8 else 0)
|
|
if entries > 256:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} declares {entries} palette entries; a PNG palette holds at most 256",
|
|
code="asset_bmp_unsupported",
|
|
)
|
|
return _BmpHeader(
|
|
header_size=header_size,
|
|
width=width,
|
|
height=-height if top_down else height,
|
|
top_down=top_down,
|
|
bits=bits,
|
|
compression=compression,
|
|
entries=entries,
|
|
offbits=offbits,
|
|
)
|
|
|
|
|
|
def _bmp_palette(data: bytes, head: _BmpHeader, *, name: str) -> bytes:
|
|
"""The colour table as PNG wants it: RGB triples, from the file's BGRA.
|
|
|
|
The order matters and getting it wrong is invisible in every structural
|
|
check: a palette read as RGB gives a picture with red and blue swapped,
|
|
the right size, the right number of colours, and the wrong answer.
|
|
"""
|
|
entries = head.entries or 256
|
|
start = 14 + head.header_size
|
|
table = data[start : start + entries * 4]
|
|
if len(table) < entries * 4:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} declares {entries} palette entries and holds {len(table) // 4}",
|
|
code="asset_samples_invalid",
|
|
)
|
|
palette = bytearray()
|
|
for index in range(entries):
|
|
blue, green, red = table[index * 4], table[index * 4 + 1], table[index * 4 + 2]
|
|
palette += bytes((red, green, blue))
|
|
return bytes(palette)
|
|
|
|
|
|
def _bmp_flat_rows(data: bytes, head: _BmpHeader, *, per_pixel: int, name: str) -> bytes:
|
|
"""Uncompressed rows, unpadded and in top-down order.
|
|
|
|
A BMP row is padded to a 4-byte boundary and stored bottom-up unless the
|
|
declared height is negative. Both are read off the header rather than
|
|
assumed, because either mistake produces a picture that is the right size
|
|
and is sheared or upside down.
|
|
"""
|
|
stride = ((head.width * per_pixel + 3) // 4) * 4
|
|
needed = stride * head.height
|
|
available = len(data) - head.offbits
|
|
if available < needed:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} needs {needed} bytes of pixel data and holds {available} "
|
|
"-- refusing to pad, because a short buffer means the header was read wrong",
|
|
code="asset_samples_invalid",
|
|
)
|
|
rows = [
|
|
data[head.offbits + index * stride : head.offbits + index * stride + head.width * per_pixel]
|
|
for index in range(head.height)
|
|
]
|
|
if not head.top_down:
|
|
rows.reverse()
|
|
return b"".join(rows)
|
|
|
|
|
|
def _bmp_rle8_rows(data: bytes, head: _BmpHeader, *, name: str) -> bytes:
|
|
"""The RLE8 opcode stream painted into a frame of the DECLARED size.
|
|
|
|
Five opcodes, and a decoder implementing only the first is wrong on real
|
|
files: an encoded run, an absolute run (whose byte count is padded to an
|
|
even length), a delta that SKIPS pixels, end-of-line and end-of-bitmap.
|
|
Skipped pixels keep index 0, which is what the format says and what an
|
|
independent decoder produces.
|
|
|
|
THE COST IS THE FRAME, NOT THE STREAM. The rows are allocated once from the
|
|
declared size -- already bounded by `check_size` before this is called --
|
|
and each run is written as one CLIPPED slice. Painting pixel by pixel would
|
|
leave the memory bounded and the CPU unbounded: a megabyte of `FF` runs is
|
|
a hundred million paint steps against a frame of 32 pixels.
|
|
|
|
A STREAM THAT NEVER SAYS IT IS FINISHED IS REFUSED, and that is the
|
|
difference between a skipped pixel and a missing one. Running out of bytes
|
|
leaves the rest of the frame at index 0 -- indistinguishable, in the
|
|
output, from a delta that skipped it, which is why every decoder agrees on
|
|
the wrong picture: measured 2026-09-19 on a real 352x548 R761 asset, a
|
|
stream cut to 90 % was carried with 13 923 pixels wrong, to 50 % with
|
|
95 890, to 10 % with 166 525, with no code and no row. So the loop may end
|
|
HERE and nowhere else, and `_bmp_flat_rows` refuses the same shape with the
|
|
same code.
|
|
|
|
The terminator is checked rather than `biSizeImage`, which is a claim by
|
|
the same untrusted header. Read off the corpus: over the 25 RLE8 BMPs
|
|
the frozen R761 delivery ships (24 distinct; the bundle carries 19 of
|
|
them, the rest being an unpointed duplicate and four no concept names),
|
|
25 of 25 end at an explicit
|
|
end-of-bitmap, on 25 of 25 it is the stream's LAST two bytes, and on 25 of
|
|
25 `biSizeImage` equals the available bytes -- so requiring it costs
|
|
nothing measured here, and a whole stream that omits it is refused
|
|
alongside a cut one because from the reader's side they are the same bytes.
|
|
|
|
AND THE TERMINATOR ALONE IS NOT A COVERAGE PROOF, because a stream may say
|
|
it is finished anywhere: measured 2026-09-19, one whose FIRST two bytes are
|
|
the end-of-bitmap escape was carried with every pixel of the frame never
|
|
decoded. So the cursor must also stand at or past the end of the last row.
|
|
|
|
THE LINE IS THE CURSOR AND NOT THE PIXELS, and that is a format argument
|
|
rather than a corpus one -- the corpus cannot choose between the two, since
|
|
25 of 25 of those files paint every pixel, 25 of 25 reach the end of the
|
|
frame and 0 of 25 use a delta. A delta escape and an end-of-line escape
|
|
STATE their skip, so the pixels they pass over keep index 0 and every
|
|
decoder produces the same picture; a pixel-coverage count would refuse both
|
|
constructions the format defines. Pixels the stream never reached have no
|
|
agreed value at all, which is why an independent decoder refuses the file:
|
|
measured over eight streams for one frame, Pillow reads the five whose
|
|
cursor reaches the end and refuses the three whose does not, one of them
|
|
short by a single pixel (`tests/test_asset_viewable.py`).
|
|
"""
|
|
width, height = head.width, head.height
|
|
rows = [bytearray(width) for _ in range(height)]
|
|
position = head.offbits
|
|
end = len(data)
|
|
x = 0
|
|
y = 0
|
|
finished = False
|
|
while position + 1 < end:
|
|
count = data[position]
|
|
value = data[position + 1]
|
|
position += 2
|
|
if count:
|
|
if 0 <= y < height and x < width:
|
|
stop = min(x + count, width)
|
|
rows[y][x:stop] = bytes((value,)) * (stop - x)
|
|
x += count
|
|
continue
|
|
if value == 0:
|
|
x = 0
|
|
y += 1
|
|
elif value == 1:
|
|
finished = True
|
|
break
|
|
elif value == 2:
|
|
if position + 2 > end:
|
|
break
|
|
x += data[position]
|
|
y += data[position + 1]
|
|
position += 2
|
|
else:
|
|
run = data[position : position + value]
|
|
position += value + (value & 1)
|
|
if 0 <= y < height and x < width:
|
|
stop = min(x + len(run), width)
|
|
rows[y][x:stop] = run[: stop - x]
|
|
x += value
|
|
if not finished:
|
|
raise ExtractionError(
|
|
f"the RLE8 stream in {name!r} ends after {position - head.offbits} of "
|
|
f"{len(data) - head.offbits} bytes without an end-of-bitmap escape "
|
|
"-- refusing to carry a frame whose remaining pixels were never decoded",
|
|
code="asset_samples_invalid",
|
|
)
|
|
if y < height - 1 or (y == height - 1 and x < width):
|
|
raise ExtractionError(
|
|
f"the RLE8 stream in {name!r} ends at row {y} column {x} of a {width}x{height} "
|
|
"frame -- refusing to carry a picture whose last rows the stream never reached",
|
|
code="asset_samples_invalid",
|
|
)
|
|
if not head.top_down:
|
|
rows.reverse()
|
|
return b"".join(bytes(row) for row in rows)
|
|
|
|
|
|
def bmp_to_png(data: bytes, *, name: str) -> bytes:
|
|
"""A BMP as a PNG with the same pixels, using nothing but the stdlib.
|
|
|
|
WHY A READER HERE AND NOT PILLOW, which this tree already carries
|
|
transitively under `pdfplumber`. Two reasons, measured rather than
|
|
preferred. First, `read_image` is on the CORE path: `.html` and `.xml` are
|
|
stdlib file types that carry images with no `[extract]` extra installed, so
|
|
a Pillow-based converter would either make a core path depend on an
|
|
optional binary wheel or buy this package its second runtime dependency.
|
|
Second, and decisive: an asset's name is its content digest, so the bytes
|
|
this function emits are part of the bundle's identity. Encoding through an
|
|
installed library would make that identity move with the library's version
|
|
-- the exact property 0.10.0 felled page rasterisation over. `encode_png`
|
|
already writes a PNG from samples with `zlib` alone; this adds the reader in
|
|
front of it.
|
|
|
|
Pillow is still the INDEPENDENT decoder in the tests, which is the job it
|
|
is good for here: 19 of 19 of R761's real RLE8 assets decode to identical
|
|
RGB through both paths (measured 2026-09-19, before this was written).
|
|
"""
|
|
head = _bmp_header(data, name=name)
|
|
channels = 3 if head.bits == 24 else 1
|
|
# THE CEILING FIRST, on the DECLARATION, before one row is allocated. The
|
|
# rows below are `width * height` bytes of an untrusted document's claim.
|
|
check_size(head.width, head.height, name=name, channels=channels, bits=8)
|
|
if head.bits == 8 and head.compression in (_BMP_RGB, _BMP_RLE8):
|
|
palette = _bmp_palette(data, head, name=name)
|
|
if head.compression == _BMP_RLE8:
|
|
if head.top_down:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} declares a top-down RLE8 image, which the format "
|
|
"does not define",
|
|
code="asset_bmp_unsupported",
|
|
)
|
|
samples = _bmp_rle8_rows(data, head, name=name)
|
|
else:
|
|
samples = _bmp_flat_rows(data, head, per_pixel=1, name=name)
|
|
limit = len(palette) // 3
|
|
if samples and max(samples) >= limit:
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} uses palette index {max(samples)} and declares {limit} "
|
|
"entries; carrying it would invent a colour",
|
|
code="asset_samples_invalid",
|
|
)
|
|
return encode_png(head.width, head.height, samples, channels=1, palette=palette)
|
|
if head.bits == 24 and head.compression == _BMP_RGB:
|
|
raw = _bmp_flat_rows(data, head, per_pixel=3, name=name)
|
|
swapped = bytearray(raw)
|
|
# BGR on disk, RGB in a PNG. Two slices rather than a loop, and read
|
|
# from `raw` both times so the first assignment cannot feed the second.
|
|
swapped[0::3] = raw[2::3]
|
|
swapped[2::3] = raw[0::3]
|
|
return encode_png(head.width, head.height, bytes(swapped), channels=3)
|
|
raise ExtractionError(
|
|
f"the BMP {name!r} stores {head.bits}-bit samples under compression "
|
|
f"{head.compression}; this reader expresses 8-bit (uncompressed and RLE8) and "
|
|
"24-bit uncompressed, and will not guess at the rest",
|
|
code="asset_bmp_unsupported",
|
|
)
|
|
|
|
|
|
#: Source media type -> the function that makes a viewable file of it. A map
|
|
#: rather than a branch, so what this package can convert is one readable line
|
|
#: and adding a format is adding a row.
|
|
_CONVERTERS = {"image/bmp": bmp_to_png}
|
|
|
|
|
|
def to_viewable(data: bytes, *, media_type: str, name: str) -> bytes:
|
|
"""Bytes a model can be shown, or a coded refusal. Never a silent carry."""
|
|
convert = _CONVERTERS.get(media_type)
|
|
if convert is None:
|
|
raise ExtractionError(
|
|
f"the image {name!r} is {media_type}, which no model can be shown, and this "
|
|
"package has no lossless conversion for it; refused rather than carried in a "
|
|
"format nothing reads",
|
|
code="asset_not_viewable",
|
|
)
|
|
return convert(data, name=name)
|
|
|
|
|
|
def read_image(data: bytes, *, name: str, label: str | None = None) -> ExtractedImage:
|
|
"""One image, typed by its bytes, or a coded refusal.
|
|
|
|
Raises :class:`ExtractionError` with `asset_type_unknown` when the bytes
|
|
are not an image this package recognises. The caller records that as an
|
|
:class:`AssetRejection` and keeps going -- a document is not lost over one
|
|
picture.
|
|
"""
|
|
kind = sniff(data)
|
|
if kind is None:
|
|
raise ExtractionError(
|
|
f"the bytes behind {name!r} are not an image format this package "
|
|
f"recognises (first bytes {data[:8]!r})",
|
|
code="asset_type_unknown",
|
|
)
|
|
media_type, suffix = kind
|
|
# THE VIEWABILITY GATE, and it stands before the size is read because the
|
|
# size that matters is the one the CARRIED file has. A BMP that becomes a
|
|
# PNG is measured as the PNG a consumer will open.
|
|
converted_from: str | None = None
|
|
source_sha256: str | None = None
|
|
if media_type not in VIEWABLE_MEDIA_TYPES:
|
|
converted_from = media_type
|
|
source_sha256 = digest(data)
|
|
data = to_viewable(data, media_type=media_type, name=name)
|
|
media_type, suffix = "image/png", ".png"
|
|
size = dimensions(data)
|
|
# THE BOUND HOLDS FOR A FILE CARRIED VERBATIM TOO. This package does not
|
|
# decode one, so it pays nothing for it -- but writing a 7 000 x 7 000 PNG
|
|
# of 47 705 bytes into a bundle hands the consumer the same bomb with
|
|
# `7000x7000 px` printed beside it, and the README's first sentence about
|
|
# this bound says such an image is refused. Sniffed from the header rather
|
|
# than claimed, like the type beside it. Measured: the largest of the
|
|
# 4 828 objects in the reference corpus is 18.6 MP, so nothing anyone has
|
|
# measured is refused here.
|
|
if size is not None:
|
|
check_size(size[0], size[1], name=name)
|
|
return ExtractedImage(
|
|
data=data,
|
|
name=name,
|
|
media_type=media_type,
|
|
suffix=suffix,
|
|
width=size[0] if size else None,
|
|
height=size[1] if size else None,
|
|
label=label,
|
|
converted_from=converted_from,
|
|
source_sha256=source_sha256,
|
|
)
|
|
|
|
|
|
def digest(data: bytes) -> str:
|
|
return hashlib.sha256(data).hexdigest()
|
|
|
|
|
|
def conversion(image: ExtractedImage) -> tuple[str, str] | None:
|
|
"""`(the source's digest, the carried asset's digest)`, or `None`.
|
|
|
|
THE RUN'S OWN RECORD OF WHAT IT REWROTE, for a reader that must not have
|
|
to take the bundle's word for it. `render_block` states the same pair on
|
|
the pointer's second line, which is where a person reads it -- but that
|
|
line is markdown in a concept body, and measured by PM 2026-09-19 an
|
|
ordinary HTML document with two `<p>` elements produces the same two
|
|
lines. A judge reading the claim off the bundle text is therefore reading
|
|
an untrusted document; a judge reading it off the accounting is reading
|
|
this function's output, which no document can reach.
|
|
"""
|
|
if image.converted_from is None or image.source_sha256 is None:
|
|
return None
|
|
return (image.source_sha256, digest(image.data))
|
|
|
|
|
|
def _reduce(text: str) -> str:
|
|
return _SEPARATOR_RUN.sub("-", unicodedata.normalize("NFC", text).lower()).strip("-")
|
|
|
|
|
|
def asset_name(image: ExtractedImage) -> str:
|
|
"""`<sha256[:12]>-<reduced original BASENAME><sniffed suffix>`.
|
|
|
|
DEDUP IS ON CONTENT, and the digest is what makes it so: the same bytes
|
|
arriving twice are one file, whichever document dropped them. The residual
|
|
is stated rather than claimed away -- identical bytes shipped under two
|
|
different base names are two files in the bundle, both correct and both
|
|
holding the same picture. Measured on the fixture inbox and on the R761
|
|
delivery, that case occurs 0 times; a rule that could not produce it at all
|
|
would have to drop the readable tail entirely, and the tail is what makes
|
|
`assets/` legible to the person checking a bundle by hand.
|
|
|
|
|
|
The suffix comes from the bytes and the stem from the name, reduced to the
|
|
same grammar every generated filename in this package uses. A link target
|
|
sits inside `](...)`, which a space or a closing parenthesis terminates --
|
|
and the R761 delivery's own hrefs carry both (`25-0143 - Tabeller -
|
|
Prosesskoden (R761-R762).jpg`), so reducing is what makes the pointer
|
|
followable rather than merely tidy.
|
|
"""
|
|
# The BASENAME, never the path the document reached it through. Measured
|
|
# on the fixture inbox: one image pointed at as `graphics/figur-84-1.png`
|
|
# from an HTML document and as `figur-84-1.png` from an STS one was written
|
|
# twice, under two names, in one run -- with the digest in both announcing
|
|
# that the bytes were identical. The path is a property of the pointer, not
|
|
# of the picture, and the full original survives on the pointer's own line.
|
|
base = image.name.rsplit("/", 1)[-1]
|
|
stem = _reduce(base.rsplit(".", 1)[0] if "." in base else base)
|
|
head = digest(image.data)[:DIGEST_PREFIX]
|
|
if not stem:
|
|
return f"{head}{image.suffix}"
|
|
return f"{head}-{stem[:NAME_TAIL_MAX].rstrip('-')}{image.suffix}"
|
|
|
|
|
|
def asset_href(image: ExtractedImage) -> str:
|
|
"""The bundle-absolute path SPEC SS 6.2 allows.
|
|
|
|
Absolute rather than relative because a segmented bundle puts concepts at
|
|
different depths: `assets/x.png` resolves to two different places from two
|
|
concepts of one document, and `/assets/x.png` to one place from every
|
|
concept in the bundle.
|
|
"""
|
|
return f"/{ASSETS_DIR}/{asset_name(image)}"
|
|
|
|
|
|
#: `sha256:` immediately in front of 64 hex digits -- the CHECKSUM FIELD this
|
|
#: module writes on a pointer's second line, and the grammar the content
|
|
#: accounting gate reads a conversion claim with.
|
|
_CHECKSUM_FIELD = re.compile(r"sha256:(?=[0-9a-fA-F]{64})")
|
|
|
|
|
|
def _inline(value: str) -> str:
|
|
"""A label, made safe for the one line it is written on.
|
|
|
|
`[` and `]` are the link grammar's own delimiters and a newline would open
|
|
a third line in a two-line block, so both are replaced rather than escaped:
|
|
Door B refuses a title containing a bracket outright (`inbox_title_invalid`)
|
|
and this text reaches a title through no route, but the pointer is body text
|
|
a proposer reads, and a half-open link there is a pointer that resolves
|
|
nowhere.
|
|
|
|
WHERE THE BOUNDARY RUNS. Everything this function returns came from the
|
|
DOCUMENT -- an `alt` attribute, an STS `<caption>`, a file name a publisher
|
|
chose. Everything `render_block` appends after it came from the run: the
|
|
size it measured, the type it sniffed, the digests it computed. The second
|
|
line carries both, so document text must not be able to emit the metadata
|
|
grammar the run writes there. Measured by PM 2026-09-19: an `alt` attribute
|
|
stating `converted from ... sha256:<a> to ... sha256:<b>` made the content
|
|
accounting gate report a picture as carried that was refused
|
|
`asset_too_large` and is not in `assets/` at all. A checksum field is
|
|
therefore disarmed here -- the digits are kept, because a reader is owed
|
|
what the document said, and the colon that makes them a FIELD is not.
|
|
"""
|
|
collapsed = " ".join(value.split())
|
|
return _CHECKSUM_FIELD.sub("sha256 ", collapsed.replace("[", "(").replace("]", ")"))
|
|
|
|
|
|
def render_block(image: ExtractedImage) -> str:
|
|
"""The two lines that stand where the image stands.
|
|
|
|
Line one is markdown, so a reader that renders the concept sees the picture
|
|
and a reader that does not sees the label. Line two states what the first
|
|
line cannot: the name the SOURCE gave the file, and the size in pixels --
|
|
the two facts a person checking the bundle against the original needs, and
|
|
the two a transcription in step 2 has to be judged against.
|
|
"""
|
|
label = image.label or image.name or asset_name(image)
|
|
size = (
|
|
f"{image.width}x{image.height} px"
|
|
if image.width is not None and image.height is not None
|
|
else "dimensions unknown"
|
|
)
|
|
detail = f"Image: {_inline(image.name or asset_name(image))} ({size})"
|
|
if image.label:
|
|
detail += f" -- {_inline(image.label)}"
|
|
# WHAT THE CONVERSION DID, on the line the rest of the asset metadata is
|
|
# already on. Both digests in full: the asset's file name carries only the
|
|
# first 12 hex characters of the new one, and a checksum a reader cannot
|
|
# paste into `shasum -a 256` is decoration. Written LAST so a labelled
|
|
# image that was not converted keeps the bytes it has today.
|
|
if image.converted_from and image.source_sha256:
|
|
detail += (
|
|
f" -- converted from {image.converted_from} sha256:{image.source_sha256}"
|
|
f" to {image.media_type} sha256:{digest(image.data)}"
|
|
)
|
|
return f"})\n{detail}"
|
|
|
|
|
|
def encode_png(
|
|
width: int,
|
|
height: int,
|
|
samples: bytes,
|
|
*,
|
|
channels: int,
|
|
palette: bytes | None = None,
|
|
alpha: bytes | None = None,
|
|
) -> bytes:
|
|
"""8-bit samples as a PNG, using nothing but `zlib`.
|
|
|
|
Refuses a size over :data:`MAX_IMAGE_PIXELS` on its own rather than
|
|
trusting the caller to have checked: this function is what allocates
|
|
`width * height * channels` bytes twice over.
|
|
|
|
A PDF image is usually not a file: `FlateDecode` hands back raw samples
|
|
with the colour model in the dictionary beside them, so carrying one at all
|
|
means encoding it. Doing that with the stdlib rather than with a renderer
|
|
is what keeps the `pdf` row free of a new dependency AND keeps an asset's
|
|
bytes -- and therefore its content-addressed name -- independent of which
|
|
version of a rasteriser happened to be installed. `OCR_DPI`'s docstring
|
|
states the opposite property for OCR text, and the difference is deliberate.
|
|
|
|
`channels` is 1 (grey, or indexed when `palette` is given) or 3 (RGB).
|
|
`alpha` is one byte per pixel, from a PDF `SMask`; absent means opaque.
|
|
"""
|
|
check_size(width, height, name=f"{width}x{height}", channels=channels)
|
|
if channels not in (1, 3):
|
|
raise ExtractionError(
|
|
f"PNG encoding supports 1 or 3 channels, not {channels}",
|
|
code="asset_samples_invalid",
|
|
)
|
|
if width <= 0 or height <= 0:
|
|
raise ExtractionError(
|
|
f"PNG encoding needs positive dimensions, got {width}x{height}",
|
|
code="asset_samples_invalid",
|
|
)
|
|
expected = width * height * channels
|
|
if len(samples) < expected:
|
|
raise ExtractionError(
|
|
f"the sample buffer holds {len(samples)} bytes where {width}x{height} "
|
|
f"at {channels} channels needs {expected} -- refusing to pad, because a "
|
|
"short buffer means the image dictionary was read wrong",
|
|
code="asset_samples_invalid",
|
|
)
|
|
if alpha is not None and len(alpha) < width * height:
|
|
raise ExtractionError(
|
|
f"the alpha buffer holds {len(alpha)} bytes where {width}x{height} needs "
|
|
f"{width * height}",
|
|
code="asset_samples_invalid",
|
|
)
|
|
if palette is not None:
|
|
if channels != 1:
|
|
raise ExtractionError(
|
|
"a palette applies to single-channel samples only",
|
|
code="asset_samples_invalid",
|
|
)
|
|
if not palette or len(palette) % 3:
|
|
raise ExtractionError(
|
|
f"a palette must be whole RGB triples, got {len(palette)} bytes",
|
|
code="asset_samples_invalid",
|
|
)
|
|
colour_type = 3
|
|
elif alpha is not None:
|
|
colour_type = 6 if channels == 3 else 4
|
|
else:
|
|
colour_type = 2 if channels == 3 else 0
|
|
|
|
rows = bytearray()
|
|
for row in range(height):
|
|
# Filter type 0 (None) on every row. A filter would shrink the file and
|
|
# would make the bytes depend on a heuristic; this encoder's output has
|
|
# to be reproducible from the samples alone, for as long as the bundle
|
|
# is quoted by its digest.
|
|
rows.append(0)
|
|
start = row * width * channels
|
|
line = samples[start : start + width * channels]
|
|
if alpha is None:
|
|
rows += line
|
|
else:
|
|
for pixel in range(width):
|
|
rows += line[pixel * channels : (pixel + 1) * channels]
|
|
rows.append(alpha[row * width + pixel])
|
|
|
|
def chunk(kind: bytes, payload: bytes) -> bytes:
|
|
return (
|
|
len(payload).to_bytes(4, "big")
|
|
+ kind
|
|
+ payload
|
|
+ zlib.crc32(kind + payload).to_bytes(4, "big")
|
|
)
|
|
|
|
ihdr = struct.pack(">IIBBBBB", width, height, 8, colour_type, 0, 0, 0)
|
|
body = b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", ihdr)
|
|
if palette is not None:
|
|
body += chunk(b"PLTE", palette)
|
|
# Level 9 with no filtering: `zlib` is stdlib and its output for a given
|
|
# level is stable within a Python build, which is the same guarantee the
|
|
# rest of this package's byte-determinism rests on.
|
|
body += chunk(b"IDAT", zlib.compress(bytes(rows), 9))
|
|
return body + chunk(b"IEND", b"")
|
|
|
|
|
|
def render_missing(
|
|
name: str,
|
|
*,
|
|
reason: str,
|
|
label: str | None = None,
|
|
href: str | None = None,
|
|
) -> str:
|
|
"""A pointer to an image the bundle does NOT carry, and why.
|
|
|
|
An image this package found and could not carry is stated in the concept,
|
|
not dropped. The reader of the bundle is the person who has to decide
|
|
whether the missing picture mattered, and they cannot decide about an
|
|
absence they were never shown -- which is precisely the shape of the defect
|
|
this whole capability exists to close.
|
|
|
|
The href is kept when the source had one, so a remote figure says WHERE it
|
|
was. Extraction never opens a socket: the network gate is an explicit
|
|
per-run opt-in and extraction is not on that path, so a remote source is
|
|
carried as a name and never as bytes.
|
|
|
|
`label` is the alt text or the figure caption, and it is written for the
|
|
same reason the line exists at all: the reader deciding whether the missing
|
|
picture mattered is much better served by "Figur 84-1 Tverrprofil" than by
|
|
a file name. 0.10.1 dropped it while closing the live-link defect -- the
|
|
parameter stayed in the signature and no branch read it -- which an
|
|
independent review measured as a regression against 0.10.0.
|
|
"""
|
|
# INERT, never ``. A remote reference is an address the
|
|
# document's author chose, and 0.10.0 wrote it as a live markdown image: a
|
|
# consumer that renders the bundle, or an agent that fetches what it
|
|
# renders, turns "this bundle was opened" into a beacon to them -- and a
|
|
# server-side consumer into an SSRF. This package opens no socket, which is
|
|
# not the same as the pointer being harmless.
|
|
#
|
|
# The address is STATED, because a reader has to know what stood there --
|
|
# and stated ONCE, inside a code span. 0.10.1 wrote it twice: once in the
|
|
# span and once as the bare `name`, and a GFM/linkify renderer autolinks a
|
|
# bare URL into `<a href="...">`. It takes a click rather than a render, so
|
|
# it is weaker than `![..]()` -- but "inert" was half the truth, and half
|
|
# is what this line exists not to be.
|
|
if href:
|
|
address = _inline(href).replace("`", "'")
|
|
shown = f"`{address}`"
|
|
else:
|
|
shown = _inline(name or "image")
|
|
detail = f"Image: {shown} (not carried: {_inline(reason)})"
|
|
if label:
|
|
detail += f" -- {_inline(label)}"
|
|
return detail
|