open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(assets): budget every link by what its decoder COSTS (0.10.1)

Browse source 
Round 3 of the 0.10.1 review, and the finding is the pattern the three rounds
share: each bound an OUTPUT, and the bomb stepped one link along. The
declared size, then the first `FlateDecode`, then every `FlateDecode` -- and
then a link this package had documented as safe.

`ASCII85Decode` was classed as bounded "by its own input because it shrinks".
It quadruples: `z` is the shorthand for four zero bytes. And the output was
never the cost -- `base64.a85decode` appends one 4-byte object per group to a
list, about a hundred bytes of memory per byte of INPUT (101.4x at 1 MiB,
96.1x at 4 MiB, 94.5x at 16 MiB on CPython 3.14).

Paired subprocesses, idle machine, both sides from PINNED trees, the document
built once by a third process and read from a file because `ru_maxrss` never
falls and `b"z" * 64 MiB` alone costs 171 MB:

  [/Fl /A85]      z x 32 Mi  33 475 B   CARRIED 3 261 599 744 -> too_large 42 070 016
  [/Fl /A85]      z x 64 Mi  66 090 B   CARRIED 6 461 558 784 -> too_large 40 280 064
  [/A85]          z x  8 Mi   8.4 MB    CARRIED   933 085 184 -> too_large 62 484 480
  [/Fl /A85 /Fl]  z x 32 Mi  33 488 B  samples_invalid 3 519 180 800 -> too_large 43 438 080

The picture was CARRIED in three of the four: not a bound that fired late, no
bound at all. Doubling the `z` run trebles the old cost and leaves the new one
where it was.

WHY THIS FORM. `assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE
link may cost -- a separate number from `MAX_IMAGE_BYTES`, because that one
bounds the picture and this one bounds producing it. `FlateDecode` is measured
as it is paid; every other permitted filter carries a MEASURED cost ratio
(`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its decoder
is called, since those decoders take a whole string and return a whole string.
A filter with no ratio is refused unread. The budget TRAVELS: a deflate link
is inflated under the smaller of the picture's bound and what the next link's
decoder may be handed, or `[/Fl /A85]` pays 256 MiB for a refusal.

A chunked ASCII85 decoder written here was the alternative and was FELLED: it
would bound `_check_stream_cost` and not the run, because `stream.get_data()`
decodes the whole chain again with pdfminer's own decoder, and it would make
this package rather than pdfminer the authority on an image's bytes. The cap
is the only number that bounds that. `resource.setrlimit(RLIMIT_AS)` was
MEASURED before anything was built on it, as the order required, and is not
usable: Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit`
and does not enforce it. No child-process cap exists.

THE CAP IS READ OFF THE CORPORA, the posture `MAX_IMAGE_PIXELS` has: over the
9 668 image objects of the 77 PDFs on this machine, 16 decode through an
ASCII85 link and the largest input to one is 450 739 bytes, against a cap of
about 5.0 MB.

A PROPERTY TEST REPLACES THE LIST OF KNOWN SHAPES: every chain of length 1-3
over the ten filters pdfminer decodes, 1 110 of 1 110, both payload fills,
each delivered under the bound or refused with a published code and never paid
for on the way (`tracemalloc`, which counts allocations and is not disturbed
by load). Known-positive beside it: 258 of 258 chains over the permitted
filters still carry a small image.

MAJOR -- the backstop had no test. `check_payload` at the end of
`_check_stream_cost` could be deleted with the whole suite green, because the
second one after `get_data()` gives the same code one step later. The two
differ in whether the payment was made, so the test asserts `get_data` was
never called.

10 OF 10 MUTANTS KILLED, control green, each killer named in the report. Four
survived a first pass and two tests exist because of it.

NOT ONE PICTURE CHANGES HANDS, MEASURED BY NAME: `_pdf_images` over every PDF
on this machine from both pinned trees -- 9 306 -> 9 306 carried over 77
files, 50 -> 50 on R761, 0 of 78 files moving a count and 0 moving a code.
R761 also settles a question raised while this order was open: 50 objects, 29
[/DCTDecode], 21 [/FlateDecode], 0 ASCII85 links -- so round 2's count of 580
`[/FlateDecode /ASCII85Decode]` objects is reproducible from nothing on this
machine. It changes no decision; a bomb shape does not need a corpus.

Version stays 0.10.1, no tag. README, CHANGELOG, CLAUDE.md and errors.py
corrected TO what the code does; the round-2 report carries a correction block
rather than a rewrite. Report:
docs/2026-09-18-utgangsbudsjett-per-ledd.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 19:00:38 +02:00
commit 3b3b8ae0ca
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
9 changed files with 784 additions and 126 deletions
Download patch file Download diff file Expand all files Collapse all files

118
src/llm_ingestion_okf/assets.py
View file

@ -198,6 +198,124 @@ def check_payload(size: int, *, name: str) -> None:
)
#: What decoding ONE link of a PDF filter chain may cost this package, in bytes
#: of memory. A SEPARATE number from `MAX_IMAGE_BYTES`, and the distinction is
#: the whole of round 3: that one bounds the picture this package will carry,
#: this one bounds what producing it costs on the way. Three rounds of this
#: review each bound an output and the bomb moved one link along, because a
#: decoder's working set is not its output. Twice `MAX_IMAGE_BYTES`, so a run
#: may hold the stream it was given and one stage of decoding at once and no
#: more.
MAX_FILTER_DECODE_BYTES = 512 * 1024 * 1024
#: MEASURED peak memory per byte of INPUT, for each filter this package lets an
#: image be reached through. `None` means the decoder is driven a chunk at a
#: time here, so the cost is measured as it is paid and no ratio is needed --
#: today that is `FlateDecode` alone (`_inflate`).
#:
#: The numbers are read off CPython 3.14 on 2026-09-18, worst case per filter:
#:
#: * `ASCII85Decode` 101.4x at 1 MiB of input, 96.1x at 4 MiB, 94.5x at 16 MiB.
#: `z` is the shorthand for four zero bytes, so `base64.a85decode` appends one
#: 4-byte object per INPUT byte to a list -- the output ratio is 4, the cost
#: ratio is a hundred, and 0.10.1 documented this filter as "bounded by its
#: own input because it shrinks". The constant sits above the worst of the
#: three, and `test_the_ascii85_cost_ratio_is_not_below_the_one_this_package
#: _measured` re-measures it so it cannot rot when CPython changes.
#: * `ASCIIHexDecode` 1.5x at 16 MiB: it strips whitespace into a copy and
#: `unhexlify`s that, and its output is half its input.
#: * `DCTDecode`, `JPXDecode` and `JBIG2Decode` are pass-through in pdfminer --
#: the bytes are handed to the image reader unchanged -- so the ratio is 1.
#:
#: A filter that is not in this table has no measured ratio and is refused
#: unread (`asset_pdf_unbounded`). That is the same decision `corpus.resolve
#: _gate` takes for an unknown gate name: a fallback reproduces the defect with
#: an extra step.
PDF_FILTER_COST_RATIO: dict[str, float | None] = {
"FlateDecode": None,
"ASCII85Decode": 104.0,
"ASCIIHexDecode": 2.0,
"DCTDecode": 1.0,
"JPXDecode": 1.0,
"JBIG2Decode": 1.0,
}
#: The largest OUTPUT each of those filters can produce per byte of input, used
#: to carry a bound forward when the bytes themselves have been discarded.
#: `ASCII85Decode` is 4 (one `z`), `ASCIIHexDecode` 0.5 (two digits to a byte),
#: pass-through 1. `FlateDecode` has no such number, which is why it is the one
#: filter measured a chunk at a time.
PDF_FILTER_OUTPUT_RATIO: dict[str, float | None] = {
"FlateDecode": None,
"ASCII85Decode": 4.0,
"ASCIIHexDecode": 0.5,
"DCTDecode": 1.0,
"JPXDecode": 1.0,
"JBIG2Decode": 1.0,
}
def filter_input_limit(canonical: str) -> int | None:
"""The largest input this package will hand to `canonical`'s decoder.
`None` for a filter decoded a chunk at a time, which needs no input limit
because its cost is measured while it is paid.
The number this produces for `ASCII85Decode` -- about 5.0 MB -- is READ OFF
the corpora the way `MAX_IMAGE_PIXELS` is: over the 9 668 image objects of
the 77 PDFs on this machine (2026-09-18), 16 decode through an
`ASCII85Decode` link and the largest input to one is 450 739 bytes, so the
limit stands more than ten times above anything measured.
"""
ratio = PDF_FILTER_COST_RATIO.get(canonical)
if ratio is None:
return None
return int(MAX_FILTER_DECODE_BYTES // ratio)
def check_filter_cost(size: int, *, canonical: str, name: str) -> None:
"""Refuse a link whose decoder would cost more than the budget, BEFORE it
decodes anything.
This is the half `inflated_size` cannot cover. That one drives zlib a chunk
at a time and stops the moment the running total crosses the bound, which
is only possible because zlib hands its output over incrementally. Nothing
else in a PDF filter chain does: `base64.a85decode` is asked for a whole
string and gives back a whole string, so by the time its output could be
measured the memory has been spent. For those the cost is PREDICTED from a
measured ratio and the input size, and predicted before the call.
"""
limit = filter_input_limit(canonical)
if limit is None or size <= limit:
return
ratio = PDF_FILTER_COST_RATIO[canonical]
raise ExtractionError(
f"the image {name!r} hands {size} bytes to {canonical}, whose decoder costs "
f"about {ratio} bytes of memory per byte of input -- over this package's "
f"budget of {MAX_FILTER_DECODE_BYTES} bytes for one link; refused before "
"the decode, because a bound on what a link OUTPUTS is not a bound on "
"what producing it costs",
code="asset_too_large",
)
def inflate_limit_for(canonical: str | None) -> int:
"""How much a `FlateDecode` link may produce, given what comes AFTER it.
The picture's own bound is `MAX_IMAGE_BYTES`, but a link's output is the
next link's input, and a decoder with a cost ratio cannot be handed more
than `filter_input_limit` allows. Carrying the budget down the chain this
way is what stops `[/FlateDecode /ASCII85Decode]` from inflating 256 MiB of
`z` before the link behind it is asked anything.
"""
limit = MAX_IMAGE_BYTES
if canonical is not None:
behind = filter_input_limit(canonical)
if behind is not None:
limit = min(limit, behind)
return limit
def inflated_size(raw: bytes, *, name: str, limit: int | None = None) -> int:
"""What a deflate stream costs to decompress, measured without paying it.