fix(assets): budget every link by what its decoder COSTS (0.10.1)

Round 3 of the 0.10.1 review, and the finding is the pattern the three rounds
share: each bound an OUTPUT, and the bomb stepped one link along. The
declared size, then the first `FlateDecode`, then every `FlateDecode` -- and
then a link this package had documented as safe.

`ASCII85Decode` was classed as bounded "by its own input because it shrinks".
It quadruples: `z` is the shorthand for four zero bytes. And the output was
never the cost -- `base64.a85decode` appends one 4-byte object per group to a
list, about a hundred bytes of memory per byte of INPUT (101.4x at 1 MiB,
96.1x at 4 MiB, 94.5x at 16 MiB on CPython 3.14).

Paired subprocesses, idle machine, both sides from PINNED trees, the document
built once by a third process and read from a file because `ru_maxrss` never
falls and `b"z" * 64 MiB` alone costs 171 MB:

  [/Fl /A85]      z x 32 Mi  33 475 B   CARRIED 3 261 599 744 -> too_large 42 070 016
  [/Fl /A85]      z x 64 Mi  66 090 B   CARRIED 6 461 558 784 -> too_large 40 280 064
  [/A85]          z x  8 Mi   8.4 MB    CARRIED   933 085 184 -> too_large 62 484 480
  [/Fl /A85 /Fl]  z x 32 Mi  33 488 B  samples_invalid 3 519 180 800 -> too_large 43 438 080

The picture was CARRIED in three of the four: not a bound that fired late, no
bound at all. Doubling the `z` run trebles the old cost and leaves the new one
where it was.

WHY THIS FORM. `assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE
link may cost -- a separate number from `MAX_IMAGE_BYTES`, because that one
bounds the picture and this one bounds producing it. `FlateDecode` is measured
as it is paid; every other permitted filter carries a MEASURED cost ratio
(`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its decoder
is called, since those decoders take a whole string and return a whole string.
A filter with no ratio is refused unread. The budget TRAVELS: a deflate link
is inflated under the smaller of the picture's bound and what the next link's
decoder may be handed, or `[/Fl /A85]` pays 256 MiB for a refusal.

A chunked ASCII85 decoder written here was the alternative and was FELLED: it
would bound `_check_stream_cost` and not the run, because `stream.get_data()`
decodes the whole chain again with pdfminer's own decoder, and it would make
this package rather than pdfminer the authority on an image's bytes. The cap
is the only number that bounds that. `resource.setrlimit(RLIMIT_AS)` was
MEASURED before anything was built on it, as the order required, and is not
usable: Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit`
and does not enforce it. No child-process cap exists.

THE CAP IS READ OFF THE CORPORA, the posture `MAX_IMAGE_PIXELS` has: over the
9 668 image objects of the 77 PDFs on this machine, 16 decode through an
ASCII85 link and the largest input to one is 450 739 bytes, against a cap of
about 5.0 MB.

A PROPERTY TEST REPLACES THE LIST OF KNOWN SHAPES: every chain of length 1-3
over the ten filters pdfminer decodes, 1 110 of 1 110, both payload fills,
each delivered under the bound or refused with a published code and never paid
for on the way (`tracemalloc`, which counts allocations and is not disturbed
by load). Known-positive beside it: 258 of 258 chains over the permitted
filters still carry a small image.

MAJOR -- the backstop had no test. `check_payload` at the end of
`_check_stream_cost` could be deleted with the whole suite green, because the
second one after `get_data()` gives the same code one step later. The two
differ in whether the payment was made, so the test asserts `get_data` was
never called.

10 OF 10 MUTANTS KILLED, control green, each killer named in the report. Four
survived a first pass and two tests exist because of it.

NOT ONE PICTURE CHANGES HANDS, MEASURED BY NAME: `_pdf_images` over every PDF
on this machine from both pinned trees -- 9 306 -> 9 306 carried over 77
files, 50 -> 50 on R761, 0 of 78 files moving a count and 0 moving a code.
R761 also settles a question raised while this order was open: 50 objects, 29
[/DCTDecode], 21 [/FlateDecode], 0 ASCII85 links -- so round 2's count of 580
`[/FlateDecode /ASCII85Decode]` objects is reproducible from nothing on this
machine. It changes no decision; a bomb shape does not need a corpus.

Version stays 0.10.1, no tag. README, CHANGELOG, CLAUDE.md and errors.py
corrected TO what the code does; the round-2 report carries a correction block
rather than a rewrite. Report:
docs/2026-09-18-utgangsbudsjett-per-ledd.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 19:00:38 +02:00
commit 3b3b8ae0ca
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
9 changed files with 784 additions and 126 deletions

View file

@ -339,14 +339,33 @@ decodes a stream through a *list* of filters, and `/Filter [/FlateDecode
/FlateDecode]` puts the whole expansion in the second one: measured, 1 636
bytes of file cost 886 554 624 bytes of peak RSS when only the first link was
measured (52 367 360 with every link measured), and the picture was still refused at the end — after the memory had
been spent. `FlateDecode` is measured, `ASCII85Decode` and `ASCIIHexDecode`
are bounded by their own input because they shrink, and `DCTDecode`,
`JPXDecode` and `JBIG2Decode` pass through unchanged. Any other filter —
`LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything written
after this — expands by an amount no chunked measurement can reach, so an
image behind one is refused UNREAD with its own code, `asset_pdf_unbounded`,
rather than decoded to find out what it costs. An encrypted stream is
deciphered first and then measured like any other.
been spent. An encrypted stream is deciphered first and then measured like any
other.
**And what a link COSTS is bounded, not the size of its output.** Bounding
every `FlateDecode` was still not a bound, because `ASCII85Decode` had been
classed as safe "because it shrinks" and it does not: `z` is that encoding's
shorthand for four zero bytes, so the filter quadruples its input, and
`base64.a85decode` holds about a hundred bytes of memory per byte of input.
Measured in paired subprocesses on an idle machine, the document built once
and read from a file so the fixture is not what is measured: a 33 475-byte PDF
decoding through `[/FlateDecode /ASCII85Decode]` cost 3 261 599 744 bytes of
peak RSS and the picture was CARRIED; bounded it is 42 070 016 and
`asset_too_large`. Doubling the run of `z` takes the old cost to
6 461 558 784 and the bounded one to 40 280 064 — the cost no longer follows
the bomb. So `FlateDecode` is measured a chunk at a time as it is paid, and
every other permitted filter carries a MEASURED worst-case cost per byte of
input which is checked against the budget *before* its decoder is called.
Any other filter — `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`,
`/Crypt`, anything written after this — has no measured ratio, so an image
behind one is refused UNREAD with its own code, `asset_pdf_unbounded`, rather
than decoded to find out what it costs. The cap that falls out for
`ASCII85Decode` is read off the corpora the way the pixel bound is: over the
9 668 image objects of the 77 PDFs measured, 16 decode through such a link and
the largest input to one is 450 739 bytes, more than ten times under the cap.
A property test runs **every** chain of length 13 over the ten filters
pdfminer decodes — 1 110 of them — and requires each to be delivered under the
bound or refused with a published code, never paid for on the way.
**What the stream bound does NOT reach**, stated because the difference
matters: a stream something else has already decoded, where the memory was