fix(assets): budget every link by what its decoder COSTS (0.10.1)
Round 3 of the 0.10.1 review, and the finding is the pattern the three rounds share: each bound an OUTPUT, and the bomb stepped one link along. The declared size, then the first `FlateDecode`, then every `FlateDecode` -- and then a link this package had documented as safe. `ASCII85Decode` was classed as bounded "by its own input because it shrinks". It quadruples: `z` is the shorthand for four zero bytes. And the output was never the cost -- `base64.a85decode` appends one 4-byte object per group to a list, about a hundred bytes of memory per byte of INPUT (101.4x at 1 MiB, 96.1x at 4 MiB, 94.5x at 16 MiB on CPython 3.14). Paired subprocesses, idle machine, both sides from PINNED trees, the document built once by a third process and read from a file because `ru_maxrss` never falls and `b"z" * 64 MiB` alone costs 171 MB: [/Fl /A85] z x 32 Mi 33 475 B CARRIED 3 261 599 744 -> too_large 42 070 016 [/Fl /A85] z x 64 Mi 66 090 B CARRIED 6 461 558 784 -> too_large 40 280 064 [/A85] z x 8 Mi 8.4 MB CARRIED 933 085 184 -> too_large 62 484 480 [/Fl /A85 /Fl] z x 32 Mi 33 488 B samples_invalid 3 519 180 800 -> too_large 43 438 080 The picture was CARRIED in three of the four: not a bound that fired late, no bound at all. Doubling the `z` run trebles the old cost and leaves the new one where it was. WHY THIS FORM. `assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE link may cost -- a separate number from `MAX_IMAGE_BYTES`, because that one bounds the picture and this one bounds producing it. `FlateDecode` is measured as it is paid; every other permitted filter carries a MEASURED cost ratio (`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its decoder is called, since those decoders take a whole string and return a whole string. A filter with no ratio is refused unread. The budget TRAVELS: a deflate link is inflated under the smaller of the picture's bound and what the next link's decoder may be handed, or `[/Fl /A85]` pays 256 MiB for a refusal. A chunked ASCII85 decoder written here was the alternative and was FELLED: it would bound `_check_stream_cost` and not the run, because `stream.get_data()` decodes the whole chain again with pdfminer's own decoder, and it would make this package rather than pdfminer the authority on an image's bytes. The cap is the only number that bounds that. `resource.setrlimit(RLIMIT_AS)` was MEASURED before anything was built on it, as the order required, and is not usable: Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit` and does not enforce it. No child-process cap exists. THE CAP IS READ OFF THE CORPORA, the posture `MAX_IMAGE_PIXELS` has: over the 9 668 image objects of the 77 PDFs on this machine, 16 decode through an ASCII85 link and the largest input to one is 450 739 bytes, against a cap of about 5.0 MB. A PROPERTY TEST REPLACES THE LIST OF KNOWN SHAPES: every chain of length 1-3 over the ten filters pdfminer decodes, 1 110 of 1 110, both payload fills, each delivered under the bound or refused with a published code and never paid for on the way (`tracemalloc`, which counts allocations and is not disturbed by load). Known-positive beside it: 258 of 258 chains over the permitted filters still carry a small image. MAJOR -- the backstop had no test. `check_payload` at the end of `_check_stream_cost` could be deleted with the whole suite green, because the second one after `get_data()` gives the same code one step later. The two differ in whether the payment was made, so the test asserts `get_data` was never called. 10 OF 10 MUTANTS KILLED, control green, each killer named in the report. Four survived a first pass and two tests exist because of it. NOT ONE PICTURE CHANGES HANDS, MEASURED BY NAME: `_pdf_images` over every PDF on this machine from both pinned trees -- 9 306 -> 9 306 carried over 77 files, 50 -> 50 on R761, 0 of 78 files moving a count and 0 moving a code. R761 also settles a question raised while this order was open: 50 objects, 29 [/DCTDecode], 21 [/FlateDecode], 0 ASCII85 links -- so round 2's count of 580 `[/FlateDecode /ASCII85Decode]` objects is reproducible from nothing on this machine. It changes no decision; a bomb shape does not need a corpus. Version stays 0.10.1, no tag. README, CHANGELOG, CLAUDE.md and errors.py corrected TO what the code does; the round-2 report carries a correction block rather than a rewrite. Report: docs/2026-09-18-utgangsbudsjett-per-ledd.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
33d3269380
commit
3b3b8ae0ca
9 changed files with 784 additions and 126 deletions
35
README.md
35
README.md
|
|
@ -339,14 +339,33 @@ decodes a stream through a *list* of filters, and `/Filter [/FlateDecode
|
|||
/FlateDecode]` puts the whole expansion in the second one: measured, 1 636
|
||||
bytes of file cost 886 554 624 bytes of peak RSS when only the first link was
|
||||
measured (52 367 360 with every link measured), and the picture was still refused at the end — after the memory had
|
||||
been spent. `FlateDecode` is measured, `ASCII85Decode` and `ASCIIHexDecode`
|
||||
are bounded by their own input because they shrink, and `DCTDecode`,
|
||||
`JPXDecode` and `JBIG2Decode` pass through unchanged. Any other filter —
|
||||
`LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything written
|
||||
after this — expands by an amount no chunked measurement can reach, so an
|
||||
image behind one is refused UNREAD with its own code, `asset_pdf_unbounded`,
|
||||
rather than decoded to find out what it costs. An encrypted stream is
|
||||
deciphered first and then measured like any other.
|
||||
been spent. An encrypted stream is deciphered first and then measured like any
|
||||
other.
|
||||
|
||||
**And what a link COSTS is bounded, not the size of its output.** Bounding
|
||||
every `FlateDecode` was still not a bound, because `ASCII85Decode` had been
|
||||
classed as safe "because it shrinks" and it does not: `z` is that encoding's
|
||||
shorthand for four zero bytes, so the filter quadruples its input, and
|
||||
`base64.a85decode` holds about a hundred bytes of memory per byte of input.
|
||||
Measured in paired subprocesses on an idle machine, the document built once
|
||||
and read from a file so the fixture is not what is measured: a 33 475-byte PDF
|
||||
decoding through `[/FlateDecode /ASCII85Decode]` cost 3 261 599 744 bytes of
|
||||
peak RSS and the picture was CARRIED; bounded it is 42 070 016 and
|
||||
`asset_too_large`. Doubling the run of `z` takes the old cost to
|
||||
6 461 558 784 and the bounded one to 40 280 064 — the cost no longer follows
|
||||
the bomb. So `FlateDecode` is measured a chunk at a time as it is paid, and
|
||||
every other permitted filter carries a MEASURED worst-case cost per byte of
|
||||
input which is checked against the budget *before* its decoder is called.
|
||||
Any other filter — `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`,
|
||||
`/Crypt`, anything written after this — has no measured ratio, so an image
|
||||
behind one is refused UNREAD with its own code, `asset_pdf_unbounded`, rather
|
||||
than decoded to find out what it costs. The cap that falls out for
|
||||
`ASCII85Decode` is read off the corpora the way the pixel bound is: over the
|
||||
9 668 image objects of the 77 PDFs measured, 16 decode through such a link and
|
||||
the largest input to one is 450 739 bytes, more than ten times under the cap.
|
||||
A property test runs **every** chain of length 1–3 over the ten filters
|
||||
pdfminer decodes — 1 110 of them — and requires each to be delivered under the
|
||||
bound or refused with a published code, never paid for on the way.
|
||||
|
||||
**What the stream bound does NOT reach**, stated because the difference
|
||||
matters: a stream something else has already decoded, where the memory was
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue