fix(assets): budget every link by what its decoder COSTS (0.10.1)
Round 3 of the 0.10.1 review, and the finding is the pattern the three rounds share: each bound an OUTPUT, and the bomb stepped one link along. The declared size, then the first `FlateDecode`, then every `FlateDecode` -- and then a link this package had documented as safe. `ASCII85Decode` was classed as bounded "by its own input because it shrinks". It quadruples: `z` is the shorthand for four zero bytes. And the output was never the cost -- `base64.a85decode` appends one 4-byte object per group to a list, about a hundred bytes of memory per byte of INPUT (101.4x at 1 MiB, 96.1x at 4 MiB, 94.5x at 16 MiB on CPython 3.14). Paired subprocesses, idle machine, both sides from PINNED trees, the document built once by a third process and read from a file because `ru_maxrss` never falls and `b"z" * 64 MiB` alone costs 171 MB: [/Fl /A85] z x 32 Mi 33 475 B CARRIED 3 261 599 744 -> too_large 42 070 016 [/Fl /A85] z x 64 Mi 66 090 B CARRIED 6 461 558 784 -> too_large 40 280 064 [/A85] z x 8 Mi 8.4 MB CARRIED 933 085 184 -> too_large 62 484 480 [/Fl /A85 /Fl] z x 32 Mi 33 488 B samples_invalid 3 519 180 800 -> too_large 43 438 080 The picture was CARRIED in three of the four: not a bound that fired late, no bound at all. Doubling the `z` run trebles the old cost and leaves the new one where it was. WHY THIS FORM. `assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE link may cost -- a separate number from `MAX_IMAGE_BYTES`, because that one bounds the picture and this one bounds producing it. `FlateDecode` is measured as it is paid; every other permitted filter carries a MEASURED cost ratio (`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its decoder is called, since those decoders take a whole string and return a whole string. A filter with no ratio is refused unread. The budget TRAVELS: a deflate link is inflated under the smaller of the picture's bound and what the next link's decoder may be handed, or `[/Fl /A85]` pays 256 MiB for a refusal. A chunked ASCII85 decoder written here was the alternative and was FELLED: it would bound `_check_stream_cost` and not the run, because `stream.get_data()` decodes the whole chain again with pdfminer's own decoder, and it would make this package rather than pdfminer the authority on an image's bytes. The cap is the only number that bounds that. `resource.setrlimit(RLIMIT_AS)` was MEASURED before anything was built on it, as the order required, and is not usable: Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit` and does not enforce it. No child-process cap exists. THE CAP IS READ OFF THE CORPORA, the posture `MAX_IMAGE_PIXELS` has: over the 9 668 image objects of the 77 PDFs on this machine, 16 decode through an ASCII85 link and the largest input to one is 450 739 bytes, against a cap of about 5.0 MB. A PROPERTY TEST REPLACES THE LIST OF KNOWN SHAPES: every chain of length 1-3 over the ten filters pdfminer decodes, 1 110 of 1 110, both payload fills, each delivered under the bound or refused with a published code and never paid for on the way (`tracemalloc`, which counts allocations and is not disturbed by load). Known-positive beside it: 258 of 258 chains over the permitted filters still carry a small image. MAJOR -- the backstop had no test. `check_payload` at the end of `_check_stream_cost` could be deleted with the whole suite green, because the second one after `get_data()` gives the same code one step later. The two differ in whether the payment was made, so the test asserts `get_data` was never called. 10 OF 10 MUTANTS KILLED, control green, each killer named in the report. Four survived a first pass and two tests exist because of it. NOT ONE PICTURE CHANGES HANDS, MEASURED BY NAME: `_pdf_images` over every PDF on this machine from both pinned trees -- 9 306 -> 9 306 carried over 77 files, 50 -> 50 on R761, 0 of 78 files moving a count and 0 moving a code. R761 also settles a question raised while this order was open: 50 objects, 29 [/DCTDecode], 21 [/FlateDecode], 0 ASCII85 links -- so round 2's count of 580 `[/FlateDecode /ASCII85Decode]` objects is reproducible from nothing on this machine. It changes no decision; a bomb shape does not need a corpus. Version stays 0.10.1, no tag. README, CHANGELOG, CLAUDE.md and errors.py corrected TO what the code does; the round-2 report carries a correction block rather than a rewrite. Report: docs/2026-09-18-utgangsbudsjett-per-ledd.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
33d3269380
commit
3b3b8ae0ca
9 changed files with 784 additions and 126 deletions
42
CHANGELOG.md
42
CHANGELOG.md
|
|
@ -128,12 +128,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
because `filters[0]` is not `FlateDecode` there. Bounded, measured idle in paired subprocesses: 52 367 360 bytes at two
|
||||
links, 61 390 848 at three, and 60 403 712 where the old path cost
|
||||
2 567 204 864.
|
||||
- **A filter whose output cannot be measured before it is produced is
|
||||
refused unread**, with its own code `asset_pdf_unbounded`. `FlateDecode`
|
||||
is measured; `ASCII85Decode` and `ASCIIHexDecode` are bounded by their own
|
||||
input because they shrink; `DCTDecode`, `JPXDecode` and `JBIG2Decode` pass
|
||||
through unchanged. `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`,
|
||||
`/Crypt` and anything unknown are refused. Over the 5 142 image objects of
|
||||
- **What a link COSTS is bounded, not the size of its output.** Bounding
|
||||
every `FlateDecode` was still not a bound: the bomb moved into
|
||||
`ASCII85Decode`, which the previous fix had classed as safe "because it
|
||||
shrinks". `z` is that encoding's shorthand for four zero bytes, so the
|
||||
filter quadruples its input, and `base64.a85decode` appends one 4-byte
|
||||
object per group to a list — about a hundred bytes of memory per byte of
|
||||
INPUT (measured on CPython 3.14: 101.4x at 1 MiB, 96.1x at 4 MiB, 94.5x at
|
||||
16 MiB). Measured in paired subprocesses, idle machine, the document built
|
||||
once and read from a file: a 33 475-byte PDF decoding through
|
||||
`[/FlateDecode /ASCII85Decode]` cost 3 261 599 744 bytes of peak RSS and
|
||||
the picture was CARRIED; bounded, 42 070 016 and `asset_too_large`.
|
||||
Doubling the run of `z` takes the old cost to 6 461 558 784 and the
|
||||
bounded one to 40 280 064, so the cost no longer follows the bomb. A
|
||||
single `[/ASCII85Decode]` link went 933 085 184 → 62 484 480, and
|
||||
`[/Fl /A85 /Fl]` 3 519 180 800 → 43 438 080 (and from
|
||||
`asset_samples_invalid` to a bound's own code).
|
||||
- **Every permitted filter now carries a measured cost ratio**
|
||||
(`assets.PDF_FILTER_COST_RATIO`) and a per-link budget
|
||||
(`MAX_FILTER_DECODE_BYTES`, 512 MiB). `FlateDecode` is measured a chunk at
|
||||
a time as it is paid, under a limit that is the smaller of the picture's
|
||||
own bound and what the NEXT link's decoder may be handed, so the budget
|
||||
travels down the chain. Every other permitted filter has its cost
|
||||
PREDICTED from its input size before its decoder is called, because those
|
||||
decoders take a whole string and return a whole string. The cap that falls
|
||||
out for `ASCII85Decode` is read off the corpora: of the 9 668 image
|
||||
objects of the 77 PDFs measured, 16 decode through such a link and the
|
||||
largest input to one is 450 739 bytes, more than ten times under it.
|
||||
- **A filter with no measured ratio is refused unread**, with its own code
|
||||
`asset_pdf_unbounded`: `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`,
|
||||
`/Crypt` and anything unknown.
|
||||
- **A property test runs every chain of length 1–3** over the ten filters
|
||||
pdfminer decodes — 1 110 of them, each with an amplifying payload —
|
||||
and requires each to be delivered under the bound or refused with a code
|
||||
in the published vocabulary, never paid for on the way. The known-positive
|
||||
beside it holds that every chain over the permitted filters still carries
|
||||
a small image. Over the 5 142 image objects of
|
||||
the 78 PDFs measured, the refused class is 4 `CCITTFaxDecode` objects,
|
||||
which are 1-bit stencil masks and were already refused one step later by
|
||||
the encoder. Measured by name over the same 78 documents, carried images
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue