fix(assets): budget every link by what its decoder COSTS (0.10.1)
Round 3 of the 0.10.1 review, and the finding is the pattern the three rounds share: each bound an OUTPUT, and the bomb stepped one link along. The declared size, then the first `FlateDecode`, then every `FlateDecode` -- and then a link this package had documented as safe. `ASCII85Decode` was classed as bounded "by its own input because it shrinks". It quadruples: `z` is the shorthand for four zero bytes. And the output was never the cost -- `base64.a85decode` appends one 4-byte object per group to a list, about a hundred bytes of memory per byte of INPUT (101.4x at 1 MiB, 96.1x at 4 MiB, 94.5x at 16 MiB on CPython 3.14). Paired subprocesses, idle machine, both sides from PINNED trees, the document built once by a third process and read from a file because `ru_maxrss` never falls and `b"z" * 64 MiB` alone costs 171 MB: [/Fl /A85] z x 32 Mi 33 475 B CARRIED 3 261 599 744 -> too_large 42 070 016 [/Fl /A85] z x 64 Mi 66 090 B CARRIED 6 461 558 784 -> too_large 40 280 064 [/A85] z x 8 Mi 8.4 MB CARRIED 933 085 184 -> too_large 62 484 480 [/Fl /A85 /Fl] z x 32 Mi 33 488 B samples_invalid 3 519 180 800 -> too_large 43 438 080 The picture was CARRIED in three of the four: not a bound that fired late, no bound at all. Doubling the `z` run trebles the old cost and leaves the new one where it was. WHY THIS FORM. `assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE link may cost -- a separate number from `MAX_IMAGE_BYTES`, because that one bounds the picture and this one bounds producing it. `FlateDecode` is measured as it is paid; every other permitted filter carries a MEASURED cost ratio (`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its decoder is called, since those decoders take a whole string and return a whole string. A filter with no ratio is refused unread. The budget TRAVELS: a deflate link is inflated under the smaller of the picture's bound and what the next link's decoder may be handed, or `[/Fl /A85]` pays 256 MiB for a refusal. A chunked ASCII85 decoder written here was the alternative and was FELLED: it would bound `_check_stream_cost` and not the run, because `stream.get_data()` decodes the whole chain again with pdfminer's own decoder, and it would make this package rather than pdfminer the authority on an image's bytes. The cap is the only number that bounds that. `resource.setrlimit(RLIMIT_AS)` was MEASURED before anything was built on it, as the order required, and is not usable: Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit` and does not enforce it. No child-process cap exists. THE CAP IS READ OFF THE CORPORA, the posture `MAX_IMAGE_PIXELS` has: over the 9 668 image objects of the 77 PDFs on this machine, 16 decode through an ASCII85 link and the largest input to one is 450 739 bytes, against a cap of about 5.0 MB. A PROPERTY TEST REPLACES THE LIST OF KNOWN SHAPES: every chain of length 1-3 over the ten filters pdfminer decodes, 1 110 of 1 110, both payload fills, each delivered under the bound or refused with a published code and never paid for on the way (`tracemalloc`, which counts allocations and is not disturbed by load). Known-positive beside it: 258 of 258 chains over the permitted filters still carry a small image. MAJOR -- the backstop had no test. `check_payload` at the end of `_check_stream_cost` could be deleted with the whole suite green, because the second one after `get_data()` gives the same code one step later. The two differ in whether the payment was made, so the test asserts `get_data` was never called. 10 OF 10 MUTANTS KILLED, control green, each killer named in the report. Four survived a first pass and two tests exist because of it. NOT ONE PICTURE CHANGES HANDS, MEASURED BY NAME: `_pdf_images` over every PDF on this machine from both pinned trees -- 9 306 -> 9 306 carried over 77 files, 50 -> 50 on R761, 0 of 78 files moving a count and 0 moving a code. R761 also settles a question raised while this order was open: 50 objects, 29 [/DCTDecode], 21 [/FlateDecode], 0 ASCII85 links -- so round 2's count of 580 `[/FlateDecode /ASCII85Decode]` objects is reproducible from nothing on this machine. It changes no decision; a bomb shape does not need a corpus. Version stays 0.10.1, no tag. README, CHANGELOG, CLAUDE.md and errors.py corrected TO what the code does; the round-2 report carries a correction block rather than a rewrite. Report: docs/2026-09-18-utgangsbudsjett-per-ledd.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
33d3269380
commit
3b3b8ae0ca
9 changed files with 784 additions and 126 deletions
|
|
@ -133,13 +133,15 @@ class ExtractionError(IngestError):
|
|||
- `asset_samples_invalid` — the sample buffer does not fit the dimensions
|
||||
the image dictionary declares. Refused rather than padded: a short buffer
|
||||
means the dictionary was read wrong
|
||||
- `asset_too_large` — the picture is over this package's bound, either
|
||||
because it DECLARES a size beyond `MAX_IMAGE_PIXELS`, because the file
|
||||
itself is that large, or because the stream behind it DECOMPRESSES to
|
||||
more than `MAX_IMAGE_BYTES`. The three are one code because they are one
|
||||
decision — this run will not hold that picture — and because a consumer
|
||||
counting refusals wants the picture, not the mechanism. The bound is read
|
||||
off the corpora and sits an order of magnitude above anything measured
|
||||
- `asset_too_large` — the picture is over this package's bound: because it
|
||||
DECLARES a size beyond `MAX_IMAGE_PIXELS`, because the file itself is
|
||||
that large, because the stream behind it DECOMPRESSES to more than
|
||||
`MAX_IMAGE_BYTES`, or because one link of its filter chain would COST
|
||||
more than `MAX_FILTER_DECODE_BYTES` to decode. The four are one code
|
||||
because they are one decision — this run will not hold that picture —
|
||||
and because a consumer counting refusals wants the picture, not the
|
||||
mechanism. Each bound is read off the corpora and sits an order of
|
||||
magnitude above anything measured
|
||||
- `asset_size_invalid` — the container declares a size that is not a size:
|
||||
a zero or negative `/Width` or `/Height`. DISTINCT from
|
||||
`asset_too_large`, because the two say different things about the
|
||||
|
|
@ -150,14 +152,16 @@ class ExtractionError(IngestError):
|
|||
negative dimension multiplies to a negative pixel count, under which
|
||||
every bound reads as satisfied
|
||||
- `asset_pdf_unbounded` — the image is reached through a PDF stream filter
|
||||
whose output this package cannot measure before producing it
|
||||
(`LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything
|
||||
unknown), or through an encrypted stream it cannot decipher. DISTINCT
|
||||
from `asset_too_large`, which says a measurement was taken and came out
|
||||
over the bound: this one says no measurement was possible, so the picture
|
||||
is refused UNREAD rather than decoded to find out what it costs. Measured
|
||||
2026-09-18: bounding only the first link of a filter chain let 1 636
|
||||
bytes of PDF cost 886 554 624 bytes of peak RSS
|
||||
this package has no measured cost ratio for (`LZWDecode`,
|
||||
`RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything unknown), or
|
||||
through an encrypted stream it cannot decipher. DISTINCT from
|
||||
`asset_too_large`, which says a measurement was taken or predicted and
|
||||
came out over the bound: this one says neither was possible, so the
|
||||
picture is refused UNREAD rather than decoded to find out what it costs.
|
||||
Measured 2026-09-18: bounding only the first link of a filter chain let
|
||||
1 636 bytes of PDF cost 886 554 624 bytes of peak RSS, and bounding
|
||||
every link's OUTPUT still let 33 475 bytes cost 3 261 599 744 through a
|
||||
filter whose decoder holds a hundred bytes per byte of input
|
||||
"""
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue