open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(assets): budget every link by what its decoder COSTS (0.10.1)

Browse source 
Round 3 of the 0.10.1 review, and the finding is the pattern the three rounds
share: each bound an OUTPUT, and the bomb stepped one link along. The
declared size, then the first `FlateDecode`, then every `FlateDecode` -- and
then a link this package had documented as safe.

`ASCII85Decode` was classed as bounded "by its own input because it shrinks".
It quadruples: `z` is the shorthand for four zero bytes. And the output was
never the cost -- `base64.a85decode` appends one 4-byte object per group to a
list, about a hundred bytes of memory per byte of INPUT (101.4x at 1 MiB,
96.1x at 4 MiB, 94.5x at 16 MiB on CPython 3.14).

Paired subprocesses, idle machine, both sides from PINNED trees, the document
built once by a third process and read from a file because `ru_maxrss` never
falls and `b"z" * 64 MiB` alone costs 171 MB:

  [/Fl /A85]      z x 32 Mi  33 475 B   CARRIED 3 261 599 744 -> too_large 42 070 016
  [/Fl /A85]      z x 64 Mi  66 090 B   CARRIED 6 461 558 784 -> too_large 40 280 064
  [/A85]          z x  8 Mi   8.4 MB    CARRIED   933 085 184 -> too_large 62 484 480
  [/Fl /A85 /Fl]  z x 32 Mi  33 488 B  samples_invalid 3 519 180 800 -> too_large 43 438 080

The picture was CARRIED in three of the four: not a bound that fired late, no
bound at all. Doubling the `z` run trebles the old cost and leaves the new one
where it was.

WHY THIS FORM. `assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE
link may cost -- a separate number from `MAX_IMAGE_BYTES`, because that one
bounds the picture and this one bounds producing it. `FlateDecode` is measured
as it is paid; every other permitted filter carries a MEASURED cost ratio
(`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its decoder
is called, since those decoders take a whole string and return a whole string.
A filter with no ratio is refused unread. The budget TRAVELS: a deflate link
is inflated under the smaller of the picture's bound and what the next link's
decoder may be handed, or `[/Fl /A85]` pays 256 MiB for a refusal.

A chunked ASCII85 decoder written here was the alternative and was FELLED: it
would bound `_check_stream_cost` and not the run, because `stream.get_data()`
decodes the whole chain again with pdfminer's own decoder, and it would make
this package rather than pdfminer the authority on an image's bytes. The cap
is the only number that bounds that. `resource.setrlimit(RLIMIT_AS)` was
MEASURED before anything was built on it, as the order required, and is not
usable: Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit`
and does not enforce it. No child-process cap exists.

THE CAP IS READ OFF THE CORPORA, the posture `MAX_IMAGE_PIXELS` has: over the
9 668 image objects of the 77 PDFs on this machine, 16 decode through an
ASCII85 link and the largest input to one is 450 739 bytes, against a cap of
about 5.0 MB.

A PROPERTY TEST REPLACES THE LIST OF KNOWN SHAPES: every chain of length 1-3
over the ten filters pdfminer decodes, 1 110 of 1 110, both payload fills,
each delivered under the bound or refused with a published code and never paid
for on the way (`tracemalloc`, which counts allocations and is not disturbed
by load). Known-positive beside it: 258 of 258 chains over the permitted
filters still carry a small image.

MAJOR -- the backstop had no test. `check_payload` at the end of
`_check_stream_cost` could be deleted with the whole suite green, because the
second one after `get_data()` gives the same code one step later. The two
differ in whether the payment was made, so the test asserts `get_data` was
never called.

10 OF 10 MUTANTS KILLED, control green, each killer named in the report. Four
survived a first pass and two tests exist because of it.

NOT ONE PICTURE CHANGES HANDS, MEASURED BY NAME: `_pdf_images` over every PDF
on this machine from both pinned trees -- 9 306 -> 9 306 carried over 77
files, 50 -> 50 on R761, 0 of 78 files moving a count and 0 moving a code.
R761 also settles a question raised while this order was open: 50 objects, 29
[/DCTDecode], 21 [/FlateDecode], 0 ASCII85 links -- so round 2's count of 580
`[/FlateDecode /ASCII85Decode]` objects is reproducible from nothing on this
machine. It changes no decision; a bomb shape does not need a corpus.

Version stays 0.10.1, no tag. README, CHANGELOG, CLAUDE.md and errors.py
corrected TO what the code does; the round-2 report carries a correction block
rather than a rewrite. Report:
docs/2026-09-18-utgangsbudsjett-per-ledd.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 19:00:38 +02:00
commit 3b3b8ae0ca
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
9 changed files with 784 additions and 126 deletions
Download patch file Download diff file Expand all files Collapse all files

48
CLAUDE.md
View file

@ -935,14 +935,13 @@ and fixtures, never code.
not `FlateDecode` there. Bounded (idle machine, paired subprocesses):
**52 367 360 B** at two links, **61 390 848 B** at three, **60 403 712 B**
where the old path cost 2 567 204 864 B, and the single-link control
unmoved at 59 232 256 -> 62 017 536. **Three classes and no fourth**
(`extract.bounded_pdf_filters`, pinned by a test): `FlateDecode` MEASURED,
`ASCII85Decode`/`ASCIIHexDecode` bounded by their own input because they
SHRINK, `DCTDecode`/`JPXDecode`/`JBIG2Decode` PASS THROUGH. Everything
unmoved at 59 232 256 -> 62 017 536. Everything
else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`,
anything written later -- is refused UNREAD with its own code
`asset_pdf_unbounded`, the same decision `corpus.resolve_gate` takes for
an unknown gate name. Cost measured over the **5 142** image objects of
an unknown gate name. **The three-class split this round shipped with --
`ASCII85Decode`/`ASCIIHexDecode` "bounded by their own input because they
SHRINK" -- was FALSE and round 3 below replaced it.** Cost measured over the **5 142** image objects of
78 PDFs: the refused class is **4** `CCITTFaxDecode` objects, all 1-bit
stencil masks already refused one step later by the encoder, and **0**
objects in an encrypted document. **NOT ONE PICTURE CHANGES HANDS AND IT
@ -953,6 +952,45 @@ and fixtures, never code.
`trinn2` hold the same document). An ENCRYPTED stream is
now deciphered and then measured (deciphering does not change a length),
where `stream.decipher is not None` used to return unmeasured.
- **THE COST OF A LINK, NOT THE SIZE OF ITS OUTPUT (round 3).** Three
rounds each bound an OUTPUT and the bomb stepped one link along: the
declared size, then the first `FlateDecode`, then every `FlateDecode`.
Round 2's third class was FALSE -- `ASCII85Decode`'s `z` is the shorthand
for four zero bytes, so it QUADRUPLES its input, and `base64.a85decode`
appends one 4-byte object per group to a list, costing about **a hundred
bytes of memory per byte of INPUT** (measured on CPython 3.14: **101.4x**
at 1 MiB, **96.1x** at 4 MiB, **94.5x** at 16 MiB). Paired subprocesses,
idle machine, both sides from pinned trees, the document built once and
read from a FILE because `ru_maxrss` never falls and `b"z" * 64 MiB`
alone costs 171 MB: `[/Fl /A85]` **33 475 B of file -> 3 261 599 744 B
peak and the picture CARRIED**, now **42 070 016 B** and
`asset_too_large`; at twice the `z` run **6 461 558 784 -> 40 280 064**,
so the cost no longer follows the bomb; `[/A85]` alone **933 085 184 ->
62 484 480**; `[/Fl /A85 /Fl]` **3 519 180 800 -> 43 438 080** and from
`asset_samples_invalid` to a bound's own code. **The rule is a BUDGET per
link** (`assets.MAX_FILTER_DECODE_BYTES`, 512 MiB, a separate number from
`MAX_IMAGE_BYTES`): `FlateDecode` is measured as it is paid, every other
permitted filter has a MEASURED cost ratio
(`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its
decoder is called, and a filter with no ratio is refused unread. **The
budget TRAVELS**: a deflate link is inflated under the smaller of the
picture's bound and what the next link's decoder may be handed, or
`[/Fl /A85]` pays 256 MiB for a refusal. The ASCII85 cap (**~5.0 MB**) is
READ OFF the corpora: of **9 668** image objects over **77** PDFs, **16**
decode through such a link and the largest input is **450 739 B**, ten
times under it. **A PROPERTY TEST replaces the list of known shapes**:
every chain of length 1-3 over the ten filters pdfminer decodes, **1 110
of 1 110**, both payload fills, each delivered under the bound or refused
with a published code and never paid for on the way (`tracemalloc`, which
counts allocations and is not disturbed by load). Known-positive: **258 of
258** chains over the permitted filters still carry a small image.
**10 of 10 mutants killed**, control green. **NOT A ROW'S DIFFERENCE ON
REAL DOCUMENTS**: paired `_pdf_images` over every PDF on this machine from two PINNED trees: **9 306 -> 9 306** bårne over 77 filer plus **50 -> 50** on R761 (**9 356 -> 9 356** together), **0 of 78** files moving a count and **0** moving a code. R761 also settles the `[/Fl /A85]` question: **50 objects, 29 `[/DCTDecode]`, 21 `[/FlateDecode]`, 0 ASCII85 links** -- round 2's count of **580** is reproducible from nothing on this machine, which changes no decision but should not stay unqualified. **`resource.setrlimit(RLIMIT_AS)` was
MEASURED before anything was built on it and is NOT usable here** --
Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit` and
does not enforce it -- so no child-process cap exists and the per-link
budget is the whole bound. Report:
`docs/2026-09-18-utgangsbudsjett-per-ledd.md`.
- **THE LIMIT IS STATED RATHER THAN IMPLIED, and it is now ONE case**: a
stream something else has ALREADY decoded, where the memory is spent
before this package is asked. `check_payload(len(data))` after