open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(assets): budget every link by what its decoder COSTS (0.10.1)

Browse source 
Round 3 of the 0.10.1 review, and the finding is the pattern the three rounds
share: each bound an OUTPUT, and the bomb stepped one link along. The
declared size, then the first `FlateDecode`, then every `FlateDecode` -- and
then a link this package had documented as safe.

`ASCII85Decode` was classed as bounded "by its own input because it shrinks".
It quadruples: `z` is the shorthand for four zero bytes. And the output was
never the cost -- `base64.a85decode` appends one 4-byte object per group to a
list, about a hundred bytes of memory per byte of INPUT (101.4x at 1 MiB,
96.1x at 4 MiB, 94.5x at 16 MiB on CPython 3.14).

Paired subprocesses, idle machine, both sides from PINNED trees, the document
built once by a third process and read from a file because `ru_maxrss` never
falls and `b"z" * 64 MiB` alone costs 171 MB:

  [/Fl /A85]      z x 32 Mi  33 475 B   CARRIED 3 261 599 744 -> too_large 42 070 016
  [/Fl /A85]      z x 64 Mi  66 090 B   CARRIED 6 461 558 784 -> too_large 40 280 064
  [/A85]          z x  8 Mi   8.4 MB    CARRIED   933 085 184 -> too_large 62 484 480
  [/Fl /A85 /Fl]  z x 32 Mi  33 488 B  samples_invalid 3 519 180 800 -> too_large 43 438 080

The picture was CARRIED in three of the four: not a bound that fired late, no
bound at all. Doubling the `z` run trebles the old cost and leaves the new one
where it was.

WHY THIS FORM. `assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE
link may cost -- a separate number from `MAX_IMAGE_BYTES`, because that one
bounds the picture and this one bounds producing it. `FlateDecode` is measured
as it is paid; every other permitted filter carries a MEASURED cost ratio
(`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its decoder
is called, since those decoders take a whole string and return a whole string.
A filter with no ratio is refused unread. The budget TRAVELS: a deflate link
is inflated under the smaller of the picture's bound and what the next link's
decoder may be handed, or `[/Fl /A85]` pays 256 MiB for a refusal.

A chunked ASCII85 decoder written here was the alternative and was FELLED: it
would bound `_check_stream_cost` and not the run, because `stream.get_data()`
decodes the whole chain again with pdfminer's own decoder, and it would make
this package rather than pdfminer the authority on an image's bytes. The cap
is the only number that bounds that. `resource.setrlimit(RLIMIT_AS)` was
MEASURED before anything was built on it, as the order required, and is not
usable: Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit`
and does not enforce it. No child-process cap exists.

THE CAP IS READ OFF THE CORPORA, the posture `MAX_IMAGE_PIXELS` has: over the
9 668 image objects of the 77 PDFs on this machine, 16 decode through an
ASCII85 link and the largest input to one is 450 739 bytes, against a cap of
about 5.0 MB.

A PROPERTY TEST REPLACES THE LIST OF KNOWN SHAPES: every chain of length 1-3
over the ten filters pdfminer decodes, 1 110 of 1 110, both payload fills,
each delivered under the bound or refused with a published code and never paid
for on the way (`tracemalloc`, which counts allocations and is not disturbed
by load). Known-positive beside it: 258 of 258 chains over the permitted
filters still carry a small image.

MAJOR -- the backstop had no test. `check_payload` at the end of
`_check_stream_cost` could be deleted with the whole suite green, because the
second one after `get_data()` gives the same code one step later. The two
differ in whether the payment was made, so the test asserts `get_data` was
never called.

10 OF 10 MUTANTS KILLED, control green, each killer named in the report. Four
survived a first pass and two tests exist because of it.

NOT ONE PICTURE CHANGES HANDS, MEASURED BY NAME: `_pdf_images` over every PDF
on this machine from both pinned trees -- 9 306 -> 9 306 carried over 77
files, 50 -> 50 on R761, 0 of 78 files moving a count and 0 moving a code.
R761 also settles a question raised while this order was open: 50 objects, 29
[/DCTDecode], 21 [/FlateDecode], 0 ASCII85 links -- so round 2's count of 580
`[/FlateDecode /ASCII85Decode]` objects is reproducible from nothing on this
machine. It changes no decision; a bomb shape does not need a corpus.

Version stays 0.10.1, no tag. README, CHANGELOG, CLAUDE.md and errors.py
corrected TO what the code does; the round-2 report carries a correction block
rather than a rewrite. Report:
docs/2026-09-18-utgangsbudsjett-per-ledd.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 19:00:38 +02:00
commit 3b3b8ae0ca
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
9 changed files with 784 additions and 126 deletions
Download patch file Download diff file Expand all files Collapse all files

42
CHANGELOG.md
View file

@ -128,12 +128,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
because `filters[0]` is not `FlateDecode` there. Bounded, measured idle in paired subprocesses: 52 367 360 bytes at two
links, 61 390 848 at three, and 60 403 712 where the old path cost
2 567 204 864.
- **A filter whose output cannot be measured before it is produced is
refused unread**, with its own code `asset_pdf_unbounded`. `FlateDecode`
is measured; `ASCII85Decode` and `ASCIIHexDecode` are bounded by their own
input because they shrink; `DCTDecode`, `JPXDecode` and `JBIG2Decode` pass
through unchanged. `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`,
`/Crypt` and anything unknown are refused. Over the 5 142 image objects of
- **What a link COSTS is bounded, not the size of its output.** Bounding
every `FlateDecode` was still not a bound: the bomb moved into
`ASCII85Decode`, which the previous fix had classed as safe "because it
shrinks". `z` is that encoding's shorthand for four zero bytes, so the
filter quadruples its input, and `base64.a85decode` appends one 4-byte
object per group to a list — about a hundred bytes of memory per byte of
INPUT (measured on CPython 3.14: 101.4x at 1 MiB, 96.1x at 4 MiB, 94.5x at
16 MiB). Measured in paired subprocesses, idle machine, the document built
once and read from a file: a 33 475-byte PDF decoding through
`[/FlateDecode /ASCII85Decode]` cost 3 261 599 744 bytes of peak RSS and
the picture was CARRIED; bounded, 42 070 016 and `asset_too_large`.
Doubling the run of `z` takes the old cost to 6 461 558 784 and the
bounded one to 40 280 064, so the cost no longer follows the bomb. A
single `[/ASCII85Decode]` link went 933 085 184 → 62 484 480, and
`[/Fl /A85 /Fl]` 3 519 180 800 → 43 438 080 (and from
`asset_samples_invalid` to a bound's own code).
- **Every permitted filter now carries a measured cost ratio**
(`assets.PDF_FILTER_COST_RATIO`) and a per-link budget
(`MAX_FILTER_DECODE_BYTES`, 512 MiB). `FlateDecode` is measured a chunk at
a time as it is paid, under a limit that is the smaller of the picture's
own bound and what the NEXT link's decoder may be handed, so the budget
travels down the chain. Every other permitted filter has its cost
PREDICTED from its input size before its decoder is called, because those
decoders take a whole string and return a whole string. The cap that falls
out for `ASCII85Decode` is read off the corpora: of the 9 668 image
objects of the 77 PDFs measured, 16 decode through such a link and the
largest input to one is 450 739 bytes, more than ten times under it.
- **A filter with no measured ratio is refused unread**, with its own code
`asset_pdf_unbounded`: `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`,
`/Crypt` and anything unknown.
- **A property test runs every chain of length 1–3** over the ten filters
pdfminer decodes — 1 110 of them, each with an amplifying payload —
and requires each to be delivered under the bound or refused with a code
in the published vocabulary, never paid for on the way. The known-positive
beside it holds that every chain over the permitted filters still carries
a small image. Over the 5 142 image objects of
the 78 PDFs measured, the refused class is 4 `CCITTFaxDecode` objects,
which are 1-bit stencil masks and were already refused one step later by
the encoder. Measured by name over the same 78 documents, carried images

48
CLAUDE.md
View file

@ -935,14 +935,13 @@ and fixtures, never code.
not `FlateDecode` there. Bounded (idle machine, paired subprocesses):
**52 367 360 B** at two links, **61 390 848 B** at three, **60 403 712 B**
where the old path cost 2 567 204 864 B, and the single-link control
unmoved at 59 232 256 -> 62 017 536. **Three classes and no fourth**
(`extract.bounded_pdf_filters`, pinned by a test): `FlateDecode` MEASURED,
`ASCII85Decode`/`ASCIIHexDecode` bounded by their own input because they
SHRINK, `DCTDecode`/`JPXDecode`/`JBIG2Decode` PASS THROUGH. Everything
unmoved at 59 232 256 -> 62 017 536. Everything
else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`,
anything written later -- is refused UNREAD with its own code
`asset_pdf_unbounded`, the same decision `corpus.resolve_gate` takes for
an unknown gate name. Cost measured over the **5 142** image objects of
an unknown gate name. **The three-class split this round shipped with --
`ASCII85Decode`/`ASCIIHexDecode` "bounded by their own input because they
SHRINK" -- was FALSE and round 3 below replaced it.** Cost measured over the **5 142** image objects of
78 PDFs: the refused class is **4** `CCITTFaxDecode` objects, all 1-bit
stencil masks already refused one step later by the encoder, and **0**
objects in an encrypted document. **NOT ONE PICTURE CHANGES HANDS AND IT
@ -953,6 +952,45 @@ and fixtures, never code.
`trinn2` hold the same document). An ENCRYPTED stream is
now deciphered and then measured (deciphering does not change a length),
where `stream.decipher is not None` used to return unmeasured.
- **THE COST OF A LINK, NOT THE SIZE OF ITS OUTPUT (round 3).** Three
rounds each bound an OUTPUT and the bomb stepped one link along: the
declared size, then the first `FlateDecode`, then every `FlateDecode`.
Round 2's third class was FALSE -- `ASCII85Decode`'s `z` is the shorthand
for four zero bytes, so it QUADRUPLES its input, and `base64.a85decode`
appends one 4-byte object per group to a list, costing about **a hundred
bytes of memory per byte of INPUT** (measured on CPython 3.14: **101.4x**
at 1 MiB, **96.1x** at 4 MiB, **94.5x** at 16 MiB). Paired subprocesses,
idle machine, both sides from pinned trees, the document built once and
read from a FILE because `ru_maxrss` never falls and `b"z" * 64 MiB`
alone costs 171 MB: `[/Fl /A85]` **33 475 B of file -> 3 261 599 744 B
peak and the picture CARRIED**, now **42 070 016 B** and
`asset_too_large`; at twice the `z` run **6 461 558 784 -> 40 280 064**,
so the cost no longer follows the bomb; `[/A85]` alone **933 085 184 ->
62 484 480**; `[/Fl /A85 /Fl]` **3 519 180 800 -> 43 438 080** and from
`asset_samples_invalid` to a bound's own code. **The rule is a BUDGET per
link** (`assets.MAX_FILTER_DECODE_BYTES`, 512 MiB, a separate number from
`MAX_IMAGE_BYTES`): `FlateDecode` is measured as it is paid, every other
permitted filter has a MEASURED cost ratio
(`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its
decoder is called, and a filter with no ratio is refused unread. **The
budget TRAVELS**: a deflate link is inflated under the smaller of the
picture's bound and what the next link's decoder may be handed, or
`[/Fl /A85]` pays 256 MiB for a refusal. The ASCII85 cap (**~5.0 MB**) is
READ OFF the corpora: of **9 668** image objects over **77** PDFs, **16**
decode through such a link and the largest input is **450 739 B**, ten
times under it. **A PROPERTY TEST replaces the list of known shapes**:
every chain of length 1-3 over the ten filters pdfminer decodes, **1 110
of 1 110**, both payload fills, each delivered under the bound or refused
with a published code and never paid for on the way (`tracemalloc`, which
counts allocations and is not disturbed by load). Known-positive: **258 of
258** chains over the permitted filters still carry a small image.
**10 of 10 mutants killed**, control green. **NOT A ROW'S DIFFERENCE ON
REAL DOCUMENTS**: paired `_pdf_images` over every PDF on this machine from two PINNED trees: **9 306 -> 9 306** bårne over 77 filer plus **50 -> 50** on R761 (**9 356 -> 9 356** together), **0 of 78** files moving a count and **0** moving a code. R761 also settles the `[/Fl /A85]` question: **50 objects, 29 `[/DCTDecode]`, 21 `[/FlateDecode]`, 0 ASCII85 links** -- round 2's count of **580** is reproducible from nothing on this machine, which changes no decision but should not stay unqualified. **`resource.setrlimit(RLIMIT_AS)` was
MEASURED before anything was built on it and is NOT usable here** --
Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit` and
does not enforce it -- so no child-process cap exists and the per-link
budget is the whole bound. Report:
`docs/2026-09-18-utgangsbudsjett-per-ledd.md`.
- **THE LIMIT IS STATED RATHER THAN IMPLIED, and it is now ONE case**: a
stream something else has ALREADY decoded, where the memory is spent
before this package is asked. `check_payload(len(data))` after

35
README.md
View file

@ -339,14 +339,33 @@ decodes a stream through a *list* of filters, and `/Filter [/FlateDecode
/FlateDecode]` puts the whole expansion in the second one: measured, 1 636
bytes of file cost 886 554 624 bytes of peak RSS when only the first link was
measured (52 367 360 with every link measured), and the picture was still refused at the end — after the memory had
been spent. `FlateDecode` is measured, `ASCII85Decode` and `ASCIIHexDecode`
are bounded by their own input because they shrink, and `DCTDecode`,
`JPXDecode` and `JBIG2Decode` pass through unchanged. Any other filter —
`LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything written
after this — expands by an amount no chunked measurement can reach, so an
image behind one is refused UNREAD with its own code, `asset_pdf_unbounded`,
rather than decoded to find out what it costs. An encrypted stream is
deciphered first and then measured like any other.
been spent. An encrypted stream is deciphered first and then measured like any
other.
**And what a link COSTS is bounded, not the size of its output.** Bounding
every `FlateDecode` was still not a bound, because `ASCII85Decode` had been
classed as safe "because it shrinks" and it does not: `z` is that encoding's
shorthand for four zero bytes, so the filter quadruples its input, and
`base64.a85decode` holds about a hundred bytes of memory per byte of input.
Measured in paired subprocesses on an idle machine, the document built once
and read from a file so the fixture is not what is measured: a 33 475-byte PDF
decoding through `[/FlateDecode /ASCII85Decode]` cost 3 261 599 744 bytes of
peak RSS and the picture was CARRIED; bounded it is 42 070 016 and
`asset_too_large`. Doubling the run of `z` takes the old cost to
6 461 558 784 and the bounded one to 40 280 064 — the cost no longer follows
the bomb. So `FlateDecode` is measured a chunk at a time as it is paid, and
every other permitted filter carries a MEASURED worst-case cost per byte of
input which is checked against the budget *before* its decoder is called.
Any other filter — `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`,
`/Crypt`, anything written after this — has no measured ratio, so an image
behind one is refused UNREAD with its own code, `asset_pdf_unbounded`, rather
than decoded to find out what it costs. The cap that falls out for
`ASCII85Decode` is read off the corpora the way the pixel bound is: over the
9 668 image objects of the 77 PDFs measured, 16 decode through such a link and
the largest input to one is 450 739 bytes, more than ten times under the cap.
A property test runs **every** chain of length 1–3 over the ten filters
pdfminer decodes — 1 110 of them — and requires each to be delivered under the
bound or refused with a published code, never paid for on the way.
**What the stream bound does NOT reach**, stated because the difference
matters: a stream something else has already decoded, where the memory was

14
docs/2026-09-18-filterkjeden-og-backstoppen.md
View file

@ -1,5 +1,19 @@
# The chain, not its first link — and a backstop nothing held
> **CORRECTION, 2026-09-18 (round 3).** This report's three-class split is
> wrong on one class. It states that `ASCII85Decode` and `ASCIIHexDecode` are
> "bounded by their own input because they shrink". `z` is ASCII85's shorthand
> for four zero bytes, so that filter QUADRUPLES its input, and
> `base64.a85decode` costs about a hundred bytes of memory per byte of input.
> Measured on the pinned tree of `0c3c490`, the commit this report closes: a
> 33 475-byte PDF decoding an image through `[/FlateDecode /ASCII85Decode]`
> cost 3 261 599 744 bytes of peak RSS and the picture was CARRIED with no
> rejection. Everything else here stands — the chain walk, the backstop, the
> paired corpus numbers — and what replaced the class is a measured cost ratio
> per filter, recorded in
> [`docs/2026-09-18-utgangsbudsjett-per-ledd.md`](2026-09-18-utgangsbudsjett-per-ledd.md).
> The text below is left as it was written.
A PM checkpoint of `0f308c1` — the commit that was to make `v0.10.1` true —
read the fix for the deflate bomb and found the bound still reachable, through
a shape the fix had not considered: a PDF decodes a stream through a **list**

254
docs/2026-09-18-utgangsbudsjett-per-ledd.md Normal file
View file

@ -0,0 +1,254 @@
# A budget per link, and a property over every chain
A PM checkpoint of `0c3c490` — the commit that bound every link of a PDF
filter chain — read the fix and found the bound still reachable. This is the
third round of the same review, and the third time a bound was put on an
**output** and the bomb stepped one link along.
The two rounds before it are
[`docs/2026-09-17-bildestien-0-10-1.md`](2026-09-17-bildestien-0-10-1.md),
[`docs/2026-09-18-bildestien-holder-0-10-1.md`](2026-09-18-bildestien-holder-0-10-1.md)
and [`docs/2026-09-18-filterkjeden-og-backstoppen.md`](2026-09-18-filterkjeden-og-backstoppen.md),
which carries a correction block pointing here.
## The finding is the pattern, not the filter
| round | what was bound | where the bomb moved |
| --- | --- | --- |
| 1 | the size the dictionary DECLARES | into the stream |
| 2 | the first `FlateDecode` link | into the second link |
| 3 | every `FlateDecode` link | into a link documented as safe |
Round 3 shipped a three-class split in which `ASCII85Decode` and
`ASCIIHexDecode` were "bounded by their own input because they shrink". That
sentence is false in two independent ways:
* **ASCII85 does not shrink.** `z` is its shorthand for four zero bytes, so
one input byte becomes four. The output ratio is 4, not 0.8.
* **The output is not the cost.** `base64.a85decode` appends one 4-byte object
per group to a list, so a run of `z` costs about a hundred bytes of memory
per byte of INPUT. Measured on CPython 3.14: **101.4x** at 1 MiB of input,
**96.1x** at 4 MiB, **94.5x** at 16 MiB. Nothing in the output size says so.
The second point is the general one, and it is why this round does not add a
fourth class. A bound on what a link OUTPUTS is not a bound on what producing
it COSTS, and every round of this review has been an instance of that.
## BLOCKER — reproduced
Measured in paired subprocesses on an idle machine, both sides from pinned
trees (the `before` side from `git archive` of `0c3c490`, the `after` side from
a snapshot of the working tree), each printing the module's own `__file__` as
a control. The document is built ONCE by a third process and read from a file:
`ru_maxrss` is a high-water mark that never falls, so a child that builds its
own fixture reports the fixture — `b"z" * (64 MiB)` alone costs 171 MB, more
than the bounded extraction it would be measuring. This tripped the first
attempt at these figures.
| chain | file | before | after |
| --- | --- | --- | --- |
| `[/FlateDecode /ASCII85Decode]`, `z` × 32 Mi | 33 475 B | **carried**, 3 261 599 744 B | `asset_too_large`, **42 070 016 B** |
| `[/FlateDecode /ASCII85Decode]`, `z` × 64 Mi | 66 090 B | **carried**, 6 461 558 784 B | `asset_too_large`, **40 280 064 B** |
| `[/ASCII85Decode]`, `z` × 8 Mi | 8 389 449 B | **carried**, 933 085 184 B | `asset_too_large`, **62 484 480 B** |
| `[/Fl /A85 /Fl]`, `z` × 32 Mi | 33 488 B | `asset_samples_invalid`, 3 519 180 800 B | `asset_too_large`, **43 438 080 B** |
Two things to read off it. The picture was **carried** in three of the four
rows — this was not a bound that fired late, it was no bound at all. And the
bounded cost does not follow the bomb: doubling the run of `z` takes the old
cost from 3.26 GB to 6.46 GB and the new one from 42.1 MB to 40.3 MB.
## The rule: a budget per link, and it travels
`assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE link may cost.
It is a separate number from `MAX_IMAGE_BYTES` (256 MiB) on purpose: that one
bounds the picture this package will carry, this one bounds what producing it
costs on the way.
`assets.PDF_FILTER_COST_RATIO` gives each permitted filter a **measured**
worst-case peak memory per byte of input. `None` means the decoder is driven a
chunk at a time here, so the cost is measured as it is paid — today that is
`FlateDecode` alone.
| filter | cost ratio | measured |
| --- | --- | --- |
| `FlateDecode` | — | driven a chunk at a time (`assets._inflate`) |
| `ASCII85Decode` | 104 | 101.4x / 96.1x / 94.5x at 1 / 4 / 16 MiB of `z` |
| `ASCIIHexDecode` | 2 | 1.5x at 16 MiB |
| `DCTDecode`, `JPXDecode`, `JBIG2Decode` | 1 | pass-through in pdfminer |
Everything else — `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`,
anything written later — has no measured ratio and is refused UNREAD with
`asset_pdf_unbounded`, before any link in front of it is decoded. That is the
same decision `corpus.resolve_gate` takes for an unknown gate name: a fallback
reproduces the defect with an extra step.
The budget **travels down the chain**. A `FlateDecode` link's output is the
next link's input, so it is inflated under a limit that is the smaller of
`MAX_IMAGE_BYTES` and what the next link's decoder may be handed
(`assets.inflate_limit_for`). Without that, `[/FlateDecode /ASCII85Decode]`
would inflate 256 MiB of `z` before the link behind it was asked anything.
### Why an input cap and not a bounded ASCII85 decoder
A chunked ASCII85 decoder written here would bound `_check_stream_cost`, and
it would not bound the run: pdfminer decodes the whole chain again in
`stream.get_data()`, with its own unbounded decoder, and that is where the
memory is actually spent. The only number that bounds *that* is the size of
the input this package allows the link to be handed. Writing our own decoder
would also make this package, rather than pdfminer, the authority on what an
image's bytes are.
### Why not a hard backstop in a child process
The order asked for `resource.setrlimit(RLIMIT_AS)` to be MEASURED before
anything was built on it. It was, and it is not available here: on this
machine (Darwin 26.6.2, CPython 3.14) `setrlimit(RLIMIT_AS, (256 MiB, hard))`
raises `ValueError: current limit exceeds maximum limit` — a fresh CPython
process has already reserved far more address space than the cap, and the hard
limit reads as `RLIM_INFINITY`. The documented behaviour agrees: Darwin does
not enforce `RLIMIT_AS`, `RLIMIT_DATA` or `RLIMIT_RSS` the way Linux does. So
no child-process memory cap was built, and the bound is the per-link budget
alone.
## The cap is read off the corpora
The cap that falls out for `ASCII85Decode` is 512 MiB / 104 ≈ **5.0 MB** of
input. Measured 2026-09-18 over the **9 668 image objects of the 77 PDFs on
this machine** (enumerated through pdfminer's own page walk):
| chain | objects |
| --- | --- |
| `[/FlateDecode]` | 6 235 |
| `[/DCTDecode]` | 2 459 |
| `[/FlateDecode /DCTDecode]` | 596 |
| `[/Fl]` | 296 |
| unfiltered | 42 |
| `[/ASCII85Decode /FlateDecode]` | 16 |
| `[/JPXDecode]` | 16 |
| `[/CCITTFaxDecode]` | 8 |
**16** objects decode through an `ASCII85Decode` link, and the largest input
any of them is handed is **450 739 bytes** — more than ten times under the
cap. That is the posture `MAX_IMAGE_PIXELS` has: a number read off the corpora
and standing an order of magnitude above anything measured, so the bound costs
no picture anybody has.
Two corrections to earlier published counts fall out of this table, and both
are about ENUMERATION rather than about the documents. The round-2 report
counted 5 142 objects over 78 PDFs and **580** behind `[/FlateDecode
/ASCII85Decode]`. This walk finds 9 668 objects over 77 files and **0** behind
that chain. The denominators differ because the two walks are different (this
one recurses into `LTFigure`; the file R761 is not in this listing), so
neither number is wrong about a document — but a chain count is only readable
beside the walk that produced it, and the `[/Fl /A85]` group is not one this
machine's corpora hold. The bomb that shape carries is real regardless: a
document does not have to exist in a corpus to be handed to `okf build`.
## The property test
`tests/test_asset_limits.py::test_no_chain_of_up_to_three_filters_is_carried_over_the_bound`
generates **every** chain of length 1–3 over the ten filters pdfminer decodes
— `K = 1 110` — twice, once with a payload of zeros (the amplifying case at
both ends: it deflates to nothing and `a85encode`s to a run of `z`) and once
with a repeated non-zero byte. Each chain's stream is built by encoding the
payload BACKWARDS through the chain, so every chain over the permitted filters
is a valid document rather than a rejection by accident.
The requirement is one sentence: the picture is either delivered with its
bytes under the bound, or refused with a code in the published vocabulary;
never carried over the bound, and never paid for on the way. `tracemalloc`
measures the paying, because it counts Python's own allocations — which is
exactly where `a85decode`'s cost lives, and unlike `ru_maxrss` it is not
disturbed by other work on the machine.
Result: **1 110 of 1 110** chains pass, both payloads — `k = K`.
Beside it, `test_every_bounded_chain_still_carries_a_small_image` runs the
**258** chains over the six permitted filters with a 64-byte image and
requires none of them to be refused. A rule that refuses everything passes the
property alone; it does not pass this.
And `test_the_ascii85_cost_ratio_is_not_below_the_one_this_package_measured`
re-measures the ratio the budget rests on, in a subprocess, at two input
sizes. If CPython ever changes `a85decode` so that it costs more, the constant
is too generous and this says so before a corpus does.
## MAJOR — the backstop had no test
`check_payload(size, name=name)` at the END of `_check_stream_cost` could be
deleted with the whole suite still passing. It is what refuses a stream no
filter in the chain expands — an unfiltered one, or one behind `DCTDecode` —
and the SECOND `check_payload`, after `get_data()`, produces the same code and
the same words one step later. A test that reads the code cannot tell the two
apart.
What separates them is whether the payment was made, so the test asserts
`get_data` was never called.
## Mutants
Ten mutations, one line each, run in a scratch clone with the unmutated copy
run FIRST as a control. The peak-RSS subprocess tests are deselected for these
runs — they measure a high-water mark and the machine was running a corpus
census — so what kills a mutant here is the property test, the code
vocabulary, or an assertion about which check fired.
| mutation | one line | killed by |
| --- | --- | --- |
| `backstop-deleted` | `check_payload` at the end of `_check_stream_cost` removed | `test_the_stream_bound_refuses_before_get_data_is_ever_called` |
| `cost-check-deleted` | the per-link `check_filter_cost` call removed | the property test, both fills, + `[/ASCII85Decode]` |
| `ascii85-budget-removed` | that filter's ratio set to `None`, so it gets no budget | 9 tests, including both ratio measurements and the corpus cap |
| `ascii85-ratio-is-one` | the ratio set to 1 instead of the measured 104 | 6 tests, including both ratio measurements |
| `budget-does-not-travel` | `inflate_limit_for` returns `MAX_IMAGE_BYTES` always | `test_the_budget_travels_to_the_next_link` |
| `widest-output-ignored` | `_widest_output` returns its input unchanged | `test_a_discarded_links_size_travels_as_the_widest_it_could_become` |
| `first-flate-not-last` | the discard happens at the FIRST deflate link | 5 tests, including both round-2 chain bombs |
| `unknown-filter-passes` | a filter with no ratio is let through instead of refused | `test_a_filter_the_bound_cannot_measure_...`, `test_asset_pdf_unbounded` |
| `budget-a-hundredfold` | `MAX_FILTER_DECODE_BYTES` multiplied by 100 | 4 tests, including both new bombs |
| `cost-check-off-by-a-factor` | the comparison allows 1 000x the limit | `test_an_ascii85_link_on_its_own_is_bounded` |
**10 of 10 killed.** Four of them survived a first pass and are the reason two
of the tests above exist: `budget-does-not-travel` and `widest-output-ignored`
had no test at all, and `budget-a-hundredfold` and
`cost-check-off-by-a-factor` were only reachable through the peak-RSS
subprocess tests that first pass had deselected. A mutant that survives is a
test that was missing, not a mutation that was unfair.
## Cost to real documents
Measured by name, not by total: `_pdf_images` run over every PDF on this
machine from each of the two pinned trees, each printing the module file it
loaded as a control, and the per-file counts compared.
| corpus | files | carried before | carried after | files whose count moved |
| --- | --- | --- | --- | --- |
| `~/corpora` + `tests/fixtures` | 77 | 9 306 | **9 306** | **0** |
| R761 Prosesskoden:2025 | 1 | 50 | **50** | **0** |
| both | 78 | 9 356 | **9 356** | **0** |
The rejection codes are identical too — `asset_pdf_unsupported` 314,
`asset_pdf_unbounded` 8, `asset_samples_invalid` 40 on both sides, and **0
files** where any per-file code count moved. Not one picture changes hands.
That is what the cap being read off the corpora buys: the only new refusal is
`check_filter_cost`, and the largest `ASCII85Decode` input any of these
documents holds is 450 739 bytes against a cap of about 5.0 MB.
The two trees are `git archive` of `0c3c490` and a snapshot of the working
tree taken before the prose edits; `diff -r` between that snapshot and the
committed tree touches docstrings and comments only, so what was measured is
what shipped.
R761 also settles the `[/FlateDecode /ASCII85Decode]` question the PM raised
while this order was open: **50 image objects, 29 `[/DCTDecode]` and 21
`[/FlateDecode]`, and 0 ASCII85 links.** So that chain is in neither the 77
corpus files nor R761, and the round-2 report's count of 580 is not
reproducible from anything on this machine. It changes no decision here — the
bomb that shape carries does not need a corpus to exist in — but a published
count that cannot be reproduced should not stay unqualified.
## What is still not bounded
Unchanged from round 2, and stated rather than implied: a stream something
else has already decoded (`_pdf_stream_bytes` returns `None`), where the
memory is spent before this package is asked. `check_payload` after
`get_data()` COUNTS it — a counted refusal, not a bounded one.

118
src/llm_ingestion_okf/assets.py
View file

@ -198,6 +198,124 @@ def check_payload(size: int, *, name: str) -> None:
)
#: What decoding ONE link of a PDF filter chain may cost this package, in bytes
#: of memory. A SEPARATE number from `MAX_IMAGE_BYTES`, and the distinction is
#: the whole of round 3: that one bounds the picture this package will carry,
#: this one bounds what producing it costs on the way. Three rounds of this
#: review each bound an output and the bomb moved one link along, because a
#: decoder's working set is not its output. Twice `MAX_IMAGE_BYTES`, so a run
#: may hold the stream it was given and one stage of decoding at once and no
#: more.
MAX_FILTER_DECODE_BYTES = 512 * 1024 * 1024
#: MEASURED peak memory per byte of INPUT, for each filter this package lets an
#: image be reached through. `None` means the decoder is driven a chunk at a
#: time here, so the cost is measured as it is paid and no ratio is needed --
#: today that is `FlateDecode` alone (`_inflate`).
#:
#: The numbers are read off CPython 3.14 on 2026-09-18, worst case per filter:
#:
#: * `ASCII85Decode` 101.4x at 1 MiB of input, 96.1x at 4 MiB, 94.5x at 16 MiB.
#: `z` is the shorthand for four zero bytes, so `base64.a85decode` appends one
#: 4-byte object per INPUT byte to a list -- the output ratio is 4, the cost
#: ratio is a hundred, and 0.10.1 documented this filter as "bounded by its
#: own input because it shrinks". The constant sits above the worst of the
#: three, and `test_the_ascii85_cost_ratio_is_not_below_the_one_this_package
#: _measured` re-measures it so it cannot rot when CPython changes.
#: * `ASCIIHexDecode` 1.5x at 16 MiB: it strips whitespace into a copy and
#: `unhexlify`s that, and its output is half its input.
#: * `DCTDecode`, `JPXDecode` and `JBIG2Decode` are pass-through in pdfminer --
#: the bytes are handed to the image reader unchanged -- so the ratio is 1.
#:
#: A filter that is not in this table has no measured ratio and is refused
#: unread (`asset_pdf_unbounded`). That is the same decision `corpus.resolve
#: _gate` takes for an unknown gate name: a fallback reproduces the defect with
#: an extra step.
PDF_FILTER_COST_RATIO: dict[str, float | None] = {
"FlateDecode": None,
"ASCII85Decode": 104.0,
"ASCIIHexDecode": 2.0,
"DCTDecode": 1.0,
"JPXDecode": 1.0,
"JBIG2Decode": 1.0,
}
#: The largest OUTPUT each of those filters can produce per byte of input, used
#: to carry a bound forward when the bytes themselves have been discarded.
#: `ASCII85Decode` is 4 (one `z`), `ASCIIHexDecode` 0.5 (two digits to a byte),
#: pass-through 1. `FlateDecode` has no such number, which is why it is the one
#: filter measured a chunk at a time.
PDF_FILTER_OUTPUT_RATIO: dict[str, float | None] = {
"FlateDecode": None,
"ASCII85Decode": 4.0,
"ASCIIHexDecode": 0.5,
"DCTDecode": 1.0,
"JPXDecode": 1.0,
"JBIG2Decode": 1.0,
}
def filter_input_limit(canonical: str) -> int | None:
"""The largest input this package will hand to `canonical`'s decoder.
`None` for a filter decoded a chunk at a time, which needs no input limit
because its cost is measured while it is paid.
The number this produces for `ASCII85Decode` -- about 5.0 MB -- is READ OFF
the corpora the way `MAX_IMAGE_PIXELS` is: over the 9 668 image objects of
the 77 PDFs on this machine (2026-09-18), 16 decode through an
`ASCII85Decode` link and the largest input to one is 450 739 bytes, so the
limit stands more than ten times above anything measured.
"""
ratio = PDF_FILTER_COST_RATIO.get(canonical)
if ratio is None:
return None
return int(MAX_FILTER_DECODE_BYTES // ratio)
def check_filter_cost(size: int, *, canonical: str, name: str) -> None:
"""Refuse a link whose decoder would cost more than the budget, BEFORE it
decodes anything.
This is the half `inflated_size` cannot cover. That one drives zlib a chunk
at a time and stops the moment the running total crosses the bound, which
is only possible because zlib hands its output over incrementally. Nothing
else in a PDF filter chain does: `base64.a85decode` is asked for a whole
string and gives back a whole string, so by the time its output could be
measured the memory has been spent. For those the cost is PREDICTED from a
measured ratio and the input size, and predicted before the call.
"""
limit = filter_input_limit(canonical)
if limit is None or size <= limit:
return
ratio = PDF_FILTER_COST_RATIO[canonical]
raise ExtractionError(
f"the image {name!r} hands {size} bytes to {canonical}, whose decoder costs "
f"about {ratio} bytes of memory per byte of input -- over this package's "
f"budget of {MAX_FILTER_DECODE_BYTES} bytes for one link; refused before "
"the decode, because a bound on what a link OUTPUTS is not a bound on "
"what producing it costs",
code="asset_too_large",
)
def inflate_limit_for(canonical: str | None) -> int:
"""How much a `FlateDecode` link may produce, given what comes AFTER it.
The picture's own bound is `MAX_IMAGE_BYTES`, but a link's output is the
next link's input, and a decoder with a cost ratio cannot be handed more
than `filter_input_limit` allows. Carrying the budget down the chain this
way is what stops `[/FlateDecode /ASCII85Decode]` from inflating 256 MiB of
`z` before the link behind it is asked anything.
"""
limit = MAX_IMAGE_BYTES
if canonical is not None:
behind = filter_input_limit(canonical)
if behind is not None:
limit = min(limit, behind)
return limit
def inflated_size(raw: bytes, *, name: str, limit: int | None = None) -> int:
"""What a deflate stream costs to decompress, measured without paying it.

34
src/llm_ingestion_okf/errors.py
View file

@ -133,13 +133,15 @@ class ExtractionError(IngestError):
- `asset_samples_invalid` — the sample buffer does not fit the dimensions
the image dictionary declares. Refused rather than padded: a short buffer
means the dictionary was read wrong
- `asset_too_large` — the picture is over this package's bound, either
because it DECLARES a size beyond `MAX_IMAGE_PIXELS`, because the file
itself is that large, or because the stream behind it DECOMPRESSES to
more than `MAX_IMAGE_BYTES`. The three are one code because they are one
decision — this run will not hold that picture — and because a consumer
counting refusals wants the picture, not the mechanism. The bound is read
off the corpora and sits an order of magnitude above anything measured
- `asset_too_large` — the picture is over this package's bound: because it
DECLARES a size beyond `MAX_IMAGE_PIXELS`, because the file itself is
that large, because the stream behind it DECOMPRESSES to more than
`MAX_IMAGE_BYTES`, or because one link of its filter chain would COST
more than `MAX_FILTER_DECODE_BYTES` to decode. The four are one code
because they are one decision — this run will not hold that picture —
and because a consumer counting refusals wants the picture, not the
mechanism. Each bound is read off the corpora and sits an order of
magnitude above anything measured
- `asset_size_invalid` — the container declares a size that is not a size:
a zero or negative `/Width` or `/Height`. DISTINCT from
`asset_too_large`, because the two say different things about the
@ -150,14 +152,16 @@ class ExtractionError(IngestError):
negative dimension multiplies to a negative pixel count, under which
every bound reads as satisfied
- `asset_pdf_unbounded` — the image is reached through a PDF stream filter
whose output this package cannot measure before producing it
(`LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything
unknown), or through an encrypted stream it cannot decipher. DISTINCT
from `asset_too_large`, which says a measurement was taken and came out
over the bound: this one says no measurement was possible, so the picture
is refused UNREAD rather than decoded to find out what it costs. Measured
2026-09-18: bounding only the first link of a filter chain let 1 636
bytes of PDF cost 886 554 624 bytes of peak RSS
this package has no measured cost ratio for (`LZWDecode`,
`RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything unknown), or
through an encrypted stream it cannot decipher. DISTINCT from
`asset_too_large`, which says a measurement was taken or predicted and
came out over the bound: this one says neither was possible, so the
picture is refused UNREAD rather than decoded to find out what it costs.
Measured 2026-09-18: bounding only the first link of a filter chain let
1 636 bytes of PDF cost 886 554 624 bytes of peak RSS, and bounding
every link's OUTPUT still let 33 475 bytes cost 3 261 599 744 through a
filter whose decoder holds a hundred bytes per byte of input
"""

260
src/llm_ingestion_okf/extract.py
View file

@ -29,6 +29,7 @@ import collections
import csv
import functools
import io
import math
import re
import statistics
import tempfile
@ -43,12 +44,15 @@ from xml.etree import ElementTree
from xml.etree.ElementTree import Element
from .assets import (
PDF_FILTER_OUTPUT_RATIO,
AssetRejection,
ExtractedImage,
check_filter_cost,
check_payload,
check_size,
encode_png,
inflate_bounded,
inflate_limit_for,
inflated_size,
read_image,
render_block,
@ -1370,44 +1374,59 @@ def _pdf_alpha(attrs: dict[str, object], width: int, height: int) -> bytes | Non
def bounded_pdf_filters() -> frozenset[str]:
"""The PDF stream filters an image may be reached through, by NAME.
THREE CLASSES, and what separates them is whether a bound can be put on the
output before the decoding is paid for.
TWO CLASSES, and what separates them is HOW the cost of a link is bounded,
never whether the link is safe. Every one of them is bounded.
* `FlateDecode` is MEASURED: inflated a chunk at a time with the output
discarded, refused the moment the running total crosses the bound.
* `ASCII85Decode` and `ASCIIHexDecode` SHRINK by construction -- five
characters to four bytes, two to one -- so their output is bounded by
their input, which is already in memory as part of the file. They are
decoded here so that a `FlateDecode` BEHIND one can be measured.
* `DCTDecode`, `JPXDecode` and `JBIG2Decode` are pass-through in pdfminer:
it hands the compressed image on for the image reader to sniff, so the
size does not change.
* `FlateDecode` is MEASURED as it is paid: inflated a chunk at a time,
refused the moment the running total crosses the bound, with the output
discarded unless a link behind it has to be measured from those bytes.
* `ASCII85Decode`, `ASCIIHexDecode`, `DCTDecode`, `JPXDecode` and
`JBIG2Decode` are PREDICTED before they are paid: each carries a measured
worst-case cost per byte of input (`assets.PDF_FILTER_COST_RATIO`), and a
link whose input times that ratio is over the budget is refused before
its decoder is called. Their decoders take a whole string and return a
whole string, so there is no moment between the two at which a cost could
be observed.
0.10.1 had a third class, and it was WRONG. `ASCII85Decode` and
`ASCIIHexDecode` were called bounded "by their own input because they
shrink". `z` is ASCII85's shorthand for four zero bytes, so that filter
QUADRUPLES its input, and `base64.a85decode` appends one 4-byte object per
group to a list, so it costs about a hundred bytes of memory per byte of
input. Measured on the pinned tree, its own interpreter, idle machine: a
33 475-byte PDF decoding through `[/FlateDecode /ASCII85Decode]` cost
3 261 599 744 bytes of peak RSS and the picture was CARRIED. Under the
ratios it is 42 070 016 bytes and `asset_too_large`, and at twice the run
of `z` -- which trebled the old cost to 6 461 558 784 -- it is 40 280 064:
the cost no longer follows the bomb.
EVERYTHING ELSE IS REFUSED with `asset_pdf_unbounded` before any of the
stream is decoded -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`,
`/Crypt`, and any filter written after this one. They expand by an amount
pdfminer will only reveal by producing the whole output, so there is no
measuring them a chunk at a time, and decoding one to find out how big it
is IS the failure this bound exists to stop. Refusing an unknown name
rather than passing it through is the same decision `corpus.resolve_gate`
takes for an unknown gate name: a fallback reproduces the defect with an
extra step.
`/Crypt`, and any filter written after this one. A filter with no measured
ratio has no budget to be checked against, and decoding one to find out
what it costs IS the failure this bound exists to stop. Refusing an unknown
name rather than passing it through is the same decision
`corpus.resolve_gate` takes for an unknown gate name: a fallback reproduces
the defect with an extra step.
The cost is measured rather than assumed. Over the 5 142 image objects of
the 78 PDFs on this machine (2026-09-18), the filter chains are 1 654
`[/DCTDecode]`, 2 236 `[/FlateDecode]`, 596 `[/FlateDecode /DCTDecode]`,
580 `[/FlateDecode /ASCII85Decode]`, 40 unfiltered, 16 `[/ASCII85Decode
/FlateDecode]`, 16 `[/JPXDecode]` and 4 `[/CCITTFaxDecode]` -- so the
refused class is those 4 objects, which are 1-bit stencil masks
(`/ImageMask true`, `/BitsPerComponent 1`) and were already refused one
step later by the encoder, twice over.
The reach is measured rather than assumed. Over the 9 668 image objects of
the 77 PDFs on this machine (2026-09-18, enumerated through pdfminer's own
page walk), the filter chains are 6 235 `[/FlateDecode]`, 2 459
`[/DCTDecode]`, 596 `[/FlateDecode /DCTDecode]`, 296 `[/Fl]`, 42
unfiltered, 16 `[/ASCII85Decode /FlateDecode]`, 16 `[/JPXDecode]` and 8
`[/CCITTFaxDecode]`. The refused class is those 8 objects, 1-bit stencil
masks (`/ImageMask true`, `/BitsPerComponent 1`) already refused one step
later by the encoder. The largest input any `ASCII85Decode` link is handed
is 450 739 bytes, more than ten times under the budget's cap, which is why
the cap costs no picture the corpora hold.
"""
return _BOUNDED_PDF_FILTERS
#: The names in `bounded_pdf_filters`, as a constant the test suite pins. The
#: docstring above is the published claim; this is what the code enforces, and
#: `_pdf_filter_classes` is the same three classes as pdfminer literals.
#: `_pdf_filter_names` maps every pdfminer spelling of these onto the canonical
#: name that `assets.PDF_FILTER_COST_RATIO` budgets.
_BOUNDED_PDF_FILTERS = frozenset(
{
"FlateDecode",
@ -1420,12 +1439,14 @@ _BOUNDED_PDF_FILTERS = frozenset(
)
def _pdf_filter_classes() -> tuple[frozenset[object], frozenset[object], frozenset[object]]:
"""The three classes as pdfminer literals: measured, shrinking, unchanged.
def _pdf_filter_names() -> dict[object, str]:
"""Every pdfminer literal this package bounds, mapped to its CANONICAL name.
Read from pdfminer rather than written out here, because a filter has more
than one spelling (`/Fl` is `/FlateDecode`) and a set of names written by
hand would refuse the abbreviation a real document uses.
hand would refuse the abbreviation a real document uses. The canonical name
is the key into `assets.PDF_FILTER_COST_RATIO`, so both spellings of a
filter are budgeted by one measured number.
"""
from pdfminer.pdftypes import (
LITERALS_ASCII85_DECODE,
@ -1436,13 +1457,26 @@ def _pdf_filter_classes() -> tuple[frozenset[object], frozenset[object], frozens
LITERALS_JPX_DECODE,
)
return (
frozenset(LITERALS_FLATE_DECODE),
frozenset(LITERALS_ASCII85_DECODE) | frozenset(LITERALS_ASCIIHEX_DECODE),
frozenset(LITERALS_DCT_DECODE)
| frozenset(LITERALS_JPX_DECODE)
| frozenset(LITERALS_JBIG2_DECODE),
)
families = {
"FlateDecode": LITERALS_FLATE_DECODE,
"ASCII85Decode": LITERALS_ASCII85_DECODE,
"ASCIIHexDecode": LITERALS_ASCIIHEX_DECODE,
"DCTDecode": LITERALS_DCT_DECODE,
"JPXDecode": LITERALS_JPX_DECODE,
"JBIG2Decode": LITERALS_JBIG2_DECODE,
}
return {
literal: canonical
for canonical, literals in families.items()
for literal in literals
if canonical in _BOUNDED_PDF_FILTERS
}
#: The two canonical names whose decoders produce fewer or more bytes than they
#: were given, and which this package therefore has to run to learn the size of
#: the link behind them. Everything else in the table is pass-through.
_SHRINKING_FILTER_NAMES = frozenset({"ASCII85Decode", "ASCIIHexDecode"})
def _pdf_stream_bytes(stream: object, name: str) -> bytes | None:
@ -1504,13 +1538,30 @@ def _check_stream_cost(stream: object, name: str) -> None:
corpus objects behind an `[/ASCII85Decode /FlateDecode]` chain unmeasured,
because `filters[0]` is not `FlateDecode` there.
So every link is walked, in order. An unknown or unmeasurable one is
refused BEFORE anything is decoded (`bounded_pdf_filters` says which, and
why). A `FlateDecode` that is the last expanding link is measured and its
output discarded, which is the common case and costs exactly what 0.10.1
cost. A `FlateDecode` with another expanding link behind it is inflated
under the same bound and handed on, so the link behind it can be measured
too -- what is held is never more than the bound.
AND THE COST OF A LINK, NOT THE SIZE OF ITS OUTPUT. Bounding every
`FlateDecode` was still not a bound, because the bomb moved into a link
0.10.1 had documented as safe: `ASCII85Decode`'s `z` is the shorthand for
four zero bytes, and its decoder holds about a hundred bytes per byte of
input. Measured on the pinned tree: 33 475 bytes of file cost
3 261 599 744 bytes of peak RSS and the picture was CARRIED. Three rounds
of this review each bound an OUTPUT and the bomb stepped one link along;
what they had in common is that a decoder's working set is not its output.
So every link is walked, in order, and each is given a BUDGET
(`assets.MAX_FILTER_DECODE_BYTES`) rather than a class:
* a filter with no measured cost ratio is refused BEFORE anything is
decoded (`bounded_pdf_filters` says which, and why);
* a `FlateDecode` is measured as it is paid, under a limit that is the
smaller of the picture's own bound and what the NEXT link's decoder may
be handed -- which is how the budget travels down the chain instead of
being applied to each link in isolation;
* the last `FlateDecode` in the chain has its output counted and thrown
away, which is the common case and costs exactly what 0.10.1 cost; an
earlier one is inflated under the same limit and handed on, so the link
behind it can be measured from real bytes;
* every other filter has its cost PREDICTED from its input size and its
measured ratio, and is refused before its decoder is called.
WHAT THIS STILL DOES NOT BOUND, stated rather than implied: a stream
something else has already decoded (`_pdf_stream_bytes` returns `None`),
@ -1518,46 +1569,91 @@ def _check_stream_cost(stream: object, name: str) -> None:
by `check_payload` AFTER `get_data()`, which makes it a counted refusal
rather than a bounded one.
"""
flate, shrinking, pass_through = _pdf_filter_classes()
names = _pdf_filter_names()
data = _pdf_stream_bytes(stream, name)
if data is None:
return
filters = stream.get_filters() if hasattr(stream, "get_filters") else []
raw_filters = stream.get_filters() if hasattr(stream, "get_filters") else []
# THE WHOLE CHAIN IS READ BEFORE THE FIRST LINK IS DECODED. A filter this
# package cannot bound must be refused without having paid for the links in
# front of it, which is only possible if the refusal is decided up front.
for literal, _params in filters:
if literal not in flate and literal not in shrinking and literal not in pass_through:
chain: list[tuple[str, object]] = []
for literal, params in raw_filters:
canonical = names.get(literal)
if canonical is None:
raise ExtractionError(
f"the image {name!r} is decoded through {literal}, a filter whose "
"output this package cannot measure before producing it; refused "
"unread rather than decoded to find out what it costs",
"cost this package has no measured ratio for; refused unread "
"rather than decoded to find out what it costs",
code="asset_pdf_unbounded",
)
for index, (literal, params) in enumerate(filters):
if literal in flate:
if not any(behind in flate for behind, _ in filters[index + 1 :]):
inflated_size(data, name=name)
return
if _has_predictor(params):
raise ExtractionError(
f"the image {name!r} applies a predictor to a link that is not the "
"last one, so the bytes this package would hand to the next filter "
"are not the bytes pdfminer decodes; refused unread",
code="asset_pdf_unbounded",
)
data = inflate_bounded(data, name=name)
elif literal in shrinking:
try:
data = _shrink(literal, data)
except Exception:
# Not this function's problem: a stream that is not valid input
# for its own filter is reported by the reader behind it, in
# that reader's vocabulary. Nothing can expand from it here.
return
# A pass-through filter leaves the bytes exactly as they are.
check_payload(len(data), name=name)
chain.append((canonical, params))
# The LAST deflate link is the one whose bytes nothing behind has to be
# measured from, so it is counted and thrown away; every earlier one is
# inflated under the same bound and handed on. Deciding this by index
# rather than by a running flag is what keeps "the bytes are gone" and "a
# link still needs them" from ever being true at once.
last_flate = max(
(index for index, (canonical, _) in enumerate(chain) if canonical == "FlateDecode"),
default=-1,
)
size = len(data)
for index, (canonical, params) in enumerate(chain):
behind = [name_behind for name_behind, _ in chain[index + 1 :]]
if canonical == "FlateDecode":
limit = inflate_limit_for(behind[0] if behind else None)
if index == last_flate:
# Nothing behind has to be measured, so the output is counted
# and thrown away: the cheap common case, and what 0.10.1 cost.
size = inflated_size(data, name=name, limit=limit)
data = b""
else:
if _has_predictor(params):
raise ExtractionError(
f"the image {name!r} applies a predictor to a link that is not "
"the last one, so the bytes this package would hand to the next "
"filter are not the bytes pdfminer decodes; refused unread",
code="asset_pdf_unbounded",
)
data = inflate_bounded(data, name=name, limit=limit)
size = len(data)
else:
# PREDICTED, not measured, and predicted BEFORE the decoder is
# called: these decoders take a whole string and return a whole
# string, so there is no moment between the two at which the cost
# could be observed.
check_filter_cost(size, canonical=canonical, name=name)
if canonical in _SHRINKING_FILTER_NAMES:
if index > last_flate >= 0:
# The bytes were discarded at the last deflate link, so the
# bound travels on as the WIDEST this link could produce.
size = _widest_output(canonical, size)
else:
try:
data = _shrink(canonical, data)
except Exception:
# Not this function's problem: a stream that is not
# valid input for its own filter is reported by the
# reader behind it, in that reader's vocabulary.
return
size = len(data)
# A pass-through filter leaves the bytes exactly as they are.
check_payload(size, name=name)
def _widest_output(canonical: str, size: int) -> int:
"""The most `canonical` can produce from `size` bytes, for a link whose
bytes were discarded and whose SIZE is all that is carried forward.
Rounded up rather than down, and never below one byte: a bound that is
optimistic by a byte is not a bound.
"""
ratio = PDF_FILTER_OUTPUT_RATIO.get(canonical)
if ratio is None: # pragma: no cover - only `FlateDecode`, handled above
return size
return max(1, math.ceil(size * ratio))
def _has_predictor(params: object) -> bool:
@ -1570,13 +1666,19 @@ def _has_predictor(params: object) -> bool:
return isinstance(predictor, int) and predictor > 1
def _shrink(literal: object, data: bytes) -> bytes:
"""The two filters whose output is smaller than their input, decoded with
pdfminer's own readers so both sides agree on what the bytes are."""
from pdfminer.ascii85 import ascii85decode, asciihexdecode
from pdfminer.pdftypes import LITERALS_ASCII85_DECODE
def _shrink(canonical: str, data: bytes) -> bytes:
"""The two ASCII filters, decoded with pdfminer's own readers so both sides
agree on what the bytes are.
if literal in LITERALS_ASCII85_DECODE:
Called only after `check_filter_cost` has allowed the input size, which is
what makes handing a whole string to a decoder that returns a whole string
a bounded thing to do. The name "shrink" is kept for the pair, but only
`ASCIIHexDecode` actually shrinks: `ASCII85Decode` can quadruple its input,
which is the defect this round was opened for.
"""
from pdfminer.ascii85 import ascii85decode, asciihexdecode
if canonical == "ASCII85Decode":
return ascii85decode(data)
return asciihexdecode(data)

103
tests/test_asset_limits.py
View file

@ -819,16 +819,32 @@ def _filters_entry(chain: tuple[str, ...]) -> str:
return "[" + " ".join("/" + name for name in chain) + "]"
def _z_run(count: int) -> bytes:
"""`count` ASCII85 `z` characters -- the shorthand for four zero bytes --
written out directly rather than produced by `a85encode`.
The ENCODER costs about forty bytes of memory per byte of input, so a
fixture built with it is what a peak-RSS measurement would measure. The
same reason `_zeros_stream` deflates without ever holding the zeros.
"""
return b"<~" + b"z" * count + b"~>"
def _z_chain_stream(chain: tuple[str, ...], count: int) -> bytes:
"""The stream a document must hold for `chain` to hand a run of `count`
`z` characters to its `ASCII85Decode` link."""
stream = _z_run(count)
for literal in reversed(chain[: chain.index("ASCII85Decode")]):
stream = _encode_for(literal, stream)
return stream
_CHAIN_CHILD = """
import resource, sys
sys.path.insert(0, {tests!r})
from test_asset_limits import _bomb, _chain_stream, _filters_entry
from llm_ingestion_okf.extract import extract_document
chain = {chain!r}
document = _bomb(
1, payload=_chain_stream(chain, b"\\x00" * {payload}), filters=_filters_entry(chain)
)
document = open({path!r}, "rb").read()
extracted = extract_document("bomb.pdf", document, assets=True)
peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print(
@ -840,14 +856,27 @@ print(
"""
def _run_chain(chain: tuple[str, ...], *, payload: int) -> tuple[int, int, str, int]:
def _run_chain(
chain: tuple[str, ...], *, payload: int, tmp_path: Path
) -> tuple[int, int, str, int]:
"""One chain's bomb in its own interpreter, so peak RSS is ITS peak and not
the high-water mark of every test that ran before it."""
the high-water mark of every test that ran before it. `payload` is how many
`z` characters the chain's `ASCII85Decode` link is handed.
The document is built HERE and handed over as a file. `ru_maxrss` is a high
water mark that never falls, so a child that builds its own fixture reports
the fixture: `b"z" * (64 MiB)` alone costs 171 MB, which is more than the
bounded extraction it would be measuring.
"""
document = tmp_path / "bomb.pdf"
document.write_bytes(
_bomb(1, payload=_z_chain_stream(chain, payload), filters=_filters_entry(chain))
)
completed = subprocess.run(
[
sys.executable,
"-c",
_CHAIN_CHILD.format(tests=str(Path(__file__).parent), chain=chain, payload=payload),
_CHAIN_CHILD.format(tests=str(Path(__file__).parent), path=str(document)),
],
capture_output=True,
text=True,
@ -857,11 +886,11 @@ def _run_chain(chain: tuple[str, ...], *, payload: int) -> tuple[int, int, str,
return int(size), int(carried), codes, int(peak)
def test_an_ascii85_link_behind_a_flate_link_is_bounded_too() -> None:
def test_an_ascii85_link_behind_a_flate_link_is_bounded_too(tmp_path: Path) -> None:
"""The round-3 BLOCKER at the shipped bound, in its own interpreter."""
pytest.importorskip("pdfplumber")
size, carried, codes, peak = _run_chain(
("FlateDecode", "ASCII85Decode"), payload=128 * 1024 * 1024
("FlateDecode", "ASCII85Decode"), payload=32 * 1024 * 1024, tmp_path=tmp_path
)
assert size < 2 * 1024 * 1024, "the fixture must stay a small file, or it proves nothing"
assert carried == 0, "a run of `z` was carried as a 1x1 picture"
@ -869,11 +898,13 @@ def test_an_ascii85_link_behind_a_flate_link_is_bounded_too() -> None:
assert peak < PEAK_RSS_BOUND, f"peak RSS {peak} bytes for a {size}-byte file"
def test_an_ascii85_link_on_its_own_is_bounded() -> None:
def test_an_ascii85_link_on_its_own_is_bounded(tmp_path: Path) -> None:
"""The same amplification with no filter in front of it: the stream IS the
run of `z`, so the cost must not be a multiple of the file."""
pytest.importorskip("pdfplumber")
size, carried, codes, peak = _run_chain(("ASCII85Decode",), payload=32 * 1024 * 1024)
size, carried, codes, peak = _run_chain(
("ASCII85Decode",), payload=8 * 1024 * 1024, tmp_path=tmp_path
)
assert carried == 0
assert codes in ASSET_REJECTION_CODES, codes
assert peak < PEAK_RSS_BOUND, f"peak RSS {peak} bytes for a {size}-byte file"
@ -1082,3 +1113,51 @@ def test_the_stream_bound_refuses_before_get_data_is_ever_called() -> None:
monkey.undo()
assert excinfo.value.code == "asset_too_large"
assert calls == [], "refused only after the stream was decoded, which is the backstop"
def test_the_budget_travels_to_the_next_link() -> None:
"""A link's output is the NEXT link's input, so a deflate link is bounded
by what the decoder behind it may be handed -- not by the picture's own
bound alone.
Without this, `[/FlateDecode /ASCII85Decode]` inflates `MAX_IMAGE_BYTES` of
`z` and only then asks whether the link behind it can afford them, which
is a 256 MiB payment for a refusal. Measured: making `inflate_limit_for`
return `MAX_IMAGE_BYTES` unconditionally left the whole suite green, so
nothing held this rule until now.
"""
cap = assets.filter_input_limit("ASCII85Decode")
assert cap is not None
assert cap < assets.MAX_IMAGE_BYTES, "the cap has to bind, or there is nothing to travel"
assert assets.inflate_limit_for("ASCII85Decode") == cap
assert assets.inflate_limit_for(None) == assets.MAX_IMAGE_BYTES
assert assets.inflate_limit_for("FlateDecode") == assets.MAX_IMAGE_BYTES, (
"a deflate link behind is measured as it is paid, so it caps nothing in front"
)
def test_a_discarded_links_size_travels_as_the_widest_it_could_become() -> None:
"""When the bytes are thrown away at the last deflate link, what travels on
is a SIZE, and it has to be the widest the links behind could make of it.
`ASCII85Decode` quadruples in the worst case and `ASCIIHexDecode` halves,
so the two go opposite ways and a rule that carried the size unchanged
would be optimistic for the first. Today that optimism cannot reach the
bound -- the cost cap already holds an ASCII85 link's input under about
5.0 MB, and four times that is well under `MAX_IMAGE_BYTES` -- so this is
pinned directly rather than through a document, and the arithmetic that
makes it unreachable is pinned beside it. If either constant moves, the
second assertion says the bound started binding.
"""
from llm_ingestion_okf import extract as extract_module
assert extract_module._widest_output("ASCII85Decode", 10) == 40
assert extract_module._widest_output("ASCIIHexDecode", 10) == 5
assert extract_module._widest_output("DCTDecode", 10) == 10
assert extract_module._widest_output("ASCII85Decode", 0) == 1, "never optimistic by a byte"
cap = assets.filter_input_limit("ASCII85Decode")
assert cap is not None
assert extract_module._widest_output("ASCII85Decode", cap) < assets.MAX_IMAGE_BYTES, (
"the widest an ASCII85 link can produce now reaches the picture bound, so this "
"rule has started to bind and needs a document behind it, not only arithmetic"
)