fix(assets): bound every link of the filter chain, and cover the backstop

The two findings of the 18.09 PM checkpoint of `0f308c1`. Red tests landed
first in `3b587ea`; this is what turns them green.

BLOCKER -- `_check_inflated` read `filters[0]`, measured that one link and
returned, which is not a bound: a PDF decodes a stream through a LIST of
filters. Measured in paired subprocesses from two pinned trees, idle machine:

  [/FlateDecode]        400 MB  408 516 B   59 232 256 ->    62 017 536 B
  [/FlateDecode x2]     400 MB    1 636 B  886 554 624 ->    52 367 360 B
  [/FlateDecode x3]     400 MB    1 070 B  889 393 152 ->    61 390 848 B
  [/FlateDecode x2]     1,2 GB    2 927 B  2 567 204 864 ->  60 403 712 B

542 000x the file at two links, and the picture WAS refused at the end -- by
`check_payload` after `get_data()`, once the memory was spent. The single-link
row is the control and does not move. It also left the 16 corpus objects behind
an `[/ASCII85Decode /FlateDecode]` chain unmeasured, since `filters[0]` is not
`FlateDecode` there.

`_check_stream_cost` walks every link. THREE CLASSES and no fourth
(`extract.bounded_pdf_filters`, pinned by a test): `FlateDecode` MEASURED, a
link with another expanding link behind it inflated under the same bound and
handed on; `ASCII85Decode`/`ASCIIHexDecode` bounded by their own input because
they SHRINK; `DCTDecode`/`JPXDecode`/`JBIG2Decode` PASS THROUGH. Everything
else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything
written later -- is refused UNREAD with a new code `asset_pdf_unbounded`,
decided before the FIRST link is decoded so a document cannot make the run pay
for the links in front of the one we cannot bound. An encrypted stream is
deciphered and then measured, where `stream.decipher is not None` used to
return unmeasured; 0 of 5 142 objects here are in an encrypted document, which
is why nothing caught it.

NOT ONE PICTURE CHANGES HANDS, AND IT IS MEASURED BY NAME. Every PDF on this
machine -- 78 documents, K2 in both trinn1 and trinn2, the shipped fixtures and
R761 -- run through `_pdf_images` page by page from both pinned trees:

  images carried          9 356 -> 9 356
  documents losing one              0 of 78
  documents gaining one             0 of 78
  asset_pdf_unsupported     322 -> 314
  asset_pdf_unbounded         0 -> 8

The 8 are the 4 `CCITTFaxDecode` stencil masks (`/ImageMask true`,
`/BitsPerComponent 1`), counted twice because trinn1 and trinn2 hold the same
document. They were refused before and are refused now, one step earlier and
under a code that says why.

MAJOR -- `check_payload(len(data))` after `get_data()` is the counted refusal
four documentation surfaces point at, and deleting exactly that line passed all
2 132 tests. It is reachable through a stream pdfminer has ALREADY decoded
(`decode()` sets `rawdata` to `None`), which is now the ONLY case outside the
bound and has a test.

Eight mutations, one line each, every one DEAD, with the unmutated tree run
first as the control: first-link-only, loop dropped, inequality reversed,
encrypted skipped, backstop deleted, unknown filter passed through,
intermediate link not carried forward, whole check removed.

`tools/okf_accounting_gate.py` gains one line, the new code in
`REJECTION_CODES` -- what a rejection code requires and nothing more. Gate
unchanged: exit 1, GATE RED rows 2, 3, 6. Version stays 0.10.1, untagged.

Suite after `git add` against a clean tree: `uv run pytest -q` ->
2152 passed, 1 skipped (226 s). ruff, ruff format --check, mypy --strict clean.

Report: docs/2026-09-18-filterkjeden-og-backstoppen.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 15:34:07 +02:00
commit 0c3c4904ee
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
11 changed files with 564 additions and 38 deletions

View file

@ -924,11 +924,41 @@ and fixtures, never code.
it), and what a PDF image's STREAM decompresses to -- measured a chunk at
a time and DISCARDED, before `get_data()`. With it: **0 carried,
`asset_too_large`, 54 MB** at 400 MB and **62 MB** at 1,2 GB, so the cost
no longer scales with the bomb. **The limit is stated rather than
implied**: the stream measurement runs where `FlateDecode` is the first
filter and the document is not encrypted; every other chain is a check on
the decoded length AFTER the decode, which is a COUNTED refusal and not a
bounded one.
no longer scales with the bomb.
- **THE CHAIN, NOT ITS FIRST LINK.** That fix read `filters[0]`, and a PDF
decodes a stream through a LIST of filters: measured 18.09 on `0f308c1`,
`/Filter [/FlateDecode /FlateDecode]` cost **886 554 624 B peak RSS from
1 636 B of file** and three links the same from 1 070 B -- about
542 000x -- with the picture still refused at the end, by the backstop,
after the memory was spent. It also left the **16** corpus objects behind
an `[/ASCII85Decode /FlateDecode]` chain unmeasured, since `filters[0]` is
not `FlateDecode` there. Bounded (idle machine, paired subprocesses):
**52 367 360 B** at two links, **61 390 848 B** at three, **60 403 712 B**
where the old path cost 2 567 204 864 B, and the single-link control
unmoved at 59 232 256 -> 62 017 536. **Three classes and no fourth**
(`extract.bounded_pdf_filters`, pinned by a test): `FlateDecode` MEASURED,
`ASCII85Decode`/`ASCIIHexDecode` bounded by their own input because they
SHRINK, `DCTDecode`/`JPXDecode`/`JBIG2Decode` PASS THROUGH. Everything
else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`,
anything written later -- is refused UNREAD with its own code
`asset_pdf_unbounded`, the same decision `corpus.resolve_gate` takes for
an unknown gate name. Cost measured over the **5 142** image objects of
78 PDFs: the refused class is **4** `CCITTFaxDecode` objects, all 1-bit
stencil masks already refused one step later by the encoder, and **0**
objects in an encrypted document. **NOT ONE PICTURE CHANGES HANDS AND IT
IS MEASURED BY NAME**: over the same 78 PDFs, carried images are
**9 356 -> 9 356**, **0 of 78** documents lose one and 0 gain one; the
only movement is **8** objects going `asset_pdf_unsupported` ->
`asset_pdf_unbounded` (the 4 masks, counted twice because `trinn1` and
`trinn2` hold the same document). An ENCRYPTED stream is
now deciphered and then measured (deciphering does not change a length),
where `stream.decipher is not None` used to return unmeasured.
- **THE LIMIT IS STATED RATHER THAN IMPLIED, and it is now ONE case**: a
stream something else has ALREADY decoded, where the memory is spent
before this package is asked. `check_payload(len(data))` after
`get_data()` COUNTS it -- a counted refusal, not a bounded one. Deleting
exactly that line passed all **2 132** tests on `0f308c1`; it has a test
now.
- **A non-positive declared dimension is `asset_size_invalid`**, its own
code, raised before the stream is read. `-1 x 40 000 000 000` is a
NEGATIVE pixel count, under which every `>` bound reads as satisfied: the