fix(assets): bound every link of the filter chain, and cover the backstop

The two findings of the 18.09 PM checkpoint of `0f308c1`. Red tests landed
first in `3b587ea`; this is what turns them green.

BLOCKER -- `_check_inflated` read `filters[0]`, measured that one link and
returned, which is not a bound: a PDF decodes a stream through a LIST of
filters. Measured in paired subprocesses from two pinned trees, idle machine:

  [/FlateDecode]        400 MB  408 516 B   59 232 256 ->    62 017 536 B
  [/FlateDecode x2]     400 MB    1 636 B  886 554 624 ->    52 367 360 B
  [/FlateDecode x3]     400 MB    1 070 B  889 393 152 ->    61 390 848 B
  [/FlateDecode x2]     1,2 GB    2 927 B  2 567 204 864 ->  60 403 712 B

542 000x the file at two links, and the picture WAS refused at the end -- by
`check_payload` after `get_data()`, once the memory was spent. The single-link
row is the control and does not move. It also left the 16 corpus objects behind
an `[/ASCII85Decode /FlateDecode]` chain unmeasured, since `filters[0]` is not
`FlateDecode` there.

`_check_stream_cost` walks every link. THREE CLASSES and no fourth
(`extract.bounded_pdf_filters`, pinned by a test): `FlateDecode` MEASURED, a
link with another expanding link behind it inflated under the same bound and
handed on; `ASCII85Decode`/`ASCIIHexDecode` bounded by their own input because
they SHRINK; `DCTDecode`/`JPXDecode`/`JBIG2Decode` PASS THROUGH. Everything
else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything
written later -- is refused UNREAD with a new code `asset_pdf_unbounded`,
decided before the FIRST link is decoded so a document cannot make the run pay
for the links in front of the one we cannot bound. An encrypted stream is
deciphered and then measured, where `stream.decipher is not None` used to
return unmeasured; 0 of 5 142 objects here are in an encrypted document, which
is why nothing caught it.

NOT ONE PICTURE CHANGES HANDS, AND IT IS MEASURED BY NAME. Every PDF on this
machine -- 78 documents, K2 in both trinn1 and trinn2, the shipped fixtures and
R761 -- run through `_pdf_images` page by page from both pinned trees:

  images carried          9 356 -> 9 356
  documents losing one              0 of 78
  documents gaining one             0 of 78
  asset_pdf_unsupported     322 -> 314
  asset_pdf_unbounded         0 -> 8

The 8 are the 4 `CCITTFaxDecode` stencil masks (`/ImageMask true`,
`/BitsPerComponent 1`), counted twice because trinn1 and trinn2 hold the same
document. They were refused before and are refused now, one step earlier and
under a code that says why.

MAJOR -- `check_payload(len(data))` after `get_data()` is the counted refusal
four documentation surfaces point at, and deleting exactly that line passed all
2 132 tests. It is reachable through a stream pdfminer has ALREADY decoded
(`decode()` sets `rawdata` to `None`), which is now the ONLY case outside the
bound and has a test.

Eight mutations, one line each, every one DEAD, with the unmutated tree run
first as the control: first-link-only, loop dropped, inequality reversed,
encrypted skipped, backstop deleted, unknown filter passed through,
intermediate link not carried forward, whole check removed.

`tools/okf_accounting_gate.py` gains one line, the new code in
`REJECTION_CODES` -- what a rejection code requires and nothing more. Gate
unchanged: exit 1, GATE RED rows 2, 3, 6. Version stays 0.10.1, untagged.

Suite after `git add` against a clean tree: `uv run pytest -q` ->
2152 passed, 1 skipped (226 s). ruff, ruff format --check, mypy --strict clean.

Report: docs/2026-09-18-filterkjeden-og-backstoppen.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 15:34:07 +02:00
commit 0c3c4904ee
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
11 changed files with 564 additions and 38 deletions

View file

@ -53,6 +53,7 @@ import re
import struct
import unicodedata
import zlib
from collections.abc import Iterator
from dataclasses import dataclass
from .errors import ExtractionError
@ -220,6 +221,31 @@ def inflated_size(raw: bytes, *, name: str, limit: int | None = None) -> int:
reader behind it reports that in its own vocabulary, so a `zlib.error` ends
the measurement at whatever was produced up to it.
"""
return sum(len(chunk) for chunk in _inflate(raw, name=name, limit=limit))
def inflate_bounded(raw: bytes, *, name: str, limit: int | None = None) -> bytes:
"""The same measurement, KEEPING the output rather than discarding it.
One chain link is not the end of a chain: a PDF may decode a stream through
`/Filter [/FlateDecode /FlateDecode]`, and measuring the first link says
nothing about the second, which is where the cost is. To bound the second
link the first one's bytes have to exist, so this inflates under the same
running bound and hands the result on. It refuses at the same point
`inflated_size` does, so what is held is never more than the bound -- which
is what separates carrying an intermediate stage from paying for a bomb.
Measured 2026-09-18: 400 MB of zeros deflated twice is 795 bytes of stream,
and the first link of that chain inflates to 407 685 bytes. The whole
expansion lives in the LAST link, which is the one nothing measured.
"""
return b"".join(_inflate(raw, name=name, limit=limit))
def _inflate(raw: bytes, *, name: str, limit: int | None) -> Iterator[bytes]:
"""Inflate `raw` a chunk at a time, refusing the moment the running total
crosses the bound. The two callers differ only in whether they keep what
comes out."""
bound = MAX_IMAGE_BYTES if limit is None else limit
decompressor = zlib.decompressobj()
total = 0
@ -236,12 +262,12 @@ def inflated_size(raw: bytes, *, name: str, limit: int | None = None) -> int:
"container declares is a claim and this is the cost",
code="asset_too_large",
)
yield produced
pending = decompressor.unconsumed_tail
if decompressor.eof or not pending:
break
except zlib.error:
return total
return total
return
@dataclass(frozen=True)