fix(assets): bound every link of the filter chain, and cover the backstop
The two findings of the 18.09 PM checkpoint of `0f308c1`. Red tests landed first in `3b587ea`; this is what turns them green. BLOCKER -- `_check_inflated` read `filters[0]`, measured that one link and returned, which is not a bound: a PDF decodes a stream through a LIST of filters. Measured in paired subprocesses from two pinned trees, idle machine: [/FlateDecode] 400 MB 408 516 B 59 232 256 -> 62 017 536 B [/FlateDecode x2] 400 MB 1 636 B 886 554 624 -> 52 367 360 B [/FlateDecode x3] 400 MB 1 070 B 889 393 152 -> 61 390 848 B [/FlateDecode x2] 1,2 GB 2 927 B 2 567 204 864 -> 60 403 712 B 542 000x the file at two links, and the picture WAS refused at the end -- by `check_payload` after `get_data()`, once the memory was spent. The single-link row is the control and does not move. It also left the 16 corpus objects behind an `[/ASCII85Decode /FlateDecode]` chain unmeasured, since `filters[0]` is not `FlateDecode` there. `_check_stream_cost` walks every link. THREE CLASSES and no fourth (`extract.bounded_pdf_filters`, pinned by a test): `FlateDecode` MEASURED, a link with another expanding link behind it inflated under the same bound and handed on; `ASCII85Decode`/`ASCIIHexDecode` bounded by their own input because they SHRINK; `DCTDecode`/`JPXDecode`/`JBIG2Decode` PASS THROUGH. Everything else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything written later -- is refused UNREAD with a new code `asset_pdf_unbounded`, decided before the FIRST link is decoded so a document cannot make the run pay for the links in front of the one we cannot bound. An encrypted stream is deciphered and then measured, where `stream.decipher is not None` used to return unmeasured; 0 of 5 142 objects here are in an encrypted document, which is why nothing caught it. NOT ONE PICTURE CHANGES HANDS, AND IT IS MEASURED BY NAME. Every PDF on this machine -- 78 documents, K2 in both trinn1 and trinn2, the shipped fixtures and R761 -- run through `_pdf_images` page by page from both pinned trees: images carried 9 356 -> 9 356 documents losing one 0 of 78 documents gaining one 0 of 78 asset_pdf_unsupported 322 -> 314 asset_pdf_unbounded 0 -> 8 The 8 are the 4 `CCITTFaxDecode` stencil masks (`/ImageMask true`, `/BitsPerComponent 1`), counted twice because trinn1 and trinn2 hold the same document. They were refused before and are refused now, one step earlier and under a code that says why. MAJOR -- `check_payload(len(data))` after `get_data()` is the counted refusal four documentation surfaces point at, and deleting exactly that line passed all 2 132 tests. It is reachable through a stream pdfminer has ALREADY decoded (`decode()` sets `rawdata` to `None`), which is now the ONLY case outside the bound and has a test. Eight mutations, one line each, every one DEAD, with the unmutated tree run first as the control: first-link-only, loop dropped, inequality reversed, encrypted skipped, backstop deleted, unknown filter passed through, intermediate link not carried forward, whole check removed. `tools/okf_accounting_gate.py` gains one line, the new code in `REJECTION_CODES` -- what a rejection code requires and nothing more. Gate unchanged: exit 1, GATE RED rows 2, 3, 6. Version stays 0.10.1, untagged. Suite after `git add` against a clean tree: `uv run pytest -q` -> 2152 passed, 1 skipped (226 s). ruff, ruff format --check, mypy --strict clean. Report: docs/2026-09-18-filterkjeden-og-backstoppen.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
3b587ea567
commit
0c3c4904ee
11 changed files with 564 additions and 38 deletions
|
|
@ -53,6 +53,7 @@ import re
|
|||
import struct
|
||||
import unicodedata
|
||||
import zlib
|
||||
from collections.abc import Iterator
|
||||
from dataclasses import dataclass
|
||||
|
||||
from .errors import ExtractionError
|
||||
|
|
@ -220,6 +221,31 @@ def inflated_size(raw: bytes, *, name: str, limit: int | None = None) -> int:
|
|||
reader behind it reports that in its own vocabulary, so a `zlib.error` ends
|
||||
the measurement at whatever was produced up to it.
|
||||
"""
|
||||
return sum(len(chunk) for chunk in _inflate(raw, name=name, limit=limit))
|
||||
|
||||
|
||||
def inflate_bounded(raw: bytes, *, name: str, limit: int | None = None) -> bytes:
|
||||
"""The same measurement, KEEPING the output rather than discarding it.
|
||||
|
||||
One chain link is not the end of a chain: a PDF may decode a stream through
|
||||
`/Filter [/FlateDecode /FlateDecode]`, and measuring the first link says
|
||||
nothing about the second, which is where the cost is. To bound the second
|
||||
link the first one's bytes have to exist, so this inflates under the same
|
||||
running bound and hands the result on. It refuses at the same point
|
||||
`inflated_size` does, so what is held is never more than the bound -- which
|
||||
is what separates carrying an intermediate stage from paying for a bomb.
|
||||
|
||||
Measured 2026-09-18: 400 MB of zeros deflated twice is 795 bytes of stream,
|
||||
and the first link of that chain inflates to 407 685 bytes. The whole
|
||||
expansion lives in the LAST link, which is the one nothing measured.
|
||||
"""
|
||||
return b"".join(_inflate(raw, name=name, limit=limit))
|
||||
|
||||
|
||||
def _inflate(raw: bytes, *, name: str, limit: int | None) -> Iterator[bytes]:
|
||||
"""Inflate `raw` a chunk at a time, refusing the moment the running total
|
||||
crosses the bound. The two callers differ only in whether they keep what
|
||||
comes out."""
|
||||
bound = MAX_IMAGE_BYTES if limit is None else limit
|
||||
decompressor = zlib.decompressobj()
|
||||
total = 0
|
||||
|
|
@ -236,12 +262,12 @@ def inflated_size(raw: bytes, *, name: str, limit: int | None = None) -> int:
|
|||
"container declares is a claim and this is the cost",
|
||||
code="asset_too_large",
|
||||
)
|
||||
yield produced
|
||||
pending = decompressor.unconsumed_tail
|
||||
if decompressor.eof or not pending:
|
||||
break
|
||||
except zlib.error:
|
||||
return total
|
||||
return total
|
||||
return
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue