open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(assets): bound every link of the filter chain, and cover the backstop

Browse source 
The two findings of the 18.09 PM checkpoint of `0f308c1`. Red tests landed
first in `3b587ea`; this is what turns them green.

BLOCKER -- `_check_inflated` read `filters[0]`, measured that one link and
returned, which is not a bound: a PDF decodes a stream through a LIST of
filters. Measured in paired subprocesses from two pinned trees, idle machine:

  [/FlateDecode]        400 MB  408 516 B   59 232 256 ->    62 017 536 B
  [/FlateDecode x2]     400 MB    1 636 B  886 554 624 ->    52 367 360 B
  [/FlateDecode x3]     400 MB    1 070 B  889 393 152 ->    61 390 848 B
  [/FlateDecode x2]     1,2 GB    2 927 B  2 567 204 864 ->  60 403 712 B

542 000x the file at two links, and the picture WAS refused at the end -- by
`check_payload` after `get_data()`, once the memory was spent. The single-link
row is the control and does not move. It also left the 16 corpus objects behind
an `[/ASCII85Decode /FlateDecode]` chain unmeasured, since `filters[0]` is not
`FlateDecode` there.

`_check_stream_cost` walks every link. THREE CLASSES and no fourth
(`extract.bounded_pdf_filters`, pinned by a test): `FlateDecode` MEASURED, a
link with another expanding link behind it inflated under the same bound and
handed on; `ASCII85Decode`/`ASCIIHexDecode` bounded by their own input because
they SHRINK; `DCTDecode`/`JPXDecode`/`JBIG2Decode` PASS THROUGH. Everything
else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything
written later -- is refused UNREAD with a new code `asset_pdf_unbounded`,
decided before the FIRST link is decoded so a document cannot make the run pay
for the links in front of the one we cannot bound. An encrypted stream is
deciphered and then measured, where `stream.decipher is not None` used to
return unmeasured; 0 of 5 142 objects here are in an encrypted document, which
is why nothing caught it.

NOT ONE PICTURE CHANGES HANDS, AND IT IS MEASURED BY NAME. Every PDF on this
machine -- 78 documents, K2 in both trinn1 and trinn2, the shipped fixtures and
R761 -- run through `_pdf_images` page by page from both pinned trees:

  images carried          9 356 -> 9 356
  documents losing one              0 of 78
  documents gaining one             0 of 78
  asset_pdf_unsupported     322 -> 314
  asset_pdf_unbounded         0 -> 8

The 8 are the 4 `CCITTFaxDecode` stencil masks (`/ImageMask true`,
`/BitsPerComponent 1`), counted twice because trinn1 and trinn2 hold the same
document. They were refused before and are refused now, one step earlier and
under a code that says why.

MAJOR -- `check_payload(len(data))` after `get_data()` is the counted refusal
four documentation surfaces point at, and deleting exactly that line passed all
2 132 tests. It is reachable through a stream pdfminer has ALREADY decoded
(`decode()` sets `rawdata` to `None`), which is now the ONLY case outside the
bound and has a test.

Eight mutations, one line each, every one DEAD, with the unmutated tree run
first as the control: first-link-only, loop dropped, inequality reversed,
encrypted skipped, backstop deleted, unknown filter passed through,
intermediate link not carried forward, whole check removed.

`tools/okf_accounting_gate.py` gains one line, the new code in
`REJECTION_CODES` -- what a rejection code requires and nothing more. Gate
unchanged: exit 1, GATE RED rows 2, 3, 6. Version stays 0.10.1, untagged.

Suite after `git add` against a clean tree: `uv run pytest -q` ->
2152 passed, 1 skipped (226 s). ruff, ruff format --check, mypy --strict clean.

Report: docs/2026-09-18-filterkjeden-og-backstoppen.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 15:34:07 +02:00
commit 0c3c4904ee
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
11 changed files with 564 additions and 38 deletions
Download patch file Download diff file Expand all files Collapse all files

35
CHANGELOG.md
View file

@ -118,10 +118,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
CARRIED, with no rejection, at 892 MB of peak RSS. With the bound: 0
carried, `asset_too_large`, 54 MB — and 62 MB where the old path cost
2 436 MB, so the cost no longer scales with the bomb.
- The measurement runs where `FlateDecode` is the first filter and the
document is not encrypted. Any other chain is caught by a check on
`len(data)` AFTER `get_data()`, which is a counted refusal and not a
bounded one. The difference is stated in the code rather than implied.
- **Every link of the chain is measured, not only the first.** A PDF
decodes a stream through a list of filters, and the first fix read
`filters[0]`: `/Filter [/FlateDecode /FlateDecode]` therefore cost
886 554 624 bytes of peak RSS from 1 636 bytes of file, and three links
cost the same from 1 070 — about 542 000x, with the picture still refused
at the end, after the memory had been spent. It also left the 16 corpus
image objects behind an `[/ASCII85Decode /FlateDecode]` chain unmeasured,
because `filters[0]` is not `FlateDecode` there. Bounded, measured idle in paired subprocesses: 52 367 360 bytes at two
links, 61 390 848 at three, and 60 403 712 where the old path cost
2 567 204 864.
- **A filter whose output cannot be measured before it is produced is
refused unread**, with its own code `asset_pdf_unbounded`. `FlateDecode`
is measured; `ASCII85Decode` and `ASCIIHexDecode` are bounded by their own
input because they shrink; `DCTDecode`, `JPXDecode` and `JBIG2Decode` pass
through unchanged. `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`,
`/Crypt` and anything unknown are refused. Over the 5 142 image objects of
the 78 PDFs measured, the refused class is 4 `CCITTFaxDecode` objects,
which are 1-bit stencil masks and were already refused one step later by
the encoder. Measured by name over the same 78 documents, carried images
are 9 356 before and 9 356 after: no document loses a picture, and the 8
objects that move code (4 masks, counted twice) were refused on both sides.
- **An encrypted stream is deciphered and then measured.** Deciphering does
not change a stream's length, so this does what pdfminer's own `decode()`
does; before, `stream.decipher is not None` returned without measuring,
which made "the document declares encryption" a way past the bound.
- What remains outside the bound is a stream something else has already
decoded, where the memory is spent before this package is asked. That one
is caught by a check on `len(data)` AFTER `get_data()`, which is a counted
refusal and not a bounded one. The difference is stated in the code rather
than implied — and, since this change, held by a test: deleting exactly
that line passed all 2 132 tests before it.
- **A declared size that is not a size is refused with its own code
(0.10.1).** `/Width -1 /Height 40000000000` multiplies to a NEGATIVE pixel
count, under which every bound read as satisfied: the check returned

40
CLAUDE.md
View file

@ -924,11 +924,41 @@ and fixtures, never code.
it), and what a PDF image's STREAM decompresses to -- measured a chunk at
a time and DISCARDED, before `get_data()`. With it: **0 carried,
`asset_too_large`, 54 MB** at 400 MB and **62 MB** at 1,2 GB, so the cost
no longer scales with the bomb. **The limit is stated rather than
implied**: the stream measurement runs where `FlateDecode` is the first
filter and the document is not encrypted; every other chain is a check on
the decoded length AFTER the decode, which is a COUNTED refusal and not a
bounded one.
no longer scales with the bomb.
- **THE CHAIN, NOT ITS FIRST LINK.** That fix read `filters[0]`, and a PDF
decodes a stream through a LIST of filters: measured 18.09 on `0f308c1`,
`/Filter [/FlateDecode /FlateDecode]` cost **886 554 624 B peak RSS from
1 636 B of file** and three links the same from 1 070 B -- about
542 000x -- with the picture still refused at the end, by the backstop,
after the memory was spent. It also left the **16** corpus objects behind
an `[/ASCII85Decode /FlateDecode]` chain unmeasured, since `filters[0]` is
not `FlateDecode` there. Bounded (idle machine, paired subprocesses):
**52 367 360 B** at two links, **61 390 848 B** at three, **60 403 712 B**
where the old path cost 2 567 204 864 B, and the single-link control
unmoved at 59 232 256 -> 62 017 536. **Three classes and no fourth**
(`extract.bounded_pdf_filters`, pinned by a test): `FlateDecode` MEASURED,
`ASCII85Decode`/`ASCIIHexDecode` bounded by their own input because they
SHRINK, `DCTDecode`/`JPXDecode`/`JBIG2Decode` PASS THROUGH. Everything
else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`,
anything written later -- is refused UNREAD with its own code
`asset_pdf_unbounded`, the same decision `corpus.resolve_gate` takes for
an unknown gate name. Cost measured over the **5 142** image objects of
78 PDFs: the refused class is **4** `CCITTFaxDecode` objects, all 1-bit
stencil masks already refused one step later by the encoder, and **0**
objects in an encrypted document. **NOT ONE PICTURE CHANGES HANDS AND IT
IS MEASURED BY NAME**: over the same 78 PDFs, carried images are
**9 356 -> 9 356**, **0 of 78** documents lose one and 0 gain one; the
only movement is **8** objects going `asset_pdf_unsupported` ->
`asset_pdf_unbounded` (the 4 masks, counted twice because `trinn1` and
`trinn2` hold the same document). An ENCRYPTED stream is
now deciphered and then measured (deciphering does not change a length),
where `stream.decipher is not None` used to return unmeasured.
- **THE LIMIT IS STATED RATHER THAN IMPLIED, and it is now ONE case**: a
stream something else has ALREADY decoded, where the memory is spent
before this package is asked. `check_payload(len(data))` after
`get_data()` COUNTS it -- a counted refusal, not a bounded one. Deleting
exactly that line passed all **2 132** tests on `0f308c1`; it has a test
now.
- **A non-positive declared dimension is `asset_size_invalid`**, its own
code, raised before the stream is read. `-1 x 40 000 000 000` is a
NEGATIVE pixel count, under which every `>` bound reads as satisfied: the

22
README.md
View file

@ -334,11 +334,25 @@ it. Measured: that document is 408 516 bytes and cost 892 MB of peak RSS with
only the declared size bounded; with the stream bounded it is refused at
54 MB, and a three-times-larger bomb costs 62 MB rather than 2 436 MB.
**Every link of the filter chain is bounded, not only the first.** A PDF
decodes a stream through a *list* of filters, and `/Filter [/FlateDecode
/FlateDecode]` puts the whole expansion in the second one: measured, 1 636
bytes of file cost 886 554 624 bytes of peak RSS when only the first link was
measured (52 367 360 with every link measured), and the picture was still refused at the end — after the memory had
been spent. `FlateDecode` is measured, `ASCII85Decode` and `ASCIIHexDecode`
are bounded by their own input because they shrink, and `DCTDecode`,
`JPXDecode` and `JBIG2Decode` pass through unchanged. Any other filter —
`LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything written
after this — expands by an amount no chunked measurement can reach, so an
image behind one is refused UNREAD with its own code, `asset_pdf_unbounded`,
rather than decoded to find out what it costs. An encrypted stream is
deciphered first and then measured like any other.
**What the stream bound does NOT reach**, stated because the difference
matters: it runs where `FlateDecode` is the first filter and the document is
not encrypted. Any other filter chain is caught by a check on the decoded
length AFTER the decode, which makes it a counted refusal rather than a
bounded one — the memory is spent and then the picture is dropped.
matters: a stream something else has already decoded, where the memory was
spent before this package was asked. That one is caught by a check on the
decoded length AFTER the decode, which makes it a counted refusal rather than
a bounded one — the picture is dropped by count, not by bound.
**A declared size that is not a size** — a zero or negative `/Width` or
`/Height` — is refused with its own code, `asset_size_invalid`, before the

15
docs/2026-09-18-bildestien-holder-0-10-1.md
View file

@ -36,6 +36,21 @@ not the stream is bounded.
## What changed: three numbers are bounded, and the limit is stated
> **CORRECTION, 2026-09-18 (same day, later) — point 3 below bounded ONE LINK
> of a filter chain, not the chain.** A PM checkpoint of `0f308c1` measured
> `/Filter [/FlateDecode /FlateDecode]`: 1 636 bytes of file, 889 573 376
> bytes of peak RSS, still refused at the end by the backstop after the memory
> was spent. The sentence below that the measurement "runs before
> `get_data()`" is true; the sentence in the section after it, that the
> measurement covers the case where "`FlateDecode` is the first filter", was a
> bound on the first link and was therefore not a bound. The round that closes
> it, with the three classes of filter and the refusal for the ones no chunked
> measurement can reach, is
> [`docs/2026-09-18-filterkjeden-og-backstoppen.md`](2026-09-18-filterkjeden-og-backstoppen.md).
> That round also found the backstop this paragraph leans on had **no test at
> all**: deleting it passed all 2 132 tests.
1. **What the container DECLARES** — unchanged from the first round.
2. **What a carried FILE measures** — new. `read_image` now checks the size it
sniffs out of the header. This package never decodes such a file, so it pays

200
docs/2026-09-18-filterkjeden-og-backstoppen.md Normal file
View file

@ -0,0 +1,200 @@
# The chain, not its first link — and a backstop nothing held
A PM checkpoint of `0f308c1` — the commit that was to make `v0.10.1` true —
read the fix for the deflate bomb and found the bound still reachable, through
a shape the fix had not considered: a PDF decodes a stream through a **list**
of filters, and the fix measured `filters[0]`.
This report records what was measured, what changed, and what the new rule
costs on real documents. The two rounds it follows are
[`docs/2026-09-17-bildestien-0-10-1.md`](2026-09-17-bildestien-0-10-1.md) and
[`docs/2026-09-18-bildestien-holder-0-10-1.md`](2026-09-18-bildestien-holder-0-10-1.md).
## BLOCKER — the bound measured one link of a chain
`_check_inflated` did this:
```python
filters = stream.get_filters()
if not filters or filters[0][0] not in LITERALS_FLATE_DECODE:
return
inflated_size(raw, name=name)
```
Two holes, and the second was invisible because the first looked like the
whole rule.
1. **A chain of two `FlateDecode` links passes the check.** The first link of
`/Filter [/FlateDecode /FlateDecode]` inflates 795 bytes to 407 685 — well
under the bound — and the check returns. `get_data()` then applies **both**
links and produces 400 MB.
2. **A chain whose first link is not `FlateDecode` is not measured at all.**
`[/ASCII85Decode /FlateDecode]` returns on the first line.
Measured on `0f308c1` in its own interpreter (peak RSS is `RUSAGE_SELF` of a
subprocess, not the high-water mark of the test session):
| chain | file | peak RSS | carried |
|---|---:|---:|---|
| `[/FlateDecode]`, 400 MB | 408 516 B | 59 232 256 B | 0, `asset_too_large` |
| `[/FlateDecode /FlateDecode]`, 400 MB | 1 636 B | **886 554 624 B** | 0, `asset_too_large` |
| `[/FlateDecode /FlateDecode /FlateDecode]`, 400 MB | 1 070 B | **889 393 152 B** | 0, `asset_too_large` |
| `[/FlateDecode /FlateDecode]`, 1,2 GB | 2 927 B | **2 567 204 864 B** | 0, `asset_too_large` |
About 543 000x the file size at two links. Note the last column: the picture
**is** refused — by `check_payload` after `get_data()`, which is the counted
refusal, not the bounded one. A test reading only the rejection code is green
on this defect, which is why two of the tests written here assert **which**
check fired, by its message.
The pre-fix figures were measured from a `git archive` of `3b587ea` on
`PYTHONPATH`, not from the editable tree, and the census below prints the
imported module's `__file__` as its own control.
## The chain is not a hypothetical
Every image XObject of the 78 PDFs on this machine, by filter chain
(2026-09-18, 5 142 objects, `get_filters()` as pdfminer resolves it):
| chain | objects |
|---|---:|
| `[/DCTDecode]` | 1 654 |
| `[/FlateDecode]` | 2 236 |
| `[/FlateDecode /DCTDecode]` | 596 |
| `[/FlateDecode /ASCII85Decode]` | 580 |
| (no filter) | 40 |
| `[/ASCII85Decode /FlateDecode]` | 16 |
| `[/JPXDecode]` | 16 |
| `[/CCITTFaxDecode]` | 4 |
1 192 real pictures are reached through a chain, so refusing every chain was
not available. 16 of them sit behind an `ASCII85Decode` and were unmeasured.
## What the rule is now
`_check_stream_cost` walks **every** link, in order, and the filters fall in
three classes (`extract.bounded_pdf_filters`, pinned by a test):
- `FlateDecode` is **measured** — inflated a chunk at a time, output
discarded, refused the moment the running total crosses the bound. A link
with another expanding link behind it is inflated **under the same bound**
and handed on, so what is held is never more than the bound.
- `ASCII85Decode` and `ASCIIHexDecode` **shrink** by construction (five
characters to four bytes, two to one), so their output is bounded by their
input, which is already in memory as part of the file. They are decoded here
so a `FlateDecode` behind one can be measured.
- `DCTDecode`, `JPXDecode` and `JBIG2Decode` are **pass-through** in pdfminer:
it hands the compressed image on for the reader to sniff, and the size does
not change.
Everything else — `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`,
and any filter written after this — expands by an amount pdfminer will only
reveal by producing the whole output. On this machine that class is **4 of
5 142** objects, all `CCITTFaxDecode`, and all four are 1-bit stencil masks
(`/ImageMask true`, `/BitsPerComponent 1`) that the encoder already refused one
step later, twice over — so no picture anyone holds changes hands. An image behind one is refused
**unread**, with its own code `asset_pdf_unbounded`, and the refusal is
decided **before the first link is decoded**, so a document cannot make this
package pay for the links in front of the one it cannot bound. Refusing an
unknown name rather than passing it through is the same decision
`corpus.resolve_gate` takes for an unknown gate name: a fallback reproduces
the defect with an extra step.
An **encrypted** stream is deciphered and then measured. Deciphering does not
change a stream's length, so this is exactly what pdfminer's own `decode()`
does; before, `stream.decipher is not None` returned unmeasured, which made
"the document declares encryption" a way past the bound. That one is a
guarantee about the code and not a measured gain: **0 of the 5 142** image
objects on this machine sit in an encrypted document, so nobody here has ever
walked that path — which is exactly why nothing caught it.
Bounded, same fixtures, same machine:
| chain | file | peak RSS before | peak RSS after |
|---|---:|---:|---:|
| `[/FlateDecode]`, 400 MB | 408 516 B | 59 232 256 B | 62 017 536 B |
| `[/FlateDecode /FlateDecode]`, 400 MB | 1 636 B | 886 554 624 B | **52 367 360 B** |
| `[/FlateDecode x3]`, 400 MB | 1 070 B | 889 393 152 B | **61 390 848 B** |
| `[/FlateDecode /FlateDecode]`, 1,2 GB | 2 927 B | 2 567 204 864 B | **60 403 712 B** |
Both columns were measured on an otherwise idle machine, in paired
subprocesses, from the two pinned trees. The single-link row is the control:
it was already bounded and does not move. The cost no longer scales with the
bomb — tripling the stream leaves the bounded run where it was, because what
grows is the compressed input, which was already in memory.
An earlier pass of the same measurement, taken while two corpus censuses were
saturating this machine's memory, read 889 573 376 / 888 401 920 /
2 579 718 144 before and 94 748 672 / 91 258 880 / 107 921 408 after. The
unbounded column is the same number either way; the bounded one is not, which
is what a peak-RSS figure taken under load is worth. The figures published
above are the idle ones, and the test's own bar (`PEAK_RSS_BOUND`, 256 MiB)
sits above both.
## MAJOR — the backstop was uncovered
`check_payload(len(data), name=name)` after `get_data()` is the counted
refusal that four documentation surfaces point at. Deleting exactly that line
passed all 2 132 tests on `0f308c1`.
It is reachable: pdfminer's `decode()` sets `rawdata` to `None`, so a stream
something else has already decoded leaves nothing to measure, and the memory
is spent before this package is asked anything. That is now the **only** case
outside the bound, and it has a test. Under the deletion the refusal becomes
`asset_pdf_unsupported` — a code about a sample buffer, for a document that is
simply too large — which is what the test asserts against.
## The cost on real documents
Every PDF on this machine — 78 documents, the K2 reference corpus in both
`trinn1` and `trinn2`, the shipped fixtures and R761 Prosesskoden:2025 — was
run through `_pdf_images` page by page on both sides, and each side's census
was run from a **pinned tree** (`git archive 3b587ea` for before, a copy for
after) with the imported module's own `__file__` printed as the control. The
editable install was not on either path.
| | before | after |
|---|---:|---:|
| documents | 78 | 78 |
| images carried | **9 356** | **9 356** |
| `asset_pdf_unsupported` | 322 | 314 |
| `asset_pdf_unbounded` | 0 | **8** |
| `asset_samples_invalid` | 40 | 40 |
| documents losing an image they carried | — | **0** |
| documents gaining one | — | **0** |
Not one document loses a picture, and the comparison is by NAME rather than by
count, so a swap would show. The eight that move code are the four
`CCITTFaxDecode` stencil masks, counted twice because `trinn1` and `trinn2`
hold the same document: they were refused before and are refused now, one step
earlier and under a code that says why. No other rejection moves.
## Mutants
Eight mutations, one line each, in the class the order named. The harness runs
the unmutated tree first and asserts it passes, because a harness whose
command is broken reports every mutant dead and has measured nothing.
| mutation | verdict |
|---|---|
| only the first link is measured (the `0.10.1` rule restored) | DEAD |
| the link loop is dropped: only `filters[0]` is looked at | DEAD |
| the running total is compared the other way round | DEAD |
| an encrypted stream is skipped again instead of deciphered | DEAD |
| the backstop after `get_data()` is deleted | DEAD |
| a filter the bound cannot measure is passed through | DEAD |
| the intermediate link is measured but not carried forward | DEAD |
| the whole cost check is removed from the image path | DEAD |
## What this round did not do
- **No tag, no push.** The version stays `0.10.1` and untagged; re-measuring
is the PM's.
- **The accounting gate gained one line** — `asset_pdf_unbounded` in
`REJECTION_CODES` — which is what a new rejection code requires and nothing
more.
- **The 43-document reference corpus was not rebuilt.** The image census above
is per document and per image object, which is what this change can move;
whether any concept text moves is `render_missing`'s open question from the
previous round and is unchanged here.

30
src/llm_ingestion_okf/assets.py
View file

@ -53,6 +53,7 @@ import re
import struct
import unicodedata
import zlib
from collections.abc import Iterator
from dataclasses import dataclass
from .errors import ExtractionError
@ -220,6 +221,31 @@ def inflated_size(raw: bytes, *, name: str, limit: int | None = None) -> int:
reader behind it reports that in its own vocabulary, so a `zlib.error` ends
the measurement at whatever was produced up to it.
"""
return sum(len(chunk) for chunk in _inflate(raw, name=name, limit=limit))
def inflate_bounded(raw: bytes, *, name: str, limit: int | None = None) -> bytes:
"""The same measurement, KEEPING the output rather than discarding it.
One chain link is not the end of a chain: a PDF may decode a stream through
`/Filter [/FlateDecode /FlateDecode]`, and measuring the first link says
nothing about the second, which is where the cost is. To bound the second
link the first one's bytes have to exist, so this inflates under the same
running bound and hands the result on. It refuses at the same point
`inflated_size` does, so what is held is never more than the bound -- which
is what separates carrying an intermediate stage from paying for a bomb.
Measured 2026-09-18: 400 MB of zeros deflated twice is 795 bytes of stream,
and the first link of that chain inflates to 407 685 bytes. The whole
expansion lives in the LAST link, which is the one nothing measured.
"""
return b"".join(_inflate(raw, name=name, limit=limit))
def _inflate(raw: bytes, *, name: str, limit: int | None) -> Iterator[bytes]:
"""Inflate `raw` a chunk at a time, refusing the moment the running total
crosses the bound. The two callers differ only in whether they keep what
comes out."""
bound = MAX_IMAGE_BYTES if limit is None else limit
decompressor = zlib.decompressobj()
total = 0
@ -236,12 +262,12 @@ def inflated_size(raw: bytes, *, name: str, limit: int | None = None) -> int:
"container declares is a claim and this is the cost",
code="asset_too_large",
)
yield produced
pending = decompressor.unconsumed_tail
if decompressor.eof or not pending:
break
except zlib.error:
return total
return total
return
@dataclass(frozen=True)

9
src/llm_ingestion_okf/errors.py
View file

@ -149,6 +149,15 @@ class ExtractionError(IngestError):
statistic about the first untrue. Refused before the stream is read: a
negative dimension multiplies to a negative pixel count, under which
every bound reads as satisfied
- `asset_pdf_unbounded` — the image is reached through a PDF stream filter
whose output this package cannot measure before producing it
(`LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything
unknown), or through an encrypted stream it cannot decipher. DISTINCT
from `asset_too_large`, which says a measurement was taken and came out
over the bound: this one says no measurement was possible, so the picture
is refused UNREAD rather than decoded to find out what it costs. Measured
2026-09-18: bounding only the first link of a filter chain let 1 636
bytes of PDF cost 886 554 624 bytes of peak RSS
"""

227
src/llm_ingestion_okf/extract.py
View file

@ -48,6 +48,7 @@ from .assets import (
check_payload,
check_size,
encode_png,
inflate_bounded,
inflated_size,
read_image,
render_block,
@ -1366,8 +1367,124 @@ def _pdf_alpha(attrs: dict[str, object], width: int, height: int) -> bytes | Non
return alpha if len(alpha) >= width * height else False
def _check_inflated(stream: object, name: str) -> None:
"""Refuse a PDF image stream whose DECOMPRESSED size is over the bound.
def bounded_pdf_filters() -> frozenset[str]:
"""The PDF stream filters an image may be reached through, by NAME.
THREE CLASSES, and what separates them is whether a bound can be put on the
output before the decoding is paid for.
* `FlateDecode` is MEASURED: inflated a chunk at a time with the output
discarded, refused the moment the running total crosses the bound.
* `ASCII85Decode` and `ASCIIHexDecode` SHRINK by construction -- five
characters to four bytes, two to one -- so their output is bounded by
their input, which is already in memory as part of the file. They are
decoded here so that a `FlateDecode` BEHIND one can be measured.
* `DCTDecode`, `JPXDecode` and `JBIG2Decode` are pass-through in pdfminer:
it hands the compressed image on for the image reader to sniff, so the
size does not change.
EVERYTHING ELSE IS REFUSED with `asset_pdf_unbounded` before any of the
stream is decoded -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`,
`/Crypt`, and any filter written after this one. They expand by an amount
pdfminer will only reveal by producing the whole output, so there is no
measuring them a chunk at a time, and decoding one to find out how big it
is IS the failure this bound exists to stop. Refusing an unknown name
rather than passing it through is the same decision `corpus.resolve_gate`
takes for an unknown gate name: a fallback reproduces the defect with an
extra step.
The cost is measured rather than assumed. Over the 5 142 image objects of
the 78 PDFs on this machine (2026-09-18), the filter chains are 1 654
`[/DCTDecode]`, 2 236 `[/FlateDecode]`, 596 `[/FlateDecode /DCTDecode]`,
580 `[/FlateDecode /ASCII85Decode]`, 40 unfiltered, 16 `[/ASCII85Decode
/FlateDecode]`, 16 `[/JPXDecode]` and 4 `[/CCITTFaxDecode]` -- so the
refused class is those 4 objects, which are 1-bit stencil masks
(`/ImageMask true`, `/BitsPerComponent 1`) and were already refused one
step later by the encoder, twice over.
"""
return _BOUNDED_PDF_FILTERS
#: The names in `bounded_pdf_filters`, as a constant the test suite pins. The
#: docstring above is the published claim; this is what the code enforces, and
#: `_pdf_filter_classes` is the same three classes as pdfminer literals.
_BOUNDED_PDF_FILTERS = frozenset(
{
"FlateDecode",
"ASCII85Decode",
"ASCIIHexDecode",
"DCTDecode",
"JPXDecode",
"JBIG2Decode",
}
)
def _pdf_filter_classes() -> tuple[frozenset[object], frozenset[object], frozenset[object]]:
"""The three classes as pdfminer literals: measured, shrinking, unchanged.
Read from pdfminer rather than written out here, because a filter has more
than one spelling (`/Fl` is `/FlateDecode`) and a set of names written by
hand would refuse the abbreviation a real document uses.
"""
from pdfminer.pdftypes import (
LITERALS_ASCII85_DECODE,
LITERALS_ASCIIHEX_DECODE,
LITERALS_DCT_DECODE,
LITERALS_FLATE_DECODE,
LITERALS_JBIG2_DECODE,
LITERALS_JPX_DECODE,
)
return (
frozenset(LITERALS_FLATE_DECODE),
frozenset(LITERALS_ASCII85_DECODE) | frozenset(LITERALS_ASCIIHEX_DECODE),
frozenset(LITERALS_DCT_DECODE)
| frozenset(LITERALS_JPX_DECODE)
| frozenset(LITERALS_JBIG2_DECODE),
)
def _pdf_stream_bytes(stream: object, name: str) -> bytes | None:
"""The stream's raw bytes, DECIPHERED where the document is encrypted.
`None` means there are none left to measure: pdfminer's `decode()` sets
`rawdata` to `None`, so a stream something else has already decoded was
paid for before this package was asked anything. That is the one path the
backstop after `get_data()` exists for.
Until this commit an encrypted stream RETURNED here without being measured,
which made "the document declares encryption" a way past the bound.
Deciphering does not change a stream's length, so this does what pdfminer's
own `decode()` does -- decipher first, then read the filters -- and the
bound applies to an encrypted document exactly as it does to any other.
"""
raw = stream.get_rawdata() if hasattr(stream, "get_rawdata") else None
if raw is None:
return None
decipher = getattr(stream, "decipher", None)
if decipher is None:
return bytes(raw)
objid = getattr(stream, "objid", None)
genno = getattr(stream, "genno", None)
if objid is None or genno is None:
raise ExtractionError(
f"the encrypted stream behind {name!r} carries no object number, so its "
"bytes cannot be deciphered before they are decoded; refused rather "
"than decoded to find out what it costs",
code="asset_pdf_unbounded",
)
try:
return bytes(decipher(objid, genno, raw, getattr(stream, "attrs", {})))
except Exception as exc:
raise ExtractionError(
f"the encrypted stream behind {name!r} could not be deciphered: {exc}",
code="asset_pdf_unbounded",
) from exc
def _check_stream_cost(stream: object, name: str) -> None:
"""Refuse a PDF image stream whose DECODED size is over the bound.
`check_size` reads `/Width` and `/Height`, which an untrusted document
writes and which say nothing about what `get_data()` will return: `/Length`
@ -1377,27 +1494,91 @@ def _check_inflated(stream: object, name: str) -> None:
and cost 892 MB of peak RSS -- the same failure mode the 0.10.0 review
named, reached through the other number.
WHAT THIS BOUNDS, and what it does not. The measurement runs on the RAW
stream, so it applies where `FlateDecode` is the first filter -- the
overwhelming case, and the one every deflate bomb needs. It does not run
when the stream is already decoded, when the document is encrypted (the raw
bytes are ciphertext until pdfminer deciphers them), or when a chain puts
another filter in front. Those are caught by `check_payload` AFTER
`get_data()`, which makes them a counted refusal rather than a bounded one:
the memory is spent and then the picture is dropped. Stated rather than
implied, because the difference is exactly what the review found missing.
"""
from pdfminer.pdftypes import LITERALS_FLATE_DECODE
THE CHAIN, NOT ITS FIRST LINK. 0.10.1 measured `filters[0]` and returned,
which is not a bound: a PDF decodes a stream through a LIST of filters, and
`/Filter [/FlateDecode /FlateDecode]` puts the whole expansion in the
second one. Measured 2026-09-18 on `0f308c1`: 1 636 bytes of file became
886 554 624 bytes of peak RSS, and three links did the same from 1 070
bytes -- about 542 000x the file, and the picture WAS refused at the end,
by `check_payload`, after the memory had been spent. It also left the 16
corpus objects behind an `[/ASCII85Decode /FlateDecode]` chain unmeasured,
because `filters[0]` is not `FlateDecode` there.
if getattr(stream, "decipher", None) is not None:
return
raw = stream.get_rawdata() if hasattr(stream, "get_rawdata") else None
if not raw:
So every link is walked, in order. An unknown or unmeasurable one is
refused BEFORE anything is decoded (`bounded_pdf_filters` says which, and
why). A `FlateDecode` that is the last expanding link is measured and its
output discarded, which is the common case and costs exactly what 0.10.1
cost. A `FlateDecode` with another expanding link behind it is inflated
under the same bound and handed on, so the link behind it can be measured
too -- what is held is never more than the bound.
WHAT THIS STILL DOES NOT BOUND, stated rather than implied: a stream
something else has already decoded (`_pdf_stream_bytes` returns `None`),
where the memory is spent before this package is asked. That one is caught
by `check_payload` AFTER `get_data()`, which makes it a counted refusal
rather than a bounded one.
"""
flate, shrinking, pass_through = _pdf_filter_classes()
data = _pdf_stream_bytes(stream, name)
if data is None:
return
filters = stream.get_filters() if hasattr(stream, "get_filters") else []
if not filters or filters[0][0] not in LITERALS_FLATE_DECODE:
return
inflated_size(raw, name=name)
# THE WHOLE CHAIN IS READ BEFORE THE FIRST LINK IS DECODED. A filter this
# package cannot bound must be refused without having paid for the links in
# front of it, which is only possible if the refusal is decided up front.
for literal, _params in filters:
if literal not in flate and literal not in shrinking and literal not in pass_through:
raise ExtractionError(
f"the image {name!r} is decoded through {literal}, a filter whose "
"output this package cannot measure before producing it; refused "
"unread rather than decoded to find out what it costs",
code="asset_pdf_unbounded",
)
for index, (literal, params) in enumerate(filters):
if literal in flate:
if not any(behind in flate for behind, _ in filters[index + 1 :]):
inflated_size(data, name=name)
return
if _has_predictor(params):
raise ExtractionError(
f"the image {name!r} applies a predictor to a link that is not the "
"last one, so the bytes this package would hand to the next filter "
"are not the bytes pdfminer decodes; refused unread",
code="asset_pdf_unbounded",
)
data = inflate_bounded(data, name=name)
elif literal in shrinking:
try:
data = _shrink(literal, data)
except Exception:
# Not this function's problem: a stream that is not valid input
# for its own filter is reported by the reader behind it, in
# that reader's vocabulary. Nothing can expand from it here.
return
# A pass-through filter leaves the bytes exactly as they are.
check_payload(len(data), name=name)
def _has_predictor(params: object) -> bool:
"""Whether a `DecodeParms` entry asks for a predictor other than `1`."""
if not isinstance(params, dict) or "Predictor" not in params:
return False
from pdfminer.pdftypes import resolve1
predictor = resolve1(params["Predictor"])
return isinstance(predictor, int) and predictor > 1
def _shrink(literal: object, data: bytes) -> bytes:
"""The two filters whose output is smaller than their input, decoded with
pdfminer's own readers so both sides agree on what the bytes are."""
from pdfminer.ascii85 import ascii85decode, asciihexdecode
from pdfminer.pdftypes import LITERALS_ASCII85_DECODE
if literal in LITERALS_ASCII85_DECODE:
return ascii85decode(data)
return asciihexdecode(data)
def _pdf_image(stream: object, name: str) -> ExtractedImage:
@ -1433,7 +1614,7 @@ def _pdf_image(stream: object, name: str) -> ExtractedImage:
# colour space is not resolved until further down and one channel is the
# LOWEST estimate of the cost -- a bound that under-counts refuses nothing
# it should carry, and the pixel count beside it does not depend on it.
# What that leaves unbounded is the stream, which `_check_inflated` reads
# What that leaves unbounded is the stream, which `_check_stream_cost` reads
# for what it actually costs rather than for what it claims.
declared = dict(getattr(stream, "attrs", {}))
declared_width = resolve1(declared.get("Width"))
@ -1444,7 +1625,7 @@ def _pdf_image(stream: object, name: str) -> ExtractedImage:
name=name,
channels=1,
)
_check_inflated(stream, name)
_check_stream_cost(stream, name)
try:
data = stream.get_data() # type: ignore[attr-defined]
except Exception as exc:
@ -1453,7 +1634,7 @@ def _pdf_image(stream: object, name: str) -> ExtractedImage:
code="asset_pdf_unsupported",
) from exc
# THE BACKSTOP, and it is a weaker guarantee than the one above it: this
# one COUNTS a stream that `_check_inflated` could not bound in advance
# one COUNTS a stream that `_check_stream_cost` could not bound in advance
# (see its docstring for which those are), after the memory has been spent.
check_payload(len(data), name=name)
if data and sniff(data) is not None:

2
tests/test_asset_limits.py
View file

@ -502,7 +502,7 @@ def test_the_caption_of_a_remote_reference_is_still_stated() -> None:
# end by `check_payload`, after the memory has been spent; one 3 KB document
# takes a batch build with it.
#
# The corpora say the chain is not a hypothetical shape. Over the 5 092 image
# The corpora say the chain is not a hypothetical shape. Over the 5 142 image
# objects of the 78 PDFs on this machine (measured 2026-09-18), 596 decode
# through `[/FlateDecode /DCTDecode]`, 580 through `[/FlateDecode
# /ASCII85Decode]` and 16 through `[/ASCII85Decode /FlateDecode]` -- so a rule

23
tests/test_error_codes.py
View file

@ -750,6 +750,29 @@ def test_asset_too_large() -> None:
assert excinfo.value.code == "asset_too_large"
def test_asset_pdf_unbounded() -> None:
"""A filter whose output cannot be measured before it is produced.
Its own code because it says something different from `asset_too_large`:
that one reports a measurement that came out over the bound, this one
reports that no measurement was possible, so the picture was refused
unread. Reached through the real PDF path rather than a helper, because
the decision is which FILTER the stream declares.
"""
from llm_ingestion_okf.extract import extract_document
pytest.importorskip("pdfplumber")
sys.path.insert(0, str(Path(__file__).parent))
from test_asset_limits import _bomb
extracted = extract_document(
"lzw.pdf", _bomb(4, payload=b"\xff" * 512, filters="/LZWDecode"), assets=True
)
assert [item.code for item in extracted.rejected] == ["asset_pdf_unbounded"]
registry = ExtractionError.__doc__ or ""
assert "`asset_pdf_unbounded`" in registry
def test_asset_size_invalid() -> None:
"""A declared size that is not a size. Its own code because it says
something different about the document than `asset_too_large` does."""

1
tools/okf_accounting_gate.py
View file

@ -392,6 +392,7 @@ def asset_holds(build: Build, source: Path) -> bool:
REJECTION_CODES: frozenset[str] = frozenset(
{
"asset_collision",
"asset_pdf_unbounded",
"asset_pdf_unsupported",
"asset_remote",
"asset_samples_invalid",