llm-ingestion-okf/docs/2026-09-18-filterkjeden-og-backstoppen.md

The chain, not its first link — and a backstop nothing held

CORRECTION, 2026-09-18 (round 3). This report's three-class split is wrong on one class. It states that ASCII85Decode and ASCIIHexDecode are "bounded by their own input because they shrink". z is ASCII85's shorthand for four zero bytes, so that filter QUADRUPLES its input, and base64.a85decode costs about a hundred bytes of memory per byte of input. Measured on the pinned tree of 0c3c490, the commit this report closes: a 33 475-byte PDF decoding an image through [/FlateDecode /ASCII85Decode] cost 3 261 599 744 bytes of peak RSS and the picture was CARRIED with no rejection. Everything else here stands — the chain walk, the backstop, the paired corpus numbers — and what replaced the class is a measured cost ratio per filter, recorded in docs/2026-09-18-utgangsbudsjett-per-ledd.md. The text below is left as it was written.

A PM checkpoint of 0f308c1 — the commit that was to make v0.10.1 true — read the fix for the deflate bomb and found the bound still reachable, through a shape the fix had not considered: a PDF decodes a stream through a list of filters, and the fix measured filters[0].

This report records what was measured, what changed, and what the new rule costs on real documents. The two rounds it follows are docs/2026-09-17-bildestien-0-10-1.md and docs/2026-09-18-bildestien-holder-0-10-1.md.

BLOCKER — the bound measured one link of a chain

_check_inflated did this:

filters = stream.get_filters()
if not filters or filters[0][0] not in LITERALS_FLATE_DECODE:
    return
inflated_size(raw, name=name)

Two holes, and the second was invisible because the first looked like the whole rule.

1. A chain of two FlateDecode links passes the check. The first link of /Filter [/FlateDecode /FlateDecode] inflates 795 bytes to 407 685 — well under the bound — and the check returns. get_data() then applies both links and produces 400 MB.
2. A chain whose first link is not FlateDecode is not measured at all. [/ASCII85Decode /FlateDecode] returns on the first line.

Measured on 0f308c1 in its own interpreter (peak RSS is RUSAGE_SELF of a subprocess, not the high-water mark of the test session):

chain	file	peak RSS	carried
[/FlateDecode], 400 MB	408 516 B	59 232 256 B	0, asset_too_large
[/FlateDecode /FlateDecode], 400 MB	1 636 B	886 554 624 B	0, asset_too_large
[/FlateDecode /FlateDecode /FlateDecode], 400 MB	1 070 B	889 393 152 B	0, asset_too_large
[/FlateDecode /FlateDecode], 1,2 GB	2 927 B	2 567 204 864 B	0, asset_too_large

About 543 000x the file size at two links. Note the last column: the picture is refused — by check_payload after get_data(), which is the counted refusal, not the bounded one. A test reading only the rejection code is green on this defect, which is why two of the tests written here assert which check fired, by its message.

The pre-fix figures were measured from a git archive of 3b587ea on PYTHONPATH, not from the editable tree, and the census below prints the imported module's __file__ as its own control.

The chain is not a hypothetical

Every image XObject of the 78 PDFs on this machine, by filter chain (2026-09-18, 5 142 objects, get_filters() as pdfminer resolves it):

chain	objects
[/DCTDecode]	1 654
[/FlateDecode]	2 236
[/FlateDecode /DCTDecode]	596
[/FlateDecode /ASCII85Decode]	580
(no filter)	40
[/ASCII85Decode /FlateDecode]	16
[/JPXDecode]	16
[/CCITTFaxDecode]	4

1 192 real pictures are reached through a chain, so refusing every chain was not available. 16 of them sit behind an ASCII85Decode and were unmeasured.

What the rule is now

_check_stream_cost walks every link, in order, and the filters fall in three classes (extract.bounded_pdf_filters, pinned by a test):

• FlateDecode is measured — inflated a chunk at a time, output discarded, refused the moment the running total crosses the bound. A link with another expanding link behind it is inflated under the same bound and handed on, so what is held is never more than the bound.
• ASCII85Decode and ASCIIHexDecode shrink by construction (five characters to four bytes, two to one), so their output is bounded by their input, which is already in memory as part of the file. They are decoded here so a FlateDecode behind one can be measured.
• DCTDecode, JPXDecode and JBIG2Decode are pass-through in pdfminer: it hands the compressed image on for the reader to sniff, and the size does not change.

Everything else — LZWDecode, RunLengthDecode, CCITTFaxDecode, /Crypt, and any filter written after this — expands by an amount pdfminer will only reveal by producing the whole output. On this machine that class is 4 of 5 142 objects, all CCITTFaxDecode, and all four are 1-bit stencil masks (/ImageMask true, /BitsPerComponent 1) that the encoder already refused one step later, twice over — so no picture anyone holds changes hands. An image behind one is refused unread, with its own code asset_pdf_unbounded, and the refusal is decided before the first link is decoded, so a document cannot make this package pay for the links in front of the one it cannot bound. Refusing an unknown name rather than passing it through is the same decision corpus.resolve_gate takes for an unknown gate name: a fallback reproduces the defect with an extra step.

An encrypted stream is deciphered and then measured. Deciphering does not change a stream's length, so this is exactly what pdfminer's own decode() does; before, stream.decipher is not None returned unmeasured, which made "the document declares encryption" a way past the bound. That one is a guarantee about the code and not a measured gain: 0 of the 5 142 image objects on this machine sit in an encrypted document, so nobody here has ever walked that path — which is exactly why nothing caught it.

Bounded, same fixtures, same machine:

chain	file	peak RSS before	peak RSS after
[/FlateDecode], 400 MB	408 516 B	59 232 256 B	62 017 536 B
[/FlateDecode /FlateDecode], 400 MB	1 636 B	886 554 624 B	52 367 360 B
[/FlateDecode x3], 400 MB	1 070 B	889 393 152 B	61 390 848 B
[/FlateDecode /FlateDecode], 1,2 GB	2 927 B	2 567 204 864 B	60 403 712 B

Both columns were measured on an otherwise idle machine, in paired subprocesses, from the two pinned trees. The single-link row is the control: it was already bounded and does not move. The cost no longer scales with the bomb — tripling the stream leaves the bounded run where it was, because what grows is the compressed input, which was already in memory.

An earlier pass of the same measurement, taken while two corpus censuses were saturating this machine's memory, read 889 573 376 / 888 401 920 / 2 579 718 144 before and 94 748 672 / 91 258 880 / 107 921 408 after. The unbounded column is the same number either way; the bounded one is not, which is what a peak-RSS figure taken under load is worth. The figures published above are the idle ones, and the test's own bar (PEAK_RSS_BOUND, 256 MiB) sits above both.

MAJOR — the backstop was uncovered

check_payload(len(data), name=name) after get_data() is the counted refusal that four documentation surfaces point at. Deleting exactly that line passed all 2 132 tests on 0f308c1.

It is reachable: pdfminer's decode() sets rawdata to None, so a stream something else has already decoded leaves nothing to measure, and the memory is spent before this package is asked anything. That is now the only case outside the bound, and it has a test. Under the deletion the refusal becomes asset_pdf_unsupported — a code about a sample buffer, for a document that is simply too large — which is what the test asserts against.

The cost on real documents

Every PDF on this machine — 78 documents, the K2 reference corpus in both trinn1 and trinn2, the shipped fixtures and a 701-page reference standard — was run through _pdf_images page by page on both sides, and each side's census was run from a pinned tree (git archive 3b587ea for before, a copy for after) with the imported module's own __file__ printed as the control. The editable install was not on either path.

	before	after
documents	78	78
images carried	9 356	9 356
asset_pdf_unsupported	322	314
asset_pdf_unbounded	0	8
asset_samples_invalid	40	40
documents losing an image they carried	—	0
documents gaining one	—	0

Not one document loses a picture, and the comparison is by NAME rather than by count, so a swap would show. The eight that move code are the four CCITTFaxDecode stencil masks, counted twice because trinn1 and trinn2 hold the same document: they were refused before and are refused now, one step earlier and under a code that says why. No other rejection moves.

Mutants

Eight mutations, one line each, in the class the order named. The harness runs the unmutated tree first and asserts it passes, because a harness whose command is broken reports every mutant dead and has measured nothing.

mutation	verdict
only the first link is measured (the 0.10.1 rule restored)	DEAD
the link loop is dropped: only filters[0] is looked at	DEAD
the running total is compared the other way round	DEAD
an encrypted stream is skipped again instead of deciphered	DEAD
the backstop after get_data() is deleted	DEAD
a filter the bound cannot measure is passed through	DEAD
the intermediate link is measured but not carried forward	DEAD
the whole cost check is removed from the image path	DEAD

What this round did not do

• No tag, no push. The version stays 0.10.1 and untagged; re-measuring is the PM's.
• The accounting gate gained one line — asset_pdf_unbounded in REJECTION_CODES — which is what a new rejection code requires and nothing more.
• The 43-document reference corpus was not rebuilt. The image census above is per document and per image object, which is what this change can move; whether any concept text moves is render_missing's open question from the previous round and is unchanged here.