open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(assets): budget every link by what its decoder COSTS (0.10.1)

Browse source 
Round 3 of the 0.10.1 review, and the finding is the pattern the three rounds
share: each bound an OUTPUT, and the bomb stepped one link along. The
declared size, then the first `FlateDecode`, then every `FlateDecode` -- and
then a link this package had documented as safe.

`ASCII85Decode` was classed as bounded "by its own input because it shrinks".
It quadruples: `z` is the shorthand for four zero bytes. And the output was
never the cost -- `base64.a85decode` appends one 4-byte object per group to a
list, about a hundred bytes of memory per byte of INPUT (101.4x at 1 MiB,
96.1x at 4 MiB, 94.5x at 16 MiB on CPython 3.14).

Paired subprocesses, idle machine, both sides from PINNED trees, the document
built once by a third process and read from a file because `ru_maxrss` never
falls and `b"z" * 64 MiB` alone costs 171 MB:

  [/Fl /A85]      z x 32 Mi  33 475 B   CARRIED 3 261 599 744 -> too_large 42 070 016
  [/Fl /A85]      z x 64 Mi  66 090 B   CARRIED 6 461 558 784 -> too_large 40 280 064
  [/A85]          z x  8 Mi   8.4 MB    CARRIED   933 085 184 -> too_large 62 484 480
  [/Fl /A85 /Fl]  z x 32 Mi  33 488 B  samples_invalid 3 519 180 800 -> too_large 43 438 080

The picture was CARRIED in three of the four: not a bound that fired late, no
bound at all. Doubling the `z` run trebles the old cost and leaves the new one
where it was.

WHY THIS FORM. `assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE
link may cost -- a separate number from `MAX_IMAGE_BYTES`, because that one
bounds the picture and this one bounds producing it. `FlateDecode` is measured
as it is paid; every other permitted filter carries a MEASURED cost ratio
(`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its decoder
is called, since those decoders take a whole string and return a whole string.
A filter with no ratio is refused unread. The budget TRAVELS: a deflate link
is inflated under the smaller of the picture's bound and what the next link's
decoder may be handed, or `[/Fl /A85]` pays 256 MiB for a refusal.

A chunked ASCII85 decoder written here was the alternative and was FELLED: it
would bound `_check_stream_cost` and not the run, because `stream.get_data()`
decodes the whole chain again with pdfminer's own decoder, and it would make
this package rather than pdfminer the authority on an image's bytes. The cap
is the only number that bounds that. `resource.setrlimit(RLIMIT_AS)` was
MEASURED before anything was built on it, as the order required, and is not
usable: Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit`
and does not enforce it. No child-process cap exists.

THE CAP IS READ OFF THE CORPORA, the posture `MAX_IMAGE_PIXELS` has: over the
9 668 image objects of the 77 PDFs on this machine, 16 decode through an
ASCII85 link and the largest input to one is 450 739 bytes, against a cap of
about 5.0 MB.

A PROPERTY TEST REPLACES THE LIST OF KNOWN SHAPES: every chain of length 1-3
over the ten filters pdfminer decodes, 1 110 of 1 110, both payload fills,
each delivered under the bound or refused with a published code and never paid
for on the way (`tracemalloc`, which counts allocations and is not disturbed
by load). Known-positive beside it: 258 of 258 chains over the permitted
filters still carry a small image.

MAJOR -- the backstop had no test. `check_payload` at the end of
`_check_stream_cost` could be deleted with the whole suite green, because the
second one after `get_data()` gives the same code one step later. The two
differ in whether the payment was made, so the test asserts `get_data` was
never called.

10 OF 10 MUTANTS KILLED, control green, each killer named in the report. Four
survived a first pass and two tests exist because of it.

NOT ONE PICTURE CHANGES HANDS, MEASURED BY NAME: `_pdf_images` over every PDF
on this machine from both pinned trees -- 9 306 -> 9 306 carried over 77
files, 50 -> 50 on R761, 0 of 78 files moving a count and 0 moving a code.
R761 also settles a question raised while this order was open: 50 objects, 29
[/DCTDecode], 21 [/FlateDecode], 0 ASCII85 links -- so round 2's count of 580
`[/FlateDecode /ASCII85Decode]` objects is reproducible from nothing on this
machine. It changes no decision; a bomb shape does not need a corpus.

Version stays 0.10.1, no tag. README, CHANGELOG, CLAUDE.md and errors.py
corrected TO what the code does; the round-2 report carries a correction block
rather than a rewrite. Report:
docs/2026-09-18-utgangsbudsjett-per-ledd.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 19:00:38 +02:00
commit 3b3b8ae0ca
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
9 changed files with 784 additions and 126 deletions
Download patch file Download diff file Expand all files Collapse all files

103
tests/test_asset_limits.py
View file

@ -819,16 +819,32 @@ def _filters_entry(chain: tuple[str, ...]) -> str:
return "[" + " ".join("/" + name for name in chain) + "]"
def _z_run(count: int) -> bytes:
"""`count` ASCII85 `z` characters -- the shorthand for four zero bytes --
written out directly rather than produced by `a85encode`.
The ENCODER costs about forty bytes of memory per byte of input, so a
fixture built with it is what a peak-RSS measurement would measure. The
same reason `_zeros_stream` deflates without ever holding the zeros.
"""
return b"<~" + b"z" * count + b"~>"
def _z_chain_stream(chain: tuple[str, ...], count: int) -> bytes:
"""The stream a document must hold for `chain` to hand a run of `count`
`z` characters to its `ASCII85Decode` link."""
stream = _z_run(count)
for literal in reversed(chain[: chain.index("ASCII85Decode")]):
stream = _encode_for(literal, stream)
return stream
_CHAIN_CHILD = """
import resource, sys
sys.path.insert(0, {tests!r})
from test_asset_limits import _bomb, _chain_stream, _filters_entry
from llm_ingestion_okf.extract import extract_document
chain = {chain!r}
document = _bomb(
1, payload=_chain_stream(chain, b"\\x00" * {payload}), filters=_filters_entry(chain)
)
document = open({path!r}, "rb").read()
extracted = extract_document("bomb.pdf", document, assets=True)
peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print(
@ -840,14 +856,27 @@ print(
"""
def _run_chain(chain: tuple[str, ...], *, payload: int) -> tuple[int, int, str, int]:
def _run_chain(
chain: tuple[str, ...], *, payload: int, tmp_path: Path
) -> tuple[int, int, str, int]:
"""One chain's bomb in its own interpreter, so peak RSS is ITS peak and not
the high-water mark of every test that ran before it."""
the high-water mark of every test that ran before it. `payload` is how many
`z` characters the chain's `ASCII85Decode` link is handed.
The document is built HERE and handed over as a file. `ru_maxrss` is a high
water mark that never falls, so a child that builds its own fixture reports
the fixture: `b"z" * (64 MiB)` alone costs 171 MB, which is more than the
bounded extraction it would be measuring.
"""
document = tmp_path / "bomb.pdf"
document.write_bytes(
_bomb(1, payload=_z_chain_stream(chain, payload), filters=_filters_entry(chain))
)
completed = subprocess.run(
[
sys.executable,
"-c",
_CHAIN_CHILD.format(tests=str(Path(__file__).parent), chain=chain, payload=payload),
_CHAIN_CHILD.format(tests=str(Path(__file__).parent), path=str(document)),
],
capture_output=True,
text=True,
@ -857,11 +886,11 @@ def _run_chain(chain: tuple[str, ...], *, payload: int) -> tuple[int, int, str,
return int(size), int(carried), codes, int(peak)
def test_an_ascii85_link_behind_a_flate_link_is_bounded_too() -> None:
def test_an_ascii85_link_behind_a_flate_link_is_bounded_too(tmp_path: Path) -> None:
"""The round-3 BLOCKER at the shipped bound, in its own interpreter."""
pytest.importorskip("pdfplumber")
size, carried, codes, peak = _run_chain(
("FlateDecode", "ASCII85Decode"), payload=128 * 1024 * 1024
("FlateDecode", "ASCII85Decode"), payload=32 * 1024 * 1024, tmp_path=tmp_path
)
assert size < 2 * 1024 * 1024, "the fixture must stay a small file, or it proves nothing"
assert carried == 0, "a run of `z` was carried as a 1x1 picture"
@ -869,11 +898,13 @@ def test_an_ascii85_link_behind_a_flate_link_is_bounded_too() -> None:
assert peak < PEAK_RSS_BOUND, f"peak RSS {peak} bytes for a {size}-byte file"
def test_an_ascii85_link_on_its_own_is_bounded() -> None:
def test_an_ascii85_link_on_its_own_is_bounded(tmp_path: Path) -> None:
"""The same amplification with no filter in front of it: the stream IS the
run of `z`, so the cost must not be a multiple of the file."""
pytest.importorskip("pdfplumber")
size, carried, codes, peak = _run_chain(("ASCII85Decode",), payload=32 * 1024 * 1024)
size, carried, codes, peak = _run_chain(
("ASCII85Decode",), payload=8 * 1024 * 1024, tmp_path=tmp_path
)
assert carried == 0
assert codes in ASSET_REJECTION_CODES, codes
assert peak < PEAK_RSS_BOUND, f"peak RSS {peak} bytes for a {size}-byte file"
@ -1082,3 +1113,51 @@ def test_the_stream_bound_refuses_before_get_data_is_ever_called() -> None:
monkey.undo()
assert excinfo.value.code == "asset_too_large"
assert calls == [], "refused only after the stream was decoded, which is the backstop"
def test_the_budget_travels_to_the_next_link() -> None:
"""A link's output is the NEXT link's input, so a deflate link is bounded
by what the decoder behind it may be handed -- not by the picture's own
bound alone.
Without this, `[/FlateDecode /ASCII85Decode]` inflates `MAX_IMAGE_BYTES` of
`z` and only then asks whether the link behind it can afford them, which
is a 256 MiB payment for a refusal. Measured: making `inflate_limit_for`
return `MAX_IMAGE_BYTES` unconditionally left the whole suite green, so
nothing held this rule until now.
"""
cap = assets.filter_input_limit("ASCII85Decode")
assert cap is not None
assert cap < assets.MAX_IMAGE_BYTES, "the cap has to bind, or there is nothing to travel"
assert assets.inflate_limit_for("ASCII85Decode") == cap
assert assets.inflate_limit_for(None) == assets.MAX_IMAGE_BYTES
assert assets.inflate_limit_for("FlateDecode") == assets.MAX_IMAGE_BYTES, (
"a deflate link behind is measured as it is paid, so it caps nothing in front"
)
def test_a_discarded_links_size_travels_as_the_widest_it_could_become() -> None:
"""When the bytes are thrown away at the last deflate link, what travels on
is a SIZE, and it has to be the widest the links behind could make of it.
`ASCII85Decode` quadruples in the worst case and `ASCIIHexDecode` halves,
so the two go opposite ways and a rule that carried the size unchanged
would be optimistic for the first. Today that optimism cannot reach the
bound -- the cost cap already holds an ASCII85 link's input under about
5.0 MB, and four times that is well under `MAX_IMAGE_BYTES` -- so this is
pinned directly rather than through a document, and the arithmetic that
makes it unreachable is pinned beside it. If either constant moves, the
second assertion says the bound started binding.
"""
from llm_ingestion_okf import extract as extract_module
assert extract_module._widest_output("ASCII85Decode", 10) == 40
assert extract_module._widest_output("ASCIIHexDecode", 10) == 5
assert extract_module._widest_output("DCTDecode", 10) == 10
assert extract_module._widest_output("ASCII85Decode", 0) == 1, "never optimistic by a byte"
cap = assets.filter_input_limit("ASCII85Decode")
assert cap is not None
assert extract_module._widest_output("ASCII85Decode", cap) < assets.MAX_IMAGE_BYTES, (
"the widest an ASCII85 link can produce now reaches the picture bound, so this "
"rule has started to bind and needs a document behind it, not only arithmetic"
)