open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(assets): budget every link by what its decoder COSTS (0.10.1)

Browse source 
Round 3 of the 0.10.1 review, and the finding is the pattern the three rounds
share: each bound an OUTPUT, and the bomb stepped one link along. The
declared size, then the first `FlateDecode`, then every `FlateDecode` -- and
then a link this package had documented as safe.

`ASCII85Decode` was classed as bounded "by its own input because it shrinks".
It quadruples: `z` is the shorthand for four zero bytes. And the output was
never the cost -- `base64.a85decode` appends one 4-byte object per group to a
list, about a hundred bytes of memory per byte of INPUT (101.4x at 1 MiB,
96.1x at 4 MiB, 94.5x at 16 MiB on CPython 3.14).

Paired subprocesses, idle machine, both sides from PINNED trees, the document
built once by a third process and read from a file because `ru_maxrss` never
falls and `b"z" * 64 MiB` alone costs 171 MB:

  [/Fl /A85]      z x 32 Mi  33 475 B   CARRIED 3 261 599 744 -> too_large 42 070 016
  [/Fl /A85]      z x 64 Mi  66 090 B   CARRIED 6 461 558 784 -> too_large 40 280 064
  [/A85]          z x  8 Mi   8.4 MB    CARRIED   933 085 184 -> too_large 62 484 480
  [/Fl /A85 /Fl]  z x 32 Mi  33 488 B  samples_invalid 3 519 180 800 -> too_large 43 438 080

The picture was CARRIED in three of the four: not a bound that fired late, no
bound at all. Doubling the `z` run trebles the old cost and leaves the new one
where it was.

WHY THIS FORM. `assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE
link may cost -- a separate number from `MAX_IMAGE_BYTES`, because that one
bounds the picture and this one bounds producing it. `FlateDecode` is measured
as it is paid; every other permitted filter carries a MEASURED cost ratio
(`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its decoder
is called, since those decoders take a whole string and return a whole string.
A filter with no ratio is refused unread. The budget TRAVELS: a deflate link
is inflated under the smaller of the picture's bound and what the next link's
decoder may be handed, or `[/Fl /A85]` pays 256 MiB for a refusal.

A chunked ASCII85 decoder written here was the alternative and was FELLED: it
would bound `_check_stream_cost` and not the run, because `stream.get_data()`
decodes the whole chain again with pdfminer's own decoder, and it would make
this package rather than pdfminer the authority on an image's bytes. The cap
is the only number that bounds that. `resource.setrlimit(RLIMIT_AS)` was
MEASURED before anything was built on it, as the order required, and is not
usable: Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit`
and does not enforce it. No child-process cap exists.

THE CAP IS READ OFF THE CORPORA, the posture `MAX_IMAGE_PIXELS` has: over the
9 668 image objects of the 77 PDFs on this machine, 16 decode through an
ASCII85 link and the largest input to one is 450 739 bytes, against a cap of
about 5.0 MB.

A PROPERTY TEST REPLACES THE LIST OF KNOWN SHAPES: every chain of length 1-3
over the ten filters pdfminer decodes, 1 110 of 1 110, both payload fills,
each delivered under the bound or refused with a published code and never paid
for on the way (`tracemalloc`, which counts allocations and is not disturbed
by load). Known-positive beside it: 258 of 258 chains over the permitted
filters still carry a small image.

MAJOR -- the backstop had no test. `check_payload` at the end of
`_check_stream_cost` could be deleted with the whole suite green, because the
second one after `get_data()` gives the same code one step later. The two
differ in whether the payment was made, so the test asserts `get_data` was
never called.

10 OF 10 MUTANTS KILLED, control green, each killer named in the report. Four
survived a first pass and two tests exist because of it.

NOT ONE PICTURE CHANGES HANDS, MEASURED BY NAME: `_pdf_images` over every PDF
on this machine from both pinned trees -- 9 306 -> 9 306 carried over 77
files, 50 -> 50 on R761, 0 of 78 files moving a count and 0 moving a code.
R761 also settles a question raised while this order was open: 50 objects, 29
[/DCTDecode], 21 [/FlateDecode], 0 ASCII85 links -- so round 2's count of 580
`[/FlateDecode /ASCII85Decode]` objects is reproducible from nothing on this
machine. It changes no decision; a bomb shape does not need a corpus.

Version stays 0.10.1, no tag. README, CHANGELOG, CLAUDE.md and errors.py
corrected TO what the code does; the round-2 report carries a correction block
rather than a rewrite. Report:
docs/2026-09-18-utgangsbudsjett-per-ledd.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 19:00:38 +02:00
commit 3b3b8ae0ca
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
9 changed files with 784 additions and 126 deletions
Download patch file Download diff file Expand all files Collapse all files

118
src/llm_ingestion_okf/assets.py
View file

@ -198,6 +198,124 @@ def check_payload(size: int, *, name: str) -> None:
)
#: What decoding ONE link of a PDF filter chain may cost this package, in bytes
#: of memory. A SEPARATE number from `MAX_IMAGE_BYTES`, and the distinction is
#: the whole of round 3: that one bounds the picture this package will carry,
#: this one bounds what producing it costs on the way. Three rounds of this
#: review each bound an output and the bomb moved one link along, because a
#: decoder's working set is not its output. Twice `MAX_IMAGE_BYTES`, so a run
#: may hold the stream it was given and one stage of decoding at once and no
#: more.
MAX_FILTER_DECODE_BYTES = 512 * 1024 * 1024
#: MEASURED peak memory per byte of INPUT, for each filter this package lets an
#: image be reached through. `None` means the decoder is driven a chunk at a
#: time here, so the cost is measured as it is paid and no ratio is needed --
#: today that is `FlateDecode` alone (`_inflate`).
#:
#: The numbers are read off CPython 3.14 on 2026-09-18, worst case per filter:
#:
#: * `ASCII85Decode` 101.4x at 1 MiB of input, 96.1x at 4 MiB, 94.5x at 16 MiB.
#: `z` is the shorthand for four zero bytes, so `base64.a85decode` appends one
#: 4-byte object per INPUT byte to a list -- the output ratio is 4, the cost
#: ratio is a hundred, and 0.10.1 documented this filter as "bounded by its
#: own input because it shrinks". The constant sits above the worst of the
#: three, and `test_the_ascii85_cost_ratio_is_not_below_the_one_this_package
#: _measured` re-measures it so it cannot rot when CPython changes.
#: * `ASCIIHexDecode` 1.5x at 16 MiB: it strips whitespace into a copy and
#: `unhexlify`s that, and its output is half its input.
#: * `DCTDecode`, `JPXDecode` and `JBIG2Decode` are pass-through in pdfminer --
#: the bytes are handed to the image reader unchanged -- so the ratio is 1.
#:
#: A filter that is not in this table has no measured ratio and is refused
#: unread (`asset_pdf_unbounded`). That is the same decision `corpus.resolve
#: _gate` takes for an unknown gate name: a fallback reproduces the defect with
#: an extra step.
PDF_FILTER_COST_RATIO: dict[str, float | None] = {
"FlateDecode": None,
"ASCII85Decode": 104.0,
"ASCIIHexDecode": 2.0,
"DCTDecode": 1.0,
"JPXDecode": 1.0,
"JBIG2Decode": 1.0,
}
#: The largest OUTPUT each of those filters can produce per byte of input, used
#: to carry a bound forward when the bytes themselves have been discarded.
#: `ASCII85Decode` is 4 (one `z`), `ASCIIHexDecode` 0.5 (two digits to a byte),
#: pass-through 1. `FlateDecode` has no such number, which is why it is the one
#: filter measured a chunk at a time.
PDF_FILTER_OUTPUT_RATIO: dict[str, float | None] = {
"FlateDecode": None,
"ASCII85Decode": 4.0,
"ASCIIHexDecode": 0.5,
"DCTDecode": 1.0,
"JPXDecode": 1.0,
"JBIG2Decode": 1.0,
}
def filter_input_limit(canonical: str) -> int | None:
"""The largest input this package will hand to `canonical`'s decoder.
`None` for a filter decoded a chunk at a time, which needs no input limit
because its cost is measured while it is paid.
The number this produces for `ASCII85Decode` -- about 5.0 MB -- is READ OFF
the corpora the way `MAX_IMAGE_PIXELS` is: over the 9 668 image objects of
the 77 PDFs on this machine (2026-09-18), 16 decode through an
`ASCII85Decode` link and the largest input to one is 450 739 bytes, so the
limit stands more than ten times above anything measured.
"""
ratio = PDF_FILTER_COST_RATIO.get(canonical)
if ratio is None:
return None
return int(MAX_FILTER_DECODE_BYTES // ratio)
def check_filter_cost(size: int, *, canonical: str, name: str) -> None:
"""Refuse a link whose decoder would cost more than the budget, BEFORE it
decodes anything.
This is the half `inflated_size` cannot cover. That one drives zlib a chunk
at a time and stops the moment the running total crosses the bound, which
is only possible because zlib hands its output over incrementally. Nothing
else in a PDF filter chain does: `base64.a85decode` is asked for a whole
string and gives back a whole string, so by the time its output could be
measured the memory has been spent. For those the cost is PREDICTED from a
measured ratio and the input size, and predicted before the call.
"""
limit = filter_input_limit(canonical)
if limit is None or size <= limit:
return
ratio = PDF_FILTER_COST_RATIO[canonical]
raise ExtractionError(
f"the image {name!r} hands {size} bytes to {canonical}, whose decoder costs "
f"about {ratio} bytes of memory per byte of input -- over this package's "
f"budget of {MAX_FILTER_DECODE_BYTES} bytes for one link; refused before "
"the decode, because a bound on what a link OUTPUTS is not a bound on "
"what producing it costs",
code="asset_too_large",
)
def inflate_limit_for(canonical: str | None) -> int:
"""How much a `FlateDecode` link may produce, given what comes AFTER it.
The picture's own bound is `MAX_IMAGE_BYTES`, but a link's output is the
next link's input, and a decoder with a cost ratio cannot be handed more
than `filter_input_limit` allows. Carrying the budget down the chain this
way is what stops `[/FlateDecode /ASCII85Decode]` from inflating 256 MiB of
`z` before the link behind it is asked anything.
"""
limit = MAX_IMAGE_BYTES
if canonical is not None:
behind = filter_input_limit(canonical)
if behind is not None:
limit = min(limit, behind)
return limit
def inflated_size(raw: bytes, *, name: str, limit: int | None = None) -> int:
"""What a deflate stream costs to decompress, measured without paying it.

34
src/llm_ingestion_okf/errors.py
View file

@ -133,13 +133,15 @@ class ExtractionError(IngestError):
- `asset_samples_invalid` — the sample buffer does not fit the dimensions
the image dictionary declares. Refused rather than padded: a short buffer
means the dictionary was read wrong
- `asset_too_large` — the picture is over this package's bound, either
because it DECLARES a size beyond `MAX_IMAGE_PIXELS`, because the file
itself is that large, or because the stream behind it DECOMPRESSES to
more than `MAX_IMAGE_BYTES`. The three are one code because they are one
decision — this run will not hold that picture — and because a consumer
counting refusals wants the picture, not the mechanism. The bound is read
off the corpora and sits an order of magnitude above anything measured
- `asset_too_large` — the picture is over this package's bound: because it
DECLARES a size beyond `MAX_IMAGE_PIXELS`, because the file itself is
that large, because the stream behind it DECOMPRESSES to more than
`MAX_IMAGE_BYTES`, or because one link of its filter chain would COST
more than `MAX_FILTER_DECODE_BYTES` to decode. The four are one code
because they are one decision — this run will not hold that picture —
and because a consumer counting refusals wants the picture, not the
mechanism. Each bound is read off the corpora and sits an order of
magnitude above anything measured
- `asset_size_invalid` — the container declares a size that is not a size:
a zero or negative `/Width` or `/Height`. DISTINCT from
`asset_too_large`, because the two say different things about the
@ -150,14 +152,16 @@ class ExtractionError(IngestError):
negative dimension multiplies to a negative pixel count, under which
every bound reads as satisfied
- `asset_pdf_unbounded` — the image is reached through a PDF stream filter
whose output this package cannot measure before producing it
(`LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything
unknown), or through an encrypted stream it cannot decipher. DISTINCT
from `asset_too_large`, which says a measurement was taken and came out
over the bound: this one says no measurement was possible, so the picture
is refused UNREAD rather than decoded to find out what it costs. Measured
2026-09-18: bounding only the first link of a filter chain let 1 636
bytes of PDF cost 886 554 624 bytes of peak RSS
this package has no measured cost ratio for (`LZWDecode`,
`RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything unknown), or
through an encrypted stream it cannot decipher. DISTINCT from
`asset_too_large`, which says a measurement was taken or predicted and
came out over the bound: this one says neither was possible, so the
picture is refused UNREAD rather than decoded to find out what it costs.
Measured 2026-09-18: bounding only the first link of a filter chain let
1 636 bytes of PDF cost 886 554 624 bytes of peak RSS, and bounding
every link's OUTPUT still let 33 475 bytes cost 3 261 599 744 through a
filter whose decoder holds a hundred bytes per byte of input
"""

260
src/llm_ingestion_okf/extract.py
View file

@ -29,6 +29,7 @@ import collections
import csv
import functools
import io
import math
import re
import statistics
import tempfile
@ -43,12 +44,15 @@ from xml.etree import ElementTree
from xml.etree.ElementTree import Element
from .assets import (
PDF_FILTER_OUTPUT_RATIO,
AssetRejection,
ExtractedImage,
check_filter_cost,
check_payload,
check_size,
encode_png,
inflate_bounded,
inflate_limit_for,
inflated_size,
read_image,
render_block,
@ -1370,44 +1374,59 @@ def _pdf_alpha(attrs: dict[str, object], width: int, height: int) -> bytes | Non
def bounded_pdf_filters() -> frozenset[str]:
"""The PDF stream filters an image may be reached through, by NAME.
THREE CLASSES, and what separates them is whether a bound can be put on the
output before the decoding is paid for.
TWO CLASSES, and what separates them is HOW the cost of a link is bounded,
never whether the link is safe. Every one of them is bounded.
* `FlateDecode` is MEASURED: inflated a chunk at a time with the output
discarded, refused the moment the running total crosses the bound.
* `ASCII85Decode` and `ASCIIHexDecode` SHRINK by construction -- five
characters to four bytes, two to one -- so their output is bounded by
their input, which is already in memory as part of the file. They are
decoded here so that a `FlateDecode` BEHIND one can be measured.
* `DCTDecode`, `JPXDecode` and `JBIG2Decode` are pass-through in pdfminer:
it hands the compressed image on for the image reader to sniff, so the
size does not change.
* `FlateDecode` is MEASURED as it is paid: inflated a chunk at a time,
refused the moment the running total crosses the bound, with the output
discarded unless a link behind it has to be measured from those bytes.
* `ASCII85Decode`, `ASCIIHexDecode`, `DCTDecode`, `JPXDecode` and
`JBIG2Decode` are PREDICTED before they are paid: each carries a measured
worst-case cost per byte of input (`assets.PDF_FILTER_COST_RATIO`), and a
link whose input times that ratio is over the budget is refused before
its decoder is called. Their decoders take a whole string and return a
whole string, so there is no moment between the two at which a cost could
be observed.
0.10.1 had a third class, and it was WRONG. `ASCII85Decode` and
`ASCIIHexDecode` were called bounded "by their own input because they
shrink". `z` is ASCII85's shorthand for four zero bytes, so that filter
QUADRUPLES its input, and `base64.a85decode` appends one 4-byte object per
group to a list, so it costs about a hundred bytes of memory per byte of
input. Measured on the pinned tree, its own interpreter, idle machine: a
33 475-byte PDF decoding through `[/FlateDecode /ASCII85Decode]` cost
3 261 599 744 bytes of peak RSS and the picture was CARRIED. Under the
ratios it is 42 070 016 bytes and `asset_too_large`, and at twice the run
of `z` -- which trebled the old cost to 6 461 558 784 -- it is 40 280 064:
the cost no longer follows the bomb.
EVERYTHING ELSE IS REFUSED with `asset_pdf_unbounded` before any of the
stream is decoded -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`,
`/Crypt`, and any filter written after this one. They expand by an amount
pdfminer will only reveal by producing the whole output, so there is no
measuring them a chunk at a time, and decoding one to find out how big it
is IS the failure this bound exists to stop. Refusing an unknown name
rather than passing it through is the same decision `corpus.resolve_gate`
takes for an unknown gate name: a fallback reproduces the defect with an
extra step.
`/Crypt`, and any filter written after this one. A filter with no measured
ratio has no budget to be checked against, and decoding one to find out
what it costs IS the failure this bound exists to stop. Refusing an unknown
name rather than passing it through is the same decision
`corpus.resolve_gate` takes for an unknown gate name: a fallback reproduces
the defect with an extra step.
The cost is measured rather than assumed. Over the 5 142 image objects of
the 78 PDFs on this machine (2026-09-18), the filter chains are 1 654
`[/DCTDecode]`, 2 236 `[/FlateDecode]`, 596 `[/FlateDecode /DCTDecode]`,
580 `[/FlateDecode /ASCII85Decode]`, 40 unfiltered, 16 `[/ASCII85Decode
/FlateDecode]`, 16 `[/JPXDecode]` and 4 `[/CCITTFaxDecode]` -- so the
refused class is those 4 objects, which are 1-bit stencil masks
(`/ImageMask true`, `/BitsPerComponent 1`) and were already refused one
step later by the encoder, twice over.
The reach is measured rather than assumed. Over the 9 668 image objects of
the 77 PDFs on this machine (2026-09-18, enumerated through pdfminer's own
page walk), the filter chains are 6 235 `[/FlateDecode]`, 2 459
`[/DCTDecode]`, 596 `[/FlateDecode /DCTDecode]`, 296 `[/Fl]`, 42
unfiltered, 16 `[/ASCII85Decode /FlateDecode]`, 16 `[/JPXDecode]` and 8
`[/CCITTFaxDecode]`. The refused class is those 8 objects, 1-bit stencil
masks (`/ImageMask true`, `/BitsPerComponent 1`) already refused one step
later by the encoder. The largest input any `ASCII85Decode` link is handed
is 450 739 bytes, more than ten times under the budget's cap, which is why
the cap costs no picture the corpora hold.
"""
return _BOUNDED_PDF_FILTERS
#: The names in `bounded_pdf_filters`, as a constant the test suite pins. The
#: docstring above is the published claim; this is what the code enforces, and
#: `_pdf_filter_classes` is the same three classes as pdfminer literals.
#: `_pdf_filter_names` maps every pdfminer spelling of these onto the canonical
#: name that `assets.PDF_FILTER_COST_RATIO` budgets.
_BOUNDED_PDF_FILTERS = frozenset(
{
"FlateDecode",
@ -1420,12 +1439,14 @@ _BOUNDED_PDF_FILTERS = frozenset(
)
def _pdf_filter_classes() -> tuple[frozenset[object], frozenset[object], frozenset[object]]:
"""The three classes as pdfminer literals: measured, shrinking, unchanged.
def _pdf_filter_names() -> dict[object, str]:
"""Every pdfminer literal this package bounds, mapped to its CANONICAL name.
Read from pdfminer rather than written out here, because a filter has more
than one spelling (`/Fl` is `/FlateDecode`) and a set of names written by
hand would refuse the abbreviation a real document uses.
hand would refuse the abbreviation a real document uses. The canonical name
is the key into `assets.PDF_FILTER_COST_RATIO`, so both spellings of a
filter are budgeted by one measured number.
"""
from pdfminer.pdftypes import (
LITERALS_ASCII85_DECODE,
@ -1436,13 +1457,26 @@ def _pdf_filter_classes() -> tuple[frozenset[object], frozenset[object], frozens
LITERALS_JPX_DECODE,
)
return (
frozenset(LITERALS_FLATE_DECODE),
frozenset(LITERALS_ASCII85_DECODE) | frozenset(LITERALS_ASCIIHEX_DECODE),
frozenset(LITERALS_DCT_DECODE)
| frozenset(LITERALS_JPX_DECODE)
| frozenset(LITERALS_JBIG2_DECODE),
)
families = {
"FlateDecode": LITERALS_FLATE_DECODE,
"ASCII85Decode": LITERALS_ASCII85_DECODE,
"ASCIIHexDecode": LITERALS_ASCIIHEX_DECODE,
"DCTDecode": LITERALS_DCT_DECODE,
"JPXDecode": LITERALS_JPX_DECODE,
"JBIG2Decode": LITERALS_JBIG2_DECODE,
}
return {
literal: canonical
for canonical, literals in families.items()
for literal in literals
if canonical in _BOUNDED_PDF_FILTERS
}
#: The two canonical names whose decoders produce fewer or more bytes than they
#: were given, and which this package therefore has to run to learn the size of
#: the link behind them. Everything else in the table is pass-through.
_SHRINKING_FILTER_NAMES = frozenset({"ASCII85Decode", "ASCIIHexDecode"})
def _pdf_stream_bytes(stream: object, name: str) -> bytes | None:
@ -1504,13 +1538,30 @@ def _check_stream_cost(stream: object, name: str) -> None:
corpus objects behind an `[/ASCII85Decode /FlateDecode]` chain unmeasured,
because `filters[0]` is not `FlateDecode` there.
So every link is walked, in order. An unknown or unmeasurable one is
refused BEFORE anything is decoded (`bounded_pdf_filters` says which, and
why). A `FlateDecode` that is the last expanding link is measured and its
output discarded, which is the common case and costs exactly what 0.10.1
cost. A `FlateDecode` with another expanding link behind it is inflated
under the same bound and handed on, so the link behind it can be measured
too -- what is held is never more than the bound.
AND THE COST OF A LINK, NOT THE SIZE OF ITS OUTPUT. Bounding every
`FlateDecode` was still not a bound, because the bomb moved into a link
0.10.1 had documented as safe: `ASCII85Decode`'s `z` is the shorthand for
four zero bytes, and its decoder holds about a hundred bytes per byte of
input. Measured on the pinned tree: 33 475 bytes of file cost
3 261 599 744 bytes of peak RSS and the picture was CARRIED. Three rounds
of this review each bound an OUTPUT and the bomb stepped one link along;
what they had in common is that a decoder's working set is not its output.
So every link is walked, in order, and each is given a BUDGET
(`assets.MAX_FILTER_DECODE_BYTES`) rather than a class:
* a filter with no measured cost ratio is refused BEFORE anything is
decoded (`bounded_pdf_filters` says which, and why);
* a `FlateDecode` is measured as it is paid, under a limit that is the
smaller of the picture's own bound and what the NEXT link's decoder may
be handed -- which is how the budget travels down the chain instead of
being applied to each link in isolation;
* the last `FlateDecode` in the chain has its output counted and thrown
away, which is the common case and costs exactly what 0.10.1 cost; an
earlier one is inflated under the same limit and handed on, so the link
behind it can be measured from real bytes;
* every other filter has its cost PREDICTED from its input size and its
measured ratio, and is refused before its decoder is called.
WHAT THIS STILL DOES NOT BOUND, stated rather than implied: a stream
something else has already decoded (`_pdf_stream_bytes` returns `None`),
@ -1518,46 +1569,91 @@ def _check_stream_cost(stream: object, name: str) -> None:
by `check_payload` AFTER `get_data()`, which makes it a counted refusal
rather than a bounded one.
"""
flate, shrinking, pass_through = _pdf_filter_classes()
names = _pdf_filter_names()
data = _pdf_stream_bytes(stream, name)
if data is None:
return
filters = stream.get_filters() if hasattr(stream, "get_filters") else []
raw_filters = stream.get_filters() if hasattr(stream, "get_filters") else []
# THE WHOLE CHAIN IS READ BEFORE THE FIRST LINK IS DECODED. A filter this
# package cannot bound must be refused without having paid for the links in
# front of it, which is only possible if the refusal is decided up front.
for literal, _params in filters:
if literal not in flate and literal not in shrinking and literal not in pass_through:
chain: list[tuple[str, object]] = []
for literal, params in raw_filters:
canonical = names.get(literal)
if canonical is None:
raise ExtractionError(
f"the image {name!r} is decoded through {literal}, a filter whose "
"output this package cannot measure before producing it; refused "
"unread rather than decoded to find out what it costs",
"cost this package has no measured ratio for; refused unread "
"rather than decoded to find out what it costs",
code="asset_pdf_unbounded",
)
for index, (literal, params) in enumerate(filters):
if literal in flate:
if not any(behind in flate for behind, _ in filters[index + 1 :]):
inflated_size(data, name=name)
return
if _has_predictor(params):
raise ExtractionError(
f"the image {name!r} applies a predictor to a link that is not the "
"last one, so the bytes this package would hand to the next filter "
"are not the bytes pdfminer decodes; refused unread",
code="asset_pdf_unbounded",
)
data = inflate_bounded(data, name=name)
elif literal in shrinking:
try:
data = _shrink(literal, data)
except Exception:
# Not this function's problem: a stream that is not valid input
# for its own filter is reported by the reader behind it, in
# that reader's vocabulary. Nothing can expand from it here.
return
# A pass-through filter leaves the bytes exactly as they are.
check_payload(len(data), name=name)
chain.append((canonical, params))
# The LAST deflate link is the one whose bytes nothing behind has to be
# measured from, so it is counted and thrown away; every earlier one is
# inflated under the same bound and handed on. Deciding this by index
# rather than by a running flag is what keeps "the bytes are gone" and "a
# link still needs them" from ever being true at once.
last_flate = max(
(index for index, (canonical, _) in enumerate(chain) if canonical == "FlateDecode"),
default=-1,
)
size = len(data)
for index, (canonical, params) in enumerate(chain):
behind = [name_behind for name_behind, _ in chain[index + 1 :]]
if canonical == "FlateDecode":
limit = inflate_limit_for(behind[0] if behind else None)
if index == last_flate:
# Nothing behind has to be measured, so the output is counted
# and thrown away: the cheap common case, and what 0.10.1 cost.
size = inflated_size(data, name=name, limit=limit)
data = b""
else:
if _has_predictor(params):
raise ExtractionError(
f"the image {name!r} applies a predictor to a link that is not "
"the last one, so the bytes this package would hand to the next "
"filter are not the bytes pdfminer decodes; refused unread",
code="asset_pdf_unbounded",
)
data = inflate_bounded(data, name=name, limit=limit)
size = len(data)
else:
# PREDICTED, not measured, and predicted BEFORE the decoder is
# called: these decoders take a whole string and return a whole
# string, so there is no moment between the two at which the cost
# could be observed.
check_filter_cost(size, canonical=canonical, name=name)
if canonical in _SHRINKING_FILTER_NAMES:
if index > last_flate >= 0:
# The bytes were discarded at the last deflate link, so the
# bound travels on as the WIDEST this link could produce.
size = _widest_output(canonical, size)
else:
try:
data = _shrink(canonical, data)
except Exception:
# Not this function's problem: a stream that is not
# valid input for its own filter is reported by the
# reader behind it, in that reader's vocabulary.
return
size = len(data)
# A pass-through filter leaves the bytes exactly as they are.
check_payload(size, name=name)
def _widest_output(canonical: str, size: int) -> int:
"""The most `canonical` can produce from `size` bytes, for a link whose
bytes were discarded and whose SIZE is all that is carried forward.
Rounded up rather than down, and never below one byte: a bound that is
optimistic by a byte is not a bound.
"""
ratio = PDF_FILTER_OUTPUT_RATIO.get(canonical)
if ratio is None: # pragma: no cover - only `FlateDecode`, handled above
return size
return max(1, math.ceil(size * ratio))
def _has_predictor(params: object) -> bool:
@ -1570,13 +1666,19 @@ def _has_predictor(params: object) -> bool:
return isinstance(predictor, int) and predictor > 1
def _shrink(literal: object, data: bytes) -> bytes:
"""The two filters whose output is smaller than their input, decoded with
pdfminer's own readers so both sides agree on what the bytes are."""
from pdfminer.ascii85 import ascii85decode, asciihexdecode
from pdfminer.pdftypes import LITERALS_ASCII85_DECODE
def _shrink(canonical: str, data: bytes) -> bytes:
"""The two ASCII filters, decoded with pdfminer's own readers so both sides
agree on what the bytes are.
if literal in LITERALS_ASCII85_DECODE:
Called only after `check_filter_cost` has allowed the input size, which is
what makes handing a whole string to a decoder that returns a whole string
a bounded thing to do. The name "shrink" is kept for the pair, but only
`ASCIIHexDecode` actually shrinks: `ASCII85Decode` can quadruple its input,
which is the defect this round was opened for.
"""
from pdfminer.ascii85 import ascii85decode, asciihexdecode
if canonical == "ASCII85Decode":
return ascii85decode(data)
return asciihexdecode(data)