fix(assets): bound what the run pays, not what the document claims (0.10.1)
A second independent review read `230d1cb` -- the commit that closed the `v0.10.0` review's two MAJOR findings -- and found one of them open. The bound read `/Width` and `/Height`, which an untrusted document writes, while `get_data()` pays for the stream beside them; `/Length` is the COMPRESSED length and the two numbers are independent. Re-measured here on `ed8d9d7` before anything changed, in its own interpreter: a 408 516-byte PDF declaring 1x1 and carrying 400 MB of deflated zeros was CARRIED, no rejection, 891 904 000 B peak RSS. After: 0 carried, `asset_too_large`, 57 065 472 B. At 1,2 GB of zeros, 2 436 MB -> 64 569 344 B -- the cost no longer scales with the bomb. End to end through the CLI with the shipped defaults: 838 000 640 B and an asset written -> exit 0, 79 650 816 B, `0 carried of 1 found`, no `assets/`. Three numbers are bounded now, not one: what a container DECLARES, what a carried FILE measures (`read_image`, so a 49 MP PNG of 47 705 bytes is not passed on to a consumer), and what a PDF stream DECOMPRESSES to (`assets.inflated_size`, a chunk at a time, output discarded, before `get_data()`). The limit is stated rather than implied: the stream measurement runs where `FlateDecode` is the first filter and the document is not encrypted; every other chain is a check on the decoded length AFTER the decode, a counted refusal and not a bounded one. A non-positive declared dimension is `asset_size_invalid`, its own code, raised before the stream is read. `-1 x 40000000000` is a NEGATIVE pixel count, under which every `>` bound read as satisfied, so the check returned silently and the refusal arrived from `encode_png` as `asset_samples_invalid`. Its own code because a publisher shipping a picture bigger than this package carries and a dictionary written to be read wrong are different facts about a document. Two smaller findings in the line that says what is missing, both introduced by the first fix: the address was written twice, once bare, and a linkifying renderer autolinks a bare URL -- written once now, in one code span; and `label` became a dead parameter, so the figure's caption was dropped, a regression against 0.10.0. It is written again in the `-- <label>` form a carried pointer uses. Version bumped to 0.10.1 across all ten places. Nine were unbound and stale: four README install lines naming the previous release, two prose lines, the "current tag" entry, `uv.lock`, and a CHANGELOG whose 0.10.1 content sat under `[Unreleased]`. Two new packaging tests bind them to `__version__`, and the README's guard tag to `[tool.uv.sources]`. Every test was red first. The fate of every image is identical with and without the new bound on three K2 PDFs carrying 800 images (464/464, 311/311 with the same 12 rejections, 25/25), and the second inflate is below the noise floor there. 0 shipped artifacts move: no bundle under `examples/`, `skills/` or `tests/fixtures/` carries an image pointer at all, measured against a known-positive control. `asset_too_large` was undocumented in the error registry; both codes are there now. `tools/okf_accounting_gate.py` gains the new code in its closed list -- one string, no behaviour change, stated because that file belongs to another order. Suite 2141 passed / 1 skipped, ruff + format + mypy --strict clean. Report: docs/2026-09-18-bildestien-holder-0-10-1.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
ed8d9d709f
commit
0f308c1f56
15 changed files with 842 additions and 81 deletions
|
|
@ -48,6 +48,7 @@ from .assets import (
|
|||
check_payload,
|
||||
check_size,
|
||||
encode_png,
|
||||
inflated_size,
|
||||
read_image,
|
||||
render_block,
|
||||
render_missing,
|
||||
|
|
@ -1365,6 +1366,40 @@ def _pdf_alpha(attrs: dict[str, object], width: int, height: int) -> bytes | Non
|
|||
return alpha if len(alpha) >= width * height else False
|
||||
|
||||
|
||||
def _check_inflated(stream: object, name: str) -> None:
|
||||
"""Refuse a PDF image stream whose DECOMPRESSED size is over the bound.
|
||||
|
||||
`check_size` reads `/Width` and `/Height`, which an untrusted document
|
||||
writes and which say nothing about what `get_data()` will return: `/Length`
|
||||
is the COMPRESSED length, and a dictionary declaring 1x1 may hang 400 MB of
|
||||
deflated zeros off it. Measured by an independent review of 0.10.1 on
|
||||
`230d1cb`: that document is 408 516 bytes, was carried without a rejection,
|
||||
and cost 892 MB of peak RSS -- the same failure mode the 0.10.0 review
|
||||
named, reached through the other number.
|
||||
|
||||
WHAT THIS BOUNDS, and what it does not. The measurement runs on the RAW
|
||||
stream, so it applies where `FlateDecode` is the first filter -- the
|
||||
overwhelming case, and the one every deflate bomb needs. It does not run
|
||||
when the stream is already decoded, when the document is encrypted (the raw
|
||||
bytes are ciphertext until pdfminer deciphers them), or when a chain puts
|
||||
another filter in front. Those are caught by `check_payload` AFTER
|
||||
`get_data()`, which makes them a counted refusal rather than a bounded one:
|
||||
the memory is spent and then the picture is dropped. Stated rather than
|
||||
implied, because the difference is exactly what the review found missing.
|
||||
"""
|
||||
from pdfminer.pdftypes import LITERALS_FLATE_DECODE
|
||||
|
||||
if getattr(stream, "decipher", None) is not None:
|
||||
return
|
||||
raw = stream.get_rawdata() if hasattr(stream, "get_rawdata") else None
|
||||
if not raw:
|
||||
return
|
||||
filters = stream.get_filters() if hasattr(stream, "get_filters") else []
|
||||
if not filters or filters[0][0] not in LITERALS_FLATE_DECODE:
|
||||
return
|
||||
inflated_size(raw, name=name)
|
||||
|
||||
|
||||
def _pdf_image(stream: object, name: str) -> ExtractedImage:
|
||||
"""One image XObject, carried verbatim where it already is a file.
|
||||
|
||||
|
|
@ -1394,16 +1429,22 @@ def _pdf_image(stream: object, name: str) -> ExtractedImage:
|
|||
|
||||
# THE DECLARED SIZE IS READ FIRST, and the stream is not touched until it
|
||||
# is within the bound: `get_data()` decompresses, so a check after it has
|
||||
# already paid for a picture of compressed zeros.
|
||||
# already paid for a picture of compressed zeros. `channels=1` because the
|
||||
# colour space is not resolved until further down and one channel is the
|
||||
# LOWEST estimate of the cost -- a bound that under-counts refuses nothing
|
||||
# it should carry, and the pixel count beside it does not depend on it.
|
||||
# What that leaves unbounded is the stream, which `_check_inflated` reads
|
||||
# for what it actually costs rather than for what it claims.
|
||||
declared = dict(getattr(stream, "attrs", {}))
|
||||
declared_width = resolve1(declared.get("Width"))
|
||||
declared_height = resolve1(declared.get("Height"))
|
||||
check_size(
|
||||
resolve1(declared.get("Width")) if isinstance(resolve1(declared.get("Width")), int) else 0,
|
||||
resolve1(declared.get("Height"))
|
||||
if isinstance(resolve1(declared.get("Height")), int)
|
||||
else 0,
|
||||
declared_width if isinstance(declared_width, int) else None,
|
||||
declared_height if isinstance(declared_height, int) else None,
|
||||
name=name,
|
||||
channels=1,
|
||||
)
|
||||
_check_inflated(stream, name)
|
||||
try:
|
||||
data = stream.get_data() # type: ignore[attr-defined]
|
||||
except Exception as exc:
|
||||
|
|
@ -1411,6 +1452,10 @@ def _pdf_image(stream: object, name: str) -> ExtractedImage:
|
|||
f"the PDF image stream behind {name!r} could not be decoded: {exc}",
|
||||
code="asset_pdf_unsupported",
|
||||
) from exc
|
||||
# THE BACKSTOP, and it is a weaker guarantee than the one above it: this
|
||||
# one COUNTS a stream that `_check_inflated` could not bound in advance
|
||||
# (see its docstring for which those are), after the memory has been spent.
|
||||
check_payload(len(data), name=name)
|
||||
if data and sniff(data) is not None:
|
||||
return read_image(data, name=name)
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue