llm-ingestion-okf/docs/2026-09-17-bildestien-0-10-1.md
Kjell Tore Guttormsen 230d1cbccd fix(assets): a remote reference is inert and a declared size is bounded
Two MAJOR findings of the independent v0.10.0 review, both with the
shipped defaults, both new in 0.10.0. Repros rebuilt as tests first.

- A remote <img src>/xlink:href became a LIVE markdown image link in the
  persisted concept, with the address and query string chosen by whoever
  wrote the document. Extraction opens no socket; a consumer rendering
  the bundle does. Now inert text with the address in a code span,
  pinned by a property over the readers rather than by one string. The
  tier asymmetry (user-upload refuses, trusted-source persisted) went to
  the guard repo with the repro.
- Nothing bounded a declared image size: 9.6 KB of PDF declaring
  3000x3000 grayscale zeros took 83 MB peak RSS, linear in pixels.
  MAX_IMAGE_PIXELS (40 000 000) and MAX_IMAGE_BYTES (256 MiB) are read
  off the corpora (largest measured 18.6 MP on K2, 1.4 MP on R761) and
  checked on what the container declares, before any decompression;
  over them is asset_too_large, counted. The same bound closes the
  inline data: URI, which the review flagged and did not measure.

Also fixed, added by PM to this order: an inline PDF image was named
from id() of a Python object, so two concept files of the reference
corpus differed between builds. It is now named from its position.

R761 unchanged: 50 carried of 50 found, assets diff -rq clean.

Report: docs/2026-09-17-bildestien-0-10-1.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 00:32:50 +02:00

5.9 KiB

Two findings in the image path, closed before push (0.10.1)

Order 20260917T135034Z-4840022792, from an independent review of v0.10.0 (~/.claude/docs/2026-09-17-review-okf-0.10.0.local.md, LOCAL-ONLY). The review's verdict was "safe to push, with reservations": no blocker, and byte-identity, determinism and --no-assets parity all hold. Both findings land with the shipped defaults (--assets on, --gate guard-trusted-source), and both are new in 0.10.0, because before it no reader read an <img> attribute or opened an image stream.

Both repros were rebuilt here as tests before anything was fixed (tests/test_asset_limits.py, 17 tests).

Reproduced:

render_missing('https://collect.example.net/p.gif?u=S', href=…)
-> '![fig](https://collect.example.net/p.gif?u=S)\nImage: … (not carried: …)'

The first line is a live image link to an address the document's author controls, query string included. This package opens no socket. A consumer that renders the bundle, or an agent that fetches what it renders, does — which turns "this bundle was opened" into a beacon (and a server-side consumer into an SSRF). Measured with guard 1.4.0: PRESET_USER_UPLOAD fails the document secure, PRESET_TRUSTED_SOURCE — the build's default — persists it.

Fixed in render_missing: a reference with an href is written as Image: <name> (not carried: <reason>) address:

`` — inert text, with the address still stated, because a reader has to know what stood there.

Pinned as a property, not a string. FOREIGN_IMAGE_LINK matches any markdown image whose target is not this bundle's own assets/. It is asserted over the two readers that resolve references (HTML, STS), over three shapes of remote address (https:, //host, upper-case scheme), and over a whole built bundle of the shipped fixture inbox. The known-positive beside it: a local image still produces a pointer block that IMAGE_POINTER matches, and a data: URI image is still carried.

The tier asymmetry is the guard's own question and was sent to llm-ingestion-pipeline-security with the repro (20260917T221801Z-428505178). Nothing was built there.

MAJOR-2: nothing bounded a declared image size

Reproduced with the review's own generator, rebuilt in the test file: a PDF declaring one grayscale image of compressed zeros.

declared PDF file peak RSS
3 000 x 3 000 9.6 KB 83 MB
8 000 x 8 000 63 KB 276 MB (review's measurement)

The cost is linear in the pixel count, so 50 000 x 50 000 is several GB. One document — malicious, defective, or a legitimately enormous scan — could take a whole batch build with it, before any gate, because the guard never sees image bytes.

The bound is read off the corpora, not chosen. Over the 4 828 image objects of the 43-document reference corpus the largest is 4 515 x 4 128 (18.6 MP, a landscape drawing). Over R761's 109 delivered pictures the largest is 2 072 x 656 (1.4 MP).

  • MAX_IMAGE_PIXELS = 40 000 000 (2.1x the largest measured).
  • MAX_IMAGE_BYTES = 256 MiB of samples.
  • Over either: asset_too_large, counted like every other refusal, with the declared size in the reason. Never a silent skip and never a killed build.

Checked on what the container DECLARES, before anything is decompressed. stream.get_data() is what pays for the bomb, so the declared Width and Height are read first. The order is observable, not asserted: the test feeds a PDF whose image stream is corrupt AND whose declared size is over the bound. Decoding first gives asset_pdf_unsupported; reading the size first gives asset_too_large. encode_png refuses the same size on its own, so the encoder does not trust its caller.

The data: URI, which the review flagged and did not measure, is closed by the same bound, checked on the payload length before decoding (base64 expands by 3/4). Measured: refused with asset_too_large as a row.

Not bounded, and stated rather than hidden: an image FILE carried verbatim is never decoded, so its cost is its own file size.

The determinism defect PM added to this order

pdfminer names an inline image (BI … EI) from id() of a Python object. Measured 2026-09-17: two concept files of the reference corpus differed between two builds of one commit, in the pointer line (Image: page-3-6344095824 against page-3-4555034320). That breaks the bit-exact rebuild invariant.

Fixed: a name pdfminer derived from id() — all digits, no resource name — is replaced by the image's position on the page (page-3-inline-2). The test builds a PDF with an inline image and compares two extractions with the _pdf_pages cache cleared between them.

The three MINOR findings

  • images: N counts pointer blocks, not unique pictures (12 pointers to 2 files is images: 12). Documented in the README rather than changed: the number is the count of places a picture stands, dedup is on content, and changing it would move bundle bytes and lose the placement count.
  • A concept that is only a pointer block is persisted as substantive. Documented: "degenerate" means zero characters after stripping whitespace, and a pointer block is text. Changing the definition is a spec question.
  • data: URI: closed, see above.

Verification

  • tests/test_asset_limits.py: 17 tests, both repros red first.
  • Full suite 2 045 passed, 1 skipped; ruff, ruff format --check and mypy --strict clean. Re-run after git add against a clean tree.
  • R761 unchanged (PM's measurement, repeated): --gate none gives 50 carried of 50 found, 50 files in assets/, and diff -rq against the bundle built before this change reports no difference.
  • tools/okf_accounting_gate.py: green on all six rows.

Proposal

Version 0.10.1. No tag, no release, no push — the content accounting sits in the same tree and PM wants the order of the two settled first.