fix(assets): a remote reference is inert and a declared size is bounded
Two MAJOR findings of the independent v0.10.0 review, both with the shipped defaults, both new in 0.10.0. Repros rebuilt as tests first. - A remote <img src>/xlink:href became a LIVE markdown image link in the persisted concept, with the address and query string chosen by whoever wrote the document. Extraction opens no socket; a consumer rendering the bundle does. Now inert text with the address in a code span, pinned by a property over the readers rather than by one string. The tier asymmetry (user-upload refuses, trusted-source persisted) went to the guard repo with the repro. - Nothing bounded a declared image size: 9.6 KB of PDF declaring 3000x3000 grayscale zeros took 83 MB peak RSS, linear in pixels. MAX_IMAGE_PIXELS (40 000 000) and MAX_IMAGE_BYTES (256 MiB) are read off the corpora (largest measured 18.6 MP on K2, 1.4 MP on R761) and checked on what the container declares, before any decompression; over them is asset_too_large, counted. The same bound closes the inline data: URI, which the review flagged and did not measure. Also fixed, added by PM to this order: an inline PDF image was named from id() of a Python object, so two concept files of the reference corpus differed between builds. It is now named from its position. R761 unchanged: 50 carried of 50 found, assets diff -rq clean. Report: docs/2026-09-17-bildestien-0-10-1.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
864570b320
commit
230d1cbccd
8 changed files with 580 additions and 3 deletions
121
docs/2026-09-17-bildestien-0-10-1.md
Normal file
121
docs/2026-09-17-bildestien-0-10-1.md
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
# Two findings in the image path, closed before push (0.10.1)
|
||||
|
||||
Order `20260917T135034Z-4840022792`, from an independent review of `v0.10.0`
|
||||
(`~/.claude/docs/2026-09-17-review-okf-0.10.0.local.md`, LOCAL-ONLY). The
|
||||
review's verdict was "safe to push, with reservations": no blocker, and
|
||||
byte-identity, determinism and `--no-assets` parity all hold. Both findings
|
||||
land with the shipped defaults (`--assets` on, `--gate guard-trusted-source`),
|
||||
and both are new in 0.10.0, because before it no reader read an `<img>`
|
||||
attribute or opened an image stream.
|
||||
|
||||
Both repros were rebuilt here as tests before anything was fixed
|
||||
(`tests/test_asset_limits.py`, 17 tests).
|
||||
|
||||
## MAJOR-1: a remote reference was a live markdown image link
|
||||
|
||||
Reproduced:
|
||||
|
||||
render_missing('https://collect.example.net/p.gif?u=S', href=…)
|
||||
-> '\nImage: … (not carried: …)'
|
||||
|
||||
The first line is a live image link to an address the document's author
|
||||
controls, query string included. This package opens no socket. A consumer that
|
||||
renders the bundle, or an agent that fetches what it renders, does — which
|
||||
turns "this bundle was opened" into a beacon (and a server-side consumer into
|
||||
an SSRF). Measured with guard 1.4.0: `PRESET_USER_UPLOAD` fails the document
|
||||
secure, `PRESET_TRUSTED_SOURCE` — the build's default — persists it.
|
||||
|
||||
**Fixed in `render_missing`:** a reference with an `href` is written as
|
||||
`Image: <name> (not carried: <reason>) address: `<address>`` — inert text,
|
||||
with the address still stated, because a reader has to know what stood there.
|
||||
|
||||
**Pinned as a property, not a string.** `FOREIGN_IMAGE_LINK` matches any
|
||||
markdown image whose target is not this bundle's own `assets/`. It is asserted
|
||||
over the two readers that resolve references (HTML, STS), over three shapes of
|
||||
remote address (`https:`, `//host`, upper-case scheme), and over a whole built
|
||||
bundle of the shipped fixture inbox. The known-positive beside it: a local
|
||||
image still produces a pointer block that `IMAGE_POINTER` matches, and a
|
||||
`data:` URI image is still carried.
|
||||
|
||||
The tier asymmetry is the guard's own question and was sent to
|
||||
`llm-ingestion-pipeline-security` with the repro
|
||||
(`20260917T221801Z-428505178`). Nothing was built there.
|
||||
|
||||
## MAJOR-2: nothing bounded a declared image size
|
||||
|
||||
Reproduced with the review's own generator, rebuilt in the test file: a PDF
|
||||
declaring one grayscale image of compressed zeros.
|
||||
|
||||
| declared | PDF file | peak RSS |
|
||||
|---|---:|---:|
|
||||
| 3 000 x 3 000 | 9.6 KB | 83 MB |
|
||||
| 8 000 x 8 000 | 63 KB | 276 MB (review's measurement) |
|
||||
|
||||
The cost is linear in the pixel count, so 50 000 x 50 000 is several GB. One
|
||||
document — malicious, defective, or a legitimately enormous scan — could take
|
||||
a whole batch build with it, before any gate, because the guard never sees
|
||||
image bytes.
|
||||
|
||||
**The bound is read off the corpora, not chosen.** Over the 4 828 image
|
||||
objects of the 43-document reference corpus the largest is 4 515 x 4 128
|
||||
(18.6 MP, a landscape drawing). Over R761's 109 delivered pictures the largest
|
||||
is 2 072 x 656 (1.4 MP).
|
||||
|
||||
- `MAX_IMAGE_PIXELS = 40 000 000` (2.1x the largest measured).
|
||||
- `MAX_IMAGE_BYTES = 256 MiB` of samples.
|
||||
- Over either: `asset_too_large`, counted like every other refusal, with the
|
||||
declared size in the reason. Never a silent skip and never a killed build.
|
||||
|
||||
**Checked on what the container DECLARES, before anything is decompressed.**
|
||||
`stream.get_data()` is what pays for the bomb, so the declared `Width` and
|
||||
`Height` are read first. The order is observable, not asserted: the test feeds
|
||||
a PDF whose image stream is corrupt AND whose declared size is over the bound.
|
||||
Decoding first gives `asset_pdf_unsupported`; reading the size first gives
|
||||
`asset_too_large`. `encode_png` refuses the same size on its own, so the
|
||||
encoder does not trust its caller.
|
||||
|
||||
**The `data:` URI, which the review flagged and did not measure,** is closed by
|
||||
the same bound, checked on the payload length before decoding (base64 expands
|
||||
by 3/4). Measured: refused with `asset_too_large` as a row.
|
||||
|
||||
Not bounded, and stated rather than hidden: an image FILE carried verbatim is
|
||||
never decoded, so its cost is its own file size.
|
||||
|
||||
## The determinism defect PM added to this order
|
||||
|
||||
pdfminer names an inline image (`BI … EI`) from `id()` of a Python object.
|
||||
Measured 2026-09-17: two concept files of the reference corpus differed
|
||||
between two builds of one commit, in the pointer line
|
||||
(`Image: page-3-6344095824` against `page-3-4555034320`). That breaks the
|
||||
bit-exact rebuild invariant.
|
||||
|
||||
**Fixed:** a name pdfminer derived from `id()` — all digits, no resource name —
|
||||
is replaced by the image's position on the page (`page-3-inline-2`). The test
|
||||
builds a PDF with an inline image and compares two extractions with the
|
||||
`_pdf_pages` cache cleared between them.
|
||||
|
||||
## The three MINOR findings
|
||||
|
||||
- **`images: N` counts pointer blocks, not unique pictures** (12 pointers to
|
||||
2 files is `images: 12`). Documented in the README rather than changed:
|
||||
the number is the count of places a picture stands, dedup is on content, and
|
||||
changing it would move bundle bytes and lose the placement count.
|
||||
- **A concept that is only a pointer block is persisted as substantive.**
|
||||
Documented: "degenerate" means zero characters after stripping whitespace,
|
||||
and a pointer block is text. Changing the definition is a spec question.
|
||||
- **`data:` URI:** closed, see above.
|
||||
|
||||
## Verification
|
||||
|
||||
- `tests/test_asset_limits.py`: 17 tests, both repros red first.
|
||||
- Full suite 2 045 passed, 1 skipped; `ruff`, `ruff format --check` and
|
||||
`mypy --strict` clean. Re-run after `git add` against a clean tree.
|
||||
- **R761 unchanged (PM's measurement, repeated):** `--gate none` gives
|
||||
`50 carried of 50 found`, 50 files in `assets/`, and `diff -rq` against the
|
||||
bundle built before this change reports no difference.
|
||||
- `tools/okf_accounting_gate.py`: green on all six rows.
|
||||
|
||||
## Proposal
|
||||
|
||||
Version `0.10.1`. No tag, no release, no push — the content accounting sits in
|
||||
the same tree and PM wants the order of the two settled first.
|
||||
Loading…
Add table
Add a link
Reference in a new issue