Three STS fixtures still carried the section titles and labels of one real
reference document, and three identifiers were copies of its codes with a
letter or a word swapped. They now describe an invented kitchen counter and
cookbook series: the titles, labels and descriptions of sts-identity.xml,
sts-inherit.xml and sts-empty-label.xml, the P350/P351 document codes, the
99-0001 delivery prefix and chapter 7 of the image and accounting corpora.
Generated fixtures are regenerated and the witness inventory's per-document
totals are identical before and after; only names and text move.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every fixture, test document, tool example and document now uses an invented
kitchen-and-baking handbook series, written in this repository. The package's
behaviour is unchanged; src/ changes are comments and help text only.
- Generated fixtures are regenerated from their generators. Their structural
counts are identical before and after: elements, images, rows, cells,
headings, bookmarks and the witness inventory's per-document totals. The
image-inbox and accounting documents are renamed kapittel-84-*.
- tools/okf_accounting_gate.py: the two options that named one real corpus
each are replaced by a generic, repeatable --corpus PATH with no default.
Row 5 compares the PDF pair alone. Gate verdict unchanged: RED rows 2, 3, 6.
- tools/okf_witness.py: the STS JSON reader for one publisher's delivery is
removed, along with its three twins and five tests. The mutation harness
loses W09.
- docs/: 13 dated reports that documented runs on a retired reference corpus
are removed, and 40 are neutralized. Dead links are removed, and no new
dangling path is introduced.
- The synthetic MCP-gate corpus and the residual probe words are neutral.
Valgt: keep the `okf quality --fasit` bar value (the measured fraction, one corpus) and
rewrite only its provenance, because the verdict stays unchanged and the
number names nothing.
Term check with the local list: 0 of 411 tracked files, 0 file names, 0 of
27 binary fixtures. Suite after git add: 2457 passed, 1 skipped. The base
tree had 2460 passed and 2 skipped; five tests went with the JSON reader and
four were added by the term check. ruff, ruff format and mypy --strict src/
are clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 3 of the 0.10.1 review, and the finding is the pattern the three rounds
share: each bound an OUTPUT, and the bomb stepped one link along. The
declared size, then the first `FlateDecode`, then every `FlateDecode` -- and
then a link this package had documented as safe.
`ASCII85Decode` was classed as bounded "by its own input because it shrinks".
It quadruples: `z` is the shorthand for four zero bytes. And the output was
never the cost -- `base64.a85decode` appends one 4-byte object per group to a
list, about a hundred bytes of memory per byte of INPUT (101.4x at 1 MiB,
96.1x at 4 MiB, 94.5x at 16 MiB on CPython 3.14).
Paired subprocesses, idle machine, both sides from PINNED trees, the document
built once by a third process and read from a file because `ru_maxrss` never
falls and `b"z" * 64 MiB` alone costs 171 MB:
[/Fl /A85] z x 32 Mi 33 475 B CARRIED 3 261 599 744 -> too_large 42 070 016
[/Fl /A85] z x 64 Mi 66 090 B CARRIED 6 461 558 784 -> too_large 40 280 064
[/A85] z x 8 Mi 8.4 MB CARRIED 933 085 184 -> too_large 62 484 480
[/Fl /A85 /Fl] z x 32 Mi 33 488 B samples_invalid 3 519 180 800 -> too_large 43 438 080
The picture was CARRIED in three of the four: not a bound that fired late, no
bound at all. Doubling the `z` run trebles the old cost and leaves the new one
where it was.
WHY THIS FORM. `assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE
link may cost -- a separate number from `MAX_IMAGE_BYTES`, because that one
bounds the picture and this one bounds producing it. `FlateDecode` is measured
as it is paid; every other permitted filter carries a MEASURED cost ratio
(`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its decoder
is called, since those decoders take a whole string and return a whole string.
A filter with no ratio is refused unread. The budget TRAVELS: a deflate link
is inflated under the smaller of the picture's bound and what the next link's
decoder may be handed, or `[/Fl /A85]` pays 256 MiB for a refusal.
A chunked ASCII85 decoder written here was the alternative and was FELLED: it
would bound `_check_stream_cost` and not the run, because `stream.get_data()`
decodes the whole chain again with pdfminer's own decoder, and it would make
this package rather than pdfminer the authority on an image's bytes. The cap
is the only number that bounds that. `resource.setrlimit(RLIMIT_AS)` was
MEASURED before anything was built on it, as the order required, and is not
usable: Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit`
and does not enforce it. No child-process cap exists.
THE CAP IS READ OFF THE CORPORA, the posture `MAX_IMAGE_PIXELS` has: over the
9 668 image objects of the 77 PDFs on this machine, 16 decode through an
ASCII85 link and the largest input to one is 450 739 bytes, against a cap of
about 5.0 MB.
A PROPERTY TEST REPLACES THE LIST OF KNOWN SHAPES: every chain of length 1-3
over the ten filters pdfminer decodes, 1 110 of 1 110, both payload fills,
each delivered under the bound or refused with a published code and never paid
for on the way (`tracemalloc`, which counts allocations and is not disturbed
by load). Known-positive beside it: 258 of 258 chains over the permitted
filters still carry a small image.
MAJOR -- the backstop had no test. `check_payload` at the end of
`_check_stream_cost` could be deleted with the whole suite green, because the
second one after `get_data()` gives the same code one step later. The two
differ in whether the payment was made, so the test asserts `get_data` was
never called.
10 OF 10 MUTANTS KILLED, control green, each killer named in the report. Four
survived a first pass and two tests exist because of it.
NOT ONE PICTURE CHANGES HANDS, MEASURED BY NAME: `_pdf_images` over every PDF
on this machine from both pinned trees -- 9 306 -> 9 306 carried over 77
files, 50 -> 50 on R761, 0 of 78 files moving a count and 0 moving a code.
R761 also settles a question raised while this order was open: 50 objects, 29
[/DCTDecode], 21 [/FlateDecode], 0 ASCII85 links -- so round 2's count of 580
`[/FlateDecode /ASCII85Decode]` objects is reproducible from nothing on this
machine. It changes no decision; a bomb shape does not need a corpus.
Version stays 0.10.1, no tag. README, CHANGELOG, CLAUDE.md and errors.py
corrected TO what the code does; the round-2 report carries a correction block
rather than a rewrite. Report:
docs/2026-09-18-utgangsbudsjett-per-ledd.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Round 3 of the 0.10.1 review. Rounds 1-3 each bound a NUMBER and the bomb
moved to the next one: the declared size, then the first `FlateDecode`, then
every `FlateDecode`. It now lives in a link this package had documented as
safe. `ASCII85Decode` was called bounded "by its own input because it
shrinks"; `z` is ASCII85's shorthand for four zero bytes, so one input byte
becomes four, and `base64.a85decode` appends one 4-byte object per group to a
list, so the DECODER costs about a hundred bytes of memory per byte of input
(measured on CPython 3.14: 101.4x at 1 MiB, 96.1x at 4 MiB, 94.5x at 16 MiB).
Measured on the pinned `0c3c490` tree, its own interpreter, idle machine: a
33 475-byte PDF decoding an image through `/Filter [/FlateDecode
/ASCII85Decode]` cost 3 827 003 392 bytes of peak RSS -- 114 000x the file --
and the picture was CARRIED, with no rejection at all.
Seven tests red, three green:
* the two bombs above, in their own interpreters, at the shipped bound;
* the cost ratio as a re-measurable known-negative, so the constant cannot
rot the first time CPython changes `a85decode`;
* the input cap against the corpora (9 668 image objects over 77 PDFs, 16
behind an `ASCII85Decode` link, largest input 450 739 bytes);
* THE PROPERTY, both payload fills: every chain of length 1-3 over the ten
filters pdfminer decodes -- 1 110 of them -- is either delivered under the
bound or refused with a code in the published vocabulary, and never paid
for on the way, which `tracemalloc` measures because that is where
`a85decode`'s cost lives.
Green but previously uncovered, which is the MAJOR of the same checkpoint:
`check_payload` at the END of `_check_stream_cost` could be deleted with the
whole suite still passing, because the second one after `get_data()` gives
the same code one step later. The two differ in whether the payment was made,
so the test asserts `get_data` was never called. The known-positive beside
the property -- every bounded chain still carries a 64-byte image -- is green
too, and a rule that refuses everything would pass the property alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two findings of the 18.09 PM checkpoint of `0f308c1`. Red tests landed
first in `3b587ea`; this is what turns them green.
BLOCKER -- `_check_inflated` read `filters[0]`, measured that one link and
returned, which is not a bound: a PDF decodes a stream through a LIST of
filters. Measured in paired subprocesses from two pinned trees, idle machine:
[/FlateDecode] 400 MB 408 516 B 59 232 256 -> 62 017 536 B
[/FlateDecode x2] 400 MB 1 636 B 886 554 624 -> 52 367 360 B
[/FlateDecode x3] 400 MB 1 070 B 889 393 152 -> 61 390 848 B
[/FlateDecode x2] 1,2 GB 2 927 B 2 567 204 864 -> 60 403 712 B
542 000x the file at two links, and the picture WAS refused at the end -- by
`check_payload` after `get_data()`, once the memory was spent. The single-link
row is the control and does not move. It also left the 16 corpus objects behind
an `[/ASCII85Decode /FlateDecode]` chain unmeasured, since `filters[0]` is not
`FlateDecode` there.
`_check_stream_cost` walks every link. THREE CLASSES and no fourth
(`extract.bounded_pdf_filters`, pinned by a test): `FlateDecode` MEASURED, a
link with another expanding link behind it inflated under the same bound and
handed on; `ASCII85Decode`/`ASCIIHexDecode` bounded by their own input because
they SHRINK; `DCTDecode`/`JPXDecode`/`JBIG2Decode` PASS THROUGH. Everything
else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything
written later -- is refused UNREAD with a new code `asset_pdf_unbounded`,
decided before the FIRST link is decoded so a document cannot make the run pay
for the links in front of the one we cannot bound. An encrypted stream is
deciphered and then measured, where `stream.decipher is not None` used to
return unmeasured; 0 of 5 142 objects here are in an encrypted document, which
is why nothing caught it.
NOT ONE PICTURE CHANGES HANDS, AND IT IS MEASURED BY NAME. Every PDF on this
machine -- 78 documents, K2 in both trinn1 and trinn2, the shipped fixtures and
R761 -- run through `_pdf_images` page by page from both pinned trees:
images carried 9 356 -> 9 356
documents losing one 0 of 78
documents gaining one 0 of 78
asset_pdf_unsupported 322 -> 314
asset_pdf_unbounded 0 -> 8
The 8 are the 4 `CCITTFaxDecode` stencil masks (`/ImageMask true`,
`/BitsPerComponent 1`), counted twice because trinn1 and trinn2 hold the same
document. They were refused before and are refused now, one step earlier and
under a code that says why.
MAJOR -- `check_payload(len(data))` after `get_data()` is the counted refusal
four documentation surfaces point at, and deleting exactly that line passed all
2 132 tests. It is reachable through a stream pdfminer has ALREADY decoded
(`decode()` sets `rawdata` to `None`), which is now the ONLY case outside the
bound and has a test.
Eight mutations, one line each, every one DEAD, with the unmutated tree run
first as the control: first-link-only, loop dropped, inequality reversed,
encrypted skipped, backstop deleted, unknown filter passed through,
intermediate link not carried forward, whole check removed.
`tools/okf_accounting_gate.py` gains one line, the new code in
`REJECTION_CODES` -- what a rejection code requires and nothing more. Gate
unchanged: exit 1, GATE RED rows 2, 3, 6. Version stays 0.10.1, untagged.
Suite after `git add` against a clean tree: `uv run pytest -q` ->
2152 passed, 1 skipped (226 s). ruff, ruff format --check, mypy --strict clean.
Report: docs/2026-09-18-filterkjeden-og-backstoppen.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Eight tests for the two findings of the 18.09 PM checkpoint of `0f308c1`.
Seven are RED here and one is green-but-uncovered; the fix is the next commit.
BLOCKER -- `_check_inflated` read `filters[0]`, measured that one link and
returned, so a stream decoded through `/Filter [/FlateDecode /FlateDecode]`
was bounded by nothing. Reproduced here in its own interpreter, same commit:
links=1 408 516 B file -> 77 971 456 B peak RSS (bounded, 0.10.1)
links=2 1 636 B file -> 889 573 376 B peak RSS (unbounded)
links=3 1 070 B file -> 888 401 920 B peak RSS (unbounded)
543 000x the file size at two links, and the image IS refused at the end --
by `check_payload`, after the memory has been spent. PM measured the same
shape at 1 606 B -> 835 MB and 2 839 B -> 2 439 MB.
The chain is not a hypothetical. Over the 5 092 image objects of the 78 PDFs
on this machine (measured 2026-09-18): 1 625 `[/DCTDecode]`, 2 215
`[/FlateDecode]`, 596 `[/FlateDecode /DCTDecode]`, 580 `[/FlateDecode
/ASCII85Decode]`, 40 unfiltered, 16 `[/ASCII85Decode /FlateDecode]`, 16
`[/JPXDecode]`, 4 `[/CCITTFaxDecode]`. So refusing every chain would cost
1 192 real pictures, and bounding only the first link leaves those 16
unmeasured -- `filters[0]` is not `FlateDecode` there, so nothing ran at all.
Two of the cheap tests assert WHICH check fired ("decompresses to more than"),
because the backstop refuses the same document by code and a test reading only
the code is green on the defect.
MAJOR -- `check_payload(len(data), name=name)` after `get_data()` is the
counted refusal that four documentation surfaces point at, and deleting
exactly that line passed all 2 132 tests. The path that reaches it is a stream
pdfminer has ALREADY decoded (`decode()` sets `rawdata` to `None`), so there is
no raw stream left to measure. That test is green here and RED under the
deletion, measured before this commit: `asset_too_large` -> `asset_pdf_unsupported`.
Each red test carries a known-positive beside it: the two chains the corpora
hold still deliver their 64x64 picture, and an already-decoded stream under
the bound is still read.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A second independent review read `230d1cb` -- the commit that closed the
`v0.10.0` review's two MAJOR findings -- and found one of them open. The
bound read `/Width` and `/Height`, which an untrusted document writes, while
`get_data()` pays for the stream beside them; `/Length` is the COMPRESSED
length and the two numbers are independent.
Re-measured here on `ed8d9d7` before anything changed, in its own
interpreter: a 408 516-byte PDF declaring 1x1 and carrying 400 MB of deflated
zeros was CARRIED, no rejection, 891 904 000 B peak RSS. After: 0 carried,
`asset_too_large`, 57 065 472 B. At 1,2 GB of zeros, 2 436 MB -> 64 569 344 B
-- the cost no longer scales with the bomb. End to end through the CLI with
the shipped defaults: 838 000 640 B and an asset written -> exit 0,
79 650 816 B, `0 carried of 1 found`, no `assets/`.
Three numbers are bounded now, not one: what a container DECLARES, what a
carried FILE measures (`read_image`, so a 49 MP PNG of 47 705 bytes is not
passed on to a consumer), and what a PDF stream DECOMPRESSES to
(`assets.inflated_size`, a chunk at a time, output discarded, before
`get_data()`). The limit is stated rather than implied: the stream
measurement runs where `FlateDecode` is the first filter and the document is
not encrypted; every other chain is a check on the decoded length AFTER the
decode, a counted refusal and not a bounded one.
A non-positive declared dimension is `asset_size_invalid`, its own code,
raised before the stream is read. `-1 x 40000000000` is a NEGATIVE pixel
count, under which every `>` bound read as satisfied, so the check returned
silently and the refusal arrived from `encode_png` as
`asset_samples_invalid`. Its own code because a publisher shipping a picture
bigger than this package carries and a dictionary written to be read wrong
are different facts about a document.
Two smaller findings in the line that says what is missing, both introduced
by the first fix: the address was written twice, once bare, and a linkifying
renderer autolinks a bare URL -- written once now, in one code span; and
`label` became a dead parameter, so the figure's caption was dropped, a
regression against 0.10.0. It is written again in the `-- <label>` form a
carried pointer uses.
Version bumped to 0.10.1 across all ten places. Nine were unbound and stale:
four README install lines naming the previous release, two prose lines, the
"current tag" entry, `uv.lock`, and a CHANGELOG whose 0.10.1 content sat
under `[Unreleased]`. Two new packaging tests bind them to `__version__`, and
the README's guard tag to `[tool.uv.sources]`.
Every test was red first. The fate of every image is identical with and
without the new bound on three K2 PDFs carrying 800 images (464/464, 311/311
with the same 12 rejections, 25/25), and the second inflate is below the
noise floor there. 0 shipped artifacts move: no bundle under `examples/`,
`skills/` or `tests/fixtures/` carries an image pointer at all, measured
against a known-positive control.
`asset_too_large` was undocumented in the error registry; both codes are
there now. `tools/okf_accounting_gate.py` gains the new code in its closed
list -- one string, no behaviour change, stated because that file belongs to
another order.
Suite 2141 passed / 1 skipped, ruff + format + mypy --strict clean.
Report: docs/2026-09-18-bildestien-holder-0-10-1.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two MAJOR findings of the independent v0.10.0 review, both with the
shipped defaults, both new in 0.10.0. Repros rebuilt as tests first.
- A remote <img src>/xlink:href became a LIVE markdown image link in the
persisted concept, with the address and query string chosen by whoever
wrote the document. Extraction opens no socket; a consumer rendering
the bundle does. Now inert text with the address in a code span,
pinned by a property over the readers rather than by one string. The
tier asymmetry (user-upload refuses, trusted-source persisted) went to
the guard repo with the repro.
- Nothing bounded a declared image size: 9.6 KB of PDF declaring
3000x3000 grayscale zeros took 83 MB peak RSS, linear in pixels.
MAX_IMAGE_PIXELS (40 000 000) and MAX_IMAGE_BYTES (256 MiB) are read
off the corpora (largest measured 18.6 MP on K2, 1.4 MP on R761) and
checked on what the container declares, before any decompression;
over them is asset_too_large, counted. The same bound closes the
inline data: URI, which the review flagged and did not measure.
Also fixed, added by PM to this order: an inline PDF image was named
from id() of a Python object, so two concept files of the reference
corpus differed between builds. It is now named from its position.
R761 unchanged: 50 carried of 50 found, assets diff -rq clean.
Report: docs/2026-09-17-bildestien-0-10-1.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
