llm-ingestion-okf/docs/2026-09-19-bmp-resten-terminator-og-pekerblokk.md
Kjell Tore Guttormsen ae441abec4
docs: the BMP rest, the terminator and the pointer block
CHANGELOG under the UNTAGGED `[0.10.1]` -- no version bump, no tag. Three
entries: the forged carry under Security, the truncated RLE8 under Fixed, and
the two small rests (the judge's fidelity limit, the lossless guard's core
arm) under "Documented, not changed".

The report carries the measurements with their denominators, including the two
checks this round introduced and then REMOVED after measuring that no mutant
can fell them -- with the numbers for each (the gate suite green without the
line anchor; 95 passed with and 95 without the length check, so the mutant the
order asked for survives).

CLAUDE.md records both defects and both rules where the viewable-asset
paragraph already stands.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-19 09:51:39 +02:00

8.9 KiB

The BMP rest: a terminator, and a claim that has to come from a pointer block

2026-09-19. Two defects PM's checkpoint on c39b666 found in the viewable-asset round, both shipped, both with the default flags. The round that introduced them was judged PARTIAL; this closes the two load-bearing rests and the two small ones. No new format, no new dependency, no version bump and no tag.

1. A truncated RLE8 stream was carried, silently

_bmp_rle8_rows painted what the stream held and left the rest of the frame at palette index 0. That is what the format says about a pixel a delta SKIPPED, so no decoder disagrees — every one of them agrees on the wrong picture. Measured by PM on a real R761 asset, 1631e526275d-7-graphic-0003.bmp, 352x548 = 192 896 pixels:

cut result
stream truncated to 90 % carried, 13 923 pixels wrong, no code, no row
to 50 % carried, 95 890 wrong
to 10 % carried, 166 525 wrong

The uncompressed path already refused the same shape (asset_samples_invalid, "refusing to pad, because a short buffer means the header was read wrong"). Eval point 6 of the round's own order says "corrupt -> published code, never silent"; this was the one place it did not hold.

The rule, read off the corpus rather than chosen

The order named two candidates — compare available bytes against biSizeImage, or require an end-of-bitmap escape. Measured first, over the 19 real RLE8 assets of the frozen R761 delivery (~/corpora/po-frosne-bundles/ r761-2025-58e1ecca6007/assets, read only):

denominator (RLE8 files): 19
stream reaches an explicit EOB:      19 of 19
the EOB is the stream's LAST 2 bytes: 19 of 19
biSizeImage == available bytes:      19 of 19
bytes after the EOB:                 Counter({0: 19})

Both candidates are satisfied by every real file, so the choice is made on what each MEANS. biSizeImage is a claim by the same untrusted header that declares the size; the terminator is a statement inside the data that the picture is complete. A third candidate, requiring every pixel to be painted, is wrong by construction: it would refuse the delta escape the format defines. So the decode may end at an explicit end-of-bitmap and nowhere else, and a stream that runs out first is refused with asset_samples_invalid.

A stream that is WHOLE but omits the terminator is refused alongside a cut one. From a reader's side they are the same bytes, and the measurement says no writer in this corpus produces the first.

Nothing real changes hands

The same 19 files still convert losslessly after the rule: 19 of 19, 2 366 365 pixels compared — PM's own figure, re-measured here with stdlib on BOTH sides (an independent BMP reader and an independent zlib + filter-reversal PNG decoder, no Pillow anywhere), with a one-byte control proving the comparison can fail.

The guard in the suite is pinned to a decoder written in the test file, which computes what each cut costs: 8, 11, 17, 0 and 0 of 32 pixels. Two of the five cuts lose no pixel at all, which is the whole reason the rule has to be the terminator and not a pixel count.

2. The judge's new route could be forged by the document

asset_holds gained a second route in the viewable-asset round, because a converted image's own bytes are not in assets/ and never will be. It reads the two digests the bundle states on the pointer line — with an expression that ran over the WHOLE bundle text. So a document could write the sentence itself. Measured by PM, and reproduced here before any code moved:

the 50000x50000 BMP was REFUSED and is not in assets/: True
judge asset_holds(refused BMP) = True          (via the image's alt text)
forged via PLAIN BODY TEXT -> asset_holds(refused BMP) = True

Before that route existed, asset_holds hashed the source file and looked in assets/ — nothing a document wrote could reach it. The gate's own first sentence is THE FASIT NEVER COMES FROM THE READER IT JUDGES, and claimed and not found had become something a document could ask to have silenced.

Two halves, and neither is sufficient alone

The judge reads the clause only from inside a pointer block — the markdown image line plus the detail line under it — and only where the clause names the asset that block points at. That closes ordinary body text and a table cell. It cannot close an image's own alt text, because a LABEL is document text that the build writes INSIDE a pointer block.

The door is assets._inline, and this is where the boundary runs: everything it returns came from the document (an alt attribute, an STS caption, a publisher's file name); everything render_block appends after it came from the run (the size it measured, the type it sniffed, the digests it computed). The second line carries both, so document text may not emit the metadata grammar the run writes there. A checksum field is disarmed: the digits are kept, because a reader is owed what the document said, and the colon that makes them a FIELD is not.

The mutants, and one finding about the harness

Three mutants in tools/okf_gate_mutants.py, one per check, each felled by its own arm: the pre-round _conversions restored verbatim (X3), the asset-name tie dropped (X4), _inline's disarming dropped (X5).

X5 survived 112 green tests on its first run, and had never been applied. The harness copies the tree and runs pytest in the copy with the venv's interpreter — which carries an EDITABLE install pointing at the working tree, so import llm_ingestion_okf in the copy resolved to the original src/. The subprocess now gets the copy's own src/ on PYTHONPATH, which wins over the editable finder, and X5 dies on the first run under it. The gate and the witness were never affected: the suite reaches those through the copy's own tools/.

killed 39 of 39, exit 0 (36 before this round).

3. Two small rests

The judge proves carriage, not fidelity. PM's M10: a mutated converter writing a BLANK PNG gives asset_holds = True, because the bundle is internally consistent — the digest it claims to have written really is the asset's digest. The suite fells that mutant by decoding both sides; the gate cannot. The docstring now says so, and the limit is asserted in BOTH directions: a test shows the blank PNG is accepted AND that the docstring names the gap. Teaching the judge pixels is a different job.

The lossless guard now runs on a core install. It decoded through Pillow, which this package neither uses nor ships and which arrives only as a transitive dependency of pdfplumber under [extract]. Measured by PM on a core environment: 4 of the 13 guards in tests/test_asset_viewable.py were SKIPPED, the lossless one among them, so a plain pip install llm-ingestion-okf never measured the property the round is named after. The new arm decodes the carried PNG with zlib and the five PNG SS 9.2 filters and compares against pixels written out in the test file, over read_image and over what the BUILD wrote. The environment is MADE rather than assumed: a sys.meta_path finder raising for every PIL name, with a known-positive that it fires. Control, run once and not committed: a Pillow-dependent line placed inside the blocked section turns the guard red.

4. Two checks this round introduced and then removed

Both were measured and found unkillable, and a guard nothing can fell is not a guard.

  • The end-of-line anchor on the judge's clause expression. With the pointer-block route and the asset-name tie in place, the only case it covers is a label forging a clause inside a block that also carries a real one — which _inline makes unreachable from a document. The whole gate suite is green without it.
  • The length check on a short absolute run in _bmp_rle8_rows. Unobservable by construction: a run shorter than its declared count means the stream ran out, so the loop cannot reach an end-of-bitmap afterwards and the picture is refused either way. Measured in a scratch copy, both directions: 95 passed with the check, 95 passed without it, so the mutant that removes it SURVIVES. The order asked for that mutant; this is the answer to it.

A fifth cut replaces them and makes an existing bound load-bearing: a stream cut in the middle of a delta escape, 17 of 32 pixels wrong. Without the two-byte bound in that branch it raises IndexError instead of a coded refusal.

5. What this round did NOT measure

  • The 43-document reference corpus was not rebuilt. The accounting gate's rows are unchanged (GATE RED: rows 2, 3, 6, row 3 identical at u = 19, d = 2, 17 of 22) — the operator-accepted losses of 2026-09-17.
  • No R761 end-to-end build was run this round; the 19-file lossless measurement is over the frozen delivery's own assets, which is the input that half of the build.
  • 16 JPEG 2000 objects are still lost with a line in the concept, and WebP is in the viewable set while sniff does not recognise one. Both are stated limits from the previous round and neither moved.