open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

docs: the BMP rest, the terminator and the pointer block

Browse source 
CHANGELOG under the UNTAGGED `[0.10.1]` -- no version bump, no tag. Three
entries: the forged carry under Security, the truncated RLE8 under Fixed, and
the two small rests (the judge's fidelity limit, the lossless guard's core
arm) under "Documented, not changed".

The report carries the measurements with their denominators, including the two
checks this round introduced and then REMOVED after measuring that no mutant
can fell them -- with the numbers for each (the gate suite green without the
line anchor; 95 passed with and 95 without the length check, so the mutant the
order asked for survives).

CLAUDE.md records both defects and both rules where the viewable-asset
paragraph already stands.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 09:51:39 +02:00
commit ae441abec4
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
3 changed files with 267 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

65
CHANGELOG.md
View file

@ -141,6 +141,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Security
- **A document can no longer forge a carry in the content-accounting gate
(0.10.1).** New in the viewable-asset round: a converted image's own bytes
are not in `assets/`, so `asset_holds` gained a second route that reads the
two digests the bundle states on the pointer line. The expression ran over
the WHOLE bundle text, so a document could simply write the sentence.
Measured by PM 2026-09-19: a BMP declaring 50 000 x 50 000, refused
`asset_too_large` and absent from `assets/`, was reported as held — through
an image's `alt` text, and through ordinary body text. Before that route
existed, `asset_holds` hashed the source file and nothing a document wrote
could reach it; the gate's own first sentence is that the fasit never comes
from the reader it judges, and `claimed and not found` could be silenced by
a document that asked for it.
- The claim now counts only inside a POINTER BLOCK this code wrote, and only
where it names the asset that block points at. That closes body text and a
table cell.
- An image's LABEL is document text written inside a pointer block, so
`assets._inline` disarms a checksum field in anything that came from the
document: the digits are kept, the colon that makes them a field is not.
That closes the `alt` route. Neither half is sufficient alone.
- Three mutants in `tools/okf_gate_mutants.py`, one per check, each felled by
its own arm; the harness now runs the copy with its own `src/` on
`PYTHONPATH`, because an editable install made a `src/` mutant resolve to
the working tree and survive without having been applied.
- **A remote image reference is no longer a live markdown image link
(0.10.1).** New in 0.10.0: before it, no reader read an `<img>` attribute at
all. A document could put `![fig](https://collect.example.net/p.gif?u=...)`
@ -258,6 +282,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **A truncated RLE8 BMP is refused instead of carried as a partly blank PNG
(0.10.1).** `_bmp_rle8_rows` painted what the stream held and left the rest
of the frame at palette index 0 — which is what the format says about a
SKIPPED pixel, so no decoder disagreed and the picture was wrong with no
code and no row. Measured by PM 2026-09-19 on a real R761 asset (352x548 =
192 896 pixels): cut to 90 % it was carried with 13 923 pixels wrong, to
50 % with 95 890, to 10 % with 166 525. The uncompressed path already
refused the same shape.
- The decode may now end at an explicit end-of-bitmap escape and nowhere
else; running out of bytes raises `asset_samples_invalid`, the code the
uncompressed path uses. NO PIXEL IS GUESSED: either every one is decoded
from the stream, or the picture is refused with a line in the concept.
- The rule is the terminator rather than `biSizeImage` (a claim by the same
untrusted header) or a coverage count (which would refuse the delta escape
the format defines), and it is read off the corpus: over the 19 real RLE8
assets of the frozen R761 delivery, **19 of 19** end at an explicit
end-of-bitmap, on **19 of 19** it is the stream's last two bytes, and on
**19 of 19** `biSizeImage` equals the available bytes. A whole stream that
omits the terminator is refused alongside a cut one.
- **Nothing real changes hands:** the same 19 files still convert losslessly
after the rule, **2 366 365 pixels** compared — this time with stdlib on
BOTH sides, an independent BMP reader and an independent `zlib` +
filter-reversal PNG decoder, with a one-byte control proving the
comparison can fail.
- **The content-accounting gate: a document refused whole is never clean.**
Its elements are all booked as coded rejections, so u = 0 and d = 0, and
`refused_whole` asks its question only for a corpus that persisted NOTHING —
@ -317,6 +366,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Documented, not changed
- **The accounting gate proves CARRIAGE, not FIDELITY.** Neither of
`asset_holds`'s routes decodes a pixel: a converter writing a blank PNG is
accepted, because the bundle is internally consistent. The suite fells that
mutant by decoding both sides; the judge cannot, and its docstring now says
so — "claiming a conversion it did not perform" means claiming one whose
FILE is missing, never one whose pixels are wrong.
- **The lossless guard now has an arm that runs on a core install.** It decoded
through Pillow, an OPTIONAL dependency here (transitive under `pdfplumber` in
`[extract]`), so 4 of the 13 guards in `tests/test_asset_viewable.py` were
SKIPPED on a plain `pip install llm-ingestion-okf` — the lossless one among
them. The new arm decodes the carried PNG with `zlib` and the five PNG SS 9.2
filters and compares against pixels written out in the test file, and it is
re-run under a `sys.meta_path` finder that makes every `PIL` import fail.
- `images: N` in a concept counts POINTER BLOCKS, not unique pictures (12
pointers to 2 files is `images: 12`). Now stated in the README.
- A concept that is only a pointer block is persisted as substantive, because

26
CLAUDE.md
View file

@ -1055,6 +1055,32 @@ and fixtures, never code.
elements, and the gate's verdict is the same `RED: rows 2, 3, 6` as before
the round. Report:
`docs/2026-09-19-hvert-baaret-bilde-kan-vises.md`.
**THAT ROUND SHIPPED TWO DEFECTS AND 0.10.1 CLOSES THEM.** A TRUNCATED RLE8
stream was carried as a partly blank PNG: the decoder painted what the
stream held and left the frame at index 0, which is what the format says
about a SKIPPED pixel, so no decoder disagreed -- measured on a real R761
asset of 192 896 pixels, a stream cut to 90 % was carried with 13 923 pixels
wrong, to 50 % with 95 890, to 10 % with 166 525, no code and no row. The
decode may now end at an explicit **end-of-bitmap escape and nowhere else**
(`asset_samples_invalid`, the code the uncompressed path already uses), and
the rule is the terminator rather than `biSizeImage` -- a claim by the same
untrusted header -- or a coverage count, which would refuse the delta the
format defines: over the 19 real RLE8 assets **19 of 19** end at an explicit
EOB, on **19 of 19** it is the last two bytes, and on **19 of 19**
`biSizeImage` equals the available bytes, so the same 19 still convert
losslessly, 2 366 365 pixels, now with stdlib on BOTH sides. And the JUDGE's
new conversion route could be **FORGED BY THE DOCUMENT**: it read the two
digests out of the WHOLE bundle text, so a BMP refused `asset_too_large` and
absent from `assets/` was reported as held, through an image's `alt` text
and through ordinary body text alike. The claim now counts only inside a
POINTER BLOCK this code wrote and only where it names that block's own
asset, and `assets._inline` disarms a checksum field in anything that came
from the document -- **the boundary is that everything `_inline` returns
came from the document and everything `render_block` appends after it came
from the run**; neither half is sufficient alone. The gate proves CARRIAGE
and not FIDELITY, which its docstring now states: a converter writing a
blank PNG is accepted, the suite fells that mutant and the judge cannot.
Report: `docs/2026-09-19-bmp-resten-terminator-og-pekerblokk.md`.
- **Account for content: `okf build --accounting PATH` and its judge
`python3 tools/okf_accounting_gate.py` (2026-09-17).**
- The gate was written RED at `0b00de4` and turned GREEN on all six rows by

176
docs/2026-09-19-bmp-resten-terminator-og-pekerblokk.md Normal file
View file

@ -0,0 +1,176 @@
# The BMP rest: a terminator, and a claim that has to come from a pointer block
2026-09-19. Two defects PM's checkpoint on `c39b666` found in the
viewable-asset round, both shipped, both with the default flags. The round
that introduced them was judged PARTIAL; this closes the two load-bearing
rests and the two small ones. No new format, no new dependency, no version
bump and no tag.
## 1. A truncated RLE8 stream was carried, silently
`_bmp_rle8_rows` painted what the stream held and left the rest of the frame
at palette index 0. That is what the format says about a pixel a delta
SKIPPED, so no decoder disagrees — every one of them agrees on the wrong
picture. Measured by PM on a real R761 asset, `1631e526275d-7-graphic-0003.bmp`,
352x548 = 192 896 pixels:
| cut | result |
|---|---|
| stream truncated to 90 % | carried, 13 923 pixels wrong, no code, no row |
| to 50 % | carried, 95 890 wrong |
| to 10 % | carried, 166 525 wrong |
The uncompressed path already refused the same shape
(`asset_samples_invalid`, "refusing to pad, because a short buffer means the
header was read wrong"). Eval point 6 of the round's own order says "corrupt
-> published code, never silent"; this was the one place it did not hold.
### The rule, read off the corpus rather than chosen
The order named two candidates — compare available bytes against
`biSizeImage`, or require an end-of-bitmap escape. Measured first, over the
19 real RLE8 assets of the frozen R761 delivery (`~/corpora/po-frosne-bundles/
r761-2025-58e1ecca6007/assets`, read only):
```
denominator (RLE8 files): 19
stream reaches an explicit EOB: 19 of 19
the EOB is the stream's LAST 2 bytes: 19 of 19
biSizeImage == available bytes: 19 of 19
bytes after the EOB: Counter({0: 19})
```
Both candidates are satisfied by every real file, so the choice is made on
what each MEANS. `biSizeImage` is a claim by the same untrusted header that
declares the size; the terminator is a statement inside the data that the
picture is complete. A third candidate, requiring every pixel to be painted,
is wrong by construction: it would refuse the delta escape the format
defines. So the decode may end at an explicit end-of-bitmap and nowhere else,
and a stream that runs out first is refused with `asset_samples_invalid`.
A stream that is WHOLE but omits the terminator is refused alongside a cut
one. From a reader's side they are the same bytes, and the measurement says
no writer in this corpus produces the first.
### Nothing real changes hands
The same 19 files still convert losslessly after the rule: **19 of 19,
2 366 365 pixels compared** — PM's own figure, re-measured here with stdlib on
BOTH sides (an independent BMP reader and an independent `zlib` +
filter-reversal PNG decoder, no Pillow anywhere), with a one-byte control
proving the comparison can fail.
The guard in the suite is pinned to a decoder written in the test file, which
computes what each cut costs: 8, 11, 17, 0 and 0 of 32 pixels. **Two of the
five cuts lose no pixel at all**, which is the whole reason the rule has to be
the terminator and not a pixel count.
## 2. The judge's new route could be forged by the document
`asset_holds` gained a second route in the viewable-asset round, because a
converted image's own bytes are not in `assets/` and never will be. It reads
the two digests the bundle states on the pointer line — with an expression
that ran over the WHOLE bundle text. So a document could write the sentence
itself. Measured by PM, and reproduced here before any code moved:
```
the 50000x50000 BMP was REFUSED and is not in assets/: True
judge asset_holds(refused BMP) = True (via the image's alt text)
forged via PLAIN BODY TEXT -> asset_holds(refused BMP) = True
```
Before that route existed, `asset_holds` hashed the source file and looked in
`assets/` — nothing a document wrote could reach it. The gate's own first
sentence is THE FASIT NEVER COMES FROM THE READER IT JUDGES, and `claimed and
not found` had become something a document could ask to have silenced.
### Two halves, and neither is sufficient alone
**The judge** reads the clause only from inside a pointer block — the markdown
image line plus the detail line under it — and only where the clause names the
asset that block points at. That closes ordinary body text and a table cell.
It cannot close an image's own `alt` text, because a LABEL is document text
that the build writes INSIDE a pointer block.
**The door** is `assets._inline`, and this is where the boundary runs:
everything it returns came from the document (an `alt` attribute, an STS
caption, a publisher's file name); everything `render_block` appends after it
came from the run (the size it measured, the type it sniffed, the digests it
computed). The second line carries both, so document text may not emit the
metadata grammar the run writes there. A checksum field is disarmed: the
digits are kept, because a reader is owed what the document said, and the
colon that makes them a FIELD is not.
### The mutants, and one finding about the harness
Three mutants in `tools/okf_gate_mutants.py`, one per check, each felled by
its own arm: the pre-round `_conversions` restored verbatim (X3), the
asset-name tie dropped (X4), `_inline`'s disarming dropped (X5).
**X5 survived 112 green tests on its first run, and had never been applied.**
The harness copies the tree and runs pytest in the copy with the venv's
interpreter — which carries an EDITABLE install pointing at the working tree,
so `import llm_ingestion_okf` in the copy resolved to the original `src/`. The
subprocess now gets the copy's own `src/` on `PYTHONPATH`, which wins over the
editable finder, and X5 dies on the first run under it. The gate and the
witness were never affected: the suite reaches those through the copy's own
`tools/`.
`killed 39 of 39, exit 0` (36 before this round).
## 3. Two small rests
**The judge proves carriage, not fidelity.** PM's M10: a mutated converter
writing a BLANK PNG gives `asset_holds = True`, because the bundle is
internally consistent — the digest it claims to have written really is the
asset's digest. The suite fells that mutant by decoding both sides; the gate
cannot. The docstring now says so, and the limit is asserted in BOTH
directions: a test shows the blank PNG is accepted AND that the docstring
names the gap. Teaching the judge pixels is a different job.
**The lossless guard now runs on a core install.** It decoded through Pillow,
which this package neither uses nor ships and which arrives only as a
transitive dependency of `pdfplumber` under `[extract]`. Measured by PM on a
core environment: 4 of the 13 guards in `tests/test_asset_viewable.py` were
SKIPPED, the lossless one among them, so a plain `pip install
llm-ingestion-okf` never measured the property the round is named after. The
new arm decodes the carried PNG with `zlib` and the five PNG SS 9.2 filters
and compares against pixels written out in the test file, over `read_image`
and over what the BUILD wrote. The environment is MADE rather than assumed: a
`sys.meta_path` finder raising for every `PIL` name, with a known-positive
that it fires. Control, run once and not committed: a Pillow-dependent line
placed inside the blocked section turns the guard red.
## 4. Two checks this round introduced and then removed
Both were measured and found unkillable, and a guard nothing can fell is not
a guard.
- The end-of-line anchor on the judge's clause expression. With the
pointer-block route and the asset-name tie in place, the only case it covers
is a label forging a clause inside a block that also carries a real one —
which `_inline` makes unreachable from a document. The whole gate suite is
green without it.
- The length check on a short absolute run in `_bmp_rle8_rows`. Unobservable
by construction: a run shorter than its declared count means the stream ran
out, so the loop cannot reach an end-of-bitmap afterwards and the picture is
refused either way. Measured in a scratch copy, both directions: **95 passed
with the check, 95 passed without it**, so the mutant that removes it
SURVIVES. The order asked for that mutant; this is the answer to it.
A fifth cut replaces them and makes an existing bound load-bearing: a stream
cut in the middle of a delta escape, 17 of 32 pixels wrong. Without the
two-byte bound in that branch it raises `IndexError` instead of a coded
refusal.
## 5. What this round did NOT measure
- The 43-document reference corpus was not rebuilt. The accounting gate's
rows are unchanged (`GATE RED: rows 2, 3, 6`, row 3 identical at u = 19,
d = 2, 17 of 22) — the operator-accepted losses of 2026-09-17.
- No R761 end-to-end build was run this round; the 19-file lossless
measurement is over the frozen delivery's own assets, which is the input
that half of the build.
- 16 JPEG 2000 objects are still lost with a line in the concept, and WebP is
in the viewable set while `sniff` does not recognise one. Both are stated
limits from the previous round and neither moved.