feat(assets): every carried image is one a model can be shown
Chosen: a stdlib BMP reader, because `read_image` is on the CORE path and an asset's name is its content digest. Measured first, as the order requires: Pillow 12.3.0 IS in this tree (transitively under `pdfplumber`) and it DOES decode RLE8 correctly -- a hand-written stdlib decoder and Pillow agree on 19 of 19 of R761's real files, RGB per pixel. So the choice does not rest on capability. It rests on two properties of this package: `.html` and `.xml` carry images with no `[extract]` extra installed, so a Pillow converter either makes a core path depend on an optional binary wheel or buys the second runtime dependency; and encoding through an installed library would make a bundle's identity move with that library's version, which is the property 0.10.0 felled page rasterisation over and `encode_png`'s docstring already defends. Pillow keeps the job it is good for: the INDEPENDENT decoder in the tests, on neither side of the conversion. The defect, measured over the frozen R761 delivery's `assets/`, denominator 50: 29 JPEG, 2 PNG and 19 RLE8 BMP. The 19 are byte-correct files nothing reads, so 19 figures were present and invisible while `images: N` reported that they had arrived. - `VIEWABLE_MEDIA_TYPES` is tested against every asset's SNIFFED type, so it is a property and not a list of formats we met. WebP is on it and `sniff` does not recognise one; the limit is stated, not implied. - `bmp_to_png`: 8-bit uncompressed, 8-bit RLE8, 24-bit uncompressed. All five RLE8 opcodes. 19 of 19 real files convert with RGB identical to Pillow's decoding of the source, 2 366 365 pixels compared. - `asset_not_viewable` and `asset_bmp_unsupported`, both published, both leaving the concept's "not carried" line. - Traceability on the pointer's second line, where the rest of the asset metadata already lives: original media type, original sha256 in full, new sha256 in full. A converted asset is ONE asset. - The ceiling is paid on the DECLARATION before a row is allocated, and an RLE run is one clipped slice -- painting pixel by pixel leaves the memory bounded and the CPU unbounded. Two repairs the change forced, each measured rather than assumed: - `tests/test_assets.py`'s "dimensions absent is absent" used a TIFF, which is now refused before `read_image` returns. The property still has a reachable case -- a JPEG whose frame header never arrives -- and uses it. - `asset_holds` in the accounting gate proved a carry by hashing the SOURCE file, which a converted image's bundle cannot satisfy. It now also reads the two digests the bundle states and HASHES THE ASSET ITSELF, so a bundle claiming a conversion it did not perform still fails. `tools/okf_asset_census.py` is the committed instrument for the known-positive: one row per image, from two pinned trees. It was caught by the rule it serves -- its first version handed `_pdf_images` the wrong page object and reported 0 images over 67 PDFs with exit 0. The attribute is asserted now and a known-positive runs before the sweep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
955ec4b2ca
commit
b0b5e71658
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
10 changed files with 826 additions and 5 deletions
156
docs/2026-09-19-hvert-baaret-bilde-kan-vises.md
Normal file
156
docs/2026-09-19-hvert-baaret-bilde-kan-vises.md
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
# Every carried image is one a model can be shown
|
||||
|
||||
`llm-ingestion-okf`, 2026-09-19. Order `20260919T033624Z-4535320370`, raised
|
||||
from PM's measurement of the frozen R761 package's `assets/` directory.
|
||||
|
||||
Nothing is tagged, pushed or released here. No new version: the entries go
|
||||
under the untagged `[0.10.1]`.
|
||||
|
||||
## 1. The defect
|
||||
|
||||
`file` over the frozen R761 delivery's own `assets/`, denominator **50**:
|
||||
|
||||
| Format | Count | A model can show it |
|
||||
|---|---|---|
|
||||
| JPEG | 29 | yes |
|
||||
| PNG | 2 | yes |
|
||||
| PC bitmap, Windows 3.x, 8-bit, compression 1 (RLE8 BMP) | **19** | **no** |
|
||||
|
||||
The 19 are byte-correct files. They are also files nothing decodes, so 19 of
|
||||
that document's figures were present and invisible at the same time — and the
|
||||
concept's own `images: N` reported that they had arrived. An absence a reader
|
||||
is shown is information; a picture that is there and unreadable is worse than
|
||||
either, because the count says the opposite.
|
||||
|
||||
Reproduced here before anything was written, with the committed census
|
||||
(§ 5): `19 of 50 CARRY image/bmp.bmp`, `29 of 50 CARRY image/jpeg.jpg`,
|
||||
`2 of 50 CARRY image/png.png`.
|
||||
|
||||
## 2. The decoder: measured, then chosen
|
||||
|
||||
The order requires the choice to rest on a measurement of what the tree
|
||||
already has. Pillow 12.3.0 is present, transitively under `pdfplumber` in the
|
||||
`[extract]` extra, and `PIL` occurs 0 times in `src/`, `tests/` and `tools/`.
|
||||
|
||||
**It decodes RLE8 correctly.** A hand-written stdlib decoder and Pillow were
|
||||
compared on all 19 real files, RGB per pixel through the palette: **19 of 19
|
||||
identical**. So the measurement's answer is yes, and the choice is not made on
|
||||
capability.
|
||||
|
||||
**Chosen: a stdlib reader. Why, and both reasons are properties of this
|
||||
package rather than preferences.**
|
||||
|
||||
1. `read_image` is on the CORE path. `.html` and `.xml` are stdlib file types
|
||||
that carry images with no `[extract]` extra installed, so a Pillow-based
|
||||
converter either makes a core path depend on an optional binary wheel or
|
||||
buys this package its second runtime dependency — against the "exactly one
|
||||
runtime dependency, ever" rule a packaging test enforces.
|
||||
2. An asset's name is its content digest, so the bytes the converter emits are
|
||||
part of the bundle's identity. Encoding through an installed library would
|
||||
make that identity move with the library's version. This is the property
|
||||
0.10.0 felled page rasterisation over, stated in `encode_png`'s own
|
||||
docstring, and it would be odd to defend it there and give it away here.
|
||||
|
||||
`encode_png` already writes a PNG from samples with `zlib` alone; this round
|
||||
adds the reader in front of it. **Pillow keeps the job it is good for: the
|
||||
INDEPENDENT decoder in the tests**, on neither side of the conversion.
|
||||
|
||||
## 3. What was built
|
||||
|
||||
- `assets.VIEWABLE_MEDIA_TYPES` — `image/png`, `image/jpeg`, `image/gif`,
|
||||
`image/webp`. `read_image` tests every asset's SNIFFED type against it. It
|
||||
is a property, not a list of formats we happened to meet. WebP is on the set
|
||||
and `sniff` does not recognise one, so a WebP is refused one step earlier as
|
||||
`asset_type_unknown`; the limit is stated rather than left implied.
|
||||
- `assets.bmp_to_png` — 8-bit uncompressed, 8-bit RLE8, 24-bit uncompressed.
|
||||
All five RLE8 opcodes: encoded run, absolute run with its pad byte, delta,
|
||||
end-of-line, end-of-bitmap. Bottom-up unless the declared height is
|
||||
negative. Palette read BGRA → RGB.
|
||||
- Two published codes. `asset_not_viewable` — a real image in a format no
|
||||
model can be shown, with no lossless conversion here (TIFF, JPEG 2000).
|
||||
`asset_bmp_unsupported` — a BMP variant this reader does not express (RLE4,
|
||||
BITFIELDS, 16/32-bit, BITMAPCOREHEADER, over 256 palette entries). Both
|
||||
leave the concept's own "not carried" line and a row in the run log.
|
||||
- `ExtractedImage.converted_from` and `.source_sha256`, written onto the
|
||||
pointer's second line where the source's file name and pixel size already
|
||||
live: original media type, original sha256 in full, new sha256 in full. Full
|
||||
digests because the asset's file name carries only 12 hex characters of the
|
||||
new one, and a checksum a reader cannot paste into `shasum -a 256` is
|
||||
decoration.
|
||||
|
||||
**A converted asset is ONE asset.** One file in `assets/`, one pointer, one
|
||||
row in the accounting — the accounting's own columns read
|
||||
`ExtractedDocument.files`, which is what the resolver resolved, and conversion
|
||||
does not touch it.
|
||||
|
||||
**The ceiling is paid before the pixels exist.** `bmp_to_png` bounds the
|
||||
DECLARED size through the same `check_size` the rest of the image path uses,
|
||||
before one row is allocated, and each RLE run is written as one CLIPPED slice.
|
||||
Painting pixel by pixel would leave the memory bounded and the CPU unbounded:
|
||||
a megabyte of `FF` runs is a hundred million paint steps against a 32-pixel
|
||||
frame.
|
||||
|
||||
## 4. The eval, red first
|
||||
|
||||
Two red commits, because the mutant survey found holes in the first one.
|
||||
|
||||
| Commit | What |
|
||||
|---|---|
|
||||
| `fea0435` | 12 guards, 8 red on their claim |
|
||||
| `9e99bb2` | 2 more guards the mutant survey found missing; 9 of 13 red |
|
||||
|
||||
The second commit is the interesting one. Walking the five mutants the order
|
||||
names against the eval as committed, **two survived**:
|
||||
|
||||
- *"the format is read from the file extension instead of the bytes"* survived
|
||||
because every image in the fixture was named after what it is: `.bmp` maps
|
||||
to `image/bmp` either way. A BMP named `graphics/figur.png` can tell the two
|
||||
apart.
|
||||
- *"the ceiling is checked after decoding"* survived because the guard only
|
||||
asserted that a refusal happens, and `encode_png`'s own `check_size` refuses
|
||||
too — one frame later, after the memory is spent. The guard already measured
|
||||
the peak; what it could not do was measure it affordably, since a
|
||||
50 000 × 50 000 frame is 2.5 GB. At 7 000 × 7 000 the declaration is still
|
||||
over the 40 MP bound and the unbounded frame is 49 MB.
|
||||
|
||||
**The lossless guard needed the same care.** Comparing "the carried asset that
|
||||
holds these pixels" against the source is green before any conversion exists,
|
||||
because a BMP carried verbatim trivially holds its own pixels. The holder has
|
||||
to be required to be VIEWABLE; with that clause the guard reads `0 carried
|
||||
assets hold these pixels` before, and passes after.
|
||||
|
||||
## 5. The measurements
|
||||
|
||||
**Lossless, on the real files.** Method: Pillow decodes the SOURCE BMP to RGB;
|
||||
`read_image` converts it; Pillow decodes the RESULT; the two sizes and two RGB
|
||||
buffers must be equal. Pillow is on neither side of the conversion.
|
||||
|
||||
- **19 of 19** lossless, 0 differing, **2 366 365 pixels** compared.
|
||||
|
||||
**Mutants, in a scratch copy of the tree.** Control green first, because
|
||||
"killed N of N" can also mean the harness cannot invoke pytest at all.
|
||||
|
||||
| Mutant | Result |
|
||||
|---|---|
|
||||
| M1 the format is read from the file extension | KILLED |
|
||||
| M2 the conversion skips 8-bit images | KILLED |
|
||||
| M3 the palette is ignored (grey out) | KILLED |
|
||||
| M4 the traceability field is omitted | KILLED |
|
||||
| M5 the ceiling is checked after decoding | KILLED |
|
||||
| M6 the BMP's BGR order is carried unswapped | KILLED |
|
||||
| M7 the RLE delta opcode is ignored | KILLED |
|
||||
|
||||
**killed 7 of 7, survived 0, not measured 0**, control green at 13 passed.
|
||||
|
||||
**The census** (`tools/okf_asset_census.py`) is the committed instrument for
|
||||
the known-positive. One row per image — the source's own sha256, the format
|
||||
read off its bytes, and what `read_image` did with it — over every image file
|
||||
and, with `--pdf`, every image object in every PDF it finds. Run from two
|
||||
pinned trees and diff the rows.
|
||||
|
||||
It was itself caught by the rule it exists to serve. Its first version handed
|
||||
`_pdf_images` a `page.page_obj` where the function reads `page.images` through
|
||||
`getattr(..., [])`, so it reported **0 images over 67 PDFs** and exit 0 — a
|
||||
clean-looking table that had measured nothing. The attribute is asserted now,
|
||||
and the known-positive (one PDF, 8 images) is run before the sweep.
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue