Chose the CURSOR over a pixel-coverage count because the corpus cannot
choose -- 25 of 25 of the R761 delivery's RLE8 BMPs paint every pixel,
25 of 25 reach the end of the frame, 0 of 25 use a delta -- and an
independent decoder can: a delta and an end-of-line escape state their
skip, so every decoder agrees on the index-0 pixels they pass over, while
a pixel count would refuse both constructions the format defines.
`_bmp_rle8_rows` now refuses (`asset_samples_invalid`) when the terminator
arrives with the cursor short of the last row. Pillow reads 5 of the 8
streams in the table and refuses the same 3, one of them short by a
single pixel.
Both docstrings the round was sent to correct are rewritten: the test no
longer claims every pixel is decoded (it is not -- a stated skip keeps
index 0), and `_bmp_rle8_rows` no longer frames the delta argument as
read off the corpus, which it never was.
R761 rebuilt: bundle `diff -r`-identical to the build before this commit,
50 assets (29 JPEG + 21 PNG), 19 of 19 conversions, SHY 71, u = 0, d = 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Chose the side channel over neutralising pointer-shaped document text,
because the second fix changes what every document SAYS in order to
defend a tool outside the build: a source quoting a bundle listing would
come out altered and existing bundles would move bytes. This reads a
file the run already writes.
`assets.conversion` names the pair, `DocumentAssets.conversions` carries
it out of the run, `DocumentAccount.conversions` books it, and the
accounting JSON states it per document. `_declared_conversions` reads it;
`_conversions` now believes a pair only when the RUN booked it AND a
pointer block confirms it for the asset it names. The confirmation can be
forged and the ledger cannot, which is why the ledger decides.
Measured through the real `okf build`: the three arms PM reproduced
(two `<p>`, one `<p>` with `<br>`, a markdown note beside the carrier)
go forged -> refused, 3 of 3, with the known-positive True in all three.
The text-level regression guard goes 3 arms to 13, the two new ones being
a perfectly written pointer block the run never booked.
R761, rebuilt: 25 BMP sources, 19 held, 19 of 19 conversions confirmed
against 19 declared, 50 assets (29 JPEG + 21 PNG, 0 BMP), SHY 71,
u = 0, d = 0, exit 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
160 passed over the three affected files. Two guards this round introduced are
removed, each because it was measured and found unkillable, and one real cut is
added in their place.
The end-of-line anchor on the judge's clause expression: with the pointer-block
route and the asset-name tie already in place, the only case it covered is a
label forging a clause inside a block that also carries a real one -- which
`assets._inline` now makes unreachable from a document. Removed with `$` gone
the whole suite stays green, which is the measurement: a guard nothing can fell
is not a guard, and it would also have made the mutant below unfaithful.
The length check on a short absolute run in `_bmp_rle8_rows`: unobservable by
construction. A run shorter than its declared count means the stream ran out,
so the loop cannot reach an end-of-bitmap afterwards and the picture is refused
either way -- the partial paint is discarded with the frame.
A FIFTH CUT REPLACES THEM, and it makes an existing bound load-bearing: a
stream cut in the middle of a delta escape, 17 of 32 pixels wrong by this
file's own decoder. Without the two-byte bound in that branch the cut raises
IndexError instead of a coded refusal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both red guards green, and the whole suite is 2204 passed / 1 skipped (2199
before this round, +5 new tests, no golden moved).
TWO HALVES, AND NEITHER IS SUFFICIENT ALONE. The judge now reads the clause
only from inside a POINTER BLOCK -- the markdown image line plus the detail
line under it -- and only where the clause names the asset that block points
at, anchored to the end of the line because the build writes it last. That
closes ordinary body text and a table cell. It cannot close an image's own
alt text, because a label is document text that the build writes INSIDE a
pointer block, which is the second half: `assets._inline` disarms a checksum
field in anything that came from the document.
WHERE THE BOUNDARY RUNS, stated in both files. Everything `_inline` returns
came from the document -- an alt attribute, an STS caption, a publisher's file
name. Everything `render_block` appends after it came from the run: the size
it measured, the type it sniffed, the digests it computed. The second line
carries both, so document text may not emit the grammar the run writes there.
The digits are kept, because a reader is owed what the document said; the
colon that makes them a FIELD is not.
The judge's expression stays restated rather than imported, for the reason
`asset_holds` already gives about the naming rule: a judge sharing the
judged's own expression agrees with it by construction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The four cuts now raise `asset_samples_invalid`, the code the UNCOMPRESSED
path already raises for a body that stops short. No pixel is guessed: the loop
may end at an explicit end-of-bitmap escape and nowhere else, and running out
of bytes -- mid encoded run, mid absolute block, or one opcode before the
terminator -- is a refusal with a row instead of a partly blank PNG.
WHY THE TERMINATOR AND NOT `biSizeImage` OR A COVERAGE COUNT. `biSizeImage` is
a claim by the same untrusted header, and a coverage count would refuse the
delta escape the format defines. Read off the corpus instead: over the 19 real
RLE8 assets of the frozen R761 delivery, 19 of 19 end at an explicit
end-of-bitmap, on 19 of 19 it is the stream's LAST two bytes, and on 19 of 19
`biSizeImage` equals the available bytes. A whole stream that omits the
terminator is refused alongside a cut one, because from a reader's side they
are the same bytes and the measurement says no real writer here produces the
first.
An absolute run shorter than its own declared count also stops the loop rather
than painting what arrived, so the refusal names the cut and not the frame.
KNOWN-POSITIVE, re-measured on the frozen delivery with stdlib on BOTH sides
(an independent BMP reader and an independent `zlib` + filter-reversal PNG
decoder, no Pillow anywhere): 19 of 19 still convert losslessly, 2 366 365
pixels compared, 0 refused, and a one-byte control confirms the comparison can
fail. 92 passed over the three asset test files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Chose the stated provenance over a second, independent counter, because a
second counter would be a number built to disagree. The door acts on the
EXTRACTED text; a counter inside `accounting.py` would have to re-read the
SOURCE bytes, and for every type extraction does not carry verbatim (pdf,
xml, html, docx) those are counts over two different strings. The gate would
then have to decide which difference is a loss and which is extraction doing
its job -- inventing a discrepancy where there is none. `accounting.py` does
not hold the source bytes at that point either; being a second reader of a
document is what `tools/okf_witness.py` exists for, outside the package.
Where an independent denominator DOES exist the tests already pin it from
outside the door: the fixture writes its own five, and R761's 71 is PM's
count over the delivery.
The `**Normalisation**` bullet now ends: "The count is the door's own, read
off the run and not recounted from the source."
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Chosen: a stdlib BMP reader, because `read_image` is on the CORE path and an
asset's name is its content digest. Measured first, as the order requires:
Pillow 12.3.0 IS in this tree (transitively under `pdfplumber`) and it DOES
decode RLE8 correctly -- a hand-written stdlib decoder and Pillow agree on
19 of 19 of R761's real files, RGB per pixel. So the choice does not rest on
capability. It rests on two properties of this package: `.html` and `.xml`
carry images with no `[extract]` extra installed, so a Pillow converter
either makes a core path depend on an optional binary wheel or buys the
second runtime dependency; and encoding through an installed library would
make a bundle's identity move with that library's version, which is the
property 0.10.0 felled page rasterisation over and `encode_png`'s docstring
already defends. Pillow keeps the job it is good for: the INDEPENDENT decoder
in the tests, on neither side of the conversion.
The defect, measured over the frozen R761 delivery's `assets/`, denominator
50: 29 JPEG, 2 PNG and 19 RLE8 BMP. The 19 are byte-correct files nothing
reads, so 19 figures were present and invisible while `images: N` reported
that they had arrived.
- `VIEWABLE_MEDIA_TYPES` is tested against every asset's SNIFFED type, so it
is a property and not a list of formats we met. WebP is on it and `sniff`
does not recognise one; the limit is stated, not implied.
- `bmp_to_png`: 8-bit uncompressed, 8-bit RLE8, 24-bit uncompressed. All five
RLE8 opcodes. 19 of 19 real files convert with RGB identical to Pillow's
decoding of the source, 2 366 365 pixels compared.
- `asset_not_viewable` and `asset_bmp_unsupported`, both published, both
leaving the concept's "not carried" line.
- Traceability on the pointer's second line, where the rest of the asset
metadata already lives: original media type, original sha256 in full, new
sha256 in full. A converted asset is ONE asset.
- The ceiling is paid on the DECLARATION before a row is allocated, and an
RLE run is one clipped slice -- painting pixel by pixel leaves the memory
bounded and the CPU unbounded.
Two repairs the change forced, each measured rather than assumed:
- `tests/test_assets.py`'s "dimensions absent is absent" used a TIFF, which
is now refused before `read_image` returns. The property still has a
reachable case -- a JPEG whose frame header never arrives -- and uses it.
- `asset_holds` in the accounting gate proved a carry by hashing the SOURCE
file, which a converted image's bundle cannot satisfy. It now also reads
the two digests the bundle states and HASHES THE ASSET ITSELF, so a bundle
claiming a conversion it did not perform still fails.
`tools/okf_asset_census.py` is the committed instrument for the
known-positive: one row per image, from two pinned trees. It was caught by
the rule it serves -- its first version handed `_pdf_images` the wrong page
object and reported 0 images over 67 PDFs with exit 0. The attribute is
asserted now and a known-positive runs before the sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operator decision 2026-09-18, and the whole of it: okf removes the soft
hyphen, the guard is untouched, and the number is BOOKED.
`normalise_extracted(text) -> (text, removed)` in `extract`, applied
once at the end of `extract_document`, so both entry points and both
sides of a plan's `text_sha256` see one string. `_pdf_units` measures
its page offsets through the same door, because that table is
CHARACTER offsets rebuilt from the pages while the text it indexes
comes back normalised -- two readings of one document, and a table
built against the other names the wrong page with full confidence.
Removing a character removes no newline, so every line-indexed rule
downstream is unmoved by construction.
`ExtractedDocument.soft_hyphens` carries the count out;
`InboxResult.normalised` carries it per document; `account_run` reads
it off the RUN rather than recounting the source, because a second
count would be a second reader. It reaches the accounting JSON as
`normalised_soft_hyphen` at both levels and `log.md` as a
`**Normalisation**` bullet naming the count, the documents touched and
that no other character is moved.
EXPOSURE, measured and not assumed: 0 of the 78 readable documents of
the K2 reference corpus carry U+00AD or any of the four real
zero-width characters (the 8 unreadable ones raise before extraction),
0 in the pinned K2 bundle's concept bodies, and 0 across
`tests/fixtures`, `examples`, `skills`, `docs`, `src`, README and
CHANGELOG. The door cannot have moved a byte anyone here has measured.
Suite 2179 passed, 1 skipped; ruff and `mypy --strict src/` clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gate gains a FIFTH column, `Unit.refused`, and it is the only one
that is not a defect in the report: the elements of a document the
build read and persisted nothing of. Their fate is declared honestly,
so `unaccounted` and `double_booked` both stay 0 -- which is exactly
why nothing else could see the loss. `refused_whole` asks its question
only for a corpus that persisted NOTHING, so one refused source beside
an accepted one, the ordinary case on a heterogeneous corpus, reached
row 3 as clean.
Row 3's reason now carries `N element(s) lost with R of D document(s)
refused whole`, and each unclean unit's detail line carries
`refused=` beside u, d, unverified and invalid, with the document's
own rejection code in the note.
On the build side `Accounting.refused` is written into the JSON and
into the `**Accounting**` bullet of `log.md` as `R of D document(s)
refused whole`. The exit code is NOT moved: it belongs to the whole
run, and a corpus holding one unreadable file among many is ordinary,
so the order's other half -- state it in the accounting -- is the one
taken. `okf build` still exits 1 when it persisted nothing at all.
`test_a_corpus_refused_whole_under_the_default_gate_is_red` kept its
point and lost its premise: the numbers still balance, and that is now
asserted as u = 0 and d = 0 rather than as a clean unit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Round 3 of the 0.10.1 review, and the finding is the pattern the three rounds
share: each bound an OUTPUT, and the bomb stepped one link along. The
declared size, then the first `FlateDecode`, then every `FlateDecode` -- and
then a link this package had documented as safe.
`ASCII85Decode` was classed as bounded "by its own input because it shrinks".
It quadruples: `z` is the shorthand for four zero bytes. And the output was
never the cost -- `base64.a85decode` appends one 4-byte object per group to a
list, about a hundred bytes of memory per byte of INPUT (101.4x at 1 MiB,
96.1x at 4 MiB, 94.5x at 16 MiB on CPython 3.14).
Paired subprocesses, idle machine, both sides from PINNED trees, the document
built once by a third process and read from a file because `ru_maxrss` never
falls and `b"z" * 64 MiB` alone costs 171 MB:
[/Fl /A85] z x 32 Mi 33 475 B CARRIED 3 261 599 744 -> too_large 42 070 016
[/Fl /A85] z x 64 Mi 66 090 B CARRIED 6 461 558 784 -> too_large 40 280 064
[/A85] z x 8 Mi 8.4 MB CARRIED 933 085 184 -> too_large 62 484 480
[/Fl /A85 /Fl] z x 32 Mi 33 488 B samples_invalid 3 519 180 800 -> too_large 43 438 080
The picture was CARRIED in three of the four: not a bound that fired late, no
bound at all. Doubling the `z` run trebles the old cost and leaves the new one
where it was.
WHY THIS FORM. `assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE
link may cost -- a separate number from `MAX_IMAGE_BYTES`, because that one
bounds the picture and this one bounds producing it. `FlateDecode` is measured
as it is paid; every other permitted filter carries a MEASURED cost ratio
(`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its decoder
is called, since those decoders take a whole string and return a whole string.
A filter with no ratio is refused unread. The budget TRAVELS: a deflate link
is inflated under the smaller of the picture's bound and what the next link's
decoder may be handed, or `[/Fl /A85]` pays 256 MiB for a refusal.
A chunked ASCII85 decoder written here was the alternative and was FELLED: it
would bound `_check_stream_cost` and not the run, because `stream.get_data()`
decodes the whole chain again with pdfminer's own decoder, and it would make
this package rather than pdfminer the authority on an image's bytes. The cap
is the only number that bounds that. `resource.setrlimit(RLIMIT_AS)` was
MEASURED before anything was built on it, as the order required, and is not
usable: Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit`
and does not enforce it. No child-process cap exists.
THE CAP IS READ OFF THE CORPORA, the posture `MAX_IMAGE_PIXELS` has: over the
9 668 image objects of the 77 PDFs on this machine, 16 decode through an
ASCII85 link and the largest input to one is 450 739 bytes, against a cap of
about 5.0 MB.
A PROPERTY TEST REPLACES THE LIST OF KNOWN SHAPES: every chain of length 1-3
over the ten filters pdfminer decodes, 1 110 of 1 110, both payload fills,
each delivered under the bound or refused with a published code and never paid
for on the way (`tracemalloc`, which counts allocations and is not disturbed
by load). Known-positive beside it: 258 of 258 chains over the permitted
filters still carry a small image.
MAJOR -- the backstop had no test. `check_payload` at the end of
`_check_stream_cost` could be deleted with the whole suite green, because the
second one after `get_data()` gives the same code one step later. The two
differ in whether the payment was made, so the test asserts `get_data` was
never called.
10 OF 10 MUTANTS KILLED, control green, each killer named in the report. Four
survived a first pass and two tests exist because of it.
NOT ONE PICTURE CHANGES HANDS, MEASURED BY NAME: `_pdf_images` over every PDF
on this machine from both pinned trees -- 9 306 -> 9 306 carried over 77
files, 50 -> 50 on R761, 0 of 78 files moving a count and 0 moving a code.
R761 also settles a question raised while this order was open: 50 objects, 29
[/DCTDecode], 21 [/FlateDecode], 0 ASCII85 links -- so round 2's count of 580
`[/FlateDecode /ASCII85Decode]` objects is reproducible from nothing on this
machine. It changes no decision; a bomb shape does not need a corpus.
Version stays 0.10.1, no tag. README, CHANGELOG, CLAUDE.md and errors.py
corrected TO what the code does; the round-2 report carries a correction block
rather than a rewrite. Report:
docs/2026-09-18-utgangsbudsjett-per-ledd.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two findings of the 18.09 PM checkpoint of `0f308c1`. Red tests landed
first in `3b587ea`; this is what turns them green.
BLOCKER -- `_check_inflated` read `filters[0]`, measured that one link and
returned, which is not a bound: a PDF decodes a stream through a LIST of
filters. Measured in paired subprocesses from two pinned trees, idle machine:
[/FlateDecode] 400 MB 408 516 B 59 232 256 -> 62 017 536 B
[/FlateDecode x2] 400 MB 1 636 B 886 554 624 -> 52 367 360 B
[/FlateDecode x3] 400 MB 1 070 B 889 393 152 -> 61 390 848 B
[/FlateDecode x2] 1,2 GB 2 927 B 2 567 204 864 -> 60 403 712 B
542 000x the file at two links, and the picture WAS refused at the end -- by
`check_payload` after `get_data()`, once the memory was spent. The single-link
row is the control and does not move. It also left the 16 corpus objects behind
an `[/ASCII85Decode /FlateDecode]` chain unmeasured, since `filters[0]` is not
`FlateDecode` there.
`_check_stream_cost` walks every link. THREE CLASSES and no fourth
(`extract.bounded_pdf_filters`, pinned by a test): `FlateDecode` MEASURED, a
link with another expanding link behind it inflated under the same bound and
handed on; `ASCII85Decode`/`ASCIIHexDecode` bounded by their own input because
they SHRINK; `DCTDecode`/`JPXDecode`/`JBIG2Decode` PASS THROUGH. Everything
else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything
written later -- is refused UNREAD with a new code `asset_pdf_unbounded`,
decided before the FIRST link is decoded so a document cannot make the run pay
for the links in front of the one we cannot bound. An encrypted stream is
deciphered and then measured, where `stream.decipher is not None` used to
return unmeasured; 0 of 5 142 objects here are in an encrypted document, which
is why nothing caught it.
NOT ONE PICTURE CHANGES HANDS, AND IT IS MEASURED BY NAME. Every PDF on this
machine -- 78 documents, K2 in both trinn1 and trinn2, the shipped fixtures and
R761 -- run through `_pdf_images` page by page from both pinned trees:
images carried 9 356 -> 9 356
documents losing one 0 of 78
documents gaining one 0 of 78
asset_pdf_unsupported 322 -> 314
asset_pdf_unbounded 0 -> 8
The 8 are the 4 `CCITTFaxDecode` stencil masks (`/ImageMask true`,
`/BitsPerComponent 1`), counted twice because trinn1 and trinn2 hold the same
document. They were refused before and are refused now, one step earlier and
under a code that says why.
MAJOR -- `check_payload(len(data))` after `get_data()` is the counted refusal
four documentation surfaces point at, and deleting exactly that line passed all
2 132 tests. It is reachable through a stream pdfminer has ALREADY decoded
(`decode()` sets `rawdata` to `None`), which is now the ONLY case outside the
bound and has a test.
Eight mutations, one line each, every one DEAD, with the unmutated tree run
first as the control: first-link-only, loop dropped, inequality reversed,
encrypted skipped, backstop deleted, unknown filter passed through,
intermediate link not carried forward, whole check removed.
`tools/okf_accounting_gate.py` gains one line, the new code in
`REJECTION_CODES` -- what a rejection code requires and nothing more. Gate
unchanged: exit 1, GATE RED rows 2, 3, 6. Version stays 0.10.1, untagged.
Suite after `git add` against a clean tree: `uv run pytest -q` ->
2152 passed, 1 skipped (226 s). ruff, ruff format --check, mypy --strict clean.
Report: docs/2026-09-18-filterkjeden-og-backstoppen.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A second independent review read `230d1cb` -- the commit that closed the
`v0.10.0` review's two MAJOR findings -- and found one of them open. The
bound read `/Width` and `/Height`, which an untrusted document writes, while
`get_data()` pays for the stream beside them; `/Length` is the COMPRESSED
length and the two numbers are independent.
Re-measured here on `ed8d9d7` before anything changed, in its own
interpreter: a 408 516-byte PDF declaring 1x1 and carrying 400 MB of deflated
zeros was CARRIED, no rejection, 891 904 000 B peak RSS. After: 0 carried,
`asset_too_large`, 57 065 472 B. At 1,2 GB of zeros, 2 436 MB -> 64 569 344 B
-- the cost no longer scales with the bomb. End to end through the CLI with
the shipped defaults: 838 000 640 B and an asset written -> exit 0,
79 650 816 B, `0 carried of 1 found`, no `assets/`.
Three numbers are bounded now, not one: what a container DECLARES, what a
carried FILE measures (`read_image`, so a 49 MP PNG of 47 705 bytes is not
passed on to a consumer), and what a PDF stream DECOMPRESSES to
(`assets.inflated_size`, a chunk at a time, output discarded, before
`get_data()`). The limit is stated rather than implied: the stream
measurement runs where `FlateDecode` is the first filter and the document is
not encrypted; every other chain is a check on the decoded length AFTER the
decode, a counted refusal and not a bounded one.
A non-positive declared dimension is `asset_size_invalid`, its own code,
raised before the stream is read. `-1 x 40000000000` is a NEGATIVE pixel
count, under which every `>` bound read as satisfied, so the check returned
silently and the refusal arrived from `encode_png` as
`asset_samples_invalid`. Its own code because a publisher shipping a picture
bigger than this package carries and a dictionary written to be read wrong
are different facts about a document.
Two smaller findings in the line that says what is missing, both introduced
by the first fix: the address was written twice, once bare, and a linkifying
renderer autolinks a bare URL -- written once now, in one code span; and
`label` became a dead parameter, so the figure's caption was dropped, a
regression against 0.10.0. It is written again in the `-- <label>` form a
carried pointer uses.
Version bumped to 0.10.1 across all ten places. Nine were unbound and stale:
four README install lines naming the previous release, two prose lines, the
"current tag" entry, `uv.lock`, and a CHANGELOG whose 0.10.1 content sat
under `[Unreleased]`. Two new packaging tests bind them to `__version__`, and
the README's guard tag to `[tool.uv.sources]`.
Every test was red first. The fate of every image is identical with and
without the new bound on three K2 PDFs carrying 800 images (464/464, 311/311
with the same 12 rejections, 25/25), and the second inflate is below the
noise floor there. 0 shipped artifacts move: no bundle under `examples/`,
`skills/` or `tests/fixtures/` carries an image pointer at all, measured
against a known-positive control.
`asset_too_large` was undocumented in the error registry; both codes are
there now. `tools/okf_accounting_gate.py` gains the new code in its closed
list -- one string, no behaviour change, stated because that file belongs to
another order.
Suite 2141 passed / 1 skipped, ruff + format + mypy --strict clean.
Report: docs/2026-09-18-bildestien-holder-0-10-1.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two MAJOR findings of the independent v0.10.0 review, both with the
shipped defaults, both new in 0.10.0. Repros rebuilt as tests first.
- A remote <img src>/xlink:href became a LIVE markdown image link in the
persisted concept, with the address and query string chosen by whoever
wrote the document. Extraction opens no socket; a consumer rendering
the bundle does. Now inert text with the address in a code span,
pinned by a property over the readers rather than by one string. The
tier asymmetry (user-upload refuses, trusted-source persisted) went to
the guard repo with the repro.
- Nothing bounded a declared image size: 9.6 KB of PDF declaring
3000x3000 grayscale zeros took 83 MB peak RSS, linear in pixels.
MAX_IMAGE_PIXELS (40 000 000) and MAX_IMAGE_BYTES (256 MiB) are read
off the corpora (largest measured 18.6 MP on K2, 1.4 MP on R761) and
checked on what the container declares, before any decompression;
over them is asset_too_large, counted. The same bound closes the
inline data: URI, which the review flagged and did not measure.
Also fixed, added by PM to this order: an inline PDF image was named
from id() of a Python object, so two concept files of the reference
corpus differed between builds. It is now named from its position.
R761 unchanged: 50 carried of 50 found, assets diff -rq clean.
Report: docs/2026-09-17-bildestien-0-10-1.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
okf build --accounting PATH inventories every source document before
extraction, in the gate's per-format vocabulary, and after the run gives
each element one fate (carried / pointer / coded rejection), written as
JSON and summarised in log.md. "carried" is checked against the written
concept bodies, so a gate that drops a line is found (test). Exit 1 on
anything unaccounted or double-booked. Opt-in: +744 s (+19 %) on the
43-document reference corpus, and that corpus fails the check on 24 real
losses (22 images on text-less PDF pages, 2 docx Title paragraphs).
Changed without the flag:
- okf build exits 1 when it extracted documents and persisted none.
Door B and corpus.measure are unchanged. One test relied on exit 0.
- An image file carried through a persisted document is its own K1b
column, no longer also extractor_unknown. The set is what the resolver
actually carried (ExtractedDocument.files), never a byte match.
tools/okf_accounting_gate.py (checks untouched) is green on all six rows,
R761 110 of 110 under both gates.
Report: docs/2026-09-17-innholdsregnskapet-bygget.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.
Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.
THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.
FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.
A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.
NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.
ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.
THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.
Also fixed, both found by measuring rather than by reading:
- a markdown image is no longer read as a cross-reference. `structure._LINK`
never looked at the character in front of the bracket, so every pointer
would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
no `assets/` at all, so every pointer named a missing file.
Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two defects reported from outside by claude-code-llm-wiki (order
20260915T202332Z-228694739), both reproduced against this tree before
anything moved.
F1 -- the packaged CLI never ran the guard. corpus.measure wired an
unconditional approve-everything stub into process_inbox and 0 of 90
add_argument calls named a gate, so the one path most people use screened
nothing while pyproject.toml made the guard a mandatory runtime dependency
and the README recommended a composition the command line could not reach.
--gate takes guard-trusted-source (default), guard-user-upload or none.
corpus.resolve_gate is the one name->callable map, with the guard imported
lazily so importing the package still does not pull it in; an unknown name
RAISES rather than falling back, because a fallback reproduces the defect
with an extra step. The gate's NAME goes into the section 9 log.md -- a
stub is only dangerous when nothing downstream can see it -- and --gate
none renders NOTHING WAS SCREENED.
The default was chosen on a measurement: over the 453 concept bodies of
the pinned reference bundle, PRESET_TRUSTED_SOURCE persists 453 of 453 and
PRESET_USER_UPLOAD holds 1, costing that concept's whole source document.
Neither tier waves anything through -- an invisible carrier and a CRITICAL
finding fail secure at both. Door B's library default is UNCHANGED at
PRESET_USER_UPLOAD: an inbox drop is an untrusted upload, an operator
pointing this command at their own folder is not. The second tier ships as
guard_adapter.inbox_gate_trusted_source, the three-line adapter that
module's docstring already described, never a preset parameter.
process_inbox(segmentations=..., gate=inbox_gate) now has a test. Before
this, `grep -rl inbox_gate tests/` gave 1 file with 0 occurrences of
`segment` -- the recommended composition was untested, which is how the
defect survived.
F2 -- a fenced code block declared structure. `# Use the opus[1m] alias`
inside a ```bash fence became a level-1 ATX heading: the document was
refused entirely where the line carried [ or ] (5 of 191 pages of the
reporter's corpus), and the concept TITLE came from somebody's shell session
on 62 of 191 (32.5 %). The fix is in the proposer and never in Door B's
title rule -- that rule is right, and a heading that was never a heading is
what has to stop being proposed. propose.fenced_lines is computed once per
text and no rule reads a fenced line, including Arm D's outline RUN, which
selects from the whole line list. Backtick and tilde fences, three leading
spaces, a closing fence at least as long as its opener, and no backtick in a
backtick fence's info string -- that last one keeps a line holding only
`okf build` from silencing a document.
MEASURED ON THE BYTES, and this is the number that decides: the 43-document
reference corpus built at b6da09c (from git archive, never the editable
tree) and rebuilt at the shipped defaults differ in log.md alone, by the one
added bullet. 865 concept files on both sides, every concept byte-identical.
Found by that control and NOT caused by this work: the pinned artifact
K2-bundle-default-20260912 was written 2026-09-09, two days before ed0418f
changed title: quoting, so it differs from what HEAD produces on 42 concept
files. test_default_bundle_pin stays green because it pins the count and the
hit@8 ranks, not the bytes. Re-pinning is the operator's call.
Suite 1896 passed / 1 skipped (+27 from 1869). ruff, ruff format and mypy
--strict clean. No version bump, no tag, no push.
Report: docs/2026-09-15-f1-f2-gaten-og-kodefencen.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bundle-only gate returned UNMEASURED and exit 3 on the very arm it was
built for: no metric computable from a bundle alone reaches boundary recall.
`boundary_share` -- declared boundaries that became a concept, over declared
boundaries -- is the one metric measured that orders the arms correctly, and it
needs the publisher's own structure, so it arrives as an input.
Measurement first, threshold after, which is what the order asked for.
P1, the normalisation, derived rather than guessed: stripping all whitespace
and lowercasing reproduces the fasit's own `norm` from its own `title` on
2 761 of 2 761 rows (alphanumerics-only scores 58). P1's own bar is 99 % on the
known-good arm and the literal reading of it reaches 22 of 2 761 -- not because
the normalisation is wrong but because okf's default route moves the numbering
token a publisher glues into a heading over into the concept id. The pair form
(concept's own directory, residual title) reaches 2 737, either reaches 2 759
(99.9 %). Both forms ship and neither is a fallback: `r761-2025-d1` is the
control in the opposite direction at 2 727 literal, 0 paired.
P2, the single corpus, is in the OUTPUT and not only in the document: the bar
is declared `corpora = 1`, every boundary row prints `N = 1 corpus`, and the
line states that `--fasit` is the caller's ASSERTION that this bundle is a
build of the document the fasit describes -- the posture `okf consume --ref`
has. Measured: the K2 reference and `n100-2023` score 0 of 2 761 and read FAIL,
which is the assertion being wrong rather than the bundle being bad.
One bar, at the pinned artifact's own value, 2 759/2 761. It is tight and the
cost is published rather than tuned away: 2 of 4 R761 builds fall under it
(2 752 and 2 727), while any bar between 41.6 % and 98.8 % separates the
known-bad arm from every R761 build measured. The known-bad arm
(`860019-mdb-100`) is 1 148 of 2 761 -- FAIL and exit 1, where the bundle-only
gate gave exit 3.
A fasit is validated at the door: not a list, a row missing `title` or `norm`,
or anything that is not JSON exits 2 with the reason, never a quiet UNMEASURED.
A fasit under five rows is UNMEASURED -- the document floor in the fasit's own
unit.
Without `--fasit` the command is byte-for-byte what it was, held by a test.
`okf check` is untouched; no version bump and no tag. 17 tests red on
assertions before the implementation, the two new doc pins each driven red and
back. Suite 1 869 passed / 1 skipped / 1 870 collected (base 5e5d01c: 1 851).
docs/2026-09-12-g37-terskler.md SS 7 carries the premises re-measured, the
seven bundles, the interval any bar could sit in, and the honesty limits --
including the correction of SS 2's own grep claim, which went false in the
commit that wrote it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Version to 0.9.0 in pyproject.toml, __init__.py and uv.lock; CHANGELOG's
[Unreleased] becomes [0.9.0] - 2026-09-13, carrying G37's `okf quality` and
K3-26's README file-type table byte-identically. MINOR rather than patch: the
release adds a command.
README's four install lines and the two tag-pairing sentences move to v0.9.0,
and the tag history gains a `v0.9.0` entry stating what ships -- three
verdicts, two bars (.pdf 8/32, .docx 2/5), both regression bars against the
pinned reference and not quality claims, `okf check` untouched at seventeen
rules.
No behaviour change in src/ beyond the version string. The guard pin stays at
v1.4.0 and the dependency floor stays at >=1.2,<2.0. Suite 1 850 passed, 1
skipped, run after `git add`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
G37. `okf check` is a CONTRACT check and a green one is not a quality gate:
measured 2026-09-10 by `vegnormal-okf`, three arms over one corpus all
returned 0 findings and exit 0 while their hit@k ranged from 6 of 6 to 0 of 6.
`okf quality <bundle>` asks the other question, per file type, with the
denominator on every line. A separate command rather than `okf check
--quality`, because the two answer different questions and a caller must not
be able to read one as the other. `okf check` is untouched.
Three verdicts and no fourth -- PASS, FAIL, UNMEASURED -- and a type with no
measured threshold is never PASS. Exit 0 judged and clean, 1 at least one
FAIL, 2 did not run, 3 nothing could be judged: exit 0 over a table of
unmeasured rows would be the silent pass this command exists to stop.
Two bars today, both `structure_null_share` (documents of a type yielding
exactly one concept), read off the pinned 43-document reference bundle: .pdf
8/32, .docx 2/5. Plus one definitional bar for every type, taken from the
harness's own degenerate-merge definition: 0 concepts with an empty body,
measured 0 of 8 602 concepts over four bundles. A bar needs five documents on
BOTH sides -- its own and the judged bundle's -- so .xlsx (2), .xml (1) and
every type with no corpus class in `extract._EVIDENCE` are UNMEASURED and
print their numbers without a verdict.
The floor on the judged bundle was found by RUNNING the gate, not by reading
it: one PDF cut into 2 182 concepts scored 0 of 1 against the 32-document
reference and read as PASS.
The gate walks the index tree and never a directory (SS 9.2; controlled
against the listing on four bundles, 453 / 2 761 / 3 206 / 446 either way),
and prints the bundle's own run log beside its counts -- a document rejected
at extraction leaves no row in the bundle, so the pinned corpus's 33 PDFs
show up as 32 and the two denominators must never be read as one.
Three of the order's five premises moved when re-measured, and they are in the
document rather than glossed: the four evidence corpora carry `source_file` on
0 of 446, 0 of 1 133, 0 of 270 and 0 of 2 756 concepts, so they name no file
type and cannot PASS; "41,6 %" is `vegnormal-okf`'s number and not in this
repository; and the same 828-document bundle carries two published hit@k
figures from two question sets.
Three candidate metrics measured and NOT shipped: duplicate titles within a
document (0 of 3 206 on the known-bad arm against 349 of 2 761 on the
known-good one) and short concepts (5.6 % against 14.6 %) order the two arms
the wrong way round; duplicate titles across the whole bundle order all four
correctly (37.8 / 16.3 / 12.6 / 5.7 %) and still ship without a bar, because
any bar separating them is read off the two bundles it would judge.
19 new tests, each rule exercised in both directions; the three README pins
were each driven red before being kept. Suite 1 850 passed, 1 skipped, 1 851
collected, run after `git add` -- +19 against a base of 1 832 collected,
measured on the stashed tree (STATE's 1 831 is one short of that).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Version to 0.8.5 in pyproject.toml, __init__.py and uv.lock; CHANGELOG's
[Unreleased] becomes [0.8.5] — 2026-09-12, carrying K3-23's instrument,
K3-25's default and K3-24's block-sources fix byte-identically, with the
two `### Changed` headings merged into one and one post written here: what
`okf.parse_frontmatter` now returns for a block `sources:` is a READING
projection PyYAML reads back on 0 of 4 605 measured block files.
No behaviour change in src/ beyond the version string. The guard pin stays
at v1.4.0 and the dependency floor stays at >=1.2,<2.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`link_in_signal` defaults to `False` on all three entry points --
`searchable_text`, `concept_scores` and `build_payload` -- through one named
constant, `DEFAULT_LINK_IN_SIGNAL`. This carries out K3-23's recommendation
(alternative c), and its numbers are the whole warrant: of the newcomers the
door's `Enclosing section:` line ever added a question token to, 39 of 39
gained it from the bundle-absolute PATH and 0 of 39 from the link's title, and
every token the path contributed is a segment of the document's own directory
-- the saturation `shared_id_prefix` takes out of the id signal, back in
through the body.
A CONSTANT rather than three naked literals, chosen against the module: the
other eight defaults here are constants (`DEFAULT_PROFILE`, `DEFAULT_LIMIT`,
`DEFAULT_TIE_SHARED_RANK`, `DEFAULT_STEM_PREFIX`, `DEFAULT_TITLE_COVERED`,
`DEFAULT_SOURCE_QUOTA`, `DEFAULT_FOLLOW_PARENT`, `DEFAULT_K`) and this was the
one default spelled out three times. It also lets a test fall on the rule from
the module that owns it.
NO CLI flag is added and none is changed. `grep -n '"--.*link'` over `src/`
gives 0 hits before and after, and `consume.main` never passed the parameter,
so the CLI inherits the function default. A flag was considered and NOT added,
because no caller could be named that needs the older reading: the three call
sites outside `consume.py` (`skill.py:360`, `tools/okf_consume_measure.py:155`
and `:247`) all inherit it deliberately.
Measured before landing: 0 of 5 shipped bundles carry the door's line
(denominators 446, 1 133, 270, 2 756, 2 761 concept files) and 5 of 5 payloads
are byte-identical across the move. The consumer repository was read only and
its tree is unchanged, listing identical including mtimes.
The docstrings move with the code rather than after it. `concept_scores` said
this was "an INSTRUMENT, not a flag and not a default" and `build_payload` that
it "carries no CLI flag"; the first is now false and the second still true, and
each says so. `cli.DEFAULT_SHELL_PARENT` keeps its value and loses its reason:
the cost it cited -- delivered sets moving through that path -- is what this
commit removes, so the comment now states why it stays off NOW, which is N = 1
document and a consumption half nobody has judged. README and CLAUDE.md carry
the same sentence where they already describe the line, measured rather than
written on reflex: neither said anything about the ranking before.
One test changed with the code. The characterisation of what the LINE does
called `concept_scores` with no parameter, because the reading it characterises
used to be the default; it now names that reading. Same fixture, same number.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One parameter, `link_in_signal`, default `True` -- so no payload moves. `False`
scores the body with `inbox._link_enclosing`'s line removed, and removes it from
the stem vocabulary too, because `searchable_text` counts `df` over the text a
hit is scored on: taking the line out of one and leaving it in the other would
measure two different texts.
The recognition is the door's own constant plus its own place -- last in the
body, after a blank line, in the door's link form. The excerpt keeps the line
either way, so two readings of ONE bundle differ in ORDER alone, which is what
separates a ranking movement from a budget displacement.
Not a CLI flag: whether this becomes a default is K3-23's measurement, not an
implementation's decision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_render_sources` gave three measured reasons for refusing to emit the block
form. Two of them fell with `28f9a4b` and are struck rather than left
standing: a block list no longer round-trips to an empty value (4 609 of
4 609 concept files), and B6's acceptance test can now pass.
Reason 2 STANDS and now carries the emission rule alone, re-measured by
reading `portfolio-optimiser` at `6eb58e5`: `read_provenance` returns
`UnreadableProvenance(reason="block-sequence")` for a block value. It is not
the guard's objection -- guard 1.4.0 reads the block form on 4 609 of 4 609
of the same files -- so the answer to "should our own emitter write block" is
NO, with the number that carries it, and the measurement that would make it
decidable is named in STATE rather than deferred without one.
The README paragraph said this library "cannot read the block form" where one
reader could and one could not. It now separates the two by KEY: `sources` in
either form through every reader, every other block mapping still refused,
with the D1b pointer intact.
Report: 222 lines, S7 form. Five deviations first, including the acceptance
sentence naming THREE flat readers rather than one, a `title` denominator of
7 372 rather than 5 372, and the `okf check` rule answered NO with a
measurement (`contract_check` holds 0 occurrences of `sources`, and
`consume` already writes `sources_unreadable` for the state a rule would
restate; `len(RULES)` is 17). Honesty limits name the one-key-wide set, the
block `verified:` fixture still reading empty, and the rendering not being
byte-identical to the source.
Suite 1807 passed / 1 skipped, rc 0, 98 s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One grammar, four call sites. `read_block_mappings` moves out of
`consume.read_sources` -- where it was written and measured -- into
`profiles`, the module both the flat readers and `consume` already import,
and the three copies of the line-oriented frontmatter grammar now decode a
block sequence for the keys `STRUCTURED_BLOCK_KEYS` names. Two copies of a
block grammar would be two answers to one question.
The value TYPE was the real choice and it was measured, not argued.
`parse_frontmatter` is public API (`okf.parse_frontmatter`) returning
`dict[str, str]`, and a list of mappings is not a `str`. Widening the return
type to `str | list[dict[str, str]]` costs 15 `mypy --strict` errors across
four of the five modules that touch the reader, plus a signature every
caller outside this repository would have to follow. Rendering the entries
back into the flow form those same readers already round-trip costs 0. The
rendering is a READING projection and says so: it is not a claim that the
value is writable -- `yaml_flow_plain` still refuses a `?` and the guard
still refuses a quote inside a flow mapping, which is why the producer
writes block in the first place.
`STRUCTURED_BLOCK_KEYS` is one key wide. `sources` is the key `read_sources`
already knows how to read; a fixture in this tree carries a block
`verified:` that still reads as an empty value, and a test pins that state
so the next widening is a decision rather than a side effect.
Nothing nested reaches the document's namespace: the entries land inside
their own value, and the K3-20 substitution guarantee is asserted per reader
copy.
Three tests that pinned the old behaviour are rewritten to what is now true,
none weakened on its other half: the block round trip in
`test_multi_source_provenance` (the evidence behind `_render_sources`'
reason 1), the v0.2 characterization (whose key-space assertion is the half
that must never weaken), and K3-22's shipped-file known-positive, where the
one difference is counted and pinned at 1.
Suite 1807 passed / 1 skipped, rc 0, 94 s -- 1782/1 before plus 25 new.
ruff clean, `mypy --strict` clean over 21 files, `uv.lock` untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Version 0.8.3 -> 0.8.4 in pyproject.toml, __init__.py and uv.lock, and the
guard pin in [tool.uv.sources] moved from v1.3.0 to v1.4.0 (uv.lock follows
via `uv lock`: the guard's version, source and requires-dist lines, nothing
else). CHANGELOG [Unreleased] closed as [0.8.4] with K3-21's and K3-22's
entries moved as written, plus one entry for the guard pin: 1.4.0 parses a
flow sequence of plain scalars that 1.3.0 refused (26 of 28 frontmatter
blocks refused by 1.3.0, 0 of 28 by 1.4.0, on one five-document bundle).
Dependency range llm-ingestion-guard>=1.2,<2.0 unchanged.
README: the four install lines and llms.txt's one to @v0.8.4, the guard
pairing prose to v0.8.4 / v1.4.0, the pip fallback to @v1.4.0, a v0.8.4
row as the current tag (seventeen checker rules), and the suite count
(1783 collected, 1782 passed, 1 skipped) with its history extended.
No behaviour change in src/ beyond the version string.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K3-21 D. `_StsReader._walk` computed `level = min(depth, _ATX_MAX_LEVEL)`
once and wrote it into BOTH the ATX heading and the `OutlineMark`. The
heading keeps the clip -- markdown has six levels and `#######` matches
nothing -- and the mark now carries the declared `depth`: it is no heading,
the declared route builds `Candidate(level=mark.level)` from it, and
`OutlineMark`'s own docstring says the level is what the tree declares,
"reported rather than fixed up". The extracted text does not move one
character; only the mark's level does.
On one standard 9 of 2 761 titled sections sit at depth 7; `--shell-parent`
gave its two depth-7 shells (`36.73211`, `36.73212`) the ancestor one level
too high. The gate -- plan otherwise unchanged, concept count, goldens, the
regression bundles, hit@k and the other 708 pointers -- is measured in a
frozen export of this commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K3-21 C. `resolve_structure` asks `_segment_lookup` first for a `parent`
edge: `(source_file, segment_id)` -> concept name, keyed off each concept's
own frontmatter (`DocumentStructure.declared`, no file read again), so a
pointer lands only inside the pointing concept's document -- `p1` exists in
every document of a multi-document bundle. A value no segment answers to is a
document number and is looked up exactly as before; a pointer naming nothing
keeps `UNRESOLVED_MARKER`. The rendering rule is untouched: a resolved
relation renders as its subject, so `parent: p1977?` becomes `parent: p1977`.
Moved on purpose, each named: both segmented goldens' index files
(`examples/ingest-golden-segmented{,-okf-v0-2}/expected-bundle/krav/1-{1,2}/
index.md`), whose declared parents s1 and s2 -> s0 rendered `parent: s0?`
while s0 stood in the bundle -- four lines, `?` removed. The four goldens
`test_the_four_existing_goldens_are_untouched` guards are not among them.
`skills/okf-consume/` regenerated, because the golden's index bytes -- and so
its ref -- moved. `test_shell_parent`'s byte test now expects the resolved
facet. README and CLAUDE.md no longer say the index renders it unresolved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K3-21 B. `cli.DEFAULT_SHELL_PARENT`'s comment said it was off because `okf
consume` reads no `parent` key; after A that sentence was false and the
default was unmeasured in both directions. Re-measured on the one standard
with heading-only sections: hit@1, hit@8 and hit@50 stay 6/6 at both k with
the known-positive at rank 1, but the link's bundle-absolute path repeats
the document's directory in 675 bodies, and the delivered set moved on 2 of
8 questions at the default k and 3 of 8 at k 50, newly delivered shells
matching the question only through that path on 1 of 3, 4 of 7 and 6 of 23
-- the saturation K3-20 took out of the id signal, back through the body.
Off.
`consume.DEFAULT_FOLLOW_PARENT`: the same delivered set as the pointer (16
of 16 payloads), 6 of 6 correct either way on questions whose answers are
never shells, at +2 488 / +2 746 B on the 2 of 8 default-k payloads holding a
linked shell and up to +20 817 B at k 50. Off.
A test holds both defaults and that the false sentence is gone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K3-21 B. The second form of `parent`: `okf consume --follow-parent`
(`consume.attach_parent_text`) puts the enclosing concept's text inside an
excerpt's `parent`, with that concept's own `sha256` so a claim resting on it
is cited as that concept. It runs AFTER the cut, on the room the cut left, in
rank order, so the delivered set, its order, the withheld list and the
denominators are the same with the flag as without it -- inherited text
cannot displace an excerpt, the mechanism a consumer measured when copied-in
ancestor text pushed the right section to withheld place 504 and 1 069. A
text that does not fit is cut to the longest prefix that does and marked
`truncated`; a parent the payload already holds, or one a higher-ranked
excerpt already carried, travels once. OFF; the defaults are chosen on the
measurement that follows this commit.
`delivered_text` is the one normalisation an excerpt's `text` and a parent's
share. Contract SS 8 point 6 gains the MAY; the template tells the reader
what `text`, `sha256` and `truncated` mean. README and CLAUDE.md name the
flag.
Moved on purpose: the SS 7.4 known-positive again (14 455 / 14 083 / 372 ->
14 721 / 14 346 / 375), and `skills/okf-consume/` regenerated with it.
`tests/test_parent_text.py::test_no_room_means_no_text_and_no_lost_excerpt`
changed from its red form: it asked through `build_payload` at `limit ==
spent`, where the knapsack's 500 B buckets admit nothing at all
(`budget_admits_nothing`); it now holds the rule at `attach_parent_text`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K3-21 A. `okf consume` resolves a concept's `parent:` pointer -- a
`segment_id`, unique only inside one document's plan -- among the concepts
sharing its `source_file` (`consume.link_parents`, one pass, no file opened
again) and an excerpt carries `parent: { concept_id, title }`. Conditional
like `req_number`: a concept with no `parent` key moves no byte. A pointer
that lands nowhere is named `parent_unresolved: true`, never dropped.
The door writes ONE line into a heading-only body whose entry has a parent:
`Enclosing section: [<title>](/<bundle-relative path>)` (SPEC SS 5.1 lineage
through links, SS 6.1 the recommended absolute form and the kind in the
prose). Only such a body, so the segmented goldens' declared parents -- bodies
holding text -- are untouched. Appended AFTER structure derivation and
screened on its own (`_screened`, the `description` rule): read as body text
the link was derived into a second, unresolved `references` edge, measured on
the fixture. `segmentation.heading_only` is the one predicate the proposer and
the door share.
`okf check` gains its seventeenth rule, `parent_unfollowable`: a `parent`
that is not a concept_id and title, names its own excerpt, or names a concept
in neither `excerpts` nor `withheld` (together every considered concept).
Contract SS 8 point 6 added, the figure carries `parent`, and "additional
members are not read by the checker" now says the checker reads only the
members SS 8 names. The template tells the reader what `parent` is and that
SS 2.2 lets it read that one concept; `skill.CONDITIONAL_FIELDS` gains
`parent`. README and CLAUDE.md say what consume now reads.
Moved on purpose, each named: the SS 7.4 known-positive IS the contract
document, so `budget.known_positive` moves in every payload (13 238 / 12 893
/ 345 -> 14 455 / 14 083 / 372); `skills/okf-consume/` regenerated from the
segmented golden, whose plan declares s1 and s2 under s0 -- its example
payload now carries both parents; `test_bundle_identity` 16 -> 17 rules;
`test_shell_parent`'s byte test also accounts for the link line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K3-22. SPEC SS 11 point 1: "Every non-reserved `.md` file in the tree
contains a parseable YAML frontmatter block." Measured with PyYAML 6.0.3,
okf's own default K2 bundle failed safe_load on 41 of 455 blocks and the
R761 build on 1 of 2 763, every one a block scalar written verbatim.
Block (the profile emitter, every key): a value the K3-19 rule refuses as
plain is written double-quoted, `\` and `"` escaped; every other value keeps
its bytes, and a flow collection or an empty value is written as it stands.
The rule, now `profiles.yaml_block_plain`, agrees with PyYAML on every
top-level value in eleven measured trees (0 refused that it reads verbatim,
0 kept that it does not). Double, never single: 0 values in those trees are
`"`-wrapped and 11 193 are `'`-wrapped.
Flow (`sources`, Door A and Door B, and a run-stated flow value): the pinned
guard refuses ANY quote in a flow mapping (1.3.0, measured), so a leaf PyYAML
needs quoted has no form both read. `yaml_flow_plain` refuses it instead:
`,[]{}`, `?`, a quote, ": ", " #", a trailing `:`, a leading indicator -- a
leading `-` before a non-space excepted, which both readers take. The file
name is checked too, because it is the entry's `title` when the document
declares none. Existing codes: inbox_source_file_unaddressable,
inbox_source_title_unaddressable, source_reference_unquotable,
run_frontmatter_invalid.
Readers: parse_frontmatter, profiles' and structure's copies, and both
read_sources branches unquote a `"`-wrapped value (`\"` and `\\` decoded,
nothing else); `'`-wrapped values are untouched, and structure keeps the
single-quote rule it already had. The flow-mapping split is quote-aware, so
`{ title: "a, b" }` is one pair. The generated SKILL.md header goes through
the same block rule.
TWO K3-19 TESTS MOVED, deliberately: test_run_frontmatter built with
`sources=[{ resource: ...?languageCode=nb, ... }]`, the exact form PyYAML
refused on 2 761 of 2 761 frontmatters of K3-19's flagged build. The two
build tests now write an address without `?`; the flag-grammar test keeps
the `?` address (it only splits), and a new test holds that the build
refuses it with exit 2 and writes nothing.
1753 passed, 1 skipped (OKF_HTML_CORPUS, known). No golden moved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tag K3-19 and K3-20 together: a NISO-STS document's own identity names
its directory and titles its sources entry, --frontmatter KEY=VALUE and
description from a section's first spec point, and the consume fix that
keeps a leading directory every concept id shares out of the first
fusion signal. K3-19 alone dropped the known-positive from rank 1 to not
delivered; K3-20 A is what makes the identity safe to ship.
Version 0.8.2 -> 0.8.3 in pyproject.toml, __init__.py and uv.lock (uv
lock changed that one line only). CHANGELOG [Unreleased] closed as
[0.8.3] with its five entries moved unchanged. Five install lines move to
@v0.8.3, the tag history gains a v0.8.3 row, and the suite count is
re-measured: 1668 collected, 1667 passed, 1 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A plan entry whose span holds its heading alone gets `parent_id` naming the
nearest preceding entry at a smaller level whose own span holds text, passing
over an ancestor that is empty too; the door writes the existing `parent:`
key. Nothing is copied and no boundary moves. The rule reads the plan's level
and order, never the row. Off by default: `okf consume` reads no `parent`
key, so no payload ranks differently, while the flag moves the bytes of every
bundle holding a heading-only section.
Measured before building on one 2 761-concept process code: 710 concepts are
heading-only; the level route names the ancestor the document's own <sec>
nesting names on 708 of 710 (two sit at depth 7, clipped to 6, and point one
level too high), where reading section numbers gets 686 (`12` begins with
`1`); 35 have no ancestor holding text and get none.
The red test expected only concept files to move. The index is a projection
of the frontmatter and shows the key as a facet, so the test now holds both;
the facet renders a segment id as unresolved (`p1?`), because `structure`
reads `parent` as a document number -- named in README, CLAUDE.md and the
CHANGELOG, not repaired here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first fusion signal read a concept's title together with every segment
of its id. On a one-document bundle every id starts with the same directory,
and since K3-19 an STS document names that directory after its own number, so
a question naming the document matched every concept except the one whose
title already named it. Measured on a 2 761-concept bundle, the known-positive
fell from rank 1 to not delivered at the default k (13 at k = 50).
`shared_id_prefix` returns the leading directory segments EVERY id shares,
never the leaf, and the signal reads the id below them. Where the ids share no
prefix the signal reads the same string as before.
Measured on a frozen export before this commit, four forms: the chosen one
gives KP rank 1 at both k with S1-S6 6/6, and K2 (12 payloads), N100/N200/N500
(15) and the five-document folder (5) byte-identical. Dropping each concept's
own document directory instead took a K2 hit@8 row from rank 5 to not
delivered, and is not shipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K3-19 c, repaired before delivery on its own measurement. The first R761
build from de7849e wrote 2 024 descriptions, and PyYAML's safe_load refused
the frontmatter of 217 of those concepts -- every one carrying ": " -- where
the same document had parsed with 1 refusal of 2 761 before the key existed.
`inbox._yaml_plain` is checked where the description is written, so a direct
`render_inbox_concept` caller is held to it as well as `okf build`: no leading
YAML indicator, no ": ", no " #", no trailing ":", no tab or line break.
Decided by rule rather than by a parser, because the one runtime dependency is
the guard; over the 2 024 measured values the rule and PyYAML agree on every
one (217 refused, 0 refused that PyYAML reads, 0 kept that it does not).
Omitted rather than quoted or cleaned: a quoted value comes back from the
line-oriented readers here WITH its quotes, and a cleaned one is a sentence
the source does not carry.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K3-19 c. The NISO-STS reader records, per titled <sec>, the FIRST <p> of its
FIRST direct-child <sec sec-type="spec"> as `OutlineMark.description`. The
plan entry carries it (`description`, only where the source has one, so every
other row's plan keeps its bytes), `parse_segmentation_plan` refuses an empty,
multi-line or non-string value, and the door writes it as the concept's
`description` after the gate has seen it: it is document text persisted
outside the body the gate screens, so it is kept only on the non-blocking
floor and as the sanitized text.
SPEC SS 4.1 makes `description` RECOMMENDED and sets no length, in SS 4.1,
SS 8 or SS 11. The limit is ours and structural -- one paragraph, whole --
because a cut inside it writes a sentence the source never wrote. Measured on
R761: 2 026 of 2 761 titled sections carry a direct-child spec point; the
first <p> runs 17 / 109 / 273 / 521 / 942 characters (min / median / p90 /
p99 / max). A section with none gets no key; nothing is derived from the
title. A stated `--frontmatter description=...` replaces it.
The extracted text does not move: the description is read beside it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K3-19 b. `okf build --frontmatter KEY=VALUE`, repeatable, split on the FIRST
'=' (`cli.frontmatter_from_flags`) because a publisher's address carries '='
itself. The value is written verbatim on ONE line: the block form of a
`sources` list is invisible to this package's line-oriented readers, so the
flow form is the only one that survives them. Also reachable as
`build(frontmatter=...)`, `measure(concept_frontmatter_values=...)`,
`process_inbox(concept_frontmatter_values=...)` and
`render_inbox_concept(concept_frontmatter_values=...)`, keyword-only with
defaults, so every existing call site is source-compatible.
Precedence: a stated value beats what the document declares, which beats the
file name. A run may ADD any key and REPLACE only `sources` and
`description` -- the two with a derived layer below the flag. Every other key
the door writes is refused by `inbox.validate_concept_frontmatter` before a
proposal is written (`run_frontmatter_invalid`): measured from the bytes, owned
by another argument, the ownership stamp a later run reads back (including
Door A's `ingest_manifest`, which would make that door claim a Door B file),
or a derived facet whose `derived` marker would go on naming a replaced value.
A value that would not read back as stated -- empty, multi-line, or padded,
since `parse_frontmatter` strips -- is refused too.
SPEC SS 4.1 "Extensions" lets a producer add any key and SS 11 forbids a
consumer to reject one. Without the flag nothing moves: a test holds the
flagged tree to the plain one minus exactly the stated line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K3-19 a. `extract.declared_identity` reads what a NISO-STS document states
about itself -- exactly one <std-ident> (<doc-number>, <year>) and exactly one
<title-wrap> (<full>, else <main>) -- and returns None for every other row,
for XML that is not STS, for an unparseable file, and for a document that
states neither. A value stated more than once is not read: an adopted
standard carries one <std-ident> per issuing body, and picking one is a guess.
`okf build` names a document's directory from its <doc-number> through the id
grammar, replacing only the file's stem. A declared name another document in
the run also claims falls back to the file name for both, said on stderr: the
existing collision gate would refuse both with "rename one", and a name read
from inside a document is not one a rename can change.
`sources[0].title` becomes <doc-number> + <year>, then the <title-wrap>
title, then the file name -- the first that survives the gate and can be
written into the flow mapping verbatim. Measured on R761, <full> carries a
comma, which ends a flow mapping, so it is never the title there; it is never
cleaned up either. `resource` stays the inbox-relative file.
Every other row, and every profile without an address, is untouched: the
identity is asked for only where `sources` is written.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The version moves to 0.8.2 in pyproject.toml, __init__.__version__ and
uv.lock (one line; `uv lock` touched no other package). CHANGELOG's
[Unreleased] becomes [0.8.2] -- 2026-09-11 with the entries K3-15, K3-17
and K3-18 wrote, unedited. The five install lines (README x4, llms.txt)
point at @v0.8.2; the guard pairing prose names v0.8.2, whose own
[tool.uv.sources] still pins guard v1.3.0; the tag history gains a v0.8.2
row and v0.8.1 stops being "the current tag". Test count re-measured on
2026-09-11: 1603 collected, 1602 passed, 1 skipped.
What the release delivers to a consumer on a tag: `okf check`'s sixteenth
rule (bundle_mismatch), which v0.8.1 does not carry, the --title-covered
stop beneath a title that answers more, and the regenerated
skills/okf-consume/. No behaviour change in src/ beyond the version string.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The hand-filled skill predated `okf skill`, declared no bundle identity,
and was refused by the checker it tells its reader to run, against the
payload shipped beside it:
NOT conformant: 16 rules over 3 excerpts and 0 withheld entries, 1 findings
bundle_mismatch: the skill declares no readable bundle identity, ...
It is now the generator's output for
examples/ingest-golden-segmented-okf-v0-2/expected-bundle, the bundle the
payload always came from:
conformant: 16 rules over 3 excerpts and 0 withheld entries, 0 findings
The payload's bytes do not move. --force and --example-question are both
required: without the question the payload is a different one (spent
2125 -> 1420, delivered 3 -> 2, withheld 0 -> 1) and the byte test fails.
`okf skill` writes the bundle root and the skill path absolute when --out
is not under .claude/skills/, so the checkout prefix is stripped in one
mechanical step that a test holds; the generator is unchanged.
The frontmatter name becomes b-golden-segmented-okf-v0-2-consume. Claude
Code takes a project skill's command from its directory, which stays
okf-consume, and nothing in this repository named the skill by name.
references/README.md is rewritten by hand with the full regeneration
command; living prose in README, CLAUDE.md, the test comment and the
skill.py docstring is corrected, dated journals are left as written.
Report: docs/2026-09-11-k3-runde18-konsumskillen-regenerert.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Round 16's partition read every concept whose WHOLE title the question
accounts for before everything the fusion ranked above it. That is a claim
about the covered title's PRECISION, and it overrode the fusion even against
a title answering MORE of the question. Measured on a 26-concept bundle of
five documents: the question names a section by three title tokens and holds
a neighbour's whole one-token title (1 of 9 question tokens); the fusion put
the named section at rank 1, the partition moved the neighbour over it.
A covered concept now RISES through the fusion's order and stops beneath the
first concept whose title answers more question tokens, by equality, than it
holds, or beneath a covered concept the fusion put above it. With nothing
above it answering more it reaches the top exactly as before. Same flag
(--title-covered / --no-title-covered), no new parameter, no new constant.
Measured before this commit, delivered ranks from build_payload:
- known-negative: rank 2 -> 1; the payload equals --no-title-covered's
- three own probes on that bundle: 1, 1, 1 (unchanged from round 16)
- R761 XML, 2 761 concepts: hit@1/8/50 6/6 - 6/6 - 6/6 at default k and at
--k 50, KP rank 1; 8 of 8 payloads byte-identical to 7cca9e0 at BOTH k
- payloads byte-identical to 7cca9e0: K2 pinned 6/6, Arm B 6/6, N100/N200/
N500 15/15; tests/test_default_bundle_pin.py 7 passed, file untouched
- candidates measured beside it: min title length (R761 hit@1 3/6), share of
the question (holds only for 1/9 < s <= 1/6), order inside the group
(group of one: no effect), stop list (no title involved is one)
Suite on the staged set: 1600 passed, 1 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`okf check` had fifteen rules and none asked whether the skill and the payload
were talking about the same bundle. Reproduced on this HEAD before any code
moved: three pairs reported `conformant: 15 rules over 8 excerpts and 438
withheld entries, 0 findings` -- a skill generated from one corpus against
another corpus's payload, the unfilled template against that payload, and a
payload sharing the skill's `bundle_id` at a foreign `ref`. All three now exit 1
with one `bundle_mismatch` finding over 16 rules.
BOTH halves are compared and the `ref` half is load-bearing: three distinct
builds on this machine carry one `bundle_id`, so an id comparison would pass a
stale skill. SS 3.3: "a version is the producer's assertion; a ref is a fact
about bytes". An identity the rule cannot read is a finding, never a silent
pass -- that is what refuses the unfilled template.
No new field: the identity was already in the generated skill's prose, now
factored into `skill.identity_line` and read back by
`contract_check.skill_identity`. Generated skill bytes unchanged, measured on
both tracked bundles on one interpreter.
The rule's first real find is this repository's own hand-made
`skills/okf-consume/SKILL.md`, which predates `okf skill` and declares no
identity a reader can act on: 1 of 1. Nine tests that asserted the old, false
conformance now pair a skill with its own bundle's payload.
Measured, nothing else moved: `~/okf-test/dokumenter` `diff -r` empty old
source vs new on one interpreter (52 files, 26 concepts), `okf project` still
byte-equal to `okf build`, K2 pin unmodified and green (453 concepts, ranks
1,1,1,1,1,5), known-negative `{}` unchanged at 9 findings.
Report: docs/2026-09-10-k3-runde15-bundle-mismatch.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A patch release carrying round 16's ranking rule and nothing else. No
behaviour changes in `src/`: the version string is the only line touched
there, and it is touched because it is the second half of the version --
`tests/test_packaging.py` holds `pyproject.toml`'s declaration and
`__init__.__version__` equal, and a tag whose package reports the previous
number lies to every consumer that installs it.
- `pyproject.toml` and `src/llm_ingestion_okf/__init__.py`: 0.8.0 -> 0.8.1.
- `CHANGELOG.md`: round 16's own entry closes as `[0.8.1]`, with a new empty
`[Unreleased]` above it and no comparison link, because the file has
published none since `[0.6.0]`. The entry gains one bullet it did not
carry: the rule's MEASURED DOWNSIDE, reproduced here on a five-document
folder -- a constructed known-negative question demotes the answering
section from delivered rank 1 to rank 2 because a neighbouring concept
titled with a single common process word has its WHOLE title accounted
for. Written as a known limitation of short generic titles, not as a fix.
- The five install lines (`README.md` 4, `llms.txt` 1) move to `@v0.8.1`.
Measured after: `okf.git@v0.8.0` 0 hits, `okf.git@v0.8.1` 5 hits as the
known-positive control, and the two remaining bare `v0.8.0` occurrences
are the tag-history row and the test-count history -- both are statements
about the past and both stay.
- The tag-history list gains `v0.8.1` as the current tag and demotes the
`v0.8.0` row to what it was; `v0.7.0` is untouched, and `docs/` is not
rewritten at all.
- The test-count history sentence named the `v0.8.0` release as the last
one and would have become false on this tag: it now records 1582 through
`v0.8.1` as well. The published figures 1583 collected / 1582 passed /
1 skipped were re-measured here and stand.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The report, the CHANGELOG entry under [Unreleased] and the README's ranking
section. No version bump and no tag: v0.8.0 was released this same day.
R761, delivered ranks, before -> after: hit@1/8/50 3/6 - 5/6 - 5/6 -> 6/6 -
6/6 - 6/6 at default k and 3/6 - 5/6 - 6/6 -> 6/6 - 6/6 - 6/6 at --k 50, with
the known-positive at rank 1 on all four and the known-negative still not a
hit. --no-title-covered on the new binary reproduces the old row exactly.
Nothing else moves, measured on the bytes: the pinned K2 bundle and Arm B keep
(1,1,1,1,1,5) with payloads byte-identical on 6 of 6 questions each,
n100/n200/n500 payloads byte-identical on 5 each, the 828-file HTML corpus at
828 plans / 0 unreadable / 6 015 md with diff -rq empty, the five-document
folder diff -r-identical at 26/52, and okf project still byte-equal to okf
build.
README's test count corrected to what this round measures: 1583 collected,
1582 passed, 1 skipped. The published 1575 was the passed count.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`title_covered_hits` names the concepts whose every title token is a token of
the question, and `concept_scores` reads them before the rest -- a PARTITION
below the lookup partition, never a fourth RRF signal.
The arithmetic that rules a signal out is the same one `lookup_hits` was made
a partition on: RRF consumes ranks alone, so with shared ranks a rule whose
positive group is SMALL separates least of all. Measured on a 2 761-concept
bundle of one standard, the group is 1, 1, 1, 1, 9, 2, 0 and 0 concepts over
the eight questions -- as a signal it moved hit@1 not at all.
By EQUALITY and never by shared prefix: four shared leading characters take
the group from 1 to 6 on one question and 9 to 31 on another, and the
answering section falls to candidate rank 6 with the known-positive to 2.
ON by default with the opt-out `--no-title-covered`. Suite: 1580 passed,
1 skipped; the pinned K2 bundle green and unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Round 13 (the `.xml` core file type read as NISO-STS, and the PDF arm's
collision counter) and round 14 (a section the source DECLARES takes the
declared-structure route: `.xml` goes from 15 of 2 761 to 2 761 of 2 761
boundaries and from 23 to 2 761 concepts at the shipped defaults, hit@1/8/50
0/6 - 0/6 - 0/6 to 3/6 - 5/6 - 6/6) are both landed. This commit adds no
functionality: it sets the version, closes the CHANGELOG entry, and points
every install line at the new tag.
- `pyproject.toml` and `src/llm_ingestion_okf/__init__.py`: 0.7.0 -> 0.8.0.
The second is the only line in `src/` this release touches. It is not a
code change but the other half of the version, written without a `v`
prefix, so a search for `v0.7.0` cannot find it;
`test_packaging.py::test_the_declared_version_agrees_with_the_packaged_one`
is what did. Left alone, the tag would report the previous release to every
consumer that installs it.
- `CHANGELOG.md`: `[Unreleased]` becomes `[0.8.0] - 2026-09-10`, with a new
empty `[Unreleased]` above it. The entries are round 13's and round 14's own
words, unchanged. No compare link is added: this file has carried none since
`[0.6.0]`, and inventing one here would be a claim about a URL nobody checked.
- The five install lines and the two prose lines naming the current tag move to
`v0.8.0`. The tag history list gains a `v0.8.0` row as the current tag and
KEEPS the `v0.7.0` row: that list states it is not install lines, so a
rewrite would delete history rather than update it.
- README's test count was 1515, measured 2026-09-09; this tree measures 1575
passed / 1 skipped with ruff 0.16.6. The surrounding sentence about the
earlier figure is repaired too, because changing the date alone would have
made it false.
`v0.7.0` stays on 1260fac. No lock change, no history rewrite in `docs/`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`extract.xml_outline` reports the marks the XML reader wrote itself -- no
bridge, no tolerance constant, no unresolved bucket, because the reader
appended the line it is naming. `find_candidates` gains `outline_rule` so the
route it already had for a PDF bookmark tree can carry a second reader's name,
and `build_plan` chooses it by the ROW (`DECLARED_STRUCTURE_IDS`), never by the
text: the same markdown from a `.md` file is still a guess and still keeps
`rule:heading`.
`rule:xml-section` is orphan-exempt for the reason the bookmark arm is -- the
check asks whether a guess was a heading, and a container section is not a
false positive.
MEASURED on R761 at SHIPPED DEFAULTS, no flag: 23 concepts -> 2761, and
2761 of 2761 declared sections became a concept with the directory and the
title the source states (0 unmatched, 0 concepts matching no declaration).
`a)`-points as concepts 0 of 4954, table blocks 10 of 10.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GREEN. A pending label prefix is now flushed as its own line when the next
thing out is a heading, when a second prefix would replace it, and at the end
of the document. Measured on R761: non-whitespace preservation goes from
1 283 395 against 1 283 393 to **1 283 395 against 1 283 395, ratio
1.000000** -- exact, which is what the invariant claims. ATX heading lines
2 761 of 2 761, unchanged.
Known-negatives held: `~/okf-test/dokumenter` still byte-identical against the
frozen `e1f4faa` export (`diff -r` empty), and that folder has 0 `.xml` files
of its 5, so it is a real control and not merely an unchanged number.
pytest -q: 1567 passed, 1 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GREEN on the twelve tests from the two commits before it. `.xml` joins
`_CORE_EXTRACTORS` beside `.html`, and 110 of 110 unreadable becomes a bundle.
A CORE type, not an `[extract]` one, because the parser is stdlib: putting it
behind the extra would make a pure-stdlib file type binary-dependent for no
gain. And never through the converter, which the existing fence test now says
in one more line: a file routed there is read by a second parser that never
sees this reader's DTD refusal, so the hardening would be true of the code and
false of the file.
TWO PATHS, NAMED RATHER THAN GUESSED. STS when the root is `<standard>` or the
document holds any `<sec>`; generic otherwise. Generic XML keeps its text in
document order and gets NO structure -- an element name is never promoted to a
heading, because an RSS feed is not a numbered standard.
THE GRAMMAR IS MARKDOWN, the same markdown the office rows and round 11's HTML
row reach the proposer through, and `propose.py` is untouched. `<sec>` with a
`<title>` becomes one ATX line carrying `<label>` + space + `<title>` at its
own nesting depth; `<sec>` with only a `<label>` becomes a body line with the
label in front, the way `li` is treated in HTML -- 4 954 of R761's 7 715 are
lettered points and one heading each would bury its own 2 761; `<table-wrap>`
becomes its label plus ONE markdown table through this package's own
`render_table`, separator line included, which is what makes it a block.
TWO CHOICES THAT ARE MEASUREMENTS, not preferences:
- Inline by ALLOWLIST, block by default -- the inverse of the HTML reader,
because block-versus-inline is a property of HTML and XML has no such
universal. Assuming block is safe (an extra break never removes text and a
boundary needs a line that matches a grammar); assuming inline is not. The
allowlist is load-bearing: that document carries 1 701 `<italic>` and
1 396 `<bold>` inside its prose.
- The ATX ceiling is 6 and STS nesting reaches 7. The depth is CLIPPED, not
dropped: 9 of the 2 761 titled sections sit at depth 7 and `#######` matches
nothing, so dropping loses the section while clipping keeps the boundary and
states the nesting one level too shallow.
A DTD IS REFUSED UNPARSED, and that is a guarantee about this code rather than
about the machine. Measured on this interpreter (3.14.0, pyexpat 2.7.3): an
external SYSTEM entity is refused by the stdlib and never fetched, but the
amplification limit that stops a billion-laughs comes from libexpat >= 2.4.0
and NOT from Python -- five levels still expand -- while `pyproject.toml`
requires only `>=3.10` and no lockfile pins an interpreter. `XMLParser` exposes
no `.parser` attribute on the C accelerator, so the handler route is not
portable either. NO new dependency: `defusedxml` and `lxml` both occur 0 times
in `uv.lock` and still do.
pytest -q: 1566 passed, 1 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GREEN on the two tests from the previous commit. `PdfOutline` gains
`collided`, and the identity it buys is the point rather than the counter:
NODES IN == len(marks) + unresolved + collided. Measured on R761 with the
shipped function, 2 763 == 2 762 + 0 + 1, where before the same document
reported 2 762 marks and 0 unresolved and said nothing about the third
number. The lost node was `SVV - Forside`, sharing line 0 with the tree's own
root node.
THE RULE WAS CHOSEN BY MEASUREMENT, NOT BY ARGUMENT. Keeping both nodes as
sibling marks was run through `find_candidates` on R761's own text and text:
the candidate list goes 2762 -> 2763, and the first candidate closes with an
EMPTY span (0, 0) because two candidates open at one offset. The orphan check
deletes an empty span in silence, so keeping both loses the same node one step
later and adds a front-matter title that is in no fasit. First-in-tree-order
wins, the loser is counted.
Known-negatives, all measured rather than assumed: `outlined-krav.pdf` and
`outline-broken-dest.pdf` unchanged in outcome and reporting `collided == 0`;
`~/okf-test/dokumenter` byte-identical against a frozen `e1f4faa` export
(`diff -r` empty, 52 md); the pinned K2 bundle's concept count and per-row
ranks unchanged (7 passed). The arm stays off by default and no boundary rule
moved.
pytest -q: 1556 passed, 1 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
