Commit graph

414 commits

Author SHA1 Message Date
6abe4b4358
test(mutants): a surviving mutant is not exit zero (H4, red)
The harness returned `2 if errors else 0`, so a run printing `killed 0
of 1` and naming its survivor exited 0. PM measured it on a copy
carrying only the X2 mutant. Red on the claim: the module imports, and
offers no verdict to ask.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-19 05:49:37 +02:00
d27ca503c8
fix(accounting): a document refused whole is never clean, and the run says how many (H1)
The gate gains a FIFTH column, `Unit.refused`, and it is the only one
that is not a defect in the report: the elements of a document the
build read and persisted nothing of. Their fate is declared honestly,
so `unaccounted` and `double_booked` both stay 0 -- which is exactly
why nothing else could see the loss. `refused_whole` asks its question
only for a corpus that persisted NOTHING, so one refused source beside
an accepted one, the ordinary case on a heterogeneous corpus, reached
row 3 as clean.

Row 3's reason now carries `N element(s) lost with R of D document(s)
refused whole`, and each unclean unit's detail line carries
`refused=` beside u, d, unverified and invalid, with the document's
own rejection code in the note.

On the build side `Accounting.refused` is written into the JSON and
into the `**Accounting**` bullet of `log.md` as `R of D document(s)
refused whole`. The exit code is NOT moved: it belongs to the whole
run, and a corpus holding one unreadable file among many is ordinary,
so the order's other half -- state it in the accounting -- is the one
taken. `okf build` still exits 1 when it persisted nothing at all.

`test_a_corpus_refused_whole_under_the_default_gate_is_red` kept its
point and lost its premise: the numbers still balance, and that is now
asserted as u = 0 and d = 0 rather than as a clean unit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-19 05:48:57 +02:00
5d1060d467
test(accounting): a document refused whole is never clean (H1, red)
PM's checkpoint of ed8d9d7 measured it: an inbox holding one refused
document and one accepted one gave `clean = 4 of 4`, `refused_whole =
None` and `okf build` exit 0, while three elements of `avvist.html`
were gone unseen. `refused_whole` asks its question only when the
corpus persisted NOTHING, and a refused document's own unit balances
by construction -- every element booked as a coded rejection is u = 0
and d = 0.

Three tests, all red on the claim and not on an import:

- the unit of a document the build refused is not clean, and its note
  names the code;
- the same beside a PERSISTED neighbour, which is the ordinary case on
  a heterogeneous corpus, with row 3 RED and the source named in the
  detail line;
- `okf build`'s own report says how many of M documents it refused
  whole -- the exit code belongs to the whole run, so the count has to
  stand in the accounting and in `log.md` or the loss is silent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-19 05:44:54 +02:00
3b3b8ae0ca
fix(assets): budget every link by what its decoder COSTS (0.10.1)
Round 3 of the 0.10.1 review, and the finding is the pattern the three rounds
share: each bound an OUTPUT, and the bomb stepped one link along. The
declared size, then the first `FlateDecode`, then every `FlateDecode` -- and
then a link this package had documented as safe.

`ASCII85Decode` was classed as bounded "by its own input because it shrinks".
It quadruples: `z` is the shorthand for four zero bytes. And the output was
never the cost -- `base64.a85decode` appends one 4-byte object per group to a
list, about a hundred bytes of memory per byte of INPUT (101.4x at 1 MiB,
96.1x at 4 MiB, 94.5x at 16 MiB on CPython 3.14).

Paired subprocesses, idle machine, both sides from PINNED trees, the document
built once by a third process and read from a file because `ru_maxrss` never
falls and `b"z" * 64 MiB` alone costs 171 MB:

  [/Fl /A85]      z x 32 Mi  33 475 B   CARRIED 3 261 599 744 -> too_large 42 070 016
  [/Fl /A85]      z x 64 Mi  66 090 B   CARRIED 6 461 558 784 -> too_large 40 280 064
  [/A85]          z x  8 Mi   8.4 MB    CARRIED   933 085 184 -> too_large 62 484 480
  [/Fl /A85 /Fl]  z x 32 Mi  33 488 B  samples_invalid 3 519 180 800 -> too_large 43 438 080

The picture was CARRIED in three of the four: not a bound that fired late, no
bound at all. Doubling the `z` run trebles the old cost and leaves the new one
where it was.

WHY THIS FORM. `assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE
link may cost -- a separate number from `MAX_IMAGE_BYTES`, because that one
bounds the picture and this one bounds producing it. `FlateDecode` is measured
as it is paid; every other permitted filter carries a MEASURED cost ratio
(`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its decoder
is called, since those decoders take a whole string and return a whole string.
A filter with no ratio is refused unread. The budget TRAVELS: a deflate link
is inflated under the smaller of the picture's bound and what the next link's
decoder may be handed, or `[/Fl /A85]` pays 256 MiB for a refusal.

A chunked ASCII85 decoder written here was the alternative and was FELLED: it
would bound `_check_stream_cost` and not the run, because `stream.get_data()`
decodes the whole chain again with pdfminer's own decoder, and it would make
this package rather than pdfminer the authority on an image's bytes. The cap
is the only number that bounds that. `resource.setrlimit(RLIMIT_AS)` was
MEASURED before anything was built on it, as the order required, and is not
usable: Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit`
and does not enforce it. No child-process cap exists.

THE CAP IS READ OFF THE CORPORA, the posture `MAX_IMAGE_PIXELS` has: over the
9 668 image objects of the 77 PDFs on this machine, 16 decode through an
ASCII85 link and the largest input to one is 450 739 bytes, against a cap of
about 5.0 MB.

A PROPERTY TEST REPLACES THE LIST OF KNOWN SHAPES: every chain of length 1-3
over the ten filters pdfminer decodes, 1 110 of 1 110, both payload fills,
each delivered under the bound or refused with a published code and never paid
for on the way (`tracemalloc`, which counts allocations and is not disturbed
by load). Known-positive beside it: 258 of 258 chains over the permitted
filters still carry a small image.

MAJOR -- the backstop had no test. `check_payload` at the end of
`_check_stream_cost` could be deleted with the whole suite green, because the
second one after `get_data()` gives the same code one step later. The two
differ in whether the payment was made, so the test asserts `get_data` was
never called.

10 OF 10 MUTANTS KILLED, control green, each killer named in the report. Four
survived a first pass and two tests exist because of it.

NOT ONE PICTURE CHANGES HANDS, MEASURED BY NAME: `_pdf_images` over every PDF
on this machine from both pinned trees -- 9 306 -> 9 306 carried over 77
files, 50 -> 50 on R761, 0 of 78 files moving a count and 0 moving a code.
R761 also settles a question raised while this order was open: 50 objects, 29
[/DCTDecode], 21 [/FlateDecode], 0 ASCII85 links -- so round 2's count of 580
`[/FlateDecode /ASCII85Decode]` objects is reproducible from nothing on this
machine. It changes no decision; a bomb shape does not need a corpus.

Version stays 0.10.1, no tag. README, CHANGELOG, CLAUDE.md and errors.py
corrected TO what the code does; the round-2 report carries a correction block
rather than a rewrite. Report:
docs/2026-09-18-utgangsbudsjett-per-ledd.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 19:00:38 +02:00
33d3269380
test(assets): the cost of a link, and a property over all 1 110 chains
Round 3 of the 0.10.1 review. Rounds 1-3 each bound a NUMBER and the bomb
moved to the next one: the declared size, then the first `FlateDecode`, then
every `FlateDecode`. It now lives in a link this package had documented as
safe. `ASCII85Decode` was called bounded "by its own input because it
shrinks"; `z` is ASCII85's shorthand for four zero bytes, so one input byte
becomes four, and `base64.a85decode` appends one 4-byte object per group to a
list, so the DECODER costs about a hundred bytes of memory per byte of input
(measured on CPython 3.14: 101.4x at 1 MiB, 96.1x at 4 MiB, 94.5x at 16 MiB).

Measured on the pinned `0c3c490` tree, its own interpreter, idle machine: a
33 475-byte PDF decoding an image through `/Filter [/FlateDecode
/ASCII85Decode]` cost 3 827 003 392 bytes of peak RSS -- 114 000x the file --
and the picture was CARRIED, with no rejection at all.

Seven tests red, three green:

* the two bombs above, in their own interpreters, at the shipped bound;
* the cost ratio as a re-measurable known-negative, so the constant cannot
  rot the first time CPython changes `a85decode`;
* the input cap against the corpora (9 668 image objects over 77 PDFs, 16
  behind an `ASCII85Decode` link, largest input 450 739 bytes);
* THE PROPERTY, both payload fills: every chain of length 1-3 over the ten
  filters pdfminer decodes -- 1 110 of them -- is either delivered under the
  bound or refused with a code in the published vocabulary, and never paid
  for on the way, which `tracemalloc` measures because that is where
  `a85decode`'s cost lives.

Green but previously uncovered, which is the MAJOR of the same checkpoint:
`check_payload` at the END of `_check_stream_cost` could be deleted with the
whole suite still passing, because the second one after `get_data()` gives
the same code one step later. The two differ in whether the payment was made,
so the test asserts `get_data` was never called. The known-positive beside
the property -- every bounded chain still carries a 64-byte image -- is green
too, and a rule that refuses everything would pass the property alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 17:20:51 +02:00
0c3c4904ee
fix(assets): bound every link of the filter chain, and cover the backstop
The two findings of the 18.09 PM checkpoint of `0f308c1`. Red tests landed
first in `3b587ea`; this is what turns them green.

BLOCKER -- `_check_inflated` read `filters[0]`, measured that one link and
returned, which is not a bound: a PDF decodes a stream through a LIST of
filters. Measured in paired subprocesses from two pinned trees, idle machine:

  [/FlateDecode]        400 MB  408 516 B   59 232 256 ->    62 017 536 B
  [/FlateDecode x2]     400 MB    1 636 B  886 554 624 ->    52 367 360 B
  [/FlateDecode x3]     400 MB    1 070 B  889 393 152 ->    61 390 848 B
  [/FlateDecode x2]     1,2 GB    2 927 B  2 567 204 864 ->  60 403 712 B

542 000x the file at two links, and the picture WAS refused at the end -- by
`check_payload` after `get_data()`, once the memory was spent. The single-link
row is the control and does not move. It also left the 16 corpus objects behind
an `[/ASCII85Decode /FlateDecode]` chain unmeasured, since `filters[0]` is not
`FlateDecode` there.

`_check_stream_cost` walks every link. THREE CLASSES and no fourth
(`extract.bounded_pdf_filters`, pinned by a test): `FlateDecode` MEASURED, a
link with another expanding link behind it inflated under the same bound and
handed on; `ASCII85Decode`/`ASCIIHexDecode` bounded by their own input because
they SHRINK; `DCTDecode`/`JPXDecode`/`JBIG2Decode` PASS THROUGH. Everything
else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything
written later -- is refused UNREAD with a new code `asset_pdf_unbounded`,
decided before the FIRST link is decoded so a document cannot make the run pay
for the links in front of the one we cannot bound. An encrypted stream is
deciphered and then measured, where `stream.decipher is not None` used to
return unmeasured; 0 of 5 142 objects here are in an encrypted document, which
is why nothing caught it.

NOT ONE PICTURE CHANGES HANDS, AND IT IS MEASURED BY NAME. Every PDF on this
machine -- 78 documents, K2 in both trinn1 and trinn2, the shipped fixtures and
R761 -- run through `_pdf_images` page by page from both pinned trees:

  images carried          9 356 -> 9 356
  documents losing one              0 of 78
  documents gaining one             0 of 78
  asset_pdf_unsupported     322 -> 314
  asset_pdf_unbounded         0 -> 8

The 8 are the 4 `CCITTFaxDecode` stencil masks (`/ImageMask true`,
`/BitsPerComponent 1`), counted twice because trinn1 and trinn2 hold the same
document. They were refused before and are refused now, one step earlier and
under a code that says why.

MAJOR -- `check_payload(len(data))` after `get_data()` is the counted refusal
four documentation surfaces point at, and deleting exactly that line passed all
2 132 tests. It is reachable through a stream pdfminer has ALREADY decoded
(`decode()` sets `rawdata` to `None`), which is now the ONLY case outside the
bound and has a test.

Eight mutations, one line each, every one DEAD, with the unmutated tree run
first as the control: first-link-only, loop dropped, inequality reversed,
encrypted skipped, backstop deleted, unknown filter passed through,
intermediate link not carried forward, whole check removed.

`tools/okf_accounting_gate.py` gains one line, the new code in
`REJECTION_CODES` -- what a rejection code requires and nothing more. Gate
unchanged: exit 1, GATE RED rows 2, 3, 6. Version stays 0.10.1, untagged.

Suite after `git add` against a clean tree: `uv run pytest -q` ->
2152 passed, 1 skipped (226 s). ruff, ruff format --check, mypy --strict clean.

Report: docs/2026-09-18-filterkjeden-og-backstoppen.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 15:34:07 +02:00
3b587ea567
test(assets): the chain and the backstop, written red first
Eight tests for the two findings of the 18.09 PM checkpoint of `0f308c1`.
Seven are RED here and one is green-but-uncovered; the fix is the next commit.

BLOCKER -- `_check_inflated` read `filters[0]`, measured that one link and
returned, so a stream decoded through `/Filter [/FlateDecode /FlateDecode]`
was bounded by nothing. Reproduced here in its own interpreter, same commit:

  links=1  408 516 B file ->  77 971 456 B peak RSS  (bounded, 0.10.1)
  links=2    1 636 B file -> 889 573 376 B peak RSS  (unbounded)
  links=3    1 070 B file -> 888 401 920 B peak RSS  (unbounded)

543 000x the file size at two links, and the image IS refused at the end --
by `check_payload`, after the memory has been spent. PM measured the same
shape at 1 606 B -> 835 MB and 2 839 B -> 2 439 MB.

The chain is not a hypothetical. Over the 5 092 image objects of the 78 PDFs
on this machine (measured 2026-09-18): 1 625 `[/DCTDecode]`, 2 215
`[/FlateDecode]`, 596 `[/FlateDecode /DCTDecode]`, 580 `[/FlateDecode
/ASCII85Decode]`, 40 unfiltered, 16 `[/ASCII85Decode /FlateDecode]`, 16
`[/JPXDecode]`, 4 `[/CCITTFaxDecode]`. So refusing every chain would cost
1 192 real pictures, and bounding only the first link leaves those 16
unmeasured -- `filters[0]` is not `FlateDecode` there, so nothing ran at all.

Two of the cheap tests assert WHICH check fired ("decompresses to more than"),
because the backstop refuses the same document by code and a test reading only
the code is green on the defect.

MAJOR -- `check_payload(len(data), name=name)` after `get_data()` is the
counted refusal that four documentation surfaces point at, and deleting
exactly that line passed all 2 132 tests. The path that reaches it is a stream
pdfminer has ALREADY decoded (`decode()` sets `rawdata` to `None`), so there is
no raw stream left to measure. That test is green here and RED under the
deletion, measured before this commit: `asset_too_large` -> `asset_pdf_unsupported`.

Each red test carries a known-positive beside it: the two chains the corpora
hold still deliver their 64x64 picture, and an already-decoded stream under
the bound is still read.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 14:15:31 +02:00
0f308c1f56
fix(assets): bound what the run pays, not what the document claims (0.10.1)
A second independent review read `230d1cb` -- the commit that closed the
`v0.10.0` review's two MAJOR findings -- and found one of them open. The
bound read `/Width` and `/Height`, which an untrusted document writes, while
`get_data()` pays for the stream beside them; `/Length` is the COMPRESSED
length and the two numbers are independent.

Re-measured here on `ed8d9d7` before anything changed, in its own
interpreter: a 408 516-byte PDF declaring 1x1 and carrying 400 MB of deflated
zeros was CARRIED, no rejection, 891 904 000 B peak RSS. After: 0 carried,
`asset_too_large`, 57 065 472 B. At 1,2 GB of zeros, 2 436 MB -> 64 569 344 B
-- the cost no longer scales with the bomb. End to end through the CLI with
the shipped defaults: 838 000 640 B and an asset written -> exit 0,
79 650 816 B, `0 carried of 1 found`, no `assets/`.

Three numbers are bounded now, not one: what a container DECLARES, what a
carried FILE measures (`read_image`, so a 49 MP PNG of 47 705 bytes is not
passed on to a consumer), and what a PDF stream DECOMPRESSES to
(`assets.inflated_size`, a chunk at a time, output discarded, before
`get_data()`). The limit is stated rather than implied: the stream
measurement runs where `FlateDecode` is the first filter and the document is
not encrypted; every other chain is a check on the decoded length AFTER the
decode, a counted refusal and not a bounded one.

A non-positive declared dimension is `asset_size_invalid`, its own code,
raised before the stream is read. `-1 x 40000000000` is a NEGATIVE pixel
count, under which every `>` bound read as satisfied, so the check returned
silently and the refusal arrived from `encode_png` as
`asset_samples_invalid`. Its own code because a publisher shipping a picture
bigger than this package carries and a dictionary written to be read wrong
are different facts about a document.

Two smaller findings in the line that says what is missing, both introduced
by the first fix: the address was written twice, once bare, and a linkifying
renderer autolinks a bare URL -- written once now, in one code span; and
`label` became a dead parameter, so the figure's caption was dropped, a
regression against 0.10.0. It is written again in the `-- <label>` form a
carried pointer uses.

Version bumped to 0.10.1 across all ten places. Nine were unbound and stale:
four README install lines naming the previous release, two prose lines, the
"current tag" entry, `uv.lock`, and a CHANGELOG whose 0.10.1 content sat
under `[Unreleased]`. Two new packaging tests bind them to `__version__`, and
the README's guard tag to `[tool.uv.sources]`.

Every test was red first. The fate of every image is identical with and
without the new bound on three K2 PDFs carrying 800 images (464/464, 311/311
with the same 12 rejections, 25/25), and the second inflate is below the
noise floor there. 0 shipped artifacts move: no bundle under `examples/`,
`skills/` or `tests/fixtures/` carries an image pointer at all, measured
against a known-positive control.

`asset_too_large` was undocumented in the error registry; both codes are
there now. `tools/okf_accounting_gate.py` gains the new code in its closed
list -- one string, no behaviour change, stated because that file belongs to
another order.

Suite 2141 passed / 1 skipped, ruff + format + mypy --strict clean.
Report: docs/2026-09-18-bildestien-holder-0-10-1.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 13:41:18 +02:00
ed8d9d709f
test(accounting): row 6 sees a refusal, a second real corpus, and 34 of 34 mutants
Row 6 was GREEN with R761 100 % rejected: every element of a refused document
is booked as a coded rejection, so u = 0 and d = 0 and the bundle is empty.
`refused_whole` asks that question on its own now -- the build order asked for
an honest red there, and PM re-measured the green on 2026-09-18 with
`okf build` exiting 1 unseen.

A skipped row no longer leaves the verdict unqualified (`GATE GREEN (row 6 not
run: ...)`), and the exit code is non-zero locally when a corpus source is on
the machine and its row did not run. m-2.

N200 Vegbygging:2024 joins R761 as a second real corpus. R761 holds 0 `fig`,
0 formulas and 0 references, so the only real corpus could not have found the
hole in the STS role map; N200 carries 194 citations, 49 figures and 135
footnotes. A `.json` whose root holds an STS node tree is counted as STS
rather than as keys and leaves -- the container is not the content.

M-4: the review's 26 mutants, ported to the code as it stands, plus 8 for the
new checks. 34 of 34 killed. `tools/okf_gate_mutants.py` runs on a copy of the
tree, and a pattern that does not match is an ERROR and exit 2 -- a mutant
that could not be applied was never measured. That fired once, on M13, after a
refactor moved the line it edits.

m-3: `APPROVED_EXCEPTIONS` was read by no row, so approving one changed
nothing. Each pair is now checked against the witness's own vocabulary and the
run says why it moves no denominator. The gate also prints its OWN limits
beside the verdict, m-5 among them.

The product's accounting tests state the new truth instead of the old one:
`okf build --accounting` over the fixture corpus exits 1 with SIX unaccounted
elements in its own vocabulary -- its first real finding, reachable only now
that fixtures carry the constructs. Four shared element names disagree with
the witness, each pinned with its cause; one of the four is a double count
this package makes (a text box's paragraph, once inside the box and again in
the paragraph carrying it).

Three fixture defects were found and fixed while building them, each of which
would have reported a loss the build never had: a shared string table not
related to the workbook, a `graphicData` with no `uri`, and an odt
`styles.xml` without `<office:styles/>`.

Report: docs/2026-09-18-regnskapsgaten-herdet.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 02:48:34 +02:00
e5dc21ec2f
test(accounting): the witnesses see what the formats actually hold (M-1..M-3)
Rows 2 and 3 require the build's inventory to EQUAL the witness's, so what
the witness does not count, nothing can lose visibly. An independent review
put a header and a comment in a docx, measured 0 of either in the bundle,
and the accounting still read "2 of 2 carried".

Thirteen classes are now counted, each with a red test written first:
docx header/footer, comment, endnote and text box (a box's paragraphs are
its own, or the text is booked twice) - pptx speaker note and hidden slide
(`show="0"`, no longer counted as an ordinary slide) - xlsx formula and
hidden sheet (the state lives in `workbook.xml` and is reached through the
relationship id, so the sheet part itself says nothing about it) - odt
header/footer from `styles.xml` and annotation (counted as prose, it made
the accounting demand a reader carry a note the author wrote to themselves)
- STS `mixed-citation`, `mml:math`, `fig` and its caption, measured by the
review at 4.1 % of N200's source text and 3.9 % of N100's.

M-2: the two STS witnesses had ONE role map between them, so row 5 -- "two
witnesses agree" -- could not see a hole in it. `_sts_role_xml` and
`_sts_role_json` are written apart, each for its own delivery, and a test
holds them apart.

M-3: 20 of 63 element types had a count of ZERO in their only fixture. Seven
hand-built documents close it, every element type now occurs at least once
(a test asserts it), and ALL TWENTY documents carry a hand count read off
the fixture's own bytes (four did before). `.xlsx image` -- the operator's
own proposed exception -- could not be exercised at all until now.

Every witness also states WHAT IT STILL DOES NOT COUNT, per file type, and
the gate prints that list on every run.

THE FIXTURE ROWS ARE RED NOW, AND THAT IS THE POINT. Row 2 red on .docx,
.odt, .pptx, .xlsx and .xml; row 3 at u = 25, d = 2 over the new classes,
including a footnote and four spreadsheet cells the build genuinely drops.
`0 claimed and not found` on the same run: nothing the build DOES book as
carried failed the bundle check, so the red is the build's and not the
instrument's.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 01:53:50 +02:00
656cbe5d02
test(accounting): the gate opens the bundle itself (BLOCKER B-1)
An independent review of `0b00de4` found the judge was a calculator over a
report the judged writes: `account()` compared BOOKED NUMBERS with the
witness's counts and never opened a concept file. Reproduced here first --
a report that changes not one byte of the bundle and books every element as
carried gave `GATE GREEN`, exit 0, and so did booking every element as
rejected.

The witness now gives every element THE PIECES OF TEXT IT IS MADE OF, and
the gate looks for each of them in the concept bodies the run wrote. Pieces
and not one joined string: a reader writes a heading's marker and a
picture's pointer block between the parts of a container, so a section is
never one contiguous run even when every word of it is there.

Also in the judge, each with a test driving it from both sides:

- a negative booking, a document declared persisted that no concept names,
  a document declared rejected that the bundle holds, a rejection code
  outside a closed list, and an `accounting_version` the gate does not read
  are each REFUSED rather than summed;
- a document the build PERSISTED whose report carries nothing from it is
  never clean ("everything rejected" was);
- an asset proves a carry only when its BYTES hash to the source's and it
  stands under the name the layout gives it. The check was a name check, so
  a zero-byte file called `<sha12>-x.png` read as a carry (m-1).

NOT ONE ELEMENT COUNT MOVED: the 13 fixture documents' counts are identical
before and after, so this commit changes what the gate CHECKS and nothing
about what the witness counts. `texts` is additive in the committed fasit.

The rtf text scanner reads `\uN` escapes and skips `{\fonttbl}`-class
groups, or a fixture's font table reads as the first paragraph of its prose;
xlsx cell text is resolved through `sharedStrings.xml`, where a
spreadsheet's words actually live; a PDF page carries its own text lines,
which no row could see before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 01:40:48 +02:00
751441c17d docs(accounting): record the operator's two decisions of 2026-09-17
The content accounting stays OPT-IN until the losses it reports on the
reference corpus are fixed; a default-on door would fail builds that pass
today. Of the three exceptions the gate proposed, only the PDF one
(heading, paragraph, table) is approved. Approving it moves no number,
because no witness counts a heading in a PDF: what it changes is that the
gap is a stated limit of the instrument rather than an open question
about the build. An image in a workbook, or in md/txt/csv/json/odt/rtf,
stays unaccounted and therefore stays red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 00:37:04 +02:00
230d1cbccd fix(assets): a remote reference is inert and a declared size is bounded
Two MAJOR findings of the independent v0.10.0 review, both with the
shipped defaults, both new in 0.10.0. Repros rebuilt as tests first.

- A remote <img src>/xlink:href became a LIVE markdown image link in the
  persisted concept, with the address and query string chosen by whoever
  wrote the document. Extraction opens no socket; a consumer rendering
  the bundle does. Now inert text with the address in a code span,
  pinned by a property over the readers rather than by one string. The
  tier asymmetry (user-upload refuses, trusted-source persisted) went to
  the guard repo with the repro.
- Nothing bounded a declared image size: 9.6 KB of PDF declaring
  3000x3000 grayscale zeros took 83 MB peak RSS, linear in pixels.
  MAX_IMAGE_PIXELS (40 000 000) and MAX_IMAGE_BYTES (256 MiB) are read
  off the corpora (largest measured 18.6 MP on K2, 1.4 MP on R761) and
  checked on what the container declares, before any decompression;
  over them is asset_too_large, counted. The same bound closes the
  inline data: URI, which the review flagged and did not measure.

Also fixed, added by PM to this order: an inline PDF image was named
from id() of a Python object, so two concept files of the reference
corpus differed between builds. It is now named from its position.

R761 unchanged: 50 carried of 50 found, assets diff -rq clean.

Report: docs/2026-09-17-bildestien-0-10-1.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 00:32:50 +02:00
864570b320 feat(accounting): okf build accounts for every source element
okf build --accounting PATH inventories every source document before
extraction, in the gate's per-format vocabulary, and after the run gives
each element one fate (carried / pointer / coded rejection), written as
JSON and summarised in log.md. "carried" is checked against the written
concept bodies, so a gate that drops a line is found (test). Exit 1 on
anything unaccounted or double-booked. Opt-in: +744 s (+19 %) on the
43-document reference corpus, and that corpus fails the check on 24 real
losses (22 images on text-less PDF pages, 2 docx Title paragraphs).

Changed without the flag:
- okf build exits 1 when it extracted documents and persisted none.
  Door B and corpus.measure are unchanged. One test relied on exit 0.
- An image file carried through a persisted document is its own K1b
  column, no longer also extractor_unknown. The set is what the resolver
  actually carried (ExtractedDocument.files), never a byte match.

tools/okf_accounting_gate.py (checks untouched) is green on all six rows,
R761 110 of 110 under both gates.

Report: docs/2026-09-17-innholdsregnskapet-bygget.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 18:35:20 +02:00
0b00de4408 test(accounting): content-accounting gate for okf build, written red
Capability loop step 3, no capability. tools/okf_accounting_gate.py asks,
per README file type, how many of the elements a SOURCE holds okf build
books as carried / pointer / coded rejection, with unaccounted and
double-booked both required to be 0. Exit 1 today on rows 2, 3, 4 and 6.

The fasit is tools/okf_witness.py (stdlib + pdfplumber + poppler, no
package import; tested on the live import graph), committed as
tests/fixtures/accounting/*inventory.json over one fixture per type.

Measured: no source inventory (0 of 13); two graphics/ files carried
through documents AND counted extractor_unknown (50 on R761 under
--gate none); a refused document logged "0 carried of 0 found"; R761
refused whole because guard 1.4.0 treats its 71 U+00AD soft hyphens as an
invisible carrier (asked of the security repo). The two R761 witnesses
agree once STS labels are counted by role, not tag.

Report: docs/2026-09-17-innholdsregnskapet-rod-gate.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 15:40:37 +02:00
bc39e8091f feat(assets): a bundle carries the images its sources declare (0.10.0) v0.10.0
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.

Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.

THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.

FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.

A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.

NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.

ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.

THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.

Also fixed, both found by measuring rather than by reading:

- a markdown image is no longer read as a cross-reference. `structure._LINK`
  never looked at the character in front of the bracket, so every pointer
  would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
  importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
  no `assets/` at all, so every pointer named a missing file.

Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 10:01:31 +02:00
332961a19c fix(gate,propose): okf build runs a real guard; a code fence declares no structure
Two defects reported from outside by claude-code-llm-wiki (order
20260915T202332Z-228694739), both reproduced against this tree before
anything moved.

F1 -- the packaged CLI never ran the guard. corpus.measure wired an
unconditional approve-everything stub into process_inbox and 0 of 90
add_argument calls named a gate, so the one path most people use screened
nothing while pyproject.toml made the guard a mandatory runtime dependency
and the README recommended a composition the command line could not reach.

  --gate takes guard-trusted-source (default), guard-user-upload or none.
  corpus.resolve_gate is the one name->callable map, with the guard imported
  lazily so importing the package still does not pull it in; an unknown name
  RAISES rather than falling back, because a fallback reproduces the defect
  with an extra step. The gate's NAME goes into the section 9 log.md -- a
  stub is only dangerous when nothing downstream can see it -- and --gate
  none renders NOTHING WAS SCREENED.

  The default was chosen on a measurement: over the 453 concept bodies of
  the pinned reference bundle, PRESET_TRUSTED_SOURCE persists 453 of 453 and
  PRESET_USER_UPLOAD holds 1, costing that concept's whole source document.
  Neither tier waves anything through -- an invisible carrier and a CRITICAL
  finding fail secure at both. Door B's library default is UNCHANGED at
  PRESET_USER_UPLOAD: an inbox drop is an untrusted upload, an operator
  pointing this command at their own folder is not. The second tier ships as
  guard_adapter.inbox_gate_trusted_source, the three-line adapter that
  module's docstring already described, never a preset parameter.

  process_inbox(segmentations=..., gate=inbox_gate) now has a test. Before
  this, `grep -rl inbox_gate tests/` gave 1 file with 0 occurrences of
  `segment` -- the recommended composition was untested, which is how the
  defect survived.

F2 -- a fenced code block declared structure. `# Use the opus[1m] alias`
inside a ```bash fence became a level-1 ATX heading: the document was
refused entirely where the line carried [ or ] (5 of 191 pages of the
reporter's corpus), and the concept TITLE came from somebody's shell session
on 62 of 191 (32.5 %). The fix is in the proposer and never in Door B's
title rule -- that rule is right, and a heading that was never a heading is
what has to stop being proposed. propose.fenced_lines is computed once per
text and no rule reads a fenced line, including Arm D's outline RUN, which
selects from the whole line list. Backtick and tilde fences, three leading
spaces, a closing fence at least as long as its opener, and no backtick in a
backtick fence's info string -- that last one keeps a line holding only
`okf build` from silencing a document.

MEASURED ON THE BYTES, and this is the number that decides: the 43-document
reference corpus built at b6da09c (from git archive, never the editable
tree) and rebuilt at the shipped defaults differ in log.md alone, by the one
added bullet. 865 concept files on both sides, every concept byte-identical.

Found by that control and NOT caused by this work: the pinned artifact
K2-bundle-default-20260912 was written 2026-09-09, two days before ed0418f
changed title: quoting, so it differs from what HEAD produces on 42 concept
files. test_default_bundle_pin stays green because it pins the count and the
hit@8 ranks, not the bytes. Re-pinning is the operator's call.

Suite 1896 passed / 1 skipped (+27 from 1869). ruff, ruff format and mypy
--strict clean. No version bump, no tag, no push.

Report: docs/2026-09-15-f1-f2-gaten-og-kodefencen.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-16 00:19:50 +02:00
b6da09cc97 feat(quality): okf quality --fasit, boundary recall against a declared structure
The bundle-only gate returned UNMEASURED and exit 3 on the very arm it was
built for: no metric computable from a bundle alone reaches boundary recall.
`boundary_share` -- declared boundaries that became a concept, over declared
boundaries -- is the one metric measured that orders the arms correctly, and it
needs the publisher's own structure, so it arrives as an input.

Measurement first, threshold after, which is what the order asked for.

P1, the normalisation, derived rather than guessed: stripping all whitespace
and lowercasing reproduces the fasit's own `norm` from its own `title` on
2 761 of 2 761 rows (alphanumerics-only scores 58). P1's own bar is 99 % on the
known-good arm and the literal reading of it reaches 22 of 2 761 -- not because
the normalisation is wrong but because okf's default route moves the numbering
token a publisher glues into a heading over into the concept id. The pair form
(concept's own directory, residual title) reaches 2 737, either reaches 2 759
(99.9 %). Both forms ship and neither is a fallback: `r761-2025-d1` is the
control in the opposite direction at 2 727 literal, 0 paired.

P2, the single corpus, is in the OUTPUT and not only in the document: the bar
is declared `corpora = 1`, every boundary row prints `N = 1 corpus`, and the
line states that `--fasit` is the caller's ASSERTION that this bundle is a
build of the document the fasit describes -- the posture `okf consume --ref`
has. Measured: the K2 reference and `n100-2023` score 0 of 2 761 and read FAIL,
which is the assertion being wrong rather than the bundle being bad.

One bar, at the pinned artifact's own value, 2 759/2 761. It is tight and the
cost is published rather than tuned away: 2 of 4 R761 builds fall under it
(2 752 and 2 727), while any bar between 41.6 % and 98.8 % separates the
known-bad arm from every R761 build measured. The known-bad arm
(`860019-mdb-100`) is 1 148 of 2 761 -- FAIL and exit 1, where the bundle-only
gate gave exit 3.

A fasit is validated at the door: not a list, a row missing `title` or `norm`,
or anything that is not JSON exits 2 with the reason, never a quiet UNMEASURED.
A fasit under five rows is UNMEASURED -- the document floor in the fasit's own
unit.

Without `--fasit` the command is byte-for-byte what it was, held by a test.
`okf check` is untouched; no version bump and no tag. 17 tests red on
assertions before the implementation, the two new doc pins each driven red and
back. Suite 1 869 passed / 1 skipped / 1 870 collected (base 5e5d01c: 1 851).

docs/2026-09-12-g37-terskler.md SS 7 carries the premises re-measured, the
seven bundles, the interval any bar could sit in, and the honesty limits --
including the correction of SS 2's own grep claim, which went false in the
commit that wrote it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 07:27:33 +02:00
5e5d01cc0a chore(release): 0.9.0 v0.9.0
Version to 0.9.0 in pyproject.toml, __init__.py and uv.lock; CHANGELOG's
[Unreleased] becomes [0.9.0] - 2026-09-13, carrying G37's `okf quality` and
K3-26's README file-type table byte-identically. MINOR rather than patch: the
release adds a command.

README's four install lines and the two tag-pairing sentences move to v0.9.0,
and the tag history gains a `v0.9.0` entry stating what ships -- three
verdicts, two bars (.pdf 8/32, .docx 2/5), both regression bars against the
pinned reference and not quality claims, `okf check` untouched at seventeen
rules.

No behaviour change in src/ beyond the version string. The guard pin stays at
v1.4.0 and the dependency floor stays at >=1.2,<2.0. Suite 1 850 passed, 1
skipped, run after `git add`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 00:15:51 +02:00
6ea8fcd3c7 feat(quality): okf quality, a per-file-type verdict with the denominator
G37. `okf check` is a CONTRACT check and a green one is not a quality gate:
measured 2026-09-10 by `vegnormal-okf`, three arms over one corpus all
returned 0 findings and exit 0 while their hit@k ranged from 6 of 6 to 0 of 6.
`okf quality <bundle>` asks the other question, per file type, with the
denominator on every line. A separate command rather than `okf check
--quality`, because the two answer different questions and a caller must not
be able to read one as the other. `okf check` is untouched.

Three verdicts and no fourth -- PASS, FAIL, UNMEASURED -- and a type with no
measured threshold is never PASS. Exit 0 judged and clean, 1 at least one
FAIL, 2 did not run, 3 nothing could be judged: exit 0 over a table of
unmeasured rows would be the silent pass this command exists to stop.

Two bars today, both `structure_null_share` (documents of a type yielding
exactly one concept), read off the pinned 43-document reference bundle: .pdf
8/32, .docx 2/5. Plus one definitional bar for every type, taken from the
harness's own degenerate-merge definition: 0 concepts with an empty body,
measured 0 of 8 602 concepts over four bundles. A bar needs five documents on
BOTH sides -- its own and the judged bundle's -- so .xlsx (2), .xml (1) and
every type with no corpus class in `extract._EVIDENCE` are UNMEASURED and
print their numbers without a verdict.

The floor on the judged bundle was found by RUNNING the gate, not by reading
it: one PDF cut into 2 182 concepts scored 0 of 1 against the 32-document
reference and read as PASS.

The gate walks the index tree and never a directory (SS 9.2; controlled
against the listing on four bundles, 453 / 2 761 / 3 206 / 446 either way),
and prints the bundle's own run log beside its counts -- a document rejected
at extraction leaves no row in the bundle, so the pinned corpus's 33 PDFs
show up as 32 and the two denominators must never be read as one.

Three of the order's five premises moved when re-measured, and they are in the
document rather than glossed: the four evidence corpora carry `source_file` on
0 of 446, 0 of 1 133, 0 of 270 and 0 of 2 756 concepts, so they name no file
type and cannot PASS; "41,6 %" is `vegnormal-okf`'s number and not in this
repository; and the same 828-document bundle carries two published hit@k
figures from two question sets.

Three candidate metrics measured and NOT shipped: duplicate titles within a
document (0 of 3 206 on the known-bad arm against 349 of 2 761 on the
known-good one) and short concepts (5.6 % against 14.6 %) order the two arms
the wrong way round; duplicate titles across the whole bundle order all four
correctly (37.8 / 16.3 / 12.6 / 5.7 %) and still ship without a bar, because
any bar separating them is read off the two bundles it would judge.

19 new tests, each rule exercised in both directions; the three README pins
were each driven red before being kept. Suite 1 850 passed, 1 skipped, 1 851
collected, run after `git add` -- +19 against a base of 1 832 collected,
measured on the stashed tree (STATE's 1 831 is one short of that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 00:13:00 +02:00
d310dbb34e docs(readme): one file-type table, not two
`### Binary extraction` carried its own six-row Format/Reader/Evidence table
over the same rows the pinned table now holds. It was true when written and
reachable by exactly the failure this module exists for: three evidence classes
copied into prose no test reads. That section now points at the pinned table
and keeps its prose about the extra.

A fifth assertion in `tests/test_docs_promises.py` holds the duplicate gone.
Known-positive: the same query finds 8 table lines in that section on the
previous commit, so it can go red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 21:36:28 +02:00
2d3c2707c7 docs(readme): a visible table for all 13 file types, pinned to the registry
The extractor registry reads 13 extensions. The README's opening line named
five of them, and the full list existed only in a hidden
`<!-- extract-formats: ... -->` comment, which no reader reads -- so the README
undersold what the code does and stated no evidence class anywhere a consumer
would look.

A `## Supported file types` table now carries one row per extension: reader,
dependency (core or the `[extract]` extra), the evidence class `_EVIDENCE`
records for the row, and one honest note. The three `constructed` office rows
carry their denominators (N = 1, N = 2, N = 1) in the table itself, so a row
that has met no document anyone wrote cannot read as a supported one; `.htm`
does not borrow `.html`'s 828-file class, because the code records none for it.
A `Not read today` section states the absences (`.doc`, `.epub`, `.eml`/`.msg`,
image files, source files, `.one`/`.vsd`) as facts, not as a queue.

Test first, red before the table existed: four assertions in
`tests/test_docs_promises.py` pin the table's row set to
`_CORE_EXTRACTORS | _OPTIONAL_EXTRACTORS`, each evidence cell to `_EVIDENCE`
(and to a fixed `stdlib, no corpus class` where the code records none), the
core/extra split to the registries, and the opening to the table.

No change to `extract.py` and no version bump: nothing about what is read
moved, only what the README says about it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 21:26:15 +02:00
313a1497b7 docs(release): R7, v0.8.5 tagged locally and measured from the tag
The release report in R6's form: premises re-measured, the one-line lockfile
diff, the README lines the tag would have made false, the gates, and the
acceptance gate run from a local `git archive v0.8.5` export.

Two bookkeeping figures in K3-25's report corrected after re-measuring: the
tests/ call denominator was 129 occurrences on 128 lines (not 112 across 13),
and the outside-tests figure was 19 on 17 lines in 5 files (not 129, which was
the tests/ number written into the wrong paragraph). The "exactly 1 changed
result" finding is unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 19:47:30 +02:00
64661c7198 chore(release): 0.8.5 v0.8.5
Version to 0.8.5 in pyproject.toml, __init__.py and uv.lock; CHANGELOG's
[Unreleased] becomes [0.8.5] — 2026-09-12, carrying K3-23's instrument,
K3-25's default and K3-24's block-sources fix byte-identically, with the
two `### Changed` headings merged into one and one post written here: what
`okf.parse_frontmatter` now returns for a block `sources:` is a READING
projection PyYAML reads back on 0 of 4 605 measured block files.

No behaviour change in src/ beyond the version string. The guard pin stays
at v1.4.0 and the dependency floor stays at >=1.2,<2.0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 19:38:46 +02:00
06c057ff39 docs(k3): round 25, the default reading drops the door's link line
What the default was and is, the three signatures, the naming choice against
the module's other eight constants, the CLI-flag decision with its zero, both
gates with this round's numbers beside round 23's, the consumer list with its
denominator, and the honesty limits.

The deviation is stated first and it is the order's own acceptance row: `S1
spent 28 020 B at the default k` is round 23's X column -- the flagged bundle
with the line SCORED -- and the new default is round 23's Y, whose published
value for that cell is 31 031. Measured here in one process on one bundle:
`link_in_signal=True` gives 28 020 and the default gives 31 031, the unflagged
build gives 31 031 too. 16 of 16 cells of the Y column reproduce to the byte,
so neither failure the stop-rule guards against is present; the row was
transcribed from the column being retired.

Also corrects one label in round 23's own file, which is otherwise untouched
because a report is a measurement with a date. The row reading "distinct tokens
the path ever contributed" carried 31/22/8, which is the QUESTION-token count:
the path's segments are `r761` and `prosesskoden`, and `prosess` is a question
token that reaches the second through the stem prefix rule -- measured,
`consume.tokens_match('prosess', 'prosesskoden')` is True at
`MIN_SHARED_PREFIX = 4`. The label is what was wrong, and correcting it is what
makes the table agree with the paragraph under it, which already explains
`prosess` that way. The numbers stand, and no CHANGELOG figure is touched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 19:07:55 +02:00
38320cd509 feat(consume): the door's link line is out of the default reading
`link_in_signal` defaults to `False` on all three entry points --
`searchable_text`, `concept_scores` and `build_payload` -- through one named
constant, `DEFAULT_LINK_IN_SIGNAL`. This carries out K3-23's recommendation
(alternative c), and its numbers are the whole warrant: of the newcomers the
door's `Enclosing section:` line ever added a question token to, 39 of 39
gained it from the bundle-absolute PATH and 0 of 39 from the link's title, and
every token the path contributed is a segment of the document's own directory
-- the saturation `shared_id_prefix` takes out of the id signal, back in
through the body.

A CONSTANT rather than three naked literals, chosen against the module: the
other eight defaults here are constants (`DEFAULT_PROFILE`, `DEFAULT_LIMIT`,
`DEFAULT_TIE_SHARED_RANK`, `DEFAULT_STEM_PREFIX`, `DEFAULT_TITLE_COVERED`,
`DEFAULT_SOURCE_QUOTA`, `DEFAULT_FOLLOW_PARENT`, `DEFAULT_K`) and this was the
one default spelled out three times. It also lets a test fall on the rule from
the module that owns it.

NO CLI flag is added and none is changed. `grep -n '"--.*link'` over `src/`
gives 0 hits before and after, and `consume.main` never passed the parameter,
so the CLI inherits the function default. A flag was considered and NOT added,
because no caller could be named that needs the older reading: the three call
sites outside `consume.py` (`skill.py:360`, `tools/okf_consume_measure.py:155`
and `:247`) all inherit it deliberately.

Measured before landing: 0 of 5 shipped bundles carry the door's line
(denominators 446, 1 133, 270, 2 756, 2 761 concept files) and 5 of 5 payloads
are byte-identical across the move. The consumer repository was read only and
its tree is unchanged, listing identical including mtimes.

The docstrings move with the code rather than after it. `concept_scores` said
this was "an INSTRUMENT, not a flag and not a default" and `build_payload` that
it "carries no CLI flag"; the first is now false and the second still true, and
each says so. `cli.DEFAULT_SHELL_PARENT` keeps its value and loses its reason:
the cost it cited -- delivered sets moving through that path -- is what this
commit removes, so the comment now states why it stays off NOW, which is N = 1
document and a consumption half nobody has judged. README and CLAUDE.md carry
the same sentence where they already describe the line, measured rather than
written on reflex: neither said anything about the ranking before.

One test changed with the code. The characterisation of what the LINE does
called `concept_scores` with no parameter, because the reading it characterises
used to be the default; it now names that reading. Same fixture, same number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 18:56:53 +02:00
3816ed505c test(consume): the reading without the door's link line is the default
K3-23 measured the whole decomposition and landed on one recommendation:
the cost is the PATH, not the link. 39 of 39 newcomers that won a question
token from the door's `Enclosing section:` line won it through the
bundle-absolute path and 0 of 39 through the link's title, and every token
the path ever contributed is a segment of the document's own directory --
the saturation `shared_id_prefix` took OUT of the id signal, back in through
the body.

Ten cases, each red for its own reason on `7faa380`, and each one measuring
BEHAVIOUR rather than a signature: the three entry points are called with no
parameter at all and what came back is read.

- `searchable_text`, `concept_scores` and `build_payload` with no parameter
  read a body without the door's line (3 cases).
- The declared default IS the behaviour: the constant is imported from the
  module that owns it, INSIDE the test so the red falls on the rule and not
  on collection, and then spent on a payload comparison.
- Y = Z in miniature: the flagged bundle read by default delivers the same
  list, the same ORDER and the same `spent` as the unflagged build.
- `link_in_signal=True` is still reachable and still does the old thing --
  both halves, because either alone would pass on a no-op.
- The two known-negatives restated against the DEFAULT path: a human line
  opening with the same two words, and the door's exact form anywhere but
  last, keep every byte.
- Rank, never form: the two readings differ in the delivered set while every
  concept delivered by both carries identical excerpt bytes.

Two stay green on both sides on purpose. The known-positive -- a bundle where
no body ends in the door's form is byte-identical under both readings -- is
section 4A's exposure gate in miniature and would go red only if the stripper
became eager. The characterisation of what the line does today is HEAD's
behaviour and keeps stating it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 18:51:32 +02:00
7faa380666 docs(k3): round 23, the path in the body signal, decomposed per signal
The measurement round 21 deferred. On R761, of the 39 newly delivered concepts
the door's link line ever added a question token to, 39 gained it from the
bundle-absolute PATH and 0 from the link's title; every token it ever
contributed is a segment of the document's own directory. So the cost is not
the link, it is the path inside it -- and only that statement points at a fix.

Rank movement and budget displacement are separated on one bundle by three
readings plus a scratch-only fourth: X vs Y is ranking, X vs W is bytes, and
Y equals Z on 16 of 16 rows, which is what makes the separation measured
rather than assumed. hit@1/8/50 stays 6/6 and the known-positive at rank 1
under every reading.

`--shell-parent` stays off at the current link form: two of three acceptance
conditions fail. The third exit is measured with the same numbers and
recommended -- reading the body without the line moves nothing on the 5 of 5
bundles anyone ships, and would make the flag free.

The link cost is 71 616 B / 4.45 %, not round 21's 72 265 B / 4.49 %; the rule
behind the number is stated, and round 21's file is left standing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 17:44:42 +02:00
8e82da4682 feat(consume): the ranking can read a body without the door's link line
One parameter, `link_in_signal`, default `True` -- so no payload moves. `False`
scores the body with `inbox._link_enclosing`'s line removed, and removes it from
the stem vocabulary too, because `searchable_text` counts `df` over the text a
hit is scored on: taking the line out of one and leaving it in the other would
measure two different texts.

The recognition is the door's own constant plus its own place -- last in the
body, after a blank line, in the door's link form. The excerpt keeps the line
either way, so two readings of ONE bundle differ in ORDER alone, which is what
separates a ranking movement from a budget displacement.

Not a CLI flag: whether this becomes a default is K3-23's measurement, not an
implementation's decision.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 17:25:00 +02:00
18b390399d test(consume): the door's link line reaches the ranking signal, and nothing can hold it out
`inbox._link_enclosing` appends one line to a heading-only body; `consume._body`
returns everything after the frontmatter, so the line is scored by the body
signal and counted by the lexical gate. Nothing in the code ever decided that:
SPEC SS 6.1's argument for the bundle-absolute form is about the FILE.

One characterisation, green on HEAD, holds the mechanism: a shell whose own
text answers nothing in the question still scores, on the door's line alone.
The eight red ones ask for the instrument K3-23 needs -- one bundle, two
readings, in one process -- and for its two obligatory known-negatives: a human
line opening with the same two words, and the door's exact form anywhere but
last, both keep every character they have.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 17:21:50 +02:00
2d4f56dc6c docs(k3): round 24, a reason that became false is struck and the emitter stays flow
`_render_sources` gave three measured reasons for refusing to emit the block
form. Two of them fell with `28f9a4b` and are struck rather than left
standing: a block list no longer round-trips to an empty value (4 609 of
4 609 concept files), and B6's acceptance test can now pass.

Reason 2 STANDS and now carries the emission rule alone, re-measured by
reading `portfolio-optimiser` at `6eb58e5`: `read_provenance` returns
`UnreadableProvenance(reason="block-sequence")` for a block value. It is not
the guard's objection -- guard 1.4.0 reads the block form on 4 609 of 4 609
of the same files -- so the answer to "should our own emitter write block" is
NO, with the number that carries it, and the measurement that would make it
decidable is named in STATE rather than deferred without one.

The README paragraph said this library "cannot read the block form" where one
reader could and one could not. It now separates the two by KEY: `sources` in
either form through every reader, every other block mapping still refused,
with the D1b pointer intact.

Report: 222 lines, S7 form. Five deviations first, including the acceptance
sentence naming THREE flat readers rather than one, a `title` denominator of
7 372 rather than 5 372, and the `okf check` rule answered NO with a
measurement (`contract_check` holds 0 occurrences of `sources`, and
`consume` already writes `sources_unreadable` for the state a rule would
restate; `len(RULES)` is 17). Honesty limits name the one-key-wide set, the
block `verified:` fixture still reading empty, and the rendering not being
byte-identical to the source.

Suite 1807 passed / 1 skipped, rc 0, 98 s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 16:50:26 +02:00
28f9a4b540 fix(profiles,materialize,structure,consume): a block sources sequence is decoded, not skipped
One grammar, four call sites. `read_block_mappings` moves out of
`consume.read_sources` -- where it was written and measured -- into
`profiles`, the module both the flat readers and `consume` already import,
and the three copies of the line-oriented frontmatter grammar now decode a
block sequence for the keys `STRUCTURED_BLOCK_KEYS` names. Two copies of a
block grammar would be two answers to one question.

The value TYPE was the real choice and it was measured, not argued.
`parse_frontmatter` is public API (`okf.parse_frontmatter`) returning
`dict[str, str]`, and a list of mappings is not a `str`. Widening the return
type to `str | list[dict[str, str]]` costs 15 `mypy --strict` errors across
four of the five modules that touch the reader, plus a signature every
caller outside this repository would have to follow. Rendering the entries
back into the flow form those same readers already round-trip costs 0. The
rendering is a READING projection and says so: it is not a claim that the
value is writable -- `yaml_flow_plain` still refuses a `?` and the guard
still refuses a quote inside a flow mapping, which is why the producer
writes block in the first place.

`STRUCTURED_BLOCK_KEYS` is one key wide. `sources` is the key `read_sources`
already knows how to read; a fixture in this tree carries a block
`verified:` that still reads as an empty value, and a test pins that state
so the next widening is a decision rather than a side effect.

Nothing nested reaches the document's namespace: the entries land inside
their own value, and the K3-20 substitution guarantee is asserted per reader
copy.

Three tests that pinned the old behaviour are rewritten to what is now true,
none weakened on its other half: the block round trip in
`test_multi_source_provenance` (the evidence behind `_render_sources`'
reason 1), the v0.2 characterization (whose key-space assertion is the half
that must never weaken), and K3-22's shipped-file known-positive, where the
one difference is counted and pinned at 1.

Suite 1807 passed / 1 skipped, rc 0, 94 s -- 1782/1 before plus 25 new.
ruff clean, `mypy --strict` clean over 21 files, `uv.lock` untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 16:42:22 +02:00
eb327bfd0c test(materialize,structure,profiles): a block sources sequence reaches every flat reader
`consume.read_sources` reads both YAML forms. This library's three copies of
the line-oriented frontmatter grammar read only the flow one, and on a block
sequence they return the key with an EMPTY value -- not a `KeyError` a
consumer can catch, an address that disappears.

Measured 2026-09-12 over four bundles a producer ships, denominator = files
carrying a frontmatter block: 2 756 of 2 757, 446 of 447, 1 133 of 1 134 and
270 of 271 concept files lose the address through the flat readers, while
PyYAML 6.0.3 and the pinned guard 1.4.0 both read it on 100 % of the same
files.

The bar is dict EQUALITY against two independent readers rather than "it
parses". The two disagree on one axis and it is named rather than averaged:
the guard keeps a leaf's quotes verbatim, PyYAML decodes them, and this
library follows `read_sources`' K3-22 rule.

Red: 13 failed, 12 passed. The 12 are the known-positive controls and the
known-negatives the fix may not move -- the nested-`title:` substitution trap
above all, plus the flow form, an absent key, and a shape `read_sources`
refuses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 16:35:03 +02:00
150c7266c6 docs(release): R6, v0.8.4 installed and measured from the tag
The release report for v0.8.4 (69dc51f): the lockfile moved four lines,
not three (the guard's own version follows the pin); the dependency floor
measured against guards 1.2.0 / 1.3.0 / 1.4.0 and left standing; the
acceptance rows re-measured from a local export of the tag, including
guard 1.4.0 refusing 0 of 28 frontmatter blocks where 1.3.0 refuses 26.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 15:04:21 +02:00
69dc51fa89 chore(release): 0.8.4 v0.8.4
Version 0.8.3 -> 0.8.4 in pyproject.toml, __init__.py and uv.lock, and the
guard pin in [tool.uv.sources] moved from v1.3.0 to v1.4.0 (uv.lock follows
via `uv lock`: the guard's version, source and requires-dist lines, nothing
else). CHANGELOG [Unreleased] closed as [0.8.4] with K3-21's and K3-22's
entries moved as written, plus one entry for the guard pin: 1.4.0 parses a
flow sequence of plain scalars that 1.3.0 refused (26 of 28 frontmatter
blocks refused by 1.3.0, 0 of 28 by 1.4.0, on one five-document bundle).
Dependency range llm-ingestion-guard>=1.2,<2.0 unchanged.

README: the four install lines and llms.txt's one to @v0.8.4, the guard
pairing prose to v0.8.4 / v1.4.0, the pip fallback to @v1.4.0, a v0.8.4
row as the current tag (seventeen checker rules), and the suite count
(1783 collected, 1782 passed, 1 skipped) with its history extended.

No behaviour change in src/ beyond the version string.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 14:53:50 +02:00
e98ca6d25f docs(k3): round 21, parent all the way to the reader
The record for K3-21 A-D: what an excerpt, a heading-only body, the checker
and the index now carry, and every gate with its denominator -- the pinned K2
bundle (7 passed in each export), 32 regression payloads, R761 with and
without `--shell-parent` (hit@1/8/50 6/6 at both k, the known-positive at
rank 1, 675 of 710 shells with exactly one link and 0 broken, 710 of 710
pointing at the source's own ancestor after D), consumption 6 of 6 in both
forms, the byte cost against the consumer's 51.0 %, the ranking cost the
link's path carries, and the 35 shells with no ancestor holding text, listed.
Deviations from the order stated first; honesty limits last.

CHANGELOG under Unreleased; README and CLAUDE.md say 710 of 710, 17 rules,
and that the outline mark keeps its depth. The README's project measurement
was re-run (26 concepts, 0 checkout paths, `okf check` 17 rules, 0 findings).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 14:28:42 +02:00
bd13c5f00b fix(extract): a declared section's outline mark keeps its depth; only the heading clips
K3-21 D. `_StsReader._walk` computed `level = min(depth, _ATX_MAX_LEVEL)`
once and wrote it into BOTH the ATX heading and the `OutlineMark`. The
heading keeps the clip -- markdown has six levels and `#######` matches
nothing -- and the mark now carries the declared `depth`: it is no heading,
the declared route builds `Candidate(level=mark.level)` from it, and
`OutlineMark`'s own docstring says the level is what the tree declares,
"reported rather than fixed up". The extracted text does not move one
character; only the mark's level does.

On one standard 9 of 2 761 titled sections sit at depth 7; `--shell-parent`
gave its two depth-7 shells (`36.73211`, `36.73212`) the ancestor one level
too high. The gate -- plan otherwise unchanged, concept count, goldens, the
regression bundles, hit@k and the other 708 pointers -- is measured in a
frozen export of this commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 14:10:30 +02:00
9822e5500c test(extract): a section below markdown's sixth level keeps its own level in the plan
K3-21 D, red. The NISO-STS reader clips `depth` to 6 once and writes that
one value into BOTH the ATX heading, where markdown has six levels, and the
`OutlineMark` the declared route builds its plan from, where nothing clips.
On one standard 9 of 2 761 titled sections sit at depth 7; the plan read them
at 6, and `--shell-parent` gave its two depth-7 shells the ancestor one level
too high.

`sts-deep.xml` (invented setting) reproduces it: the two depth-7 shells point
at the depth-5 section on e717b1c's code and must point at the depth-6 one.
Held: the mark carries depth 7; the heading stays `######` and the extracted
headings do not move one character; the shells point at their depth-6
ancestor.

2 of 3 red on 5970369; the green one is the text guard.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 14:02:49 +02:00
59703699a5 fix(structure): the index resolves a parent naming a segment of its own document
K3-21 C. `resolve_structure` asks `_segment_lookup` first for a `parent`
edge: `(source_file, segment_id)` -> concept name, keyed off each concept's
own frontmatter (`DocumentStructure.declared`, no file read again), so a
pointer lands only inside the pointing concept's document -- `p1` exists in
every document of a multi-document bundle. A value no segment answers to is a
document number and is looked up exactly as before; a pointer naming nothing
keeps `UNRESOLVED_MARKER`. The rendering rule is untouched: a resolved
relation renders as its subject, so `parent: p1977?` becomes `parent: p1977`.

Moved on purpose, each named: both segmented goldens' index files
(`examples/ingest-golden-segmented{,-okf-v0-2}/expected-bundle/krav/1-{1,2}/
index.md`), whose declared parents s1 and s2 -> s0 rendered `parent: s0?`
while s0 stood in the bundle -- four lines, `?` removed. The four goldens
`test_the_four_existing_goldens_are_untouched` guards are not among them.
`skills/okf-consume/` regenerated, because the golden's index bytes -- and so
its ref -- moved. `test_shell_parent`'s byte test now expects the resolved
facet. README and CLAUDE.md no longer say the index renders it unresolved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 13:55:03 +02:00
c188a0c749 test(structure): the index resolves a parent naming a segment of its own document
K3-21 C, red. `structure` read `parent` as a document number only; a
`segment_id` answers to no key its lookup builds, so every `--shell-parent`
pointer rendered unresolved (`parent: p1977?`, 675 of 675 on one standard)
and both declared parents in each segmented golden render `parent: s0?` --
while the concept each names stands in the bundle.

Held here: a segment pointer resolves among the concepts sharing the
pointer's `source_file`; the same id in another document is not the parent; a
pointer naming nothing still carries `UNRESOLVED_MARKER`; a parent that is a
document number resolves as before; the flagged fixture build's index renders
no parent unresolved.

3 of 5 red on 46e555d; the two green are the unchanged routes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 13:48:11 +02:00
46e555d460 docs(consume,build): both parent defaults stay off, and the reason is now a measurement
K3-21 B. `cli.DEFAULT_SHELL_PARENT`'s comment said it was off because `okf
consume` reads no `parent` key; after A that sentence was false and the
default was unmeasured in both directions. Re-measured on the one standard
with heading-only sections: hit@1, hit@8 and hit@50 stay 6/6 at both k with
the known-positive at rank 1, but the link's bundle-absolute path repeats
the document's directory in 675 bodies, and the delivered set moved on 2 of
8 questions at the default k and 3 of 8 at k 50, newly delivered shells
matching the question only through that path on 1 of 3, 4 of 7 and 6 of 23
-- the saturation K3-20 took out of the id signal, back through the body.
Off.

`consume.DEFAULT_FOLLOW_PARENT`: the same delivered set as the pointer (16
of 16 payloads), 6 of 6 correct either way on questions whose answers are
never shells, at +2 488 / +2 746 B on the 2 of 8 default-k payloads holding a
linked shell and up to +20 817 B at k 50. Off.

A test holds both defaults and that the false sentence is gone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 13:39:50 +02:00
839bd61349 feat(consume): --follow-parent carries the enclosing section's text, from room the cut left
K3-21 B. The second form of `parent`: `okf consume --follow-parent`
(`consume.attach_parent_text`) puts the enclosing concept's text inside an
excerpt's `parent`, with that concept's own `sha256` so a claim resting on it
is cited as that concept. It runs AFTER the cut, on the room the cut left, in
rank order, so the delivered set, its order, the withheld list and the
denominators are the same with the flag as without it -- inherited text
cannot displace an excerpt, the mechanism a consumer measured when copied-in
ancestor text pushed the right section to withheld place 504 and 1 069. A
text that does not fit is cut to the longest prefix that does and marked
`truncated`; a parent the payload already holds, or one a higher-ranked
excerpt already carried, travels once. OFF; the defaults are chosen on the
measurement that follows this commit.

`delivered_text` is the one normalisation an excerpt's `text` and a parent's
share. Contract SS 8 point 6 gains the MAY; the template tells the reader
what `text`, `sha256` and `truncated` mean. README and CLAUDE.md name the
flag.

Moved on purpose: the SS 7.4 known-positive again (14 455 / 14 083 / 372 ->
14 721 / 14 346 / 375), and `skills/okf-consume/` regenerated with it.
`tests/test_parent_text.py::test_no_room_means_no_text_and_no_lost_excerpt`
changed from its red form: it asked through `build_payload` at `limit ==
spent`, where the knapsack's 500 B buckets admit nothing at all
(`budget_admits_nothing`); it now holds the rule at `attach_parent_text`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 13:11:24 +02:00
ca83dadf53 test(consume): the enclosing section's text, and it never displaces an excerpt
K3-21 B, red. The second form of `parent`: `okf consume --follow-parent`
carries the enclosing concept's text inside `parent`, with that concept's own
`sha256`, placed AFTER the cut from the room it left, in rank order. Held
here: the delivered set, its order, the withheld list and the denominators
are the same with the flag as without it at k 1, 8 and 50; a text that does
not fit is clipped and marked `truncated`; with no room there is no text and
no lost excerpt; a parent the payload already holds, or one two shells share,
travels once; the checker accepts the form; the CLI takes the flag and its
opt-out.

7 of 8 red on 4f7bd61; the green one is the pointer-alone default.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 12:53:33 +02:00
4f7bd61500 feat(consume): parent reaches the reader -- excerpt field, body link, checker rule
K3-21 A. `okf consume` resolves a concept's `parent:` pointer -- a
`segment_id`, unique only inside one document's plan -- among the concepts
sharing its `source_file` (`consume.link_parents`, one pass, no file opened
again) and an excerpt carries `parent: { concept_id, title }`. Conditional
like `req_number`: a concept with no `parent` key moves no byte. A pointer
that lands nowhere is named `parent_unresolved: true`, never dropped.

The door writes ONE line into a heading-only body whose entry has a parent:
`Enclosing section: [<title>](/<bundle-relative path>)` (SPEC SS 5.1 lineage
through links, SS 6.1 the recommended absolute form and the kind in the
prose). Only such a body, so the segmented goldens' declared parents -- bodies
holding text -- are untouched. Appended AFTER structure derivation and
screened on its own (`_screened`, the `description` rule): read as body text
the link was derived into a second, unresolved `references` edge, measured on
the fixture. `segmentation.heading_only` is the one predicate the proposer and
the door share.

`okf check` gains its seventeenth rule, `parent_unfollowable`: a `parent`
that is not a concept_id and title, names its own excerpt, or names a concept
in neither `excerpts` nor `withheld` (together every considered concept).
Contract SS 8 point 6 added, the figure carries `parent`, and "additional
members are not read by the checker" now says the checker reads only the
members SS 8 names. The template tells the reader what `parent` is and that
SS 2.2 lets it read that one concept; `skill.CONDITIONAL_FIELDS` gains
`parent`. README and CLAUDE.md say what consume now reads.

Moved on purpose, each named: the SS 7.4 known-positive IS the contract
document, so `budget.known_positive` moves in every payload (13 238 / 12 893
/ 345 -> 14 455 / 14 083 / 372); `skills/okf-consume/` regenerated from the
segmented golden, whose plan declares s1 and s2 under s0 -- its example
payload now carries both parents; `test_bundle_identity` 16 -> 17 rules;
`test_shell_parent`'s byte test also accounts for the link line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 12:36:23 +02:00
a5cd7c5688 test(consume): parent reaches the reader -- payload, body link, checker
K3-21 A, red. Round 20 wrote `parent:` on 675 of 710 heading-only sections
and no reader could see it: `okf consume` did not read the key, the payload
did not carry it, the body held no link, and the pointer was a `segment_id`
a reader cannot open without enumerating the bundle.

Held here: a heading-only body carries ONE bundle-relative link to its
enclosing section (SPEC SS 5.1, SS 6.1); an excerpt carries `parent` as the
resolved concept id and title, never the raw id; an unresolvable pointer is
named `parent_unresolved`; a pointer resolves inside its own document; the
checker has 17 rules and refuses a parent a reader cannot follow, with four
known-negatives; the contract and the skill template name the field.

11 of 12 red on e717b1c; the one green is the guard that a payload with no
`parent` meets the rule as before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 12:20:23 +02:00
e717b1c87a docs(k3): round 22, frontmatter a YAML reader reads back the same
K3-22's report, CHANGELOG under [Unreleased], and two corrections of a
sentence measured false.

README and CLAUDE.md both said of the flow form "both are valid YAML, and a
real YAML consumer recovers the same structure from either". An unquoted URL
with a query string inside a flow mapping is not valid YAML for PyYAML, and
the quoted form is refused by the pinned guard; the paragraph now states the
limit. CLAUDE.md also records the K3-22 invariant and the new refusal of a
`--frontmatter` flow value with such a leaf.

Measured, fix `ed0418f` against base `0308169`, each tree built twice from
frozen exports: five-document project 0 files moved; R761 XML and HTML 1 line
each; K2 default 42 `title` lines, after which 454 of 454 frontmatters parse
and read back the same (base 413 parse, 412 read back). hit@k on R761 base =
fix, S1-S6 6/6 at hit@1/8/50 at both k, KP rank 1; okf check 32 of 32 payloads
0 findings over 16 rules; K2 pin 7 passed in the export.

Found outside the order: the pinned guard refuses nearly every segmented
concept okf writes, on its scalar flow sequences (`source_offset`,
`references`, `derived`) -- identical before and after, so older than K3-22.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 11:47:40 +02:00
ed0418f228 fix(frontmatter): write a value a YAML reader reads back, and read both forms
K3-22. SPEC SS 11 point 1: "Every non-reserved `.md` file in the tree
contains a parseable YAML frontmatter block." Measured with PyYAML 6.0.3,
okf's own default K2 bundle failed safe_load on 41 of 455 blocks and the
R761 build on 1 of 2 763, every one a block scalar written verbatim.

Block (the profile emitter, every key): a value the K3-19 rule refuses as
plain is written double-quoted, `\` and `"` escaped; every other value keeps
its bytes, and a flow collection or an empty value is written as it stands.
The rule, now `profiles.yaml_block_plain`, agrees with PyYAML on every
top-level value in eleven measured trees (0 refused that it reads verbatim,
0 kept that it does not). Double, never single: 0 values in those trees are
`"`-wrapped and 11 193 are `'`-wrapped.

Flow (`sources`, Door A and Door B, and a run-stated flow value): the pinned
guard refuses ANY quote in a flow mapping (1.3.0, measured), so a leaf PyYAML
needs quoted has no form both read. `yaml_flow_plain` refuses it instead:
`,[]{}`, `?`, a quote, ": ", " #", a trailing `:`, a leading indicator -- a
leading `-` before a non-space excepted, which both readers take. The file
name is checked too, because it is the entry's `title` when the document
declares none. Existing codes: inbox_source_file_unaddressable,
inbox_source_title_unaddressable, source_reference_unquotable,
run_frontmatter_invalid.

Readers: parse_frontmatter, profiles' and structure's copies, and both
read_sources branches unquote a `"`-wrapped value (`\"` and `\\` decoded,
nothing else); `'`-wrapped values are untouched, and structure keeps the
single-quote rule it already had. The flow-mapping split is quote-aware, so
`{ title: "a, b" }` is one pair. The generated SKILL.md header goes through
the same block rule.

TWO K3-19 TESTS MOVED, deliberately: test_run_frontmatter built with
`sources=[{ resource: ...?languageCode=nb, ... }]`, the exact form PyYAML
refused on 2 761 of 2 761 frontmatters of K3-19's flagged build. The two
build tests now write an address without `?`; the flag-grammar test keeps
the `?` address (it only splits), and a new test holds that the build
refuses it with exit 2 and writes nothing.

1753 passed, 1 skipped (OKF_HTML_CORPUS, known). No golden moved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 11:09:49 +02:00
06e61a5acf test(frontmatter): what okf writes must be YAML a YAML reader reads back the same
K3-22, red first. SPEC SS 11 point 1 requires "a parseable YAML frontmatter
block" in every file. Measured with PyYAML 6.0.3 before any code moved: the
K2 default bundle this repository pins fails safe_load on 41 of 455
frontmatter blocks, all on `title` (a leading `- `, `**` or `*`, or ": "),
and the R761 build on 1 of 2 763 (a title ending in `:`). No `sources` value
okf itself wrote failed; the 4 605 consumer failures come from that
consumer's own writer.

Each case goes through a public path (render_inbox_concept, the profile
emitter, Door A's sources renderer, skill.render) and is held to three
readers: safe_load must not raise, BaseLoader must return the same strings
as parse_frontmatter / read_sources, and the pinned guard must admit it. The
guard is why quoting inside a flow mapping is not the fix: 1.3.0 refuses any
quote in a flow mapping (measured), so a `sources` leaf PyYAML needs quoted
has no form both readers accept, and is refused instead.

57 of 85 red on 0308169; the 28 green are the known-negatives and controls.
The rest of the suite is unchanged: 1695 passed (1667 + 28), 1 skipped.

pyyaml joins [dependency-groups] dev and nothing else; uv.lock moves by
exactly the two lines that dev dependency adds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 10:56:05 +02:00
0308169c79 docs(release): R5, v0.8.3 installed and measured from the tag
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 06:21:31 +02:00
0963dfa5e5 chore(release): 0.8.3 v0.8.3
Tag K3-19 and K3-20 together: a NISO-STS document's own identity names
its directory and titles its sources entry, --frontmatter KEY=VALUE and
description from a section's first spec point, and the consume fix that
keeps a leading directory every concept id shares out of the first
fusion signal. K3-19 alone dropped the known-positive from rank 1 to not
delivered; K3-20 A is what makes the identity safe to ship.

Version 0.8.2 -> 0.8.3 in pyproject.toml, __init__.py and uv.lock (uv
lock changed that one line only). CHANGELOG [Unreleased] closed as
[0.8.3] with its five entries moved unchanged. Five install lines move to
@v0.8.3, the tag history gains a v0.8.3 row, and the suite count is
re-measured: 1668 collected, 1667 passed, 1 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-11 06:09:54 +02:00