Version 1.1.0 in pyproject, the package, uv.lock and every README install
line; the "current tag" entry moves to v1.1.0 and v1.0.0 becomes history.
The CHANGELOG states what a user notices: BM25 ranking on passages and
titles, sub-questions in one call, a map per bundle, a weak-coverage
signal, one generic skill plus the server as the standard way in over a
folder, --gate in okf project, and okf-consumption/2 as BREAKING for a
reader of withheld. Known limitations gain the vocabulary route through
the map, the retrieval gate's red rows as they stand under BM25, the
answer smoke test (no corpus content, no counts) and unmeasured speed.
eval/svar/ is ignored the way eval/soek's sets are: answers about a
consumer's corpus are inputs, never committed. No tag, no push.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`okf check --payload` takes the reply to one call over a folder as well
as a single payload: every bundle's payload is held to all 19 rules on
its own, a finding is named with its bundle, one every payload carries
alike is reported once, an answer labelled with a bundle its payload
does not describe is `answer_misattributed`, and a reply with no answer
is `payload_invalid`. No rule is added, and a single payload's report is
unchanged. Contract SS 2.5.4 names the folder run and SS 8.11 fixes the
reply; the known-positive moves to 24 620 / delta 592.
The skill text follows: the working method's steps 1 and 4 name the
folder, and the generic skill says to use the server's tools first where
they are registered, with the skill as the supplement. The folder is an
instruction in both generators, never a path: the bundle's parent
written absolute named this checkout, and the test holding generated
commands to no repository path fell on it.
v1.1 order F, part F4.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`okf project`'s closing text and the README's first screen now say it in
that order: register `okf mcp --root` once, and it answers from every
project and reaches subagents; the skill beside the bundle is for someone
who would rather register nothing; neither is made again when a bundle is
rebuilt. Two tests hold the order in both places. The README's stale
`<id>-consume` skill path is corrected to `okf-consume-any`.
v1.1 order F, part F3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`okf consume <folder>` is the server's `okf_ask` with no bundle named,
byte for byte: no ranking of its own, every excerpt carrying its bundle
id. `--bundle-id` asks one bundle under the folder. A flag that acts on
one bundle's cut is refused by name over a folder rather than dropped,
because the server takes none of them. A bundle path reads as before.
v1.1 order F, part F2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`okf card <folder>` prints `okf_list` and `okf_describe` with no bundle
named, joined by `mcp_server.overview`, and computes nothing of its own:
one source, two doors. Chosen over a separate `--root` flag because the
skill's first step must be the same command whether it was pointed at a
bundle or at a folder of them; the rule that decides is discovery's own
(a directory carrying an `index.md` is a bundle). A bundle path prints its
card as before.
v1.1 order F, part F1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
C6. `docs/consumption-contract.md` now says what the pre-pass emits:
- SS 2.5 point 2: sub-questions are written in the bundle's terms and given to
the pre-pass in ONE run where it takes several;
- SS 8's example is `okf-consumption/2` as shipped: `withheld` is the
`{total, by_rule, nearest, complete}` mapping, `coverage` carries
`absent_terms` and `weak`, an excerpt carries `text`, `text_sha256`,
`passage` and `own_title`; point 1 closes the identity on `withheld.total`;
- point 7 keeps the falsified verdicts on record and states the one reading
carried since v1.1, `weak`, with its rule: a word held in no form, or
nothing delivered -- whether a word exists at all, which does not move with
question style or corpus size the way the two shares did;
- point 8 the passage (`{start, end, of}`, `text_sha256` over the passage,
`sha256` still the file's), point 9 several sub-questions in one run (one
cut, `questions`, `subquestions`, per-sub-question coverage), point 10 a
title inherited from the heading above, with `own_title`;
- SS 10 names this library's default ranking (BM25) and `--ranking fusion`.
`okf check` holds the two new forms a reader acts on: `passage_malformed`
(not whole numbers with 0 <= start < end <= of) and `subquestions_unindexed`
(not distinct indices into `questions`, or indices in a payload listing none).
17 -> 19 rules; the two tests that pin the published count move with it.
Each rule is held against a real payload (0 findings) and against that
payload broken six ways.
Editing the contract moved the SS 7.4 known-positive, measured once after the
edit: 19 837 -> 23 672 encoded, 19 358 -> 23 092 raw, delta 479 -> 580. The
example payload and `skills/okf-consume` are regenerated by the published
recipe and `okf check` reports 19 rules, 0 findings on them.
Suite on a clean tree after `git add`: 2443 passed, 2 skipped, 4 xfailed.
ruff, ruff format, mypy --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
C5. `bundlemap.build_map` lists a bundle in its own words: one line per
source document -- its name, then the titles of its concepts in document
order -- and documents whose names differ only in their numbers (a changelog
per release, a note per week) as ONE line: the name with every number as `#`,
the count, the first and last by natural order, and the titles across the
series that are words. `SERIES_MIN` = 5, at most `TITLES_PER_LINE` = 24 titles
a line, the lines capped at `MAP_MAX_BYTES` = 48 000 together with
`lines_truncated` counting the rest. Derived on every call, never stored.
The card (`okf card`, `okf_describe`) carries it as `map` and no longer
carries `source_files`: that list named every document a second time with no
series collapsed, a quarter of the reply on a large bundle, for names the map
already carries. Chose removal over keeping both because the describe reply
has to fit a client's tool-reply limit and the map says more.
The working method now reads: take the map first (`okf card`, or
`okf_describe`), write two to four sub-questions in its words, and send them
in ONE call (`--question` repeated, or `okf_ask` `questions`). Changed in the
skill template, the generated `skills/okf-consume`, and the server
instructions (held under the 2 KB a client keeps). The regeneration recipe for
`skills/okf-consume` gains `--for-bundle`: since v1.1 the generator writes the
generic skill by default, so the recipe as published produced the other file.
A test holds a four-sub-question `okf_ask` over concepts far over the passage
size under 50 000 bytes of reply text (25 000 tokens at a pessimistic two
bytes a token). The real-collection measurements are kept in local state.
Suite on a clean tree after `git add`: 2429 passed, 2 skipped, 4 xfailed.
ruff, ruff format, mypy --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
C3, the remaining half. The proposer names a table block with no heading of
its own after the line it starts on (`Tabell linje <n>`). A position, not a
name: searched, it matched no question, and shown, it told a reader nothing
about what the table is.
`consume.inherit_table_titles` reads such a concept under the title of the
nearest concept ABOVE it in the same source document -- in reading order, the
heading the table stands under -- ordered by `source_offset`, else
`source_lines`, one key per document and never mixed. Applied where the bundle
is loaded, so the ranking, the excerpt and the near misses all see the same
title; the excerpt keeps the file's own as `own_title`, so the name shown is
never mistaken for the one in the file. A table with nothing named above it,
or a document whose concepts do not all carry one position key, is left as it
is.
Chose to do it in the reading and not in the build: no bundle byte moves, no
collection has to be rebuilt, and the proposer's goldens stay pinned.
`consume.MECHANICAL_TITLE` is held against the proposer's own output by a test.
Suite on a clean tree after `git add`: 2418 passed, 2 skipped, 4 xfailed.
ruff, ruff format, mypy --strict clean. The search gate's table for the
previous commit is kept in local state.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
C2. `consume.build_multi_payload` takes two or more questions, reads the
bundle ONCE (`bm25.prepare` builds the index a question does not depend on),
ranks and cuts each sub-question exactly as `build_payload` would alone, and
interleaves the deliveries: first excerpt of each sub-question in turn, then
the second, a concept already taken skipped, cut at the same `k` and `limit`
one question gets -- so asking four times does not buy a payload four times
the size.
Chose round-robin, not a merge by score, because two questions' BM25 totals
are not on one scale: a merge by score would let the wordiest sub-question
take every place. It is the rule the search gate measured with before the
product had it, moved unchanged.
Shape, and only for two or more questions (one question is `build_payload`'s
payload byte for byte):
- `questions` replaces `question`;
- every excerpt carries `subquestions`, the indices of every sub-question
whose own delivery named it, the one whose text (passage) it carries first;
- `coverage` holds one block per sub-question (the single shape plus its
`question`, `unanswered_in_payload` read against what the reader receives),
`weak_subquestions`, and `weak` true only when EVERY sub-question is weak;
- `withheld` is every concept the merge did not deliver: `below_k` where a
sub-question delivered it and the merge's cut did not, otherwise the rule
of the sub-question that ranked it best. `nearest` walks the rankings in the
delivery's turn order. The contract checker accepts it with 0 findings.
`okf consume --question A --question B` and `okf_ask` with `questions` (both
forms at once is `question_ambiguous`) reach it. A reservation or a fusion
widening acts on ONE cut and is refused with several questions
(`subquestions_flag_conflict`).
The search gate's series (e) and (f) now ask ONE call with every
sub-question; the gate's own merge is gone. Sets and thresholds untouched.
The gate's table for this commit is kept in local state.
Suite on a clean tree after `git add`: 2411 passed, 2 skipped, 4 xfailed.
ruff, ruff format, mypy --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
C4. `coverage` gains two keys, and every key it had keeps its bytes:
- `absent_terms`: the question's words the bundle holds in NO form -- not as
written, and not through a relative it uses (`bm25.query_groups`, the same
bridge the ranking reads through);
- `weak`: true when one such word exists or nothing was delivered.
A reading with its rule in the open, never a verdict about the bundle. It is
computed for both rankings (`bm25.absent_terms` serves the fusion). The
retrieval gate's `marked` -- the one reading both gates share -- reads `weak`
beside its own bar, never instead of it; the known-negative that strips the
payload's words now strips both readings.
Words that only FRAME a question are stopwords in both languages (`how
often`, `hvor ofte`, `hva står i`, `what does it say`), and so are the
Norwegian function words spelled without their letters (`naar`, `paa`), the way
ASCII-only text writes them. Read as topic words they would be "absent" from
any collection that never uses them, which is what the synthetic sets showed
on three answered questions before the list was extended.
The working method says what to do with it, in one sentence each: the skill
template's step 3 and the MCP server's instructions (1 253 bytes, under the
2 048 a client keeps) -- rephrase in the bundle's words, and if it stays weak,
say the bundle does not cover the question.
The search gate's table for this commit is kept in local state.
Suite on a clean tree after `git add`: 2397 passed, 2 skipped, 4 xfailed.
ruff, ruff format, mypy --strict clean. Retrieval gate unchanged at the rows
the previous commit left red.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
C1. `okf consume` and MCP's `okf_ask` now rank with BM25 (`bm25.py`) instead
of the three-signal fusion. Two signals, fused by reciprocal rank:
- passage: every body cut into 500-character windows every 250, a concept
scored by its BEST window -- a narrow question is answered in one place;
- field: title three times, the id path and source name twice, then the body
-- a broad question is answered by what a section is called.
The document prior and the rarity weight are gone from the default: the first
favoured big documents full of common words, the second gave its largest
weight to a word the collection does not hold. Under BM25 such a word weighs
exactly zero. A signal that scores a concept zero adds nothing to it, and ties
share a rank, so alphabetical order lifts nothing either.
Three rules carried over from the fusion, each with its own test, because the
suite showed what BM25 alone lost:
- a directory every concept shares is not read (K3-20's defect, one signal on);
- a number a section is known by (`4.2`, `10.2-2`) is kept as one token, or
a question naming a section by its number matches nothing in it;
- a question word the collection does NOT hold is read as the collection's
words it shares a leading word with (`consume.tokens_match`) -- Norwegian
inflection and compounding -- at that word's idf, never at its own.
The lookup and title-covered partitions are shared with the fusion
(`_partitioned`). `ranking="fusion"` / `--ranking fusion` keeps the old order
reachable; `--cost-vocabulary` and `--rarity-weight` widen only the fusion and
are refused with the default (`ranking_flag_conflict`) rather than ignored.
C3. A concept longer than `PASSAGE_CHARS` (4 000) is delivered as the span
around its best window, snapped to whole lines, under the nearest heading
above it, with `[...]` where text was left out. `passage: {start, end, of}`
says so, `text_sha256` covers what was delivered, and `sha256` stays the
file's, so the whole can be fetched by `concept_id`. 4 000 because eight
excerpts of it stay far under a tool response's limit even with several
sub-questions merged, while a 500-character window keeps 3 500 characters of
surroundings. The budget pays for the passage, not the file.
Tests moved with the default, each stated rather than silenced:
- fusion-mechanism tests (cost vocabulary, rarity weight, reservation, shared
rank, the reference-bundle pins) ask for `ranking="fusion"`, the order they
were measured on; the BM25 reading of the reference bundle is a separate
measurement, kept in local state;
- the retrieval gate still measures the shipped default. Row 1 holds. Four of
its premises were built against the fusion (a concept forced below k that
BM25 now delivers, a quota that no longer decides, mutants patching fusion
code) and are `xfail(strict=True)` until the fixtures are re-measured;
- the shipped example payload is regenerated; the shipped skill is unchanged.
README's Consume section and CLAUDE.md state the new default and that the
flags described after it belong to the fusion.
The search gate's table for this commit is kept in local state: the question
sets belong to a consumer whose content does not go on a public mirror.
Suite on a clean tree after `git add`: 2390 passed, 2 skipped, 4 xfailed.
ruff, ruff format, mypy --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operator decision 2026-09-21: nothing from that consumer's collection goes out
on the public remote. The NAME stays where it is already published -- it is a
consumer of this library, named as such, and removing it would mean rewriting
published history, which this repository does not do. What goes is everything
that describes their CONTENT.
Removed across README, CLAUDE.md, CHANGELOG, four dated reports, the
consumption contract, three source modules and three test modules: their
corpus's document and page counts, the concept count of a bundle built from
it, the byte figures of a payload built from it, the question and fasit counts
and recorded score of their evaluation set, a bundle id with two content refs,
an order id naming them, and a path into their repository.
Kept, because the argument survives without the corpus: RATIOS and
percentages. A ratio is the finding -- a withheld list that is 65.5 % of a
payload is a defect at any corpus size -- and it discloses nothing about how
large anyone's collection is. Where a claim lost its denominator it now SAYS
so rather than quietly reading as unmeasured: the gate-refusal limitation in
the README states that the corpus and its counts are deliberately withheld and
points the reader at their own build, which is the number that binds them
anyway.
One integrity pin is kept and named here rather than left to be found: the
retrieval gate still pins that set by sha256, because the pin is what refuses
a self-written file in the right shape, and a checksum discloses nothing about
what it checksums. Its recorded SCORE is gone -- that was their figure about
their own corpus, and the row now says so instead of restating it.
The known-positive constants move with the contract document, as they must.
Suite green, 2372 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
**A subagent inherits its session's MCP tools; it does not inherit its skills.**
So the method A3 put in the skill reaches the main thread and no arm running
below it, and the one place every caller sees is the server's own
`instructions` and its tool descriptions. Both are truncated by Claude Code at
2 KB, and truncation is worse than rejection here -- a reader gets the first
half of a method and no sign the rest existed -- so what travels is the SHORT
form and the long one stays in the skill, which has no such cap. A test holds
it under the limit WITH a control, so the assertion is a measurement and not a
tautology.
`okf_describe` without `bundle_id` now describes every served bundle, where it
refused and `okf_ask` in the same position fanned out. The tool a caller is
told to read FIRST was the one requiring a name it did not have yet, and a
tool that refuses the call its sibling accepts is a shape a client must be
told out of band -- the configuration this server exists to remove. The named
call's shape is byte-unchanged, and so is every one-to-one server's: the
fan-out replaces an ERROR, so no caller's bytes move.
`okf project`'s closing lines and the README's first screen carry the one line
the USER runs to register the server on user scope, verified against Claude
Code's own MCP documentation (`claude mcp add [options] <name> -- <command>`).
Nothing here starts Claude Code, and the line says whose it is. Measured:
a project bundle at `<root>/<project>/.okf/<id>` is depth 3, inside
`MAX_DISCOVERY_DEPTH`, so a `--root` server finds what `okf project` wrote --
a test builds one and discovers it rather than reasoning about the walk.
A5, and it was free: `okf skill` without `--out` now refuses in the same
`refused (<code>)` form as every other refusal in this chain. The exit code
does not move -- 2 was already right, "the run did not happen" -- what was
wrong was that a caller parsing our form got argparse's line on the one flag
everybody forgets.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The operator built a 2313-concept bundle from one project's own documentation,
asked it a question in his own words, and judged the result unusable. The
generated skill was an audit contract: all its discipline sat on the accounting
-- markings, denominators, budget lines, source pointers -- and none of it on
understanding the question, searching again, or writing one coherent answer.
Two sentences actively forbade the second of those.
**The two forbidding sentences are gone and their replacements are tested from
both sides.** "Do not go looking for context the pre-pass deliberately
withheld" read as "one run per question", and no wording of the operator's
question put the right document inside a single run's cut -- so a rule against
a second run was a rule against finding it at all. "Not something to retry with
a narrower question" generalised a budget-refusal case into the same ban.
SS 2.2 of the contract said the first of them, so the contract moved with the
skill rather than being left to disagree with it: a second pre-pass run with
other terms, and a fetch of a concept the payload NAMED, are reachable; SS 9's
two real boundaries -- directory enumeration, the verdict layer -- are not.
**Two new sections, and the checker requires them.** `## Working method`: read
the bundle's map, put the question into the bundle's own words, split a broad
question into 2-4 sub-questions, search per sub-question, read what lay just
outside the cut and search again with its words, same method across several
bundles, then assemble ONE answer ordered by sub-question, saying which source
holds and what is not covered. `## Answer form`: the questioner's language,
plain prose, no `below_k`, no digests, no budget lines, no denominators; short
textbook-style references (document + section, plus bundle where several were
read); and the audit trail written only when the questioner asks for it or
into a document that travels without the skill. `REQUIRED_SECTIONS` follows the
template and the contract's new SS 2.5 and SS 2.6 -- never the other way round.
**The generic skill becomes what `okf skill` and `okf project` write.** A
per-bundle skill's numbers go stale the moment its bundle is rebuilt, one copy
per consuming project, and a project with two bundles installs two
near-identical skills; the generic form carries no bundle's numbers and names
`okf card` for them. `--for-bundle` is the opt-in for the instantiated copy,
which still refuses out loud on a stale pairing -- safe to keep, not enough to
keep default. `rule_bundle_identity` learned to tell a generic skill from an
unfilled template by the frontmatter name the generator writes, so the template
still fails for the opposite reason: it declares no identity because it is
unfinished.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured 2026-09-20 on a 2313-concept bundle of one project's own
documentation: `withheld` held 2 305 entries = 186 440 B of compact JSON =
**65.5 % of the 284 850-byte payload**, and not one of those bytes counted
against the budget the same payload reports (`spent` was 45 192). A reader was
handed 239 658 bytes the budget line did not know about, to learn 2 305 concept
ids with nothing beside them -- the title being exactly what `--withheld-titles`
existed to buy, and which was off because buying it for 2 305 entries cost
another 37.9 %.
`withheld` is now a mapping: `total` (equal to `denominators.withheld`, so
SS 5.2's identity is unmoved and closes on the NUMBERS), `by_rule` (the same
total decomposed over the closed rule set, so "what kind of drop" is answerable
without the list), `nearest` (the best-ranked drops BY NAME, with title and
source document, so a reader who sees a near miss can ask for it) and
`complete`. The near misses are read off the ranking, not off `cut`'s output:
`cut` sorts by id so the partition is comparable, and that order says nothing
about which concept a reader might want next.
Same question, same bundle, after: **52 421 bytes, 18.4 % of the old file**.
The whole list stays reachable behind `--withheld-full`, and the two
instruments that classify EVERY miss by its rule -- the retrieval gate and
`okf_consume_measure` -- now ask for it explicitly and assert `complete`
rather than assuming it. `--withheld-nearest N` sets the cap (default 20,
which is `k` plus the next twelve). `--withheld-titles` is retired: a flag
whose only remaining effect would be to STRIP the title from a list the caller
asked for in full names no decision worth two shapes for one list.
`CONTRACT_REVISION` moves to `okf-consumption/2`, because a consumer indexing
the old key as a list would otherwise break silently. Three checker rules move
with it, and one of them is the interesting case: `parent_unfollowable` used
`excerpts` + `withheld` as the bundle's own denominator, which a truncated
block is not -- so that clause now runs only where the payload SAYS it is
complete, stated in SS 8.6 rather than left as a silence, with the other two
clauses (shape, self-reference) running either way. `Report` carries both
denominators, because a report claiming it examined 2 305 entries it never saw
is the same defect one level up.
The generated skill's "breaking point" section goes with it: it extrapolated a
concept count from the cost of ONE withheld entry, and there is no such slope
any more. It now states what this bundle's bookkeeping cost and that the block
is bounded by the cap rather than by the bundle -- an extrapolation from a
slope the code no longer has would be a measurement of the previous revision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`project.create` called `build()` with five keyword arguments and no `gate=`,
so `okf project` screened by the package default and nothing anywhere said
that was a choice rather than the only option. The gate is not a segmentation
rule -- it is a screen about whether a document may be persisted at all -- so
it is the one flag this command owns that may move a bundle's bytes, and the
module docstring, the README paragraph and the test file's own claim are
corrected rather than left standing beside the new flag.
The default is `okf build`'s default, so an unflagged `okf project` is the
bytes it always was; the byte-equality invariant against `okf build` holds
unchanged. An unknown gate name still raises `gate_invalid` rather than
falling back -- a fallback reproduces the defect with an extra step.
Tests first, all three red: the gate name read back out of the bundle's own
`log.md` (not out of the call), the flag parsed by this command's parser, and
the unknown name refused.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured 2026-09-20 on an official documentation corpus of 594 sources built
with the shipped default gate `guard-trusted-source`: 17 sources were refused
OUTRIGHT -- `fail_secure` 3, `quarantine_review` 14 -- and 16 of them were
among 197 official documentation pages, the pages on hooks, skills,
permissions, errors, env-vars and authentication among them. The summary said
only `fail_secure`: 3/594. Three of the four facts a reader needs were missing:
the COUNT of documents the gate dropped (the existing `rejected (coded)` line
sums gate refusals and extraction failures, two failures with two different
remedies), the NAMES, and the way out. Rebuilt with `--gate none`, all 17 went
through untouched, so the refusal is the gate and not the readers.
`okf build` now prints a `Documents the gate refused WHOLE` section directly
under the denominator, carrying all four: the count with its denominator, the
names capped at ten with the rest in the bundle's `log.md`, the codes, and
`--gate none` for a source you vouch for yourself. The same fact goes to stderr
in one line, built from the same field, because `okf build > report.txt` is an
ordinary thing to do. `log.md` gains one bullet naming every refused document,
uncapped.
The exit code deliberately does not move. The build is valid -- every refusal
is coded, the conservation identity holds, and the bundle is a true record of
what the gate allowed. What was wrong was the silence.
A run the gate refused nothing from is byte-identical in both places, which is
the known-negative in the new suite: no bundle this repository ships was built
with a gate refusal, so this cannot have moved a byte measured here.
Also, and measuring nothing new:
- README gains `Known limitations` high up -- the gate's refusals and the way
out, the absent ceiling on what one run pays for images (a 70 KB PDF with 16
images under the declared limit reached 851 MB peak RSS; RLIMIT_AS is not
enforceable on this platform, so the 512 MiB per-link budget is the whole
bound), the three gates of this repository that are RED today (retrieval
5/7/8/9, MCP 2, accounting 2/3/6 -- all three re-run on this commit), what
the content accounting does not count, and the rough edges nothing is
planned for.
- The two `pip install` lines under "Install in detail" install `[extract]`.
The first screen does; those two did not, so the two recipes produced
different installations and the detailed one reports `resolved converter
path: unresolved (extractor_extra_missing)`.
- Version `1.0.0`, synced across pyproject, `__version__`, `uv.lock`, the four
README install lines, the install prose, the current-tag entry and the
CHANGELOG, where the two "after the 0.10.1 notes were written, untagged"
sections are folded in. It adds no capability over `v0.10.1`; what it adds
is that the tool says what it does not do.
Suite: 2325 passed, 2 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four claims on the front page were false on this commit, and one of them was a
number no division ever produced.
**The retrieval gate.** README reported it RED on rows 3, 4, 5, 7, 8 and 9,
with row 3 at 2 of 5 and row 4 at 3 of 6. Run on this commit it is RED on rows
5, 7, 8 and 9, with row 3 at 5 of 5 and row 4 at 6 of 6: `f81683e` made a
withheld concept carry the rule that actually decided it, and `05cb190` gave
the payload a `coverage` block, and neither updated the table. Row 8 is `0 of 3
| NOT RUN` on the default run and was published as `44 of 64 questions`, which
is what it scores the day all three private sets are handed to it -- now
labelled with the day and the machine rather than printed as a row. The same
four figures were stale in `CLAUDE.md`.
**The breaking point in a generated skill.** `int(LIMIT / per_withheld) if
per_withheld else 0` printed `At roughly 0 concepts the bookkeeping alone
reaches the 120000-byte limit` whenever the generation run withheld nothing --
the absence of a measurement, rendered as one, and read as a bundle that breaks
before it holds anything. A run with no withheld entry has no slope to
extrapolate from, so the sentence is withheld with its reason. The shipped
`skills/okf-consume/SKILL.md` is generated with the question its
`references/README.md` names, withholds nothing, and carried exactly that `0`;
it is regenerated. Two arms in the test, because one would pass on an empty
set: the bundles that withhold something must still state a positive figure.
The sentence for that arm also stopped saying `**4 bytes** for 3 concepts`
where the 4 bytes were the cost of 0 withheld entries. It is now `for N of M
concepts`, which moves two generated skills' line counts and therefore the
published comparison: 280 of 312 and 310 -> 281 of 313 and 311, re-measured,
with the 62 differing lines unchanged.
**Four tools.** A single-bundle server exposes three: `okf_list` is absent
where there is nothing to list. README's table already said so in a cell; the
heading and the CHANGELOG did not.
**What `--accounting` accounts for.** The account is over the element classes
each format's vocabulary names, verified against `accounting._READERS` rather
than against the report: a file whose suffix has no reader is accounted at file
level only, `.docx` reads `document.xml` and `footnotes.xml` (so headers,
footers, endnotes and comments are outside), `.pptx` reads the slides (so
speaker notes are outside), `.xlsx` reads the worksheets (so cell comments are
outside and a cell contributes its cached value, never its formula), and `.rtf`
skips its header and footer groups. A hidden slide or sheet IS counted -- it
lives in the same part as a visible one. Nothing is built for this; the list is
what `0 unaccounted` does not claim.
Gates re-run on the commit: retrieval `GATE RED: rows 5, 7, 8, 9` (exit 1),
MCP `GATE RED: rows 2` (exit 1), both matching what is now written.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`okf_fetch` resolved a concept through `connectors.safe_resolve` from the day
the server was written. The other three ways into the same bytes did not.
`okf consume` and `okf_ask` reach `consume.build_payload`, `okf_describe`
reaches `mcp_server.card`, and both built the concept path by joining the
index's own name onto the bundle root. `consume._join` refuses a `..` segment
and an absolute target, but it is a STRING rule over the index text, and a
symlink is a fact about the filesystem that reading that text cannot see: the
index could name `lekkasje.md`, that name could be a link to a file outside the
bundle, and the file came back in the answer.
Measured before the fix, on a bundle carrying one honest concept and one
escaping link: 8 of 11 new rows red, the 3 green ones being `okf_fetch` on the
same two links and the known-positive that the clean bundle still answers. So
the suite was not red for an unrelated reason, and the fix is not "refuse every
bundle holding a link".
One place, not three copies: `consume.resolve_in_bundle` makes the check and
`consume.read_path_in_bundle` adds the file's presence. Every reader here goes
through them -- the index walk, the ref, the document prior, the payload, the
card, `okf_fetch`, and the three outside `consume` (`skill`, `quality`,
`project`) that joined the same way.
Two more failure modes in the same check, because they are the same question:
* A NAMED PIPE is not a regular file. `read_text` on one blocks for as long as
nobody writes to it, which on a server is the whole process; the red row for
it ran 60 s to a subprocess deadline and now returns in under a second.
* A DEAD INDEX LINK raised `FileNotFoundError`, and the broad handler in
`handle` wrote `{error}` into the refusal -- the SERVER's absolute path,
handed to whoever asked, over one index entry naming a file nobody wrote.
It is `concept_unreadable` now, naming the concept and not the machine.
The returned path is the JOINED one, never the resolved one: `read_concept`
derives a concept id by taking the read path relative to the bundle root, and
once containment holds the two are the same bytes.
2334 passed, 2 skipped (was 2323 + 2). `mypy --strict src/` clean over 25 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The eval was written RED at `5f1772e` with no server in the tree. This is the
capability it was written against.
`okf mcp --bundle <dir>` serves exactly one bundle, whose tools take no bundle
argument. `okf mcp --root <dir>` (repeatable) serves every bundle under the
roots and knows NONE of them by name. Four tools -- `okf_list`,
`okf_describe`, `okf_ask`, `okf_fetch` -- each carrying its reason in the
description a client actually reads.
Gate today: 1 (7/7) - 2 (83/181) - 3 (4/4) - 4 (9/9) - 5 (3/3) - 6 (6/6),
`GATE RED: rows 2`, exit 1.
THE PROTOCOL IS STDLIB, AND THAT IS THE PACKAGING INVARIANT KEPT RATHER THAN
A TASTE. An MCP SDK would be this package's second runtime dependency on the
DEFAULT install path, for four JSON-RPC methods and a newline framing, and
`test_the_only_runtime_dependency_is_the_security_boundary` pins that list
literally. Chosen hand-written because the surface needed is `initialize`,
`notifications/initialized`, `tools/list` and `tools/call`; `uv.lock` is
untouched.
NOTHING IS CACHED ACROSS CALLS, and row 3 is why. Every call re-walks the
roots and recomputes `bundle_ref`, so a bundle added, removed or rebuilt while
the process runs is seen by the next call with no restart, no configuration
edit and no code change -- 9 of 9 discovery checks over three bundles written
while the server was serving. The cost is paid per call and is published
rather than hidden: 0.75 s for the identity of a 2 756-concept bundle, 5.6 s
for one ask, 4 min 13 s for row 2's full run over four bundles.
CONTAINMENT IS TWO INDEPENDENT CHECKS: the bundle's own index must name the
concept, AND `connectors.safe_resolve` must place it inside the bundle. A
mutant removing either one alone still refuses -- with a DIFFERENT code, which
row 6 asserts by name -- and one removing both is killed. Row 6 declares a
code set per case because its first run had the 10 MB concept refused as
`concept_unknown`: the fixture had not named the file in the index, so the
size ceiling never ran and the row was green for a reason unrelated to the
attack.
`okf card <bundle>` and `okf skill --generic` are the one-to-many skill
candidate. The card is DERIVED on every run and never written into the bundle:
storing it would move the bytes of all six `examples/*/expected-bundle` trees
(23 files compared byte-for-byte) and of the pinned reference bundle, to keep
something recomputable in under a second, and a stored card is one more
artefact that can disagree with what is beside it. Measured here rather than
taken from the order: two per-bundle skills are identical on 280 of 312 and
310 lines; the 62 that differ are identity, concept count, the
conditional-field table, the whole-bundle cost and the breaking point. The
generic skill carries none of them, and `render_generic()` takes no argument,
so there is no bundle it could have read.
Row 2 decomposes into three numbers and the middle one is the finding: 99 of
181 (bundle, anchor) pairs are present in the bundles at all, 83 of those 99
were reached, and 0 of 83 were met by `okf_fetch` on the anchor as a concept
id. The set's anchors and this library's concept ids are different
vocabularies, so every pair met was met through the ranker -- 83 is a FLOOR on
the ceiling, never the ceiling.
13 mutants in a scratch copy, never in the working tree: 12 killed, 1 survived
with its mechanism printed, 0 errors, control green first. Suite 2323 passed,
2 skipped. The architecture choice between the two shapes is the OPERATOR's;
these rows are its input. Report: docs/2026-09-20-mcp-to-varianter.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`coverage` carries three lists: the terms the pre-pass read the question as,
the terms no concept in the bundle answers, and the terms no delivered
excerpt answers. Without it a reader holding eight excerpts cannot tell a
bundle that ANSWERED its question from one that merely ranked something --
the two payloads have the same shape.
FACTS, AND NO VERDICT, which is a measurement and not caution. Two readings
were built and both falsified over 81 questions (16 synthetic, 65 across the
three real sets, 2026-09-20): the share of question terms a delivered
excerpt answers separates the synthetic controls at 0.33 against 0.50 and
REVERSES on real data (covered questions down to 0.27, one genuinely
uncovered question at 0.71); the share of a bundle tying the best lexical
match is ~0.00 for every real question either way. Question style dominates
the first, corpus size the second.
The one bar this repository declares is the gate's: `UNANSWERED_BAR = 2/3`
over `unanswered_in_bundle`, swept and collapsing at both ends -- at 0.50
eleven real covered questions are marked, at 0.70 the row falls to 5 of 6,
at 2/3 the row is 6 of 6 and 0 of 65 real questions are marked. The margin
is thin (0.6087 against 0.6667) and is published that way, together with
what it does not catch: r761-sk2's own known-negative sits at 0.2857.
Row 4: 3 of 6 RED -> 6 of 6 GREEN, with the 10 answered synthetic questions
held unmarked as the known-negative. The contract's SS 8 gains point 7, the
consumption skill is told to read the block, and the SS 7.4 known-positive
moves with the document (14 721/375 -> 16 389/417). Suite 2292 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The quota filters the WHOLE candidate list, so every over-quota candidate
came back `source_quota_exceeded` -- including the ones the RANK had
already put outside k, which the quota only reached because it ran first.
`_fates_without_quota` asks the same cut what would have become of each
candidate with no quota, and the drop keeps THAT rule; only a candidate the
quota-off cut would have delivered is named as the quota's. The packer is
lifted into `_pack` and used by both, so the quota-off fate is decided by
the code the run itself uses and never by a second implementation.
The retrieval gate's row 3 goes 2 of 5 RED to 5 of 5 GREEN. Row 7 is
unchanged at 12 of 14; M01 and M02 lose their row-3 credit, which was the
lying label moving and not the ranking. Suite: 2288 passed, 0 failed; no
committed payload moves a byte.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Green: 23 of 23 in the suite that was 3 red, and the sweep's carried-here-
refused-there count goes 703 of 22 620 to 0, the drawn-differently count
1 492 to 32.
Two clauses, both refusing with the code the path already uses:
* an end-of-line escape at column 0. It closes no row, so the row it moves
the cursor over is a row the stream never wrote -- PM's four-EOL frame,
carried with 32 of 32 pixels never decoded while Pillow refuses the file.
* a delta whose horizontal offset would leave the row. The format puts that
offset inside the line; this reader keeps the cursor past the row end and a
flat decoder rolls it into the next row, which is two pictures from one
stream.
CHOSEN OVER PM's RECOMMENDATION, and the recommendation was measured first:
refusing only a stream that painted nothing leaves 512 streams carried here
and refused there, and 1 171 drawn differently. It would have narrowed this
class for the third round running instead of closing it.
WHAT IS NOT CLOSED IS STATED, in the docstring and in the test: 32 of 22 620
streams are still drawn differently, every one of them a run or absolute
block that overruns its row. Refusing those gives 0 and 0 -- and costs 15 of
the 25 real RLE8 files, which would drop 15 figures and move a pinned
bundle's bytes. Measured on the corpus first: over 11 441 files scanned
across the four raw deliveries and the K2 reference corpus, the 25 BMPs on
this machine use an end-of-line at column 0 in 0 of 25 and a delta in 0 of
25, and 25 of 25 still decode to Pillow's pixels exactly (3 117 220 pixels
compared, after the change).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Chose the CURSOR over a pixel-coverage count because the corpus cannot
choose -- 25 of 25 of the R761 delivery's RLE8 BMPs paint every pixel,
25 of 25 reach the end of the frame, 0 of 25 use a delta -- and an
independent decoder can: a delta and an end-of-line escape state their
skip, so every decoder agrees on the index-0 pixels they pass over, while
a pixel count would refuse both constructions the format defines.
`_bmp_rle8_rows` now refuses (`asset_samples_invalid`) when the terminator
arrives with the cursor short of the last row. Pillow reads 5 of the 8
streams in the table and refuses the same 3, one of them short by a
single pixel.
Both docstrings the round was sent to correct are rewritten: the test no
longer claims every pixel is decoded (it is not -- a stated skip keeps
index 0), and `_bmp_rle8_rows` no longer frames the delta argument as
read off the corpus, which it never was.
R761 rebuilt: bundle `diff -r`-identical to the build before this commit,
50 assets (29 JPEG + 21 PNG), 19 of 19 conversions, SHY 71, u = 0, d = 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Chose the side channel over neutralising pointer-shaped document text,
because the second fix changes what every document SAYS in order to
defend a tool outside the build: a source quoting a bundle listing would
come out altered and existing bundles would move bytes. This reads a
file the run already writes.
`assets.conversion` names the pair, `DocumentAssets.conversions` carries
it out of the run, `DocumentAccount.conversions` books it, and the
accounting JSON states it per document. `_declared_conversions` reads it;
`_conversions` now believes a pair only when the RUN booked it AND a
pointer block confirms it for the asset it names. The confirmation can be
forged and the ledger cannot, which is why the ledger decides.
Measured through the real `okf build`: the three arms PM reproduced
(two `<p>`, one `<p>` with `<br>`, a markdown note beside the carrier)
go forged -> refused, 3 of 3, with the known-positive True in all three.
The text-level regression guard goes 3 arms to 13, the two new ones being
a perfectly written pointer block the run never booked.
R761, rebuilt: 25 BMP sources, 19 held, 19 of 19 conversions confirmed
against 19 declared, 50 assets (29 JPEG + 21 PNG, 0 BMP), SHY 71,
u = 0, d = 0, exit 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
160 passed over the three affected files. Two guards this round introduced are
removed, each because it was measured and found unkillable, and one real cut is
added in their place.
The end-of-line anchor on the judge's clause expression: with the pointer-block
route and the asset-name tie already in place, the only case it covered is a
label forging a clause inside a block that also carries a real one -- which
`assets._inline` now makes unreachable from a document. Removed with `$` gone
the whole suite stays green, which is the measurement: a guard nothing can fell
is not a guard, and it would also have made the mutant below unfaithful.
The length check on a short absolute run in `_bmp_rle8_rows`: unobservable by
construction. A run shorter than its declared count means the stream ran out,
so the loop cannot reach an end-of-bitmap afterwards and the picture is refused
either way -- the partial paint is discarded with the frame.
A FIFTH CUT REPLACES THEM, and it makes an existing bound load-bearing: a
stream cut in the middle of a delta escape, 17 of 32 pixels wrong by this
file's own decoder. Without the two-byte bound in that branch the cut raises
IndexError instead of a coded refusal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both red guards green, and the whole suite is 2204 passed / 1 skipped (2199
before this round, +5 new tests, no golden moved).
TWO HALVES, AND NEITHER IS SUFFICIENT ALONE. The judge now reads the clause
only from inside a POINTER BLOCK -- the markdown image line plus the detail
line under it -- and only where the clause names the asset that block points
at, anchored to the end of the line because the build writes it last. That
closes ordinary body text and a table cell. It cannot close an image's own
alt text, because a label is document text that the build writes INSIDE a
pointer block, which is the second half: `assets._inline` disarms a checksum
field in anything that came from the document.
WHERE THE BOUNDARY RUNS, stated in both files. Everything `_inline` returns
came from the document -- an alt attribute, an STS caption, a publisher's file
name. Everything `render_block` appends after it came from the run: the size
it measured, the type it sniffed, the digests it computed. The second line
carries both, so document text may not emit the grammar the run writes there.
The digits are kept, because a reader is owed what the document said; the
colon that makes them a FIELD is not.
The judge's expression stays restated rather than imported, for the reason
`asset_holds` already gives about the naming rule: a judge sharing the
judged's own expression agrees with it by construction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The four cuts now raise `asset_samples_invalid`, the code the UNCOMPRESSED
path already raises for a body that stops short. No pixel is guessed: the loop
may end at an explicit end-of-bitmap escape and nowhere else, and running out
of bytes -- mid encoded run, mid absolute block, or one opcode before the
terminator -- is a refusal with a row instead of a partly blank PNG.
WHY THE TERMINATOR AND NOT `biSizeImage` OR A COVERAGE COUNT. `biSizeImage` is
a claim by the same untrusted header, and a coverage count would refuse the
delta escape the format defines. Read off the corpus instead: over the 19 real
RLE8 assets of the frozen R761 delivery, 19 of 19 end at an explicit
end-of-bitmap, on 19 of 19 it is the stream's LAST two bytes, and on 19 of 19
`biSizeImage` equals the available bytes. A whole stream that omits the
terminator is refused alongside a cut one, because from a reader's side they
are the same bytes and the measurement says no real writer here produces the
first.
An absolute run shorter than its own declared count also stops the loop rather
than painting what arrived, so the refusal names the cut and not the frame.
KNOWN-POSITIVE, re-measured on the frozen delivery with stdlib on BOTH sides
(an independent BMP reader and an independent `zlib` + filter-reversal PNG
decoder, no Pillow anywhere): 19 of 19 still convert losslessly, 2 366 365
pixels compared, 0 refused, and a one-byte control confirms the comparison can
fail. 92 passed over the three asset test files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Chose the stated provenance over a second, independent counter, because a
second counter would be a number built to disagree. The door acts on the
EXTRACTED text; a counter inside `accounting.py` would have to re-read the
SOURCE bytes, and for every type extraction does not carry verbatim (pdf,
xml, html, docx) those are counts over two different strings. The gate would
then have to decide which difference is a loss and which is extraction doing
its job -- inventing a discrepancy where there is none. `accounting.py` does
not hold the source bytes at that point either; being a second reader of a
document is what `tools/okf_witness.py` exists for, outside the package.
Where an independent denominator DOES exist the tests already pin it from
outside the door: the fixture writes its own five, and R761's 71 is PM's
count over the delivery.
The `**Normalisation**` bullet now ends: "The count is the door's own, read
off the run and not recounted from the source."
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Chosen: a stdlib BMP reader, because `read_image` is on the CORE path and an
asset's name is its content digest. Measured first, as the order requires:
Pillow 12.3.0 IS in this tree (transitively under `pdfplumber`) and it DOES
decode RLE8 correctly -- a hand-written stdlib decoder and Pillow agree on
19 of 19 of R761's real files, RGB per pixel. So the choice does not rest on
capability. It rests on two properties of this package: `.html` and `.xml`
carry images with no `[extract]` extra installed, so a Pillow converter
either makes a core path depend on an optional binary wheel or buys the
second runtime dependency; and encoding through an installed library would
make a bundle's identity move with that library's version, which is the
property 0.10.0 felled page rasterisation over and `encode_png`'s docstring
already defends. Pillow keeps the job it is good for: the INDEPENDENT decoder
in the tests, on neither side of the conversion.
The defect, measured over the frozen R761 delivery's `assets/`, denominator
50: 29 JPEG, 2 PNG and 19 RLE8 BMP. The 19 are byte-correct files nothing
reads, so 19 figures were present and invisible while `images: N` reported
that they had arrived.
- `VIEWABLE_MEDIA_TYPES` is tested against every asset's SNIFFED type, so it
is a property and not a list of formats we met. WebP is on it and `sniff`
does not recognise one; the limit is stated, not implied.
- `bmp_to_png`: 8-bit uncompressed, 8-bit RLE8, 24-bit uncompressed. All five
RLE8 opcodes. 19 of 19 real files convert with RGB identical to Pillow's
decoding of the source, 2 366 365 pixels compared.
- `asset_not_viewable` and `asset_bmp_unsupported`, both published, both
leaving the concept's "not carried" line.
- Traceability on the pointer's second line, where the rest of the asset
metadata already lives: original media type, original sha256 in full, new
sha256 in full. A converted asset is ONE asset.
- The ceiling is paid on the DECLARATION before a row is allocated, and an
RLE run is one clipped slice -- painting pixel by pixel leaves the memory
bounded and the CPU unbounded.
Two repairs the change forced, each measured rather than assumed:
- `tests/test_assets.py`'s "dimensions absent is absent" used a TIFF, which
is now refused before `read_image` returns. The property still has a
reachable case -- a JPEG whose frame header never arrives -- and uses it.
- `asset_holds` in the accounting gate proved a carry by hashing the SOURCE
file, which a converted image's bundle cannot satisfy. It now also reads
the two digests the bundle states and HASHES THE ASSET ITSELF, so a bundle
claiming a conversion it did not perform still fails.
`tools/okf_asset_census.py` is the committed instrument for the
known-positive: one row per image, from two pinned trees. It was caught by
the rule it serves -- its first version handed `_pdf_images` the wrong page
object and reported 0 images over 67 PDFs with exit 0. The attribute is
asserted now and a known-positive runs before the sweep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operator decision 2026-09-18, and the whole of it: okf removes the soft
hyphen, the guard is untouched, and the number is BOOKED.
`normalise_extracted(text) -> (text, removed)` in `extract`, applied
once at the end of `extract_document`, so both entry points and both
sides of a plan's `text_sha256` see one string. `_pdf_units` measures
its page offsets through the same door, because that table is
CHARACTER offsets rebuilt from the pages while the text it indexes
comes back normalised -- two readings of one document, and a table
built against the other names the wrong page with full confidence.
Removing a character removes no newline, so every line-indexed rule
downstream is unmoved by construction.
`ExtractedDocument.soft_hyphens` carries the count out;
`InboxResult.normalised` carries it per document; `account_run` reads
it off the RUN rather than recounting the source, because a second
count would be a second reader. It reaches the accounting JSON as
`normalised_soft_hyphen` at both levels and `log.md` as a
`**Normalisation**` bullet naming the count, the documents touched and
that no other character is moved.
EXPOSURE, measured and not assumed: 0 of the 78 readable documents of
the K2 reference corpus carry U+00AD or any of the four real
zero-width characters (the 8 unreadable ones raise before extraction),
0 in the pinned K2 bundle's concept bodies, and 0 across
`tests/fixtures`, `examples`, `skills`, `docs`, `src`, README and
CHANGELOG. The door cannot have moved a byte anyone here has measured.
Suite 2179 passed, 1 skipped; ruff and `mypy --strict src/` clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gate gains a FIFTH column, `Unit.refused`, and it is the only one
that is not a defect in the report: the elements of a document the
build read and persisted nothing of. Their fate is declared honestly,
so `unaccounted` and `double_booked` both stay 0 -- which is exactly
why nothing else could see the loss. `refused_whole` asks its question
only for a corpus that persisted NOTHING, so one refused source beside
an accepted one, the ordinary case on a heterogeneous corpus, reached
row 3 as clean.
Row 3's reason now carries `N element(s) lost with R of D document(s)
refused whole`, and each unclean unit's detail line carries
`refused=` beside u, d, unverified and invalid, with the document's
own rejection code in the note.
On the build side `Accounting.refused` is written into the JSON and
into the `**Accounting**` bullet of `log.md` as `R of D document(s)
refused whole`. The exit code is NOT moved: it belongs to the whole
run, and a corpus holding one unreadable file among many is ordinary,
so the order's other half -- state it in the accounting -- is the one
taken. `okf build` still exits 1 when it persisted nothing at all.
`test_a_corpus_refused_whole_under_the_default_gate_is_red` kept its
point and lost its premise: the numbers still balance, and that is now
asserted as u = 0 and d = 0 rather than as a clean unit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Round 3 of the 0.10.1 review, and the finding is the pattern the three rounds
share: each bound an OUTPUT, and the bomb stepped one link along. The
declared size, then the first `FlateDecode`, then every `FlateDecode` -- and
then a link this package had documented as safe.
`ASCII85Decode` was classed as bounded "by its own input because it shrinks".
It quadruples: `z` is the shorthand for four zero bytes. And the output was
never the cost -- `base64.a85decode` appends one 4-byte object per group to a
list, about a hundred bytes of memory per byte of INPUT (101.4x at 1 MiB,
96.1x at 4 MiB, 94.5x at 16 MiB on CPython 3.14).
Paired subprocesses, idle machine, both sides from PINNED trees, the document
built once by a third process and read from a file because `ru_maxrss` never
falls and `b"z" * 64 MiB` alone costs 171 MB:
[/Fl /A85] z x 32 Mi 33 475 B CARRIED 3 261 599 744 -> too_large 42 070 016
[/Fl /A85] z x 64 Mi 66 090 B CARRIED 6 461 558 784 -> too_large 40 280 064
[/A85] z x 8 Mi 8.4 MB CARRIED 933 085 184 -> too_large 62 484 480
[/Fl /A85 /Fl] z x 32 Mi 33 488 B samples_invalid 3 519 180 800 -> too_large 43 438 080
The picture was CARRIED in three of the four: not a bound that fired late, no
bound at all. Doubling the `z` run trebles the old cost and leaves the new one
where it was.
WHY THIS FORM. `assets.MAX_FILTER_DECODE_BYTES` (512 MiB) is what decoding ONE
link may cost -- a separate number from `MAX_IMAGE_BYTES`, because that one
bounds the picture and this one bounds producing it. `FlateDecode` is measured
as it is paid; every other permitted filter carries a MEASURED cost ratio
(`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its decoder
is called, since those decoders take a whole string and return a whole string.
A filter with no ratio is refused unread. The budget TRAVELS: a deflate link
is inflated under the smaller of the picture's bound and what the next link's
decoder may be handed, or `[/Fl /A85]` pays 256 MiB for a refusal.
A chunked ASCII85 decoder written here was the alternative and was FELLED: it
would bound `_check_stream_cost` and not the run, because `stream.get_data()`
decodes the whole chain again with pdfminer's own decoder, and it would make
this package rather than pdfminer the authority on an image's bytes. The cap
is the only number that bounds that. `resource.setrlimit(RLIMIT_AS)` was
MEASURED before anything was built on it, as the order required, and is not
usable: Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit`
and does not enforce it. No child-process cap exists.
THE CAP IS READ OFF THE CORPORA, the posture `MAX_IMAGE_PIXELS` has: over the
9 668 image objects of the 77 PDFs on this machine, 16 decode through an
ASCII85 link and the largest input to one is 450 739 bytes, against a cap of
about 5.0 MB.
A PROPERTY TEST REPLACES THE LIST OF KNOWN SHAPES: every chain of length 1-3
over the ten filters pdfminer decodes, 1 110 of 1 110, both payload fills,
each delivered under the bound or refused with a published code and never paid
for on the way (`tracemalloc`, which counts allocations and is not disturbed
by load). Known-positive beside it: 258 of 258 chains over the permitted
filters still carry a small image.
MAJOR -- the backstop had no test. `check_payload` at the end of
`_check_stream_cost` could be deleted with the whole suite green, because the
second one after `get_data()` gives the same code one step later. The two
differ in whether the payment was made, so the test asserts `get_data` was
never called.
10 OF 10 MUTANTS KILLED, control green, each killer named in the report. Four
survived a first pass and two tests exist because of it.
NOT ONE PICTURE CHANGES HANDS, MEASURED BY NAME: `_pdf_images` over every PDF
on this machine from both pinned trees -- 9 306 -> 9 306 carried over 77
files, 50 -> 50 on R761, 0 of 78 files moving a count and 0 moving a code.
R761 also settles a question raised while this order was open: 50 objects, 29
[/DCTDecode], 21 [/FlateDecode], 0 ASCII85 links -- so round 2's count of 580
`[/FlateDecode /ASCII85Decode]` objects is reproducible from nothing on this
machine. It changes no decision; a bomb shape does not need a corpus.
Version stays 0.10.1, no tag. README, CHANGELOG, CLAUDE.md and errors.py
corrected TO what the code does; the round-2 report carries a correction block
rather than a rewrite. Report:
docs/2026-09-18-utgangsbudsjett-per-ledd.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two findings of the 18.09 PM checkpoint of `0f308c1`. Red tests landed
first in `3b587ea`; this is what turns them green.
BLOCKER -- `_check_inflated` read `filters[0]`, measured that one link and
returned, which is not a bound: a PDF decodes a stream through a LIST of
filters. Measured in paired subprocesses from two pinned trees, idle machine:
[/FlateDecode] 400 MB 408 516 B 59 232 256 -> 62 017 536 B
[/FlateDecode x2] 400 MB 1 636 B 886 554 624 -> 52 367 360 B
[/FlateDecode x3] 400 MB 1 070 B 889 393 152 -> 61 390 848 B
[/FlateDecode x2] 1,2 GB 2 927 B 2 567 204 864 -> 60 403 712 B
542 000x the file at two links, and the picture WAS refused at the end -- by
`check_payload` after `get_data()`, once the memory was spent. The single-link
row is the control and does not move. It also left the 16 corpus objects behind
an `[/ASCII85Decode /FlateDecode]` chain unmeasured, since `filters[0]` is not
`FlateDecode` there.
`_check_stream_cost` walks every link. THREE CLASSES and no fourth
(`extract.bounded_pdf_filters`, pinned by a test): `FlateDecode` MEASURED, a
link with another expanding link behind it inflated under the same bound and
handed on; `ASCII85Decode`/`ASCIIHexDecode` bounded by their own input because
they SHRINK; `DCTDecode`/`JPXDecode`/`JBIG2Decode` PASS THROUGH. Everything
else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything
written later -- is refused UNREAD with a new code `asset_pdf_unbounded`,
decided before the FIRST link is decoded so a document cannot make the run pay
for the links in front of the one we cannot bound. An encrypted stream is
deciphered and then measured, where `stream.decipher is not None` used to
return unmeasured; 0 of 5 142 objects here are in an encrypted document, which
is why nothing caught it.
NOT ONE PICTURE CHANGES HANDS, AND IT IS MEASURED BY NAME. Every PDF on this
machine -- 78 documents, K2 in both trinn1 and trinn2, the shipped fixtures and
R761 -- run through `_pdf_images` page by page from both pinned trees:
images carried 9 356 -> 9 356
documents losing one 0 of 78
documents gaining one 0 of 78
asset_pdf_unsupported 322 -> 314
asset_pdf_unbounded 0 -> 8
The 8 are the 4 `CCITTFaxDecode` stencil masks (`/ImageMask true`,
`/BitsPerComponent 1`), counted twice because trinn1 and trinn2 hold the same
document. They were refused before and are refused now, one step earlier and
under a code that says why.
MAJOR -- `check_payload(len(data))` after `get_data()` is the counted refusal
four documentation surfaces point at, and deleting exactly that line passed all
2 132 tests. It is reachable through a stream pdfminer has ALREADY decoded
(`decode()` sets `rawdata` to `None`), which is now the ONLY case outside the
bound and has a test.
Eight mutations, one line each, every one DEAD, with the unmutated tree run
first as the control: first-link-only, loop dropped, inequality reversed,
encrypted skipped, backstop deleted, unknown filter passed through,
intermediate link not carried forward, whole check removed.
`tools/okf_accounting_gate.py` gains one line, the new code in
`REJECTION_CODES` -- what a rejection code requires and nothing more. Gate
unchanged: exit 1, GATE RED rows 2, 3, 6. Version stays 0.10.1, untagged.
Suite after `git add` against a clean tree: `uv run pytest -q` ->
2152 passed, 1 skipped (226 s). ruff, ruff format --check, mypy --strict clean.
Report: docs/2026-09-18-filterkjeden-og-backstoppen.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A second independent review read `230d1cb` -- the commit that closed the
`v0.10.0` review's two MAJOR findings -- and found one of them open. The
bound read `/Width` and `/Height`, which an untrusted document writes, while
`get_data()` pays for the stream beside them; `/Length` is the COMPRESSED
length and the two numbers are independent.
Re-measured here on `ed8d9d7` before anything changed, in its own
interpreter: a 408 516-byte PDF declaring 1x1 and carrying 400 MB of deflated
zeros was CARRIED, no rejection, 891 904 000 B peak RSS. After: 0 carried,
`asset_too_large`, 57 065 472 B. At 1,2 GB of zeros, 2 436 MB -> 64 569 344 B
-- the cost no longer scales with the bomb. End to end through the CLI with
the shipped defaults: 838 000 640 B and an asset written -> exit 0,
79 650 816 B, `0 carried of 1 found`, no `assets/`.
Three numbers are bounded now, not one: what a container DECLARES, what a
carried FILE measures (`read_image`, so a 49 MP PNG of 47 705 bytes is not
passed on to a consumer), and what a PDF stream DECOMPRESSES to
(`assets.inflated_size`, a chunk at a time, output discarded, before
`get_data()`). The limit is stated rather than implied: the stream
measurement runs where `FlateDecode` is the first filter and the document is
not encrypted; every other chain is a check on the decoded length AFTER the
decode, a counted refusal and not a bounded one.
A non-positive declared dimension is `asset_size_invalid`, its own code,
raised before the stream is read. `-1 x 40000000000` is a NEGATIVE pixel
count, under which every `>` bound read as satisfied, so the check returned
silently and the refusal arrived from `encode_png` as
`asset_samples_invalid`. Its own code because a publisher shipping a picture
bigger than this package carries and a dictionary written to be read wrong
are different facts about a document.
Two smaller findings in the line that says what is missing, both introduced
by the first fix: the address was written twice, once bare, and a linkifying
renderer autolinks a bare URL -- written once now, in one code span; and
`label` became a dead parameter, so the figure's caption was dropped, a
regression against 0.10.0. It is written again in the `-- <label>` form a
carried pointer uses.
Version bumped to 0.10.1 across all ten places. Nine were unbound and stale:
four README install lines naming the previous release, two prose lines, the
"current tag" entry, `uv.lock`, and a CHANGELOG whose 0.10.1 content sat
under `[Unreleased]`. Two new packaging tests bind them to `__version__`, and
the README's guard tag to `[tool.uv.sources]`.
Every test was red first. The fate of every image is identical with and
without the new bound on three K2 PDFs carrying 800 images (464/464, 311/311
with the same 12 rejections, 25/25), and the second inflate is below the
noise floor there. 0 shipped artifacts move: no bundle under `examples/`,
`skills/` or `tests/fixtures/` carries an image pointer at all, measured
against a known-positive control.
`asset_too_large` was undocumented in the error registry; both codes are
there now. `tools/okf_accounting_gate.py` gains the new code in its closed
list -- one string, no behaviour change, stated because that file belongs to
another order.
Suite 2141 passed / 1 skipped, ruff + format + mypy --strict clean.
Report: docs/2026-09-18-bildestien-holder-0-10-1.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two MAJOR findings of the independent v0.10.0 review, both with the
shipped defaults, both new in 0.10.0. Repros rebuilt as tests first.
- A remote <img src>/xlink:href became a LIVE markdown image link in the
persisted concept, with the address and query string chosen by whoever
wrote the document. Extraction opens no socket; a consumer rendering
the bundle does. Now inert text with the address in a code span,
pinned by a property over the readers rather than by one string. The
tier asymmetry (user-upload refuses, trusted-source persisted) went to
the guard repo with the repro.
- Nothing bounded a declared image size: 9.6 KB of PDF declaring
3000x3000 grayscale zeros took 83 MB peak RSS, linear in pixels.
MAX_IMAGE_PIXELS (40 000 000) and MAX_IMAGE_BYTES (256 MiB) are read
off the corpora (largest measured 18.6 MP on K2, 1.4 MP on R761) and
checked on what the container declares, before any decompression;
over them is asset_too_large, counted. The same bound closes the
inline data: URI, which the review flagged and did not measure.
Also fixed, added by PM to this order: an inline PDF image was named
from id() of a Python object, so two concept files of the reference
corpus differed between builds. It is now named from its position.
R761 unchanged: 50 carried of 50 found, assets diff -rq clean.
Report: docs/2026-09-17-bildestien-0-10-1.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
okf build --accounting PATH inventories every source document before
extraction, in the gate's per-format vocabulary, and after the run gives
each element one fate (carried / pointer / coded rejection), written as
JSON and summarised in log.md. "carried" is checked against the written
concept bodies, so a gate that drops a line is found (test). Exit 1 on
anything unaccounted or double-booked. Opt-in: +744 s (+19 %) on the
43-document reference corpus, and that corpus fails the check on 24 real
losses (22 images on text-less PDF pages, 2 docx Title paragraphs).
Changed without the flag:
- okf build exits 1 when it extracted documents and persisted none.
Door B and corpus.measure are unchanged. One test relied on exit 0.
- An image file carried through a persisted document is its own K1b
column, no longer also extractor_unknown. The set is what the resolver
actually carried (ExtractedDocument.files), never a byte match.
tools/okf_accounting_gate.py (checks untouched) is green on all six rows,
R761 110 of 110 under both gates.
Report: docs/2026-09-17-innholdsregnskapet-bygget.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.
Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.
THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.
FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.
A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.
NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.
ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.
THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.
Also fixed, both found by measuring rather than by reading:
- a markdown image is no longer read as a cross-reference. `structure._LINK`
never looked at the character in front of the bracket, so every pointer
would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
no `assets/` at all, so every pointer named a missing file.
Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two defects reported from outside by claude-code-llm-wiki (order
20260915T202332Z-228694739), both reproduced against this tree before
anything moved.
F1 -- the packaged CLI never ran the guard. corpus.measure wired an
unconditional approve-everything stub into process_inbox and 0 of 90
add_argument calls named a gate, so the one path most people use screened
nothing while pyproject.toml made the guard a mandatory runtime dependency
and the README recommended a composition the command line could not reach.
--gate takes guard-trusted-source (default), guard-user-upload or none.
corpus.resolve_gate is the one name->callable map, with the guard imported
lazily so importing the package still does not pull it in; an unknown name
RAISES rather than falling back, because a fallback reproduces the defect
with an extra step. The gate's NAME goes into the section 9 log.md -- a
stub is only dangerous when nothing downstream can see it -- and --gate
none renders NOTHING WAS SCREENED.
The default was chosen on a measurement: over the 453 concept bodies of
the pinned reference bundle, PRESET_TRUSTED_SOURCE persists 453 of 453 and
PRESET_USER_UPLOAD holds 1, costing that concept's whole source document.
Neither tier waves anything through -- an invisible carrier and a CRITICAL
finding fail secure at both. Door B's library default is UNCHANGED at
PRESET_USER_UPLOAD: an inbox drop is an untrusted upload, an operator
pointing this command at their own folder is not. The second tier ships as
guard_adapter.inbox_gate_trusted_source, the three-line adapter that
module's docstring already described, never a preset parameter.
process_inbox(segmentations=..., gate=inbox_gate) now has a test. Before
this, `grep -rl inbox_gate tests/` gave 1 file with 0 occurrences of
`segment` -- the recommended composition was untested, which is how the
defect survived.
F2 -- a fenced code block declared structure. `# Use the opus[1m] alias`
inside a ```bash fence became a level-1 ATX heading: the document was
refused entirely where the line carried [ or ] (5 of 191 pages of the
reporter's corpus), and the concept TITLE came from somebody's shell session
on 62 of 191 (32.5 %). The fix is in the proposer and never in Door B's
title rule -- that rule is right, and a heading that was never a heading is
what has to stop being proposed. propose.fenced_lines is computed once per
text and no rule reads a fenced line, including Arm D's outline RUN, which
selects from the whole line list. Backtick and tilde fences, three leading
spaces, a closing fence at least as long as its opener, and no backtick in a
backtick fence's info string -- that last one keeps a line holding only
`okf build` from silencing a document.
MEASURED ON THE BYTES, and this is the number that decides: the 43-document
reference corpus built at b6da09c (from git archive, never the editable
tree) and rebuilt at the shipped defaults differ in log.md alone, by the one
added bullet. 865 concept files on both sides, every concept byte-identical.
Found by that control and NOT caused by this work: the pinned artifact
K2-bundle-default-20260912 was written 2026-09-09, two days before ed0418f
changed title: quoting, so it differs from what HEAD produces on 42 concept
files. test_default_bundle_pin stays green because it pins the count and the
hit@8 ranks, not the bytes. Re-pinning is the operator's call.
Suite 1896 passed / 1 skipped (+27 from 1869). ruff, ruff format and mypy
--strict clean. No version bump, no tag, no push.
Report: docs/2026-09-15-f1-f2-gaten-og-kodefencen.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bundle-only gate returned UNMEASURED and exit 3 on the very arm it was
built for: no metric computable from a bundle alone reaches boundary recall.
`boundary_share` -- declared boundaries that became a concept, over declared
boundaries -- is the one metric measured that orders the arms correctly, and it
needs the publisher's own structure, so it arrives as an input.
Measurement first, threshold after, which is what the order asked for.
P1, the normalisation, derived rather than guessed: stripping all whitespace
and lowercasing reproduces the fasit's own `norm` from its own `title` on
2 761 of 2 761 rows (alphanumerics-only scores 58). P1's own bar is 99 % on the
known-good arm and the literal reading of it reaches 22 of 2 761 -- not because
the normalisation is wrong but because okf's default route moves the numbering
token a publisher glues into a heading over into the concept id. The pair form
(concept's own directory, residual title) reaches 2 737, either reaches 2 759
(99.9 %). Both forms ship and neither is a fallback: `r761-2025-d1` is the
control in the opposite direction at 2 727 literal, 0 paired.
P2, the single corpus, is in the OUTPUT and not only in the document: the bar
is declared `corpora = 1`, every boundary row prints `N = 1 corpus`, and the
line states that `--fasit` is the caller's ASSERTION that this bundle is a
build of the document the fasit describes -- the posture `okf consume --ref`
has. Measured: the K2 reference and `n100-2023` score 0 of 2 761 and read FAIL,
which is the assertion being wrong rather than the bundle being bad.
One bar, at the pinned artifact's own value, 2 759/2 761. It is tight and the
cost is published rather than tuned away: 2 of 4 R761 builds fall under it
(2 752 and 2 727), while any bar between 41.6 % and 98.8 % separates the
known-bad arm from every R761 build measured. The known-bad arm
(`860019-mdb-100`) is 1 148 of 2 761 -- FAIL and exit 1, where the bundle-only
gate gave exit 3.
A fasit is validated at the door: not a list, a row missing `title` or `norm`,
or anything that is not JSON exits 2 with the reason, never a quiet UNMEASURED.
A fasit under five rows is UNMEASURED -- the document floor in the fasit's own
unit.
Without `--fasit` the command is byte-for-byte what it was, held by a test.
`okf check` is untouched; no version bump and no tag. 17 tests red on
assertions before the implementation, the two new doc pins each driven red and
back. Suite 1 869 passed / 1 skipped / 1 870 collected (base 5e5d01c: 1 851).
docs/2026-09-12-g37-terskler.md SS 7 carries the premises re-measured, the
seven bundles, the interval any bar could sit in, and the honesty limits --
including the correction of SS 2's own grep claim, which went false in the
commit that wrote it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Version to 0.9.0 in pyproject.toml, __init__.py and uv.lock; CHANGELOG's
[Unreleased] becomes [0.9.0] - 2026-09-13, carrying G37's `okf quality` and
K3-26's README file-type table byte-identically. MINOR rather than patch: the
release adds a command.
README's four install lines and the two tag-pairing sentences move to v0.9.0,
and the tag history gains a `v0.9.0` entry stating what ships -- three
verdicts, two bars (.pdf 8/32, .docx 2/5), both regression bars against the
pinned reference and not quality claims, `okf check` untouched at seventeen
rules.
No behaviour change in src/ beyond the version string. The guard pin stays at
v1.4.0 and the dependency floor stays at >=1.2,<2.0. Suite 1 850 passed, 1
skipped, run after `git add`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
G37. `okf check` is a CONTRACT check and a green one is not a quality gate:
measured 2026-09-10 by `vegnormal-okf`, three arms over one corpus all
returned 0 findings and exit 0 while their hit@k ranged from 6 of 6 to 0 of 6.
`okf quality <bundle>` asks the other question, per file type, with the
denominator on every line. A separate command rather than `okf check
--quality`, because the two answer different questions and a caller must not
be able to read one as the other. `okf check` is untouched.
Three verdicts and no fourth -- PASS, FAIL, UNMEASURED -- and a type with no
measured threshold is never PASS. Exit 0 judged and clean, 1 at least one
FAIL, 2 did not run, 3 nothing could be judged: exit 0 over a table of
unmeasured rows would be the silent pass this command exists to stop.
Two bars today, both `structure_null_share` (documents of a type yielding
exactly one concept), read off the pinned 43-document reference bundle: .pdf
8/32, .docx 2/5. Plus one definitional bar for every type, taken from the
harness's own degenerate-merge definition: 0 concepts with an empty body,
measured 0 of 8 602 concepts over four bundles. A bar needs five documents on
BOTH sides -- its own and the judged bundle's -- so .xlsx (2), .xml (1) and
every type with no corpus class in `extract._EVIDENCE` are UNMEASURED and
print their numbers without a verdict.
The floor on the judged bundle was found by RUNNING the gate, not by reading
it: one PDF cut into 2 182 concepts scored 0 of 1 against the 32-document
reference and read as PASS.
The gate walks the index tree and never a directory (SS 9.2; controlled
against the listing on four bundles, 453 / 2 761 / 3 206 / 446 either way),
and prints the bundle's own run log beside its counts -- a document rejected
at extraction leaves no row in the bundle, so the pinned corpus's 33 PDFs
show up as 32 and the two denominators must never be read as one.
Three of the order's five premises moved when re-measured, and they are in the
document rather than glossed: the four evidence corpora carry `source_file` on
0 of 446, 0 of 1 133, 0 of 270 and 0 of 2 756 concepts, so they name no file
type and cannot PASS; "41,6 %" is `vegnormal-okf`'s number and not in this
repository; and the same 828-document bundle carries two published hit@k
figures from two question sets.
Three candidate metrics measured and NOT shipped: duplicate titles within a
document (0 of 3 206 on the known-bad arm against 349 of 2 761 on the
known-good one) and short concepts (5.6 % against 14.6 %) order the two arms
the wrong way round; duplicate titles across the whole bundle order all four
correctly (37.8 / 16.3 / 12.6 / 5.7 %) and still ship without a bar, because
any bar separating them is read off the two bundles it would judge.
19 new tests, each rule exercised in both directions; the three README pins
were each driven red before being kept. Suite 1 850 passed, 1 skipped, 1 851
collected, run after `git add` -- +19 against a base of 1 832 collected,
measured on the stashed tree (STATE's 1 831 is one short of that).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Version to 0.8.5 in pyproject.toml, __init__.py and uv.lock; CHANGELOG's
[Unreleased] becomes [0.8.5] — 2026-09-12, carrying K3-23's instrument,
K3-25's default and K3-24's block-sources fix byte-identically, with the
two `### Changed` headings merged into one and one post written here: what
`okf.parse_frontmatter` now returns for a block `sources:` is a READING
projection PyYAML reads back on 0 of 4 605 measured block files.
No behaviour change in src/ beyond the version string. The guard pin stays
at v1.4.0 and the dependency floor stays at >=1.2,<2.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`link_in_signal` defaults to `False` on all three entry points --
`searchable_text`, `concept_scores` and `build_payload` -- through one named
constant, `DEFAULT_LINK_IN_SIGNAL`. This carries out K3-23's recommendation
(alternative c), and its numbers are the whole warrant: of the newcomers the
door's `Enclosing section:` line ever added a question token to, 39 of 39
gained it from the bundle-absolute PATH and 0 of 39 from the link's title, and
every token the path contributed is a segment of the document's own directory
-- the saturation `shared_id_prefix` takes out of the id signal, back in
through the body.
A CONSTANT rather than three naked literals, chosen against the module: the
other eight defaults here are constants (`DEFAULT_PROFILE`, `DEFAULT_LIMIT`,
`DEFAULT_TIE_SHARED_RANK`, `DEFAULT_STEM_PREFIX`, `DEFAULT_TITLE_COVERED`,
`DEFAULT_SOURCE_QUOTA`, `DEFAULT_FOLLOW_PARENT`, `DEFAULT_K`) and this was the
one default spelled out three times. It also lets a test fall on the rule from
the module that owns it.
NO CLI flag is added and none is changed. `grep -n '"--.*link'` over `src/`
gives 0 hits before and after, and `consume.main` never passed the parameter,
so the CLI inherits the function default. A flag was considered and NOT added,
because no caller could be named that needs the older reading: the three call
sites outside `consume.py` (`skill.py:360`, `tools/okf_consume_measure.py:155`
and `:247`) all inherit it deliberately.
Measured before landing: 0 of 5 shipped bundles carry the door's line
(denominators 446, 1 133, 270, 2 756, 2 761 concept files) and 5 of 5 payloads
are byte-identical across the move. The consumer repository was read only and
its tree is unchanged, listing identical including mtimes.
The docstrings move with the code rather than after it. `concept_scores` said
this was "an INSTRUMENT, not a flag and not a default" and `build_payload` that
it "carries no CLI flag"; the first is now false and the second still true, and
each says so. `cli.DEFAULT_SHELL_PARENT` keeps its value and loses its reason:
the cost it cited -- delivered sets moving through that path -- is what this
commit removes, so the comment now states why it stays off NOW, which is N = 1
document and a consumption half nobody has judged. README and CLAUDE.md carry
the same sentence where they already describe the line, measured rather than
written on reflex: neither said anything about the ranking before.
One test changed with the code. The characterisation of what the LINE does
called `concept_scores` with no parameter, because the reading it characterises
used to be the default; it now names that reading. Same fixture, same number.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One parameter, `link_in_signal`, default `True` -- so no payload moves. `False`
scores the body with `inbox._link_enclosing`'s line removed, and removes it from
the stem vocabulary too, because `searchable_text` counts `df` over the text a
hit is scored on: taking the line out of one and leaving it in the other would
measure two different texts.
The recognition is the door's own constant plus its own place -- last in the
body, after a blank line, in the door's link form. The excerpt keeps the line
either way, so two readings of ONE bundle differ in ORDER alone, which is what
separates a ranking movement from a budget displacement.
Not a CLI flag: whether this becomes a default is K3-23's measurement, not an
implementation's decision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_render_sources` gave three measured reasons for refusing to emit the block
form. Two of them fell with `28f9a4b` and are struck rather than left
standing: a block list no longer round-trips to an empty value (4 609 of
4 609 concept files), and B6's acceptance test can now pass.
Reason 2 STANDS and now carries the emission rule alone, re-measured by
reading `portfolio-optimiser` at `6eb58e5`: `read_provenance` returns
`UnreadableProvenance(reason="block-sequence")` for a block value. It is not
the guard's objection -- guard 1.4.0 reads the block form on 4 609 of 4 609
of the same files -- so the answer to "should our own emitter write block" is
NO, with the number that carries it, and the measurement that would make it
decidable is named in STATE rather than deferred without one.
The README paragraph said this library "cannot read the block form" where one
reader could and one could not. It now separates the two by KEY: `sources` in
either form through every reader, every other block mapping still refused,
with the D1b pointer intact.
Report: 222 lines, S7 form. Five deviations first, including the acceptance
sentence naming THREE flat readers rather than one, a `title` denominator of
7 372 rather than 5 372, and the `okf check` rule answered NO with a
measurement (`contract_check` holds 0 occurrences of `sources`, and
`consume` already writes `sources_unreadable` for the state a rule would
restate; `len(RULES)` is 17). Honesty limits name the one-key-wide set, the
block `verified:` fixture still reading empty, and the rendering not being
byte-identical to the source.
Suite 1807 passed / 1 skipped, rc 0, 98 s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One grammar, four call sites. `read_block_mappings` moves out of
`consume.read_sources` -- where it was written and measured -- into
`profiles`, the module both the flat readers and `consume` already import,
and the three copies of the line-oriented frontmatter grammar now decode a
block sequence for the keys `STRUCTURED_BLOCK_KEYS` names. Two copies of a
block grammar would be two answers to one question.
The value TYPE was the real choice and it was measured, not argued.
`parse_frontmatter` is public API (`okf.parse_frontmatter`) returning
`dict[str, str]`, and a list of mappings is not a `str`. Widening the return
type to `str | list[dict[str, str]]` costs 15 `mypy --strict` errors across
four of the five modules that touch the reader, plus a signature every
caller outside this repository would have to follow. Rendering the entries
back into the flow form those same readers already round-trip costs 0. The
rendering is a READING projection and says so: it is not a claim that the
value is writable -- `yaml_flow_plain` still refuses a `?` and the guard
still refuses a quote inside a flow mapping, which is why the producer
writes block in the first place.
`STRUCTURED_BLOCK_KEYS` is one key wide. `sources` is the key `read_sources`
already knows how to read; a fixture in this tree carries a block
`verified:` that still reads as an empty value, and a test pins that state
so the next widening is a decision rather than a side effect.
Nothing nested reaches the document's namespace: the entries land inside
their own value, and the K3-20 substitution guarantee is asserted per reader
copy.
Three tests that pinned the old behaviour are rewritten to what is now true,
none weakened on its other half: the block round trip in
`test_multi_source_provenance` (the evidence behind `_render_sources`'
reason 1), the v0.2 characterization (whose key-space assertion is the half
that must never weaken), and K3-22's shipped-file known-positive, where the
one difference is counted and pinned at 1.
Suite 1807 passed / 1 skipped, rc 0, 94 s -- 1782/1 before plus 25 new.
ruff clean, `mypy --strict` clean over 21 files, `uv.lock` untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Version 0.8.3 -> 0.8.4 in pyproject.toml, __init__.py and uv.lock, and the
guard pin in [tool.uv.sources] moved from v1.3.0 to v1.4.0 (uv.lock follows
via `uv lock`: the guard's version, source and requires-dist lines, nothing
else). CHANGELOG [Unreleased] closed as [0.8.4] with K3-21's and K3-22's
entries moved as written, plus one entry for the guard pin: 1.4.0 parses a
flow sequence of plain scalars that 1.3.0 refused (26 of 28 frontmatter
blocks refused by 1.3.0, 0 of 28 by 1.4.0, on one five-document bundle).
Dependency range llm-ingestion-guard>=1.2,<2.0 unchanged.
README: the four install lines and llms.txt's one to @v0.8.4, the guard
pairing prose to v0.8.4 / v1.4.0, the pip fallback to @v1.4.0, a v0.8.4
row as the current tag (seventeen checker rules), and the suite count
(1783 collected, 1782 passed, 1 skipped) with its history extended.
No behaviour change in src/ beyond the version string.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K3-21 D. `_StsReader._walk` computed `level = min(depth, _ATX_MAX_LEVEL)`
once and wrote it into BOTH the ATX heading and the `OutlineMark`. The
heading keeps the clip -- markdown has six levels and `#######` matches
nothing -- and the mark now carries the declared `depth`: it is no heading,
the declared route builds `Candidate(level=mark.level)` from it, and
`OutlineMark`'s own docstring says the level is what the tree declares,
"reported rather than fixed up". The extracted text does not move one
character; only the mark's level does.
On one standard 9 of 2 761 titled sections sit at depth 7; `--shell-parent`
gave its two depth-7 shells (`36.73211`, `36.73212`) the ancestor one level
too high. The gate -- plan otherwise unchanged, concept count, goldens, the
regression bundles, hit@k and the other 708 pointers -- is measured in a
frozen export of this commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
K3-21 C. `resolve_structure` asks `_segment_lookup` first for a `parent`
edge: `(source_file, segment_id)` -> concept name, keyed off each concept's
own frontmatter (`DocumentStructure.declared`, no file read again), so a
pointer lands only inside the pointing concept's document -- `p1` exists in
every document of a multi-document bundle. A value no segment answers to is a
document number and is looked up exactly as before; a pointer naming nothing
keeps `UNRESOLVED_MARKER`. The rendering rule is untouched: a resolved
relation renders as its subject, so `parent: p1977?` becomes `parent: p1977`.
Moved on purpose, each named: both segmented goldens' index files
(`examples/ingest-golden-segmented{,-okf-v0-2}/expected-bundle/krav/1-{1,2}/
index.md`), whose declared parents s1 and s2 -> s0 rendered `parent: s0?`
while s0 stood in the bundle -- four lines, `?` removed. The four goldens
`test_the_four_existing_goldens_are_untouched` guards are not among them.
`skills/okf-consume/` regenerated, because the golden's index bytes -- and so
its ref -- moved. `test_shell_parent`'s byte test now expects the resolved
facet. README and CLAUDE.md no longer say the index renders it unresolved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
