# llm-ingestion-okf Turn a folder of documents into a bundle a model can answer from **with a source on every claim** — offline, deterministic, no model call anywhere in the run path. Thirteen file types are read; [Supported file types](#supported-file-types) lists each one with the evidence behind it. ## Install Python 3.10+ and [uv](https://docs.astral.sh/uv/). One line: ```sh uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v1.1.0" ``` ## Use it ```sh okf project ~/my-documents # folder in: a bundle under ./.okf, and a skill beside it ``` **The standard way in is the server.** Register it once, on user scope, pointing at the directory that holds your projects. Every project you open can then ask any bundle under it, subagents included — they inherit MCP tools and not skills — and a bundle added or rebuilt is seen by the next call. You run this line; `okf` never starts Claude Code: ```sh claude mcp add --scope user okf -- okf mcp --root ~/projects ``` **The skill is the supplement**, for when you would rather register nothing: `okf project` writes it beside the bundle, so starting `claude` in that directory is enough. It reads every bundle under `./.okf` with the same code the server runs. Neither has to be made again when a bundle is rebuilt. Then ask in plain language. Three shapes of request work, and the skill states the rules for each: - **a question** — "hva er kravene til pris?" - **a hypothesis** — "stemmer det at leverandoeren baerer risikoen for grunnforhold?" Answered per premise as `confirmed` / `refuted` / `undecidable-from-bundle`. - **a task whose answer is a document** — "lag `krav-pris.md` med alle krav til pris, ett avsnitt per krav, med dokument og kravnummer." Every claim in the written file carries its source; a paragraph with no ground is written and marked, never dropped. Everything below is detail: [Consume in Claude Code](#consume-in-claude-code) for the same thing in steps and with several bundles at once, [Build](#build) for the flags, [Requirements](#requirements) for the pip fallback and the guard pairing. ## Known limitations Read this before pointing the tool at documents you depend on. Every number here was measured; none of it is a plan. - **The default gate refuses whole documents, and they are documents you may want.** Measured 2026-09-20 against a real corpus of official documentation: `guard-trusted-source`, the shipped default, refused a minority of sources outright, under `fail_secure` and `quarantine_review`, and most of those were ordinary reference pages. Not one element of a refused document reaches the bundle. Rebuilt with `--gate none`, every one of them went through untouched, so the refusal is the gate and not the readers: a page of official documentation naturally carries commands and instruction-shaped text, and the guard reads that as something to hold for review. **The corpus, its size and the per-page counts are deliberately not published here** — it belongs to a consumer whose material this repository does not republish — so this bullet carries no denominator. Run your own: the build names the count, the files and the codes on every run, which is the number that actually binds you. The build says so now — it names the count, the files, the codes and that command — and exits 0, because the bundle is a true record of what the gate allowed. **If you vouch for the source yourself, build with `--gate none`;** the bundle then records that nothing was screened. The default was chosen on one measurement over one pinned bundle's 453 concept bodies, which is a thin denominator for a decision this consequential. - **Nothing bounds what one run pays for images.** Each decode link is capped (`MAX_FILTER_DECODE_BYTES`, 512 MiB) and an oversized picture is refused with its own code, but the run as a whole has no ceiling: measured, a 70 KB PDF carrying 16 images each under the declared limit reached **851 MB peak RSS** and every picture was carried. A hard cap outside Python was measured and is not available here — `resource.setrlimit(RLIMIT_AS)` raises on Darwin 26.6.2 and is not enforced — so the per-link budget is the whole bound. `--no-assets` takes the image path out entirely. - **Three of this repository's own gates are RED, and each red row is a stated finding rather than a bug to be surprised by.** The retrieval gate is red on rows 5 and 8, the MCP gate on row 2, and the content accounting's judge on rows 2, 3 and 6. For a user that means: no ranking change has yet been registered against a held-out set (row 5), the real-corpus row is NOT RUN without its private set and so is red rather than green by omission (row 8) -- and since 2026-09-22 its judge reads an excerpt v1.1 cuts to its answering passage as that passage, accepted only when it is the exact reconstruction from the bundle's bytes. MCP anchors and concept ids are different vocabularies so `okf_fetch` cannot be addressed with a set's anchor (row 2), and the accounting still reports real losses on the reference corpus (rows 2, 3, 6). The rows and their numbers are under [Judge the retrieval](#judge-the-retrieval-python3-toolsokf_retrieval_gatepy) and [Serve a bundle over MCP](#serve-a-bundle-over-mcp-okf-mcp). - **A question worded differently from the bundle finds it through the map.** The ranking is lexical: a question has to share words with the passages it should reach. A question in another language, or in a user's words rather than the documentation's, reaches the right concepts first when it is asked as two to four sub-questions in the words of the bundle's own map (`okf card` / `okf_describe`), which is the working method the skill and the server state. Asked raw, it may be answered from the wrong concepts, and `coverage.weak` is a reading, not a guarantee, that this happened. - **The answer quality of v1.1 was checked on five questions, not measured.** Five answers, each written by a fresh model context from one call over a folder of two bundles, graded by hand. What showed red: one answer carried the right content without the wording the grading required, and the helpers did not always keep to one call. The question the bundles do not cover was declined rather than answered. Five questions is a smoke test, not a rate; the questions, answers and counts are about a consumer's corpus and are not published here. - **Speed was not measured for v1.1**, and neither was what a folder reply costs a client's context limit on a large folder of bundles. - **The content accounting counts the element classes its vocabulary names, and no others.** `0 unaccounted` is a statement about those classes, not about the document: a file whose suffix has no reader is accounted at file level only, parts no vocabulary names (headers, footers, endnotes, comments, speaker notes, cell formulas) are outside it, and an image in `xlsx`, `md`, `txt`, `csv`, `json`, `odt` or `rtf` is unaccounted and therefore red. It is opt-in (`--accounting PATH`) for that reason. The full list is under [Build](#build). - **There is no context graph and no visualisation.** Nothing in this package draws a bundle. ## What this library is Status: phases 1–3 are implemented. Phase 1 (spec-based ingestion) covers manifest validation, the `file`/`sql`/`http` connectors, deterministic materialization, index generation, and the golden fixture suite under `examples/`. Phase 2 adds the bundle inbox (`process_inbox`) and external-bundle import (`import_bundle`), both against an **injected** persist gate, with `llm_ingestion_okf.guard_adapter` wiring that gate to the real guard (see below). Phase 3 makes the bundle contract configurable, so types, layers, frontmatter sets, index shape, and reserved-file policy are carried by a profile rather than by constants (see [Upstream OKF versions](#upstream-okf-versions)). Binary extraction runs behind the optional `[extract]` extra: `pdf` through a PDF parser, and five office formats through a vendored document converter. Three of those five office rows are **constructed** rather than measured — see [Binary extraction](#binary-extraction). Phase 4 (the Node half) is planned (see `docs/plan/`). ## Supported file types Thirteen extensions are read. The table below is pinned to the extraction registry by `tests/test_docs_promises.py` — row for row, and cell for cell on the evidence class the code records — so a type added without a row here turns that test red. | File type | Read by | Dependency | Evidence | Note | |---|---|---|---|---| | `.md` | `_extract_passthrough` | core | stdlib, no corpus class | The decoded bytes are the concept text, so `source_lines` are the original's own lines. | | `.txt` | `_extract_passthrough` | core | stdlib, no corpus class | As `.md`. A document with no headings yields no segments, which is a failed build rather than a flat bundle. | | `.csv` | `_extract_csv` | core | stdlib, no corpus class | Parsed with the stdlib reader and rendered as one markdown table; a file with no header row is refused. | | `.json` | `_extract_json` | core | stdlib, no corpus class | Fenced verbatim. No structure is derived from the keys. | | `.html` | `_extract_html` | core | measured | Block tags open their own lines and `h1`–`h6` carry the ATX marker for their own level. It does **not** go through the converter, although the converter reads HTML: that route would add CVE-2025-51591 (SSRF via an iframe in HTML input) and buy nothing. The denominator is 828 files — one product, one format, one publisher. | | `.htm` | `_extract_html` | core | stdlib, no corpus class | The same reader as `.html`. The 828-file class is recorded for `.html` alone, and this row does not borrow it. | | `.xml` | `_extract_xml` | core | measured | A NISO-STS document (`` root, or any ``) becomes one heading per titled section at the section's own nesting depth; any other XML keeps its text in document order and gets no invented structure. A `=1.2,<2.0`, which `v1.4.0` satisfies; the pairing above is read off that tag's own `[tool.uv.sources]`, not off this branch. Reading a pin off `main` and installing it against an older okf tag is the one combination that fails. ### Earlier tags, as history These are not install lines. They record what each earlier tag was, so a reader who meets one in an older document knows what they are looking at. - `v1.1.0` — the current tag. The reading side: BM25 ranking on passages and titles, several sub-questions in one call, a map per bundle, a signal for weak coverage, and one generic skill plus the server as the standard way in over a whole folder of bundles. The payload contract is `okf-consumption/2`, which breaks a reader of `withheld` as a list. See the CHANGELOG. - `v1.0.0` — the tag v1 was finished at. It adds no capability to `v0.10.1`: a document the gate refuses whole is now named in the run's own summary with its code and with the one command that carries it anyway, and the front page states what this tool does not do. Read [Known limitations](#known-limitations) before you depend on it. After this tag the library is touched for defects found in its own use, and the next round is Google OKF v0.3. - `v0.10.1` — the image path of `v0.10.0`, with the two defects an independent review found in it closed. A remote `` or `xlink:href` is inert text with the address in one code span, never a live markdown image link, and no longer loses the figure's caption. An image is bounded in three places rather than one: the size a container DECLARES, the size a carried file has, and — new in this tag — what the stream behind a PDF image actually DECOMPRESSES to, which is an independent number. A declared size that is not positive is refused with its own code, `asset_size_invalid`, before the stream is read. - `v0.10.0` — a bundle carries the IMAGES its sources declare. Five readers place them (`pdf`, the converter's office rows, `html`, `xml`), `assets/` at the bundle root holds the bytes under a content-addressed name, and the concept carries a two-line pointer where the picture stood. ON by default; `--no-assets` reproduces the pre-0.10.0 bytes, measured on the 43-document reference corpus as a one-line difference in `log.md`. The image bytes are not screened — the gate reads text — and the log says so. Door C carries the assets its merged concepts point at. - `v0.9.0` — `okf quality `, a per-file-type verdict on a bundle with the denominator on every line. Three verdicts (`PASS` / `FAIL` / `UNMEASURED`) and a type with no measured threshold is never `PASS`; two bars ship, `.pdf` 8/32 and `.docx` 2/5, both regression bars against the pinned 43-document reference rather than quality claims. `okf check` is untouched and still has seventeen rules. - `v0.8.5` — a block `sources:` sequence is decoded by all three of this library's flat frontmatter readers, where they returned the key with an empty value. `okf.parse_frontmatter` is public API, so this changes what an outside caller reads: it returns a flow string where it returned an empty one. That string is a READING projection — PyYAML reads it back on 0 of the 4 605 block files measured, because the `?` opening a query string in the source URL ends the flow scalar — and the emitter still writes flow, so no bundle bytes move. `okf consume` also stops scoring the door's own `Enclosing section:` link line, which is now the default reading; the excerpt still carries the line, so what moves is order and never an excerpt's bytes. No new functionality; `okf check` has seventeen rules. - `v0.8.4` — frontmatter this library writes is YAML a YAML reader reads back the same (a block scalar it would misread is written double-quoted; a flow leaf with no form both readers accept is refused), and `okf consume` resolves a concept's `parent:` pointer into the excerpt (`parent: { concept_id, title }`) with one enclosing-section link in a heading-only body, a pointer the index now resolves too. `okf consume --follow-parent` and `okf build --shell-parent` ship off; `okf check` has seventeen rules (`parent_unfollowable`). The guard pin moves to `v1.4.0`, which reads a flow sequence of plain scalars (`source_offset: [1, 24]`) that the previous pin refused. - `v0.8.3` — a NISO-STS document's own `` names its directory and titles its `sources` entry, `okf build --frontmatter KEY=VALUE` stamps a key on every concept of a run, and an STS section's `description` comes from its own first spec point. `okf consume` keeps a leading directory every concept id shares out of its first fusion signal, which is what keeps that identity from costing the answering section its rank. `okf build --shell-parent` ships off; `okf check` still has sixteen rules. - `v0.8.2` — `okf check` refuses a skill and a payload that name different bundles (`bundle_mismatch`, the checker's sixteenth rule; `v0.8.1` has fifteen), and a covered title no longer rises above a title that answers more of the question. No new command or flag; the shipped `skills/okf-consume/` is regenerated so it passes that check. - `v0.8.1` — a question that accounts for a concept's WHOLE title reads that concept first (`--title-covered`, on by default, opt out with `--no-title-covered`). A ranking fix, no new functionality: on one publisher's 2 761-concept bundle the answering section was delivered at rank 1 on 3 of 6 scored questions before it and 6 of 6 after, and no other measured bundle's payload changed one byte. - `v0.8.0` — `.xml` is a core file type, read as NISO-STS through the stdlib parser, and a section the source DECLARES takes the declared-structure route — one publisher's process code segments at 2 761 of 2 761 of its own declared sections at the shipped defaults. No other file type changes one byte, measured on the bytes. - `v0.7.0` — `okf project` builds the bundle `okf build` builds (they were one flag apart before it), and the generated skill states the question, hypothesis and document-task modes with relative paths. - `v0.6.0` — the first tag carrying the `okf project`, `okf consume`, `okf check` and `okf skill` subcommands. - `v0.5.0a2` — a pre-release for the named OKF v0.2 pilot set only. - `v0.4.0` — the last tag before the OKF v0.2 work; it declares `llm-ingestion-guard>=0.2,<0.3`, which only guard `v0.2.0` satisfies. **No tag yet makes OKF v0.2 generally available.** `OKF_LATEST` is unchanged and still points at `DEFAULT`; flipping that alias is the GA event and none of the tags above is it (see [Upstream OKF versions](#upstream-okf-versions)). ## Build Installing the package installs one command. A folder of documents in, an OKF bundle out: ``` okf build ./documents --bundle ./bundle --bundle-id my-bundle --okf-version 0.2 ``` It walks the folder recursively, proposes a segmentation for each document with the mechanical rules, replays those proposals through the bundle inbox, writes the bundle and its `log.md`, and prints the run's numbers. Every proposal is marked `PROPOSED` and `adjudicated: false` — the command segments nothing a human has approved, and says so in the artifact. The last line that matters is the conservation identity: `merged + coded rejections == N`, where `N` is the folder's file count read at run time. **The run exits non-zero when it does not hold**, and names the unaccounted files, so a pipeline cannot mistake a partial bundle for a complete one. Flags worth knowing: `--segments off` ingests each document as one concept and asks for no root values; `--plans-dir` keeps the proposals instead of discarding them; `--report` writes the full report to a file as well as stdout. `--ingested-at` and `--proposed-at` default to `1970-01-01T00:00:00Z` rather than the clock, so two builds of the same folder are byte-identical — a wall-clock default would break rebuild-equals-incremental for every caller who did not pass them. ### Images: what the source draws, carried (0.10.0) `okf build` carries the images its sources declare. The bytes go to `assets/` at the bundle root, named `-`, and the concept carries a two-line pointer where the picture stood: ```markdown ![Tabell 84-2 Toleranseklasser](/assets/e54e5f5da0e8-tabell-84-2.png) Image: graphics/tabell-84-2.png (120x90 px) -- Tabell 84-2 Toleranseklasser ``` The first line is markdown, so a reader that renders the concept sees the picture. The second states what the first cannot — the name the SOURCE gave the file and the size in pixels — which are the two facts a person checking the bundle against the original needs. **Why it exists.** Measured on R761 Prosesskoden:2025, a process code published as a 701-page PDF and as a NISO-STS delivery: the process text is carried in full, and 12 `Tabell N-N` and 9 `Figur N-N` captions stand over nothing, because the publisher ships those tables as raster images in both deliveries. Process 84 says "toleranseklasse ... er gitt i tabell 84-2" and table 84-2 is a JPEG. A bundle like that reads as complete and is not. **What it costs, measured on the 43-document reference corpus** (`K2/trinn1`, two builds of the same commit, 2026-09-17): | | `--no-assets` | default | |---|---|---| | concept files | 453 | 454 | | markdown files | 865 | 867 | | assets | 0 | 2 964 | | bundle size | 4.7 MB | 115 MB | | wall time | 2 414 s | 3 088 s | | peak RSS | 6.26 GB | 8.74 GB | 2 964 carried of 3 145 found; 4 622 pointers, so content de-duplication folds 1 658 repeats into the files they already are. 422 of the 865 markdown files differ. **One concept appears, and the mechanism is measured rather than guessed:** the pointers are body text, so a section holding 146 of that document's images grew from 19.0 % to 30.6 % of the extracted text and crossed `--outline-gate`'s 0.20 share clause. **`--no-assets` reproduces the pre-0.10.0 bytes**, and `log.md` then says `NOT CARRIED` rather than falling silent — a bundle nobody looked for figures in must not read like a bundle of documents that had none. **The image bytes are not screened.** The gate reads text; a picture is not text. The pointer block passes the gate like any other body line, and the file beside it passes nothing. `log.md` says so on every run that carries one. **Every carried image is one a model can be SHOWN.** A bundle that holds a picture nothing can read is worse than one that says the picture is missing: the count reports that it arrived. Measured over the frozen R761 delivery's own `assets/` (denominator 50): 29 JPEG, 2 PNG and **19 RLE8 BMP** — correct files that no model decodes. Every asset's type is read off its bytes and tested against the viewable set; a BMP is converted losslessly to PNG (8-bit uncompressed, 8-bit RLE8, 24-bit uncompressed), and anything else outside the set is refused with `asset_not_viewable` and a line in the concept saying what stood there. A BMP variant this reader does not express — RLE4, BITFIELDS, 16- or 32-bit samples, a 12-byte BITMAPCOREHEADER — is `asset_bmp_unsupported`, a different fact about the document and a different thing to go and fix. The reader is stdlib and adds no dependency. Pillow, which this tree already carries transitively under `pdfplumber`, was measured first and rejected on two counts: images are carried on the CORE path, where `.html` and `.xml` need no `[extract]` extra, and an asset's name is its content digest — encoding through an installed library would make a bundle's identity move with that library's version, which is the property page rasterisation was felled over. Pillow is the independent decoder in the tests instead, and against it **19 of 19** of R761's real RLE8 assets convert with identical RGB, 2 366 365 pixels compared. A converted asset is ONE asset: one file in `assets/`, one pointer, one row in the accounting. The pointer's second line — where the source's own file name and the size in pixels already live — states the original media type, the original sha256 in full and the new one, so a reader can take the original delivery, run `shasum -a 256` and find the row. The cost is measured per image rather than per bundle, with a committed script (`tools/okf_asset_census.py`) run from two pinned trees over 9 714 image rows: exactly **35 rows moved**. Nineteen are the BMPs, now PNG. The other **16 are JPEG 2000 objects** carried out of PDF streams — a format no model decodes either, and one no stdlib route converts, so they are refused with `asset_not_viewable` and stated in the concept instead of being carried unreadably. **9 321 of 9 321** JPEG and PNG rows are byte-identical across the move. **A size CEILING, read off the same corpora (0.10.1).** An image over `MAX_IMAGE_PIXELS` (40 000 000 pixels) or `MAX_IMAGE_BYTES` (256 MiB) is refused with `asset_too_large`, counted like every other refusal. The largest image in the 43-document reference corpus is 4 515 x 4 128 (18.6 MP) and the largest of R761's 109 pictures is 2 072 x 656 (1.4 MP), so the bound is an order of magnitude above anything measured. It exists because a few kilobytes can declare an enormous picture: a 9.6 KB PDF declaring 3 000 x 3 000 grayscale zeros took 83 MB of peak RSS, a 63 KB one declaring 8 000 x 8 000 took 276 MB, and the cost is linear in the pixel count, so one document could take a whole batch build with it — before any gate, because the guard never sees image bytes. **Three numbers are bounded, not one, because a claim is not a cost.** What a container DECLARES (`/Width` x `/Height`, a PNG header, a `data:` payload's encoded length) is read before anything is decoded. What a carried FILE measures is read the same way — this package never decodes such a file, so it pays nothing for it, but writing a 7 000 x 7 000 PNG of 47 705 bytes into a bundle would hand the consumer the same bomb with `7000x7000 px` printed beside it. And what a PDF image's STREAM decompresses to is measured before it is held, a chunk at a time and discarded, because `/Length` is the COMPRESSED length and a dictionary declaring 1x1 may hang 400 MB of deflated zeros off it. Measured: that document is 408 516 bytes and cost 892 MB of peak RSS with only the declared size bounded; with the stream bounded it is refused at 54 MB, and a three-times-larger bomb costs 62 MB rather than 2 436 MB. **Every link of the filter chain is bounded, not only the first.** A PDF decodes a stream through a *list* of filters, and `/Filter [/FlateDecode /FlateDecode]` puts the whole expansion in the second one: measured, 1 636 bytes of file cost 886 554 624 bytes of peak RSS when only the first link was measured (52 367 360 with every link measured), and the picture was still refused at the end — after the memory had been spent. An encrypted stream is deciphered first and then measured like any other. **And what a link COSTS is bounded, not the size of its output.** Bounding every `FlateDecode` was still not a bound, because `ASCII85Decode` had been classed as safe "because it shrinks" and it does not: `z` is that encoding's shorthand for four zero bytes, so the filter quadruples its input, and `base64.a85decode` holds about a hundred bytes of memory per byte of input. Measured in paired subprocesses on an idle machine, the document built once and read from a file so the fixture is not what is measured: a 33 475-byte PDF decoding through `[/FlateDecode /ASCII85Decode]` cost 3 261 599 744 bytes of peak RSS and the picture was CARRIED; bounded it is 42 070 016 and `asset_too_large`. Doubling the run of `z` takes the old cost to 6 461 558 784 and the bounded one to 40 280 064 — the cost no longer follows the bomb. So `FlateDecode` is measured a chunk at a time as it is paid, and every other permitted filter carries a MEASURED worst-case cost per byte of input which is checked against the budget *before* its decoder is called. Any other filter — `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything written after this — has no measured ratio, so an image behind one is refused UNREAD with its own code, `asset_pdf_unbounded`, rather than decoded to find out what it costs. The cap that falls out for `ASCII85Decode` is read off the corpora the way the pixel bound is: over the 9 668 image objects of the 77 PDFs measured, 16 decode through such a link and the largest input to one is 450 739 bytes, more than ten times under the cap. A property test runs **every** chain of length 1–3 over the ten filters pdfminer decodes — 1 110 of them — and requires each to be delivered under the bound or refused with a published code, never paid for on the way. **What the stream bound does NOT reach**, stated because the difference matters: a stream something else has already decoded, where the memory was spent before this package was asked. That one is caught by a check on the decoded length AFTER the decode, which makes it a counted refusal rather than a bounded one — the picture is dropped by count, not by bound. **A declared size that is not a size** — a zero or negative `/Width` or `/Height` — is refused with its own code, `asset_size_invalid`, before the stream is read. Distinct from `asset_too_large` on purpose: one is a publisher shipping a picture bigger than this package carries, the other is a dictionary written wrong or written to be read wrong. **A remote reference is INERT (0.10.1).** `` and an STS `xlink:href` to an address off this machine are written as text with the address in one code span, never as `![...](https://...)` and never as a bare URL a linkifying renderer would autolink. Extraction opens no socket, but a markdown renderer or an agent that fetches what it renders does, which would turn "this bundle was opened" into a beacon to whoever wrote the document. The address is still stated, and so is the figure's caption, because a reader has to know what stood there. **`images: N` in a concept counts POINTER BLOCKS, not unique pictures.** One image referenced twelve times in one concept is `images: 12` and one file in `assets/`. The key is a count of the places a picture stands, and dedup is on content. **No size floor, and that is a measurement too.** The obvious filter is "ignore anything under N pixels", and the distribution offers no N: over the 4 828 image objects in that corpus, 149 declare no size, 162 are under 32x32, 92 under 64x64, 406 under 128x128, 498 under 256x256, 590 under 512x512 and 2 931 are larger — a broad spread with no gap, unlike `OCR_CID_SHARE`'s, which is bimodal with nothing between the modes. A threshold read off no gap is a number this package chose, and it would silently drop somebody's small table. `--accounting PATH` accounts for the CONTENT, not only the files. Before extraction, every source document is inventoried in a per-format element vocabulary: headings, paragraphs, tables, cells, images, pages, and so on. After the run, each element gets exactly one fate: - **carried:** all of its text is in the concepts written for the document, or its image is in `assets/`; - **pointer:** a remote image, which is never fetched, or a markdown image reference kept verbatim; - **a coded rejection:** the gate's or the reader's code. The result goes to PATH as JSON and into `log.md`. The build exits 1 when an element has no fate or has two. A document the gate refuses is logged as `: elements found in the source, 0 carried: document rejected ```, and the `Images` bullet then counts what the sources declare. "Carried" means the text is present, not that it is in the right place. A short element such as a section label can be found elsewhere in the same document. The judge is `tools/okf_accounting_gate.py`, which compares the inventory against an independent witness. **The account covers the element classes the vocabulary knows, and no others.** `accounting._READERS` names twelve suffixes, each with its own tuple of classes (`.md`: heading, paragraph, table, table_row, image, code_block; `.pdf`: image and page only, which is the approved exception below). Two consequences are stated here rather than left to be discovered, because "0 unaccounted" reads like a statement about the document and is a statement about those classes: - **A file whose suffix has no reader is accounted at FILE level only** — carried, merged or rejected — never element by element. - **Parts of a document that no vocabulary names are not counted, so content there can go missing under exit 0 and `0 unaccounted`.** Verified against the readers: `.docx` reads `word/document.xml` and `word/footnotes.xml`, so headers, footers, endnotes and comments are outside; `.pptx` reads `ppt/slides/slideN.xml`, so speaker notes, masters and layouts are outside; `.xlsx` reads the worksheets, the shared strings and the drawings, so cell comments are outside and a cell contributes its cached value or inline string, never its formula; `.rtf` skips the `header`, `footer`, `info`, `pict`, `stylesheet`, `fonttbl` and `colortbl` groups. A hidden slide or sheet IS counted — it lives in the same part as a visible one. Nothing here is built for now: the list is what the account does not claim. **Two operator decisions, 2026-09-17.** The accounting stays OPT-IN until the losses it reports on the reference corpus are fixed, because a default-on door would fail builds that pass today. And of the three exceptions the gate proposed, only the PDF one is approved: a PDF without a structure tree declares no heading, paragraph or table, so no witness can count them. An image in a workbook, or in md, txt, csv, json, odt or rtf, stays unaccounted and therefore stays red. A run that refused a document whole says so in both places: the accounting carries `refused` and each document's own `status`, and `log.md` carries `R of D document(s) refused whole`. The exit code does not move for it — it belongs to the whole run, and a corpus holding one unreadable file among many is ordinary — so the count is what keeps a partial refusal from being silent. The judge treats such a document as never clean, with its elements in their own `refused` column: every one of them is booked honestly as a coded rejection, so u and d both stay 0 and nothing else could see the loss. **The soft hyphen is removed before the persist gate, and counted** (operator decision 2026-09-18). U+00AD is in `llm-ingestion-guard`'s zero-width set, and `output:zero-width-present` is an any-tier carrier: a document carrying one is `fail_secure` at every trust level. Measured on R761 Prosesskoden:2025 — 71 U+00AD, and 0 of U+200B, U+200C, U+200D, U+FEFF and U+2060 — those 71 are Norwegian hyphenation points inside words (`ar[SHY]beider`, `bitu[SHY]men`), so a 701-page process code was unreadable for the whole chain over typography. `extract.normalise_extracted` removes that one character from every extracted text and reports the count as `normalised_soft_hyphen`, per document and for the run, in the accounting JSON and in a `**Normalisation**` bullet in `log.md`. The guard is not touched and the other four characters are not touched: they carry no typographic job in running text, so removing one would be a decision about what the guard screens for, taken in the wrong repository. U+00A0 NBSP is not in the guard's set and is not touched either. Reach, measured 2026-09-19: **0 of the 78** readable documents of the reference corpus carry any of the six characters, so no bundle measured here moves. Two things hold with or without the flag: - `okf build` exits 1 when it extracted at least one document and persisted none. - An image file beside a document is counted once. If a persisted document carried it, it is in the conservation identity's own column (`merged + files carried through a document + coded rejections = N`); otherwise it is a coded rejection. `--frontmatter KEY=VALUE` (repeatable) stamps a key on every concept of the run, for what the operator knows and the document does not say — an edition, a publisher's address. It splits on the first `=` and writes the value verbatim on one line, so `'sources=[{ resource: , title: }]'` survives whole. It adds any key and replaces only `sources` and `description`, the two with a layer below them (what the document declares, else the file name); every other key the door writes itself is refused before anything is read. `--shell-parent` (**off**; opt out explicitly with `--no-shell-parent`) gives a concept whose body is its heading alone a `parent:` naming the `segment_id` of the nearest ancestor that holds text — the nearest preceding plan entry at a smaller level, passing over an ancestor that is empty too. It copies no text and moves no boundary. It exists for a document that states its points once and lets every nested section inherit them: measured on one process code, **710 of 2 761** concepts are heading-only, and the plan's level and order name the same ancestor as the document's own nesting on **710 of 710** since K3-21 (708 before: the two others sit at depth 7, and the reader clipped their level to 6 in the plan as well as in the markdown heading, so they pointed one level too high). It was off because `okf consume` did not read `parent`. Since K3-21 it does: an excerpt carries `parent` as the enclosing concept's `concept_id` and `title` (resolved inside the concept's own document, never the raw segment id), and a heading-only body gains one line, `Enclosing section: [title](/path)`, in the bundle-relative form SPEC § 6.1 recommends. `okf consume --follow-parent` carries the enclosing section's text inside `parent` as well, with that concept's own `sha256`, and only from the room the cut left — so it never displaces an excerpt. Whether either default moves is measured separately. **The ranking does not score that line (K3-25).** It reaches the excerpt and the file exactly as before; `consume.DEFAULT_LINK_IN_SIGNAL` is `False`, so the body signal and the stem vocabulary read a heading-only body without it. The measurement: of the newcomers the line ever added a question token to, **39 of 39** gained it from the bundle-absolute PATH and **0 of 39** from the link's title, every such token being a segment of the document's own directory — so a bundle built with `--shell-parent` now delivers what the unflagged build delivers, on **16 of 16** rows. It cost no shipped bundle a byte: **0 of 5** carry the line, so **5 of 5** payloads are byte-identical across the move. The index resolves the pointer too: a segment id is looked up among the concepts of its own document, so it renders `parent: p1` when the concept it names is in the bundle and keeps the `?` only when nothing answers to it. ### The segmentation flags Nine rules are reachable from `okf build`, and since 2026-09-11 **all nine are ON by default** — `--outline-run 3`, `--table-grid` and `--unit-fold` since 2026-09-08, `--drop-wrapped-outline` and `--outline-gate` since 2026-09-09, `--sheet-section-rows`, `--keep-table-heading` and `--first-span-from-zero` since 2026-09-10, and `--close-span-gaps` since 2026-09-11 — each an operator decision, and each with an explicit opt-out: `--outline-run 0`, `--no-table-grid`, `--no-unit-fold`, `--keep-wrapped-outline`, `--no-outline-gate`, `--no-sheet-section-rows`, `--no-keep-table-heading`, `--no-first-span-from-zero`, `--no-close-span-gaps`. Passing all nine reproduces the pre-2026-09-08 bytes exactly, and the last three reproduce the pre-2026-09-10 bundle byte for byte — measured with `diff -rq`, 0 differences, not asserted. Each line below carries the number it was measured at, and nothing beyond it. An **eleventh** flag, `--pdf-outline`, is **off** by default, and it is the one rule here that does not read the extracted text at all: it cuts a PDF at the boundaries the file's own `/Outlines` bookmark tree declares. It is not `--outline-run` under another name — that one is a text heuristic over numbered lines in the extracted text, while this one opens a structure index the PDF already carries and the build had never looked at. It is a segmentation arm and not a reader option: the extracted text is byte for byte the same either way, and a PDF that carries no bookmark tree builds byte-identically with the flag on. Two consequences come free with it. The concept title comes from the BOOKMARK, so it is not cut short where the page wrapped the heading across two lines; and a page that lies before the first bookmark destination is the contents listing rather than a second copy of the body, so a contents entry and the section it lists stop landing as two concepts under one id. The measurement is one 701-page process code whose publisher also ships a NISO-STS structure for it, so the fasit is the publisher's own. Under the shipped default that document gives 1967 of 2761 boundaries, none of its 28 chapters, and 794 of 794 misses have their heading text present in the text the build read. With the arm it gives 2759 of 2761 and 28 of 28. The flag stays off because reach is the open question, not quality: **1 of the 8** reference PDFs in this repository's own sample carries a usable tree, and a bookmark tree is the publisher's *claim* about its own structure — a stale or wrongly pointing one carries that error straight into the segmentation. A **tenth** flag, `--bold-title`, is **off** by default. It is the rule for the type whose container declares nothing: `rtf` has no heading style, so an author's title is bold text, and the row was measured at 0 of 0 declared headings, 0 concepts and 1368 of 1368 characters in no segment. The grammar is markdown, not `rtf` — the converter already writes that title as `**…**` in the same output every office row produces — and it is gated by the principle `--outline-gate` already carries: recovery yields to declaration. Measured over 47 readable documents, three parameters were swept and one carried; false positives are **0 of the 31 documents that declare**, and the rule reaches **2 of 39** corpus documents, both `docx`, **0 of 33 `pdf`** and **0 of 2 `xlsx`**. On the `rtf` fixture set it recovers **6 of 6 authored titles over N = 4** with **0** false titles and **0 of 1994** characters in no segment. On a five-document folder it moves 26 → 27 concepts, replacing a mechanical `tabell-linje-30` with two named concepts. **A re-run is what this costs a consumer, and it is not a small one:** on the 43-document reference corpus the default bundle goes from **629 concepts in 1108 files** (the 2026-09-03 tree) to **492 in 944** after the 2026-09-08 move, to **425 in 810** after the 2026-09-09 one and to **436 in 832** after the 2026-09-10 one (digest `8dff8a8e6c15d2f7…`). On a five-document folder the last move is **15 concepts in 30 files → 26 in 52**. The proposer's own defaults (`tools/okf_propose_segments.py`) did NOT move, so every published reproduction block still runs as written. **What the 2026-09-09 move had to clear, stated because it is the bar every later move is held to:** the twelve-position reference improves (`pdf` **2 of 8 → 7 of 8**, the sheet **5 of 12 → 10 of 12**, `docx` unchanged at 3 of 3) AND hit@8 on a K2 bundle built with it holds **5 of 6 at ranks 1,1,1,1,1,–**, with no row losing rank 1. A configuration that improved the reference and cost a rank was measured in the same session and did NOT ship; see `docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md` § 9.1. | flag | what it does | measured | |---|---|---| | `--outline-run N` (default **3**) | also propose a boundary where the document's own bare-integer numbering sustains an ascending run of at least `N`; `0` is this arm's opt-out | a tender PDF whose headings are bare integers: **no boundary** at `0`, **9 concepts** at `3`, against a reference of 9 | | `--table-grid` (**on** by default; opt out with `--no-table-grid`) | a pandoc grid-table rule line no longer closes an open table block, so one grid table is one concept | a `.docx` experience list: **21 → 6** concepts | | `--unit-fold` (**on** by default; opt out with `--no-unit-fold`) | discard a contents-list run, fold a deeper heading into its parent, fold a table into the shorter heading that introduces it. Adds no boundary, so it can only reduce a plan | on a 12-document sample scored against an operator's unit worksheet: **5 of 12** match — but that figure was measured with `--table-grid` ON, and the shipped default does not include it. Measured without it the same sample scores **2 of 12**, `docx` **0 of 3**, because the fold's table clause has no joined table to fold | | `--keep-table-heading` (**on** by default since 2026-09-10; opt out with `--no-keep-table-heading`) | keep a heading whose body is empty only because a table opens under it, and absorb that table into its span | the two spreadsheets in that corpus, and **0 of 32 `pdf` and 0 of 5 `docx`**: the concept count does not move (1 → 1), its first byte does — the concept gains the heading line it was missing | | `--sheet-section-rows` (**on** by default since 2026-09-10; opt out with `--no-sheet-section-rows`) | cut an open table block at the rows that label its sections — a run of at least three rows whose first cell is a bare numeric label. The opposite direction from `--table-grid`, which decides how far a block extends | a tender price sheet whose whole body is one table block: **1 → 12 concepts**, against a reference of 11 cost groups plus the sheet's preamble. Whole corpus: **1 of 39** readable documents changes, **0 of 32 `pdf`, 0 of 5 `docx`, 1 of 2 `xlsx`**. It reached 11 of 12 on the reference two rounds before it shipped, and was held back both times by a RETRIEVAL cost that turned out not to be its own: on a bundle built with it the gold document splits 1 → 12 concepts and row 1 of the hit@8 set fell rank 1 → 2. The repair is on the reading side (`--tie-shared-rank`, now the default), and with it in place the sheet reaches 11 of 12 with hit@8 holding **5 of 6 at ranks 1,1,1,1,1,–** | | `--drop-wrapped-outline` (**on** by default since 2026-09-09; opt out with `--keep-wrapped-outline`) | do not admit an `--outline-run` candidate whose line continues onto the next one. Judges recovered candidates only, never a heading the document declares | quoted regulation text, whose numbered paragraphs match the outline grammar exactly: **4 → 1 concepts**, the reference. Whole corpus: **5 of 39**, all `pdf`; on the 12-document sample **8 of 34** outline candidates wrap, and none of the 26 the operator kept. On the reference it carries `pdf` from **5 of 8 to 6 of 8** together with the gate below, and neither reaches 7 of 8 without the other | | `--outline-gate` (**on** by default since 2026-09-09; opt out with `--no-outline-gate`) | admit `--outline-run`'s RECOVERED headings only where the document declares none of its own, plus any one recovered heading whose span covers `OUTLINE_SHARE` (0.20) of the text. Applied at admission, before spans are closed, so the text a removed mark opened is carried by the mark above it rather than lost | on the 12-document sample: `pdf` **2 of 8 → 5 of 8** alone and **7 of 8** with the rule above, `docx` unchanged at **3 of 3**. Whole corpus: it fires on **25 of 39** readable documents, changes the plan in **15 of 39**, and removes **64 of 485** proposed entries. No plan disappears (32 → 32) | | `--first-span-from-zero` (**on** by default since 2026-09-10; opt out with `--no-first-span-from-zero`) | start the first concept at character 0, so the text above it belongs to a segment instead of to none. Adds no boundary and removes none | Measured over the 39-document corpus, the default before this rule left **207 435 characters — 11.92 %** — in no segment at all: **163 804 above the first entry** (in **32 of the 32** documents that get a plan), 26 041 *between* entries and 17 590 after the last. This rule closes the first part entirely, 79 % of the whole, leaving **43 631 characters (2.51 %) over 8 of 32 documents** with two named mechanisms of their own. It adds no boundary and the K2 concept count is identical with and without it (**425 = 425**); on the 12-position reference it changes **not one cell**, and hit@8 on a K2 bundle built with it holds **5 of 6 at ranks 1,1,1,1,1,–** under both tie-breaks | | `--close-span-gaps` (**on** by default since 2026-09-11; opt out with `--no-close-span-gaps`) | close a concept's span against the next SURVIVING concept, and the last against the end of the text. Three steps remove a candidate AFTER its neighbour's span was already closed against it — the orphan check, and `fold_units` clause 1 both between entries and on the last run — and the removed mark's text then belongs to no segment. Adds no boundary and removes none; only spans' ends move | It closes the whole remainder the rule above left: **43 631 characters, 2.51 % of the corpus over 8 of the 32 documents with a plan, to 0** — both the 26 041 between entries and the 17 590 after the last. Decomposed: orphan check **18 527** over 15 of 39 documents, clause 1 **7 514** between entries, clause 1 on the last run **all 17 590** of the tail (with `unit_fold=False` the corpus tail gap is 0). The entry count is identical (**429 = 429** on the corpus, **436 = 436** concepts on K2, **52 = 52** md on a five-document folder); on the 12-position reference it changes **not one cell** (11 of 12 under `|F|`[3]=12, 10 of 12 under `|F|`[3]=11), and hit@8 holds **5 of 6 at ranks 1,1,1,1,1,–** on the new bundle, the previous default and Arm B alike | | `--pdf-outline` (**off**; opt out is the default, opt in with the flag) | cut a PDF at the boundaries its own `/Outlines` bookmark tree declares, instead of at the ones the text rules recover. A SEGMENTATION arm, not a reader option: the extracted text is byte for byte the same either way, and a PDF that carries no tree builds byte-identically with the flag on. The title comes from the BOOKMARK, so it is not cut short at the page's line break, and a page before the first bookmark destination is the contents listing rather than a second copy of the body | one 701-page process code whose publisher also ships a NISO-STS structure for it, so the fasit is the publisher's own: boundaries **1967 of 2761 → 2759 of 2761**, chapter level **0 of 28 → 28 of 28**, concept titles identical to the source title after normalisation **2761 of 2761**, false positives **163 of 2182 → 3 of 2762**, directories carrying two concept files **132 of 2050 → 2 of 2738**, contents-copy pairs **65 → 0**. Consumption on the same eight questions: fasit present in the bundle **4 of 7 → 7 of 7**, hit@1/8/50 **1/6 · 2/6 · 4/6 → 3/6 · 5/6 · 6/6**. Cost 119.22 s → 183.31 s wall, peak RSS 3252 → 3251 MiB, no new dependency and no second parse of the pages. **Off, and the reach is why:** **1 of the 8** reference PDFs carries a usable tree at all, and a bookmark tree is the publisher's CLAIM about its own structure — a stale or wrongly pointing one carries that error straight into the segmentation | They compose, and the order above is the order they apply in. Measured on a five-document tender folder (2 `pdf`, 2 `docx`, 1 `xlsx`), concepts per document: | document | default | `--outline-run 3` | `+ --table-grid` | `+ --unit-fold` | `+ --keep-table-heading` | `+ --sheet-section-rows --drop-wrapped-outline` | |---|---|---|---|---|---|---| | tender PDF, technical requirements | 1 (no boundary) | 9 | 9 | 9 | 9 | 9 | | tender PDF, technical layout | 1 (no boundary) | 1 | 1 | 1 | 1 | 1 | | price sheet `.xlsx` | 1 | 1 | 1 | 1 | 1 | **12** | | experience list `.docx` | 21 | 21 | 6 | 3 | 3 | 3 | | agreement `.docx` | 2 | 2 | 1 | 1 | 1 | 1 | | markdown files in the bundle | 31 | 49 | 33 | 30 | 30 | 52 | Every column merged 5 of 5 with 0 rejections. The reference for the first row is 9, so the default is a full arm behind what the proposer can do on that document — which is a statement about the default, not a licence to change it here. ### The two PDF reader flags Separate from the six above, and they sit before every one of them: a segmentation flag changes how the proposer cuts a text, these change what the text says. **All three are off by default.** | flag | what it does | measured | |---|---|---| | `--pdf-headings font` | a PDF carries no heading markup, so one is inferred from typography — a line whose dominant font size is above the document's character-weighted median AND whose dominant font name says bold — and emitted as an ATX heading in the same markdown the office path produces, so the existing heading rule reads it | on a tender PDF: **9 of 9** numbered chapters found, plus 4 extra candidates. Whole corpus: **25 of 32 `pdf`** change, **0 of 5 `docx`**, **0 of 2 `xlsx`**. **Off by measurement:** against the operator's unit worksheet it takes `pdf` from **2 of 8 to 0 of 8**, losing two exact matches, because on those documents the outline rule already found the chapters and a second heading source can only add | | `--pdf-headings font-reserve` | the same typographic rule, applied ONLY to a document whose own numbering the outline arm finds no run of — typography as a second heading source where there is no first one, never on top of one. Three values of one option (`none`, `font`, `font-reserve`), so no caller can ask for two at once | reaches **4 of 39** readable corpus documents (10 of 32 `pdf` admit no outline run; 4 of those render differently at all). **Off by measurement, and the measurement is that it changes nothing measurable:** on the operator's twelve-position unit worksheet it alters **not one cell** — the five positions where it fires are one PDF whose glyphs carry no ToUnicode mapping and four office documents the PDF reader never touches. The position it was built for numbers its own chapters, so the reserve is silent there by construction | | `--ocr` | read a PDF page as an image when its own text never arrived: the page extracts empty, or as `(cid:N)` placeholder codes at or above 10 % of its characters. Needs the optional `ocr` group | on the one corpus document with the failure: **95.07 % → 0 %** cid, **44 → 2561** words of four or more letters, 17 → **18** pages with text, 3.6 s/page. Whole corpus: **16 of 834** pages qualify, in **1 of 32** files | ``` pip install "llm-ingestion-okf[extract,ocr]" ``` Without that group `--ocr` is a typed refusal (`extractor_ocr_group_missing`) per file, never a crash, and the corpus run still reports `merged + coded rejections == N`. OCR text is a model's reading of an image: it is reproducible against the model version and rendering resolution it was produced with, and no dependency pin can promise more. The full measurement, including the per-page distribution the 10 % threshold was read off, is `docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md`. Measured 2026-09-08 on a 43-file corpus (33 `pdf`, 5 `docx`, 2 `xlsx`, and three files no reader accepts), one `okf build` invocation replacing the shell loop over `tools/` that produced the same corpus's bundle on 2026-09-03: | figure | value | |---|---| | `N` (folder file count, computed) | 43 | | merged | 39/43 | | coded rejections | 4/43 (`extractor_unknown` 3, `extractor_empty_pdf` 1) | | K1b | `39 + 4 = 43 = N`, exit `0` | | files written | 1108 | | identical to the 2026-09-03 bundle | 1104/1108 | | wall time | 842.82 s total, 19.600 s per file (re-measured 2026-09-08) | The six files that differ are all in the corpus's two spreadsheet documents, and they are the change reported in `docs/2026-09-08-prisform-og-loggen-k2.md`: a spreadsheet's tables are now written as pipe tables, so each row is one line with its cells delimited rather than padded out to the widest cell in the column. Two concept files are renamed by it, two are removed under their old names, and the two documents' own `index.md` follow. The root `index.md` is identical to the stored one again, because this library no longer links the bundle's `log.md` from it. The command's own byte-identity test compares it against the two scripts at the current commit, where the two agree over the whole tree. ## Consume The other direction: a bundle plus one question in, one bounded, contract-shaped payload out. ``` python3 tools/okf_consume.py ./bundle --question "your question" --out payload.json ``` `tools/okf_consume.py` is the **pre-pass** `docs/consumption-contract.md` § 1 defines — the deterministic program that reads the bundle, ranks its concepts, cuts them to a bounded set and emits one payload. It decides nothing about the question; the skill that reads the payload does the judgement. It calls no model, opens no socket, imports nothing outside the standard library and this package, and takes no clock: the same bundle bytes and the same `(question, k, limit, cost_vocabulary, reserve_top_rank, rarity_weight)` produce byte-identical output. **The ranking is BM25 since v1.1** (`--ranking bm25`, the default). Two signals are fused by rank: each concept's best 500-character passage, and its title and id path weighted above its body. A word the bundle does not hold weighs nothing by itself; one it holds in another form — a Norwegian inflection or compound — is read as that form. A concept longer than 4 000 characters is delivered as the passage that answers, under the nearest heading above it, marked with `passage: {start, end, of}`, so the whole can be fetched by its `concept_id`. `--ranking fusion` is the older three-signal ranking; the flags below that say they widen a signal (`--cost-vocabulary`, `--rarity-weight`) belong to it and are refused without it. The rest of this section describes the fusion. **A table fragment is read under the heading it stands under.** When the builder can only name a table block by the line it starts on (`Tabell linje N`), `okf consume` reads it under the title of the nearest concept above it in the same document — in what it ranks and in what the excerpt shows — and the excerpt keeps the file's own title as `own_title`. No bundle byte moves. **The payload says when the bundle looks like it does not cover a question.** `coverage.absent_terms` lists the question's words the bundle holds in no form — not as written and not through a relative it uses — and `coverage.weak` is true when one such word exists or nothing was delivered. It is a reading with its rule in the open, not a verdict: the reader rephrases in the bundle's own words, and if it stays weak, says the bundle does not cover the question. Words that only frame a question (`how often`, `hva står i`) are not topic words. **Several sub-questions in one call.** A broad question is asked best as two to four narrow ones in the bundle's own words: `okf consume ./bundle --question "first" --question "second"`, or `okf_ask` with `questions`. The bundle is read once, each sub-question is ranked and cut as it would be alone, and the deliveries are interleaved — first excerpt of each in turn, then the second, duplicates dropped — under the same `--k` and `--limit` one question gets. The payload then carries `questions` instead of `question`, every excerpt names the `subquestions` it answered, and `coverage` has one block per sub-question, with `weak` true only when every sub-question is weak. One question gives exactly the payload it always did. The consumption contract (`docs/consumption-contract.md` § 8.8–8.10) states the three new forms — a passage, several sub-questions, a heading-inherited title — and `okf check` holds the first two, with nineteen rules: `passage_malformed` refuses a `passage` that is not whole numbers with `0 <= start < end <= of`, and `subquestions_unindexed` refuses an excerpt whose `subquestions` are not distinct indices into the payload's `questions`. `--cost-vocabulary` is off by default and widens one question class: it lets a declared list of cost/price/quantity terms bridge a question and a document that name money with different words. The gate is the question — one naming no such term gets byte-identical bytes either way — and what it does and does not close is measured in `docs/2026-09-08-blindsone-below-k-k2.md`. `--reserve-top-rank` is off by default and answers a different objection: the budget is packed by an exact knapsack, which maximises a SUM of scores and therefore has no opinion about rank, so a top-ranked excerpt costing a large share of the budget is out-summed by many small ones. Measured, that made `--k` a dial that could EVICT the concept a question was asked about. The flag gives rank one its bytes before the pack runs — after the `over_budget_alone` pre-exclusion, never before — and the payload then declares `budget.reserved`. On a 629-concept corpus it changed the delivered set in 2 of 24 measured combinations, both of them that eviction: `docs/2026-09-08-blindsone-laas2-budsjett-k2.md`. `--source-quota N` is **on** by default at **2** since 2026-09-10 (opt out with `--no-source-quota`), and it is the third widening here that alters a payload with no bundle changing. It caps how many DELIVERED places one source document may take, cutting where the shortlist is cut so the freed place goes to the next candidate and `k` is still delivered in full. The defect it repairs was measured outside this repository on a 3206-concept bundle of a published handbook: the code's own process overview contributes **28 of 3206 concepts (0.87 %)** and **8.0 % of the source characters**, and took **8 of 8** delivered places on one question and **7 of 8** on the known-positive, which was not delivered at all. Identical at 343 and 1651 concepts, so the cause is the corpus's COMPOSITION — that it holds its own table of contents — and not its size; any corpus with a contents list, a project overview or a summary document has the same property. Swept over {2, 3, 4, off} on three bundles: at 2 and 3 hit@8 goes **5 of 6 to 6 of 6 on both K2 bundles** with all five standing rank-1 rows unmoved, and at 4 and off it stays 5 of 6. On the handbook bundle hit@8 goes **2 of 6 to 4 of 6** and the dominant document's share of delivered places **8 of 8 to 2 of 8**. 2 rather than 3 on rank: the recovered rows come in at 5 and 4 rather than 7 and 5. What the gain is NOT: hit@8 asks whether the gold DOCUMENT was delivered, and a document quota directly raises how many distinct documents a payload holds, so that metric is not neutral with respect to this rule — the five rows that were already rank 1 are, and they did not move. The adverse case is named rather than found later: a bundle built from ONE document has one `source_file` on every concept, so the quota would deliver 2 excerpts instead of `k`; the shortlist is topped back up from the best-ranked over-quota candidates, which makes such a bundle byte-identical to the quota being off. `--rarity-weight` was measured against the same defect in the same session and does **not** repair it: on the handbook bundle it leaves the dominant document at 8 of 8 places on the question it floods and delivers neither that answer nor the known-positive. `--rarity-weight` is off by default and weights each lexical hit by `log(N/df)` over the bundle's own concepts instead of counting it as one, so a requirement number is not worth what a common verb is worth. The default being off is a measurement rather than a preference: on four corpora it took one gold concept from withheld to delivered and a priced sheet from candidate rank 10 to 2, left one gold rank unmoved, and cost another seven rank positions — because the four-character prefix matcher makes a unique identifier read as 135-of-446 common on that bundle. Where it cannot help is decomposed rather than guessed: RRF fuses RANKS, so a weight moves nothing on a signal the gold already leads. `docs/2026-09-08-sjeldenhetsvekt.md`. Its published figures were measured under the pre-2026-09-10 tie-break and are not re-measured. `--stem-prefix` is **on** by default since 2026-09-09 (opt out with `--no-stem-prefix`), and like `--tie-shared-rank` below it alters a payload with no bundle changing. `MIN_SHARED_PREFIX = 4` exists for Norwegian compounding, and it also matches four characters that are not a stem: measured on the pinned 453-concept bundle with the control run first, `under` occurs 79 times by equality and matches 172 concepts by prefix, while `bilateral` occurs **0** times and matched **400 of 453** through `bilag`, and `standhaftig` 0 and 219 through `standard`. Three repairs were measured and all three failed on the same row — a longer floor (5–8), a coverage share (0.5–0.8) and a long-words-only floor (≥ 8) each cost row 1 its rank on the default bundle and the whole row on Arm B, because row 1's token `prisene` reaches its gold document through `pris|sammenstilling` on four characters. The rule that works asks whether the shared prefix is a WORD the bundle uses: `bilateral` 400 → 0 and 512 → 0, `standhaftig` 219 → 56 and 235 → 33, **every hit@8 row keeping rank 1 on both bundles**. `undersjøisk` stops at 162 because `under` is a word here — a genuine Norwegian morpheme, so the residual is a different answer and not a ceiling. `--tie-shared-rank` is **on** by default since 2026-09-10 (opt out with `--no-tie-shared-rank`), and it is the one change in this library that alters a payload with no bundle changing — a consumer pinned to the previous excerpt order needs the opt-out. RRF emits a rank for every concept in every signal, including a signal that scored them all the same, and the declared tie-break then orders that group by `concept_id`; the fusion reads alphabetical order as if it were a measurement. Shared ranks make a signal that separates nothing contribute the same constant to each concept in the group. What it buys is general rather than cosmetic: a document the segmenter splits from 1 concept into 12 fills that signal's whole top tie group with its own concepts, so the one that leads the body signal takes position 11 instead of 1 and the document loses fused rank 1 to a single-concept competitor leading nothing — **the fusion was punishing fine-graining for being fine-grained**, which put the segmentation side and the retrieval side in competition over one number. It shipped OFF on 2026-09-08 because hit@8 fell 5 of 6 to 4 of 6, and that figure is real and **conditional**: swept over 2 document-prior exponents x 3 bundles x 6 rows, the lost row is lost only at exponent 1.0. The exponent moved to 0.5 on 2026-09-09 for an unrelated reason, correctly reported as moving no hit@8 row, and nobody measured the pair — so a rule sat behind a published number that had stopped being true in the same commit. A flag's "off by measurement" is a measurement of a *configuration*, not a property of the flag. `docs/2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md`. `--title-covered` is **on** by default since 2026-09-10 (opt out with `--no-title-covered`), and it is the fourth widening here that alters a payload with no bundle changing. A concept whose EVERY title token is a token of the question is read before the concepts the fusion ranked above it. The defect it repairs is that both lexical signals are unnormalised COVERAGE COUNTS: they measure how much of the question a candidate answers, and nothing measures how much of the CANDIDATE the question accounts for, so a section titled with the question's subject alone scores exactly what a narrower section titled with that subject plus a qualifier scores — and then loses on the body count, because a longer title and a longer body can only reach more of the question. Measured on a 2 761-concept bundle of one standard, where **none of the six flags above moved the number at all**: hit@1/8/50 **3 of 6 · 5 of 6 · 5 of 6 → 6 of 6 · 6 of 6 · 6 of 6** at default `k`, the same 6 of 6 at `--k 50`, the known-positive holding rank 1 at both and the known-negative still not a hit. The three recovered rows go 4 → 1, not-delivered → 1 and 3 → 1. It is a PARTITION and not a fourth RRF signal, and the arithmetic is the reason: RRF consumes ranks alone, so with shared ranks a rule whose positive group has `m` members is worth `1/61 − 1/(61 + m)`, and a rule firing on ONE concept of 2 761 is worth 0.00026 against a body-signal gap of 0.0029 — **a precise rule is worth least under this fusion, backwards from what precision is for**. Measured as a signal the same predicate moves hit@1 not at all; as a partition it reaches 6 of 6. `lookup_hits` is the same shape for the same measured reason, and it still wins: this partition lands below it. The title is read by EQUALITY, never by shared prefix — four shared leading characters take the group from 1 to 6 on one question and 9 to 31 on another, with the answering section falling to candidate rank 6 and the known-positive to 2. Two candidate repairs were measured and felled first. Pivoted length normalisation of the body signal collapses at every value swept (b = 0.25, 0.5, 0.75, 1.0 give hit@8 3, 1, 1 and 0 of 6, and at 1.0 the known-positive falls to rank 49), because the median concept holds 22 tokens against a mean of 60 — so length normalisation promotes thousands of tiny concepts over the section that treats the subject. Title *precision* as a signal reaches candidate hit@1 5 of 6 and takes the known-positive from rank 1 to 4 every time it does. The reach of what shipped is narrow and stated as such: it fires on 6 of 8 questions on that bundle and on **0 of 21** measured cells across the pinned K2 bundle, Arm B and the three N bundles, whose payloads are byte-identical either way. `docs/2026-09-10-k3-runde16-hele-tittelen-tar-ruten.md`. **Since round 17 the partition is bounded by RECALL**, under the same flag. A covered title states its PRECISION — it says nothing the question did not ask — and round 16 let that claim override the fusion even against a title answering MORE of the question: on a 26-concept bundle of five tender documents, a question naming a section by three of its title tokens also held a neighbour's whole one-token title, and the neighbour took rank 1. A covered concept now rises through the fusion's order and stops beneath the first concept whose title answers more question tokens, by equality, than it holds. On the standard's bundle nothing stood above a covered concept answering more, so its 8 payloads are byte-identical at both `k`; the 27 payloads where the rule never fires are byte-identical too. A minimum title length, a share of the question, an order inside the group and a stop list were measured beside it and not taken. 0.8.1's unbounded order is reproducible by no flag; `--no-title-covered` still gives the pre-0.8.1 order. `docs/2026-09-11-k3-runde17-dekningen-stopper-ved-en-bredere-tittel.md`. It emits the § 8 shape — `contract`, `bundle` (`bundle_id` plus a `sha256-tree:` content identity), `budget` (unit, instrument, limit, spent and a validated known-positive), `denominators`, `excerpts` and `withheld` — and every withheld concept is accounted for by the rule that dropped it, from a closed set of seven. **`withheld` is counts plus names, not one entry per concept** (revision `okf-consumption/2`). It carries the `total`, the same total decomposed `by_rule`, the best-ranked drops by name — with title and source document, so a reader who sees a near miss can ask for it — and `complete`, which says whether those names ARE the whole set. `--withheld-nearest N` sets how many are named (default 20) and `--withheld-full` names every one, which is what an instrument classifying every miss should ask for. The default moved on a measurement: on a large real bundle the flat list came to **65.5 % of the written payload**, none of it counted against the budget the same payload reported, and none of it anything a reader could act on. The same question after the change costs **18.4 %** of what it did before. `--withheld-titles` is retired by that change — it existed to buy the one field the near misses now carry. Every excerpt carries the concept's `title`, and — when the producer wrote them — `req_number`, the SPEC § 5.1 address `sources`, and **every top-level `source_*` key**, by prefix rather than by allowlist: a fixed list names the locators its author thought of, and one real bundle locates by `source_element_id` on 269 of its 274 concepts. A key the producer did not write stays absent rather than arriving empty, and an address this reader cannot decode is named (`sources_unreadable`) rather than dropped into the same silence. A concept naming the section that encloses it delivers `parent` — that concept's `concept_id` and `title` — or `parent_unresolved` when the pointer lands nowhere; `okf check` (seventeen rules) refuses a `parent` a reader could not follow. The reason is a measurement: with `concept_id` and body text alone, a delivered gold concept at rank 1 still left the answer unable to name the document it was quoting. `considered == withheld + delivered` closes by construction, and the payload is refused rather than reported when it does not. Three exit codes, not two: **0** a payload was written, **1** the run happened and refused (the budget admitted none of the concepts that answered the question, or an asserted `--ref` contradicted the bytes), **2** the run did not happen. Collapsing 2 into 1 would report an unread bundle as a failed cut. `--ref` is an **assertion**, never an override — the identity is always computed from the bytes, because labelling a payload with an identity its bytes do not have is the one thing § 3.3 exists to prevent. Check any payload against the skill that will read it: ``` python3 tools/okf_contract_check.py --skill skills/okf-consume/SKILL.md --payload payload.json ``` `skills/okf-consume/` is an instantiated consumption skill generated by `okf skill` from the golden bundle this repository ships, `examples/ingest-golden-segmented-okf-v0-2/expected-bundle`, together with the payload that proves it: `okf check` on the pair is conformant with 0 findings. It is regenerated with the command in `skills/okf-consume/references/README.md` rather than edited, and a test holds the shipped bytes to the generator's. Its numbers describe that three-concept bundle and nothing larger. The measurement behind the consumption skill as a form — hit@8 **5 of 6** questions at rank 1 on a 629-concept bundle against a chance baseline of **1.35 of 6**, with one control that failed — is `docs/2026-09-07-okf-konsumskill-maaling.md`; the copy filled by hand for that corpus cannot ship and is no longer the one here. ## Judge a bundle: `okf quality` `okf check` is a **contract** check — it asks whether a payload carries what a claim must rest on. It is not a quality gate, and that is measured rather than conceded: on 2026-09-10 three arms over one corpus all returned 0 findings and exit 0 while their hit@k ranged from 6 of 6 to 0 of 6. `okf quality ` asks the other question, **per file type and with the denominator on every line**: ```sh okf quality .okf/my-bundle ``` Three verdicts and no fourth — `PASS`, `FAIL`, `UNMEASURED` — and a type with no measured threshold is **never** `PASS`. Exit codes: **0** judged and clean, **1** at least one `FAIL`, **2** the run did not happen, **3** nothing could be judged (every row `UNMEASURED`), because exit 0 over a table of unmeasured rows would be the silent pass this command exists to stop. Two thresholds exist today, both `structure_null_share` — the share of a type's documents that yielded exactly one concept — read off the pinned 43-document reference bundle: | file type | threshold | N | |---|---|---| | `.pdf` | 8/32 | 32 documents | | `.docx` | 2/5 | 5 documents | | every type | 0 concepts with an empty body | definitional | Every other type is `UNMEASURED`, including `.xlsx` (2 documents), `.xml` (1 document) and `.html` (no bundle measured here). A threshold needs at least five documents on both sides — the bundle's and its own — because a `1/1` is not a rate. **What a `PASS` is not.** It is a regression bar against a pinned artifact, not a claim that the cut found the document's own structure. hit@k needs a question set and is not asked here; the measurement that says so, the corpora behind each number, and three candidate metrics that were measured and not shipped are in [`docs/2026-09-12-g37-terskler.md`](docs/2026-09-12-g37-terskler.md). ### Boundary recall: `--fasit` Give the command the boundaries the source itself declares and it adds one whole-bundle row, `boundary_share` — the share of them that became a concept: ```sh okf quality .okf/my-bundle --fasit declared-sections.json ``` The fasit is a JSON list whose rows carry `title` and the key they are matched on, `norm` (all whitespace stripped, lowercased). An unreadable one exits **2**, never `UNMEASURED`. A boundary counts as recovered in either of two forms — a concept whose normalised title equals `norm`, or the pair of the concept's own directory and its residual title — because the numbering token a publisher glues into a heading lands in the concept *id* on one route and in the *title* on another: measured on one 2 761-section standard, the first form alone reaches **22 of 2 761** where the two together reach **2 759**. | metric | threshold | N | |---|---|---| | `boundary_share` | 2 759/2 761 | 2 761 declared boundaries, **1 corpus** | **`--fasit` is an assertion**, the way `okf consume --ref` is: it says this bundle is a build of the document the fasit describes. A bundle of another product scores near zero and reads `FAIL` — that is the assertion being wrong, not the bundle. The bar itself rests on **one product**, which the output says on every run. Both facts, the arm it separates (1 148 of 2 761 against 2 759 of 2 761) and the interval any bar could sit in are in [`docs/2026-09-12-g37-terskler.md`](docs/2026-09-12-g37-terskler.md) § 7. ## Judge the retrieval: `python3 tools/okf_retrieval_gate.py` A separate question from `okf quality`, and a separate command: quality asks what a bundle looks like, this asks whether the payload for a question carries the fasit — and whether the payload says so when it does not know. ```bash python3 tools/okf_retrieval_gate.py # eight rows, one exit code python3 tools/okf_retrieval_gate.py --json # the same rows as JSON ``` Eight rows, exit 0 only when every one is green, 1 otherwise, 2 for wrong input. Rows 1–4, 6 and 7 run against a synthetic corpus this repository generates and six question sets it ships, pinned by sha256: no network, no private corpus, no clock. A question set is always an input — `sha256` is checked before a byte is measured and a mismatch is exit 2 — because a gold set names a consumer's documents and this repository is public. **The corpus is pinned the same way** (`SPECS_SHA256`): every row counts against those documents, so moving them without moving the pin is exit 2. **It is RED today, on rows 5 and 8**, and each of those is a finding rather than a defect in the gate: | row | what it asks | today | |---|---|---| | 1 | hit@payload, one fasit entry = one unit | 14 of 14 | | 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 | | 3 | the `rule` the payload prints for a withheld fasit is the true one | 5 of 5 | | 4 | an uncovered question comes back marked, a covered one does not | 6 of 6 | | 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 | | 6 | every delivery confirmed against the bundle's own bytes | 14 of 14 | | 7 | mechanical mutants of the ranking and the cut, felled | 15 of 15 | | 8 | the real set, from path + sha256 | 0 of 1 set, NOT RUN without `--real` | **The public rows run on invented material only** (operator decision 2026-09-21). The test track built on material tied to the operator's employer is retired rather than re-measured or frozen: its row 9 is gone, row 8 no longer reads the two sets built on it, and no test or gate here needs that material to run. Rows 3 and 4 were this gate's two findings and both are closed, which is what a gate written before the capability is for. Row 3: in a bundle built from ONE source document, every concept past the first two carries that document's `source_file`, so a concept the RANK had already lost came back withheld as `source_quota_exceeded`. A drop now keeps the rule the same cut without the quota would have given it, and only a candidate that cut would have delivered is named as the quota's — 2 of 5 to 5 of 5. Row 4: the payload had no key a consumer could read as "this bundle does not answer that", so an uncovered question came back with excerpts and no statement. `coverage` states the terms the pre-pass read, the terms no concept in the bundle answers and the terms no delivered excerpt answers — facts, no verdict, and the one bar is the gate's own `UNANSWERED_BAR = 2/3` — 3 of 6 to 6 of 6. Row 7's mutants patch the code the DEFAULT ranking runs (BM25 since v1.1), and it fells all 15. Until 2026-09-22 three survived with **0 ranks and 0 deliveries** moved -- the passage signal reading no body, the field signal weighing no title and no path, and a flattened fusion (`bm25.RRF_K`) -- because every synthetic concept was short and opened with its own title, so none of the three ever decided a delivery. Three fixtures now make each one decide one (`set-mechanisms.json`): a long concept answered in one window of its body against short concepts whose titles carry the question's words; a concept named by its path alone against decoys denser in the body; and a concept first in one signal and 21st in the other against a decoy 10th and 11th, which a flattened fusion ranks first because only the rank sum then counts. Each fixture is delivered at the shipped defaults and falls out of `k` with its own mechanism switched off; `src/` did not change. A delivery is judged against the bundle's own bytes (row 6), and since v1.1 a concept over 4 000 characters is delivered as its answering passage: the nearest heading above it, `[...]`, the span, `[...]`. The judge counts such a passage only when it is EXACTLY that reconstruction -- the span byte for byte the delivered body at the passage's own offsets, markers only where text is left out, the heading a line of the body above the span -- and it reads the citation in the span alone. A looser rule ("the span occurs somewhere in the text") would accept an invented sentence beside the span; the fifteenth mutant (M15) is exactly that, and a fixture that delivers a passage fells it. Row 8 is never green by leaving something out: it requires every named set and is NOT RUN until it has them — one set of three used to read `6 of 6 GREEN`. The set lives in another repository and is read, never written: `--real wiki ` runs it. Granularity is stated on every line and the two forms are never summed: a set naming a citation is measured at citation granularity, a set naming only a section is measured at concept granularity. **Row 8's own headline is therefore at QUESTION granularity**, with the two unit totals printed below it, each with its own denominator. ## Consume in Claude Code A folder of documents to an answer a model can cite, in **three lines**. You do not need this repository — the first line installs the command, the second builds the bundle and writes a skill beside it, the third asks. ```sh uv tool install "llm-ingestion-okf[extract] @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v1.1.0" okf project ~/my-documents claude ``` `okf project` writes the bundle to `.okf//` and a skill to `.claude/skills/okf-consume-any/` in the **current directory**, then prints what it read, what it wrote, and which documents a question cannot reach. Start `claude` in that directory and ask in plain language; the generated skill runs the pre-pass and the contract check itself and marks every claim with its source. **Running non-interactively.** In print mode the skill needs its tools named explicitly, or the model answers without ever reading the bundle and marks every premise `undecidable-from-bundle`: ```sh claude -p --allowedTools=Bash,Read,Grep,Glob "" ``` Add `Write,Edit` for the mode that produces a document. `--permission-mode acceptEdits` alone does **not** do it — measured 2026-09-09 over four runs, the `okf consume` call is refused without the explicit tool list. The interactive `claude` above needs none of this. `` is the folder's name reduced to `[a-z0-9-]`. Run it once per folder with `--id ` to have several bundles reachable at once — each skill carries its own `bundle_id`, which is what lets a model pick between them. `--out ` puts the project somewhere other than the current directory. Measured 2026-09-09 from a fresh `uv tool install` with this repository nowhere on the path: 5 documents in, **26** concepts out, a skill carrying **0** paths into any checkout, and `okf check` conformant on its own payload (17 rules, 0 findings; the count was 15 until `bundle_mismatch` landed 2026-09-10 and 16 until `parent_unfollowable` landed 2026-09-11, and the pair was re-measured each time rather than carried over -- the last time from a frozen export of this repository, with 26 concepts and 0 checkout paths again). The 2026-09-08 run of the same measurement reported 15 concepts, and that number was the defect rather than the result: `okf project` was calling `build()` as a function and reading its signature's defaults, which disagreed with argparse's on two flags. Two tests now hold the two default sets equal. Before that day the same result took a `PYTHONPATH`, a snapshot of a clone, and a generated skill that named that clone by absolute path on four lines — so it could not be moved, shared, or run by anyone else. ### The same thing in steps, if you want to see the payload ```sh okf build ./documents --bundle ./bundle --bundle-id my-bundle --okf-version 0.2 okf skill --out ./project/.claude/skills/okf-consume-any okf consume ./bundle --question "your question" --out /tmp/payload.json okf check --skill ./project/.claude/skills/okf-consume-any/SKILL.md --payload /tmp/payload.json ``` A bundle you only have read access to is fine — the generator only reads it. ### The honest limits Measured on **four questions** across two bundles, which is a demonstration and not a hit rate. The ranking is lexical, and one of the four found a topic the bundle **does** cover and did not rank it into the cut — the skill then said so with its denominator instead of answering, which is the behaviour the contract asks for, but a miss is still a miss. `docs/2026-09-08-claude-code-skill-vilkaarlig-bundle.md` has the runs. Two more things the summary tells you and this paragraph will not repeat: a document that landed **whole** (no heading, table or numbered outline to cut it on) comes back as one excerpt, which the budget often refuses and which often does not carry the answer at the place you asked about; and a document that is in the folder but **not** in the bundle cannot be quoted at all. Both cases are answered `[sourced-not-sufficient]`, and `okf project` names the documents. ### The skill that runs this for you `skills/okf-prosjekt/` in this repository is a Claude Code skill (Norwegian) that wraps the command above: it takes a folder, runs `okf project`, and reads the summary back. Install it for your user account after cloning: ```sh mkdir -p ~/.claude/skills && cp -R skills/okf-prosjekt ~/.claude/skills/ ``` ## Serve a bundle over MCP: `okf mcp` Two shapes, one implementation, and the difference is what an agent has to be told in advance. ```sh okf mcp --bundle .okf/my-bundle # one server, one bundle okf mcp --root ~/bundles --root ./.okf # one server, every bundle under the roots ``` `--bundle` serves exactly one bundle, fixed at startup; its tools take no bundle argument, because there is nothing to choose. `--root` (repeatable) serves every bundle found under the given directories and **knows none of them by name**: it discovers them per call, so a bundle you add, remove or rebuild while the server is running is picked up by the next call. No restart, no configuration edit, no code change. Four tools on a multi-bundle server and **three** on a single-bundle one — `okf_list` is absent where there is nothing to list — and each one's description says why it exists: | tool | what it answers | |---|---| | `okf_list` | which bundles are reachable right now, with each one's content identity and concept count (multi-bundle servers only) | | `okf_describe` | what one bundle is: id, ref, concept count, how many concepts carry each conditionally-written field, and its `map` — one line per source document with its section titles, a series of like-named documents as one line. Omitting `bundle_id` on a multi-bundle server describes them all, as `okf_ask` does | | `okf_ask` | one question, or two to four sub-questions in `questions`, and one bounded payload of excerpts, each with its bundle id, concept id, title and provenance locators. Omitting `bundle_id` on a multi-bundle server asks them all and splits the budget | | `okf_fetch` | one named concept, verbatim, with its frontmatter and locators | **The server carries the working method, because a subagent inherits MCP tools and not skills.** Its `instructions` and the `okf_ask` description state the short form — read the map, write two to four sub-questions in the bundle's own words and send them in ONE call, read what lay just outside the cut and ask again with its words, then write one answer in the questioner's language. Claude Code truncates both at 2 KB, so the long form stays in the skill, which has no such cap; a test holds the short one under the limit with a control, because a truncated method is worse than a missing one. **Nothing is cached between calls, and that is the design.** Every call re-reads the directories and recomputes the bundle's content identity, so the identity in an answer is a fact about the bytes at the moment of the call rather than at startup — a server that answered from yesterday's bundle is the one failure you cannot see from the outside. The cost is real and is paid per call: on a 2 756-concept bundle the identity is a 0.75 s hash of the whole concept tree, and one `okf_ask` is 5.6 s. **Refusals are loud.** A path climbing out of the bundle, a symlink leaving the served root, a bundle id nobody answers to, a directory whose manifest cannot be read, and a concept above the server's size ceiling each come back as an error with a code — never as a plausible-looking empty answer. A concept over the ceiling is refused whole rather than truncated: a truncated concept read as whole is a wrong answer that looks right. A directory that cannot be read as a bundle is **reported** in `okf_list`'s `unreadable`, not skipped. The protocol is written with the standard library only. An MCP SDK would be this package's second runtime dependency on the default install path, for four JSON-RPC methods and a newline framing — see [Requirements](#requirements). `tools/okf_mcp_gate.py` is the eval: it starts the server as a subprocess, speaks real stdio to it, and measures six rows. It was written red before the server existed, and it is red today on row 2. The measurements, the update drill and the limits are in [`docs/2026-09-20-mcp-to-varianter.md`](docs/2026-09-20-mcp-to-varianter.md). ### One skill for every bundle: `okf skill` and `okf card` **`okf skill --out ` writes one installable skill for ANY bundle. That is the default since 2026-09-20**, and `okf project` installs the same one: ```sh okf skill --out ~/.claude/skills/okf-consume-any okf card .okf/my-bundle # the per-bundle numbers, as JSON, on demand ``` The generic skill carries no bundle's id, no ref and no count; it tells its reader to run `okf card ` first. The card is **derived on every run and never written into the bundle**, so there is no second artefact that can disagree with the bytes beside it. It is therefore never stale, and one skill serves every bundle a project holds. **The card carries the bundle's map** (`map`): one line per source document — its name, then the titles of its concepts in document order — and documents whose names differ only in their numbers (a changelog per release) as one line with the count and the span. It is the bundle's own words, to write sub-questions in. The lines are capped at 48 000 bytes together (`lines_truncated` counts what a larger bundle leaves out), and a line lists at most 24 titles. The map replaced the card's flat `source_files` list. **Point it at a folder and it sees every bundle under it** (since v1.1 F): ```sh okf card ~/okf # every bundle under the folder, each with its card ``` It prints what the server's `okf_list` and `okf_describe` give with no bundle named -- the listing (id, ref, concept count, directory), the directories that look like a bundle and cannot be read as one, and every bundle's card -- and it computes nothing of its own: the command calls the server's two functions. A bundle added or rebuilt under the folder is in the next run's answer with nothing regenerated. Pointed at one bundle, it prints that bundle's card as before; the command decides which it was given by the same rule discovery uses (a directory carrying an `index.md` is a bundle). **And one question -- or several sub-questions -- asks every bundle under the folder in one call:** ```sh okf consume ~/okf --question "first sub-question" --question "second sub-question" okf consume ~/okf --question "..." --bundle-id my-bundle # just one of them ``` The reply is the server's `okf_ask` with no bundle named, byte for byte: the budget split between the bundles, one payload per bundle, and every excerpt carrying the id of the bundle it came from. There is no ranking of its own. The flags that change how ONE bundle is cut (`--ref`, `--ranking`, `--no-source-quota` and the rest) are refused over a folder, by name, rather than dropped, because the server takes none of them; point at one bundle to use them. `okf check --payload` takes that reply as well as a single payload: every bundle's payload is held to every rule on its own, a finding is named with its bundle, and an answer labelled with a bundle its payload does not describe is a finding (`answer_misattributed`). The generic skill tells its reader both forms, and says to use the server's tools first where they are registered. `okf skill --for-bundle` still writes the per-bundle form, with the identity and the numbers measured into the text — which is exactly what makes that file stale the moment the bundle is rebuilt. It refuses out loud when it was not regenerated (`bundle_mismatch`), so its cost is a stopped session rather than a wrong answer; that is why it is no longer the default. Measured on two unrelated bundles: two per-bundle skills are identical on 281 of 313 and 311 lines. The 62 lines that differ are exactly identity, concept count, the conditional-field table, the whole-bundle cost and the payload-cost section — the five things a rebuild invalidates. ## Implemented scope (v1) The library provides three entry points for getting content into an OKF bundle: 1. **Spec-based ingestion.** An implementation of the normative ingest specification owned by `portfolio-optimiser-commons`: manifest → `file`/`sql`/`http` connector → deterministic materialization of `ingest-{id}.md` concept files → index generation. Zero model calls in the run path; output is reproducible byte-for-byte against golden fixtures. 2. **Bundle inbox.** A drop directory where common file types are converted to OKF concept files. All file-type→text extraction lives in this library: `md`, `txt`, `csv`, `json`, `html` and `xml` are handled by the stdlib core; `pdf` and the five office formats (`docx`, `xlsx`, `pptx`, `odt`, `rtf`) require the optional `[extract]` extra and are rejected fail-fast without it. Extracted text passes the security gate before anything is persisted. An `xml` file that declares NISO-STS structure (`` root, or any ``) becomes one heading per titled section, at the section's own nesting depth, with the section's `