open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity
llm-ingestion-okf/CLAUDE.md
Kjell Tore Guttormsen cb0c10b421 fix(retrieval-gate): the judge reads a passage as its exact reconstruction — M15 felled, row 7 15 of 15 
Row 8's judge required every delivered text to be the whole concept's bytes
(the row-6 identity), which is older than v1.1 C1: an excerpt `as_passage` cut
to its answering passage was class e even when its span carried the citation.
PM's re-measurement of e503f6a found that the dominant miss class of row 8 on
the real set.

`passage_span` accepts a passage only when the delivered text IS what the
bundle's bytes rebuild: the span is body[start:end] byte for byte in the
DELIVERED body (the offsets land a frontmatter's length off in the file), `of`
is that body's length, `[...]` stands exactly where text is left out, and the
one other line allowed is a heading line of the body above the span, or a
prefix of one. The citation is read in the span alone. A passage that is not
its reconstruction is class e with its own detail. A quote only in the heading
or the markers is class d.

Chose to check that the heading is one of the body's own heading lines above
the span, and not to re-derive WHICH heading `as_passage` picks, because the
guarantee is "the bundle's bytes"; the nearest-heading rule is the product's
presentation and a judge that copies it agrees by construction.

M15 (a passage carrying one sentence the file does not) is row 7's fifteenth
mutant. No synthetic concept was long enough to be cut, so a new fixture
delivers one (DELIVERY, ~6 600 characters, set set-passage.json); the corpus
pin moved with it. MUTANT_BAR unchanged, src/ untouched.

Gate, synthetic: rows 1 and 6 13/13 -> 14/14, row 7 14/14 -> 15 of 15 GREEN
(M15 -> row 1, 6), rows 2/3/4 unchanged, GATE RED: rows 5, 8. Row 8 rerun on
the real set: no class e miss remains (its numbers are kept out of this public
history, per the wiki directive). Suite after git add,
FORCE_COLOR unset: 2450 passed, 1 skipped (+6). ruff, format, mypy --strict
clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-22 23:53:09 +02:00

1919 lines
136 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# llm-ingestion-okf
## Context
Shared OKF (Open Knowledge Format) ingestion library. Three entry doors,
one boundary rule:
- **Door A — spec-based ingestion:** implements the normative
`ingest-spec.md` owned by `portfolio-optimiser-commons` (manifest →
`file`/`sql`/`http` connector → deterministic materialization of
`ingest-{id}.md` → index generation; zero model calls). This repo
IMPLEMENTS the spec; commons keeps authorship. Spec changes the library
needs go via commons, never edited locally. The library ships the §11
golden fixtures (byte-exact) for the three door-A source types
(`ingest-golden-{file,sql,http}/`, shipped in `9dd86b1`).
- **Door B — bundle inbox:** converts dropped files to OKF concepts. The drop
directory is walked RECURSIVELY, sorted by relative path, and a concept's
`source_file` is that relative path (`/`-separated) while its NAME still
comes from the basename — so a nested duplicate hits the §3 collision
refusal rather than vanishing. Dot-directories and a bundle nested inside
the inbox are skipped with a code, never silently, because recursion makes
the door's own output reachable as its own input (operator 2026-09-06; the
flat listing was not a boundary, it was an absence with no denominator). All
file-type→text extraction lives HERE (the guard is text-only). v1 core:
`md`, `txt`, `csv`, `json`, `html` (stdlib). **`html` got a measured
segmentation row 2026-09-10.** Until then `_HTMLTextExtractor.text()` was
`" ".join("".join(parts).split())`, and `str.split()` with no argument splits
on newlines too, so extraction of ANY HTML file returned unconditionally ONE
line while every boundary grammar in `propose` is line-anchored -- measured
outside this repo, **828 of 828** real sections gave 0 plans and exit 2 at
every sample point, and a coarser 145-document cut gave 145 of 145. Block tags
now open their own lines and `h1`-`h6` carry the ATX marker for their OWN level
(a flat `#` would hand `_ATX` three top-level boundaries where the document
declares one section and two subsections). The output grammar is MARKDOWN, the
same the office rows reach the proposer through, so **no HTML-only heading
grammar exists**; the fix is in the extractor and **never** the converter,
because `.html` stays out of `_PANDOC_FORMATS` on CVE-2025-51591. After:
**828 of 828 plans, exit 0, 3206 concepts / 6015 md -- the markdown path's
count EXACTLY**, and the same at 414 (1651) and 83 (343). Text preservation is
an EXACT invariant and not a percentage: strip the added ATX markers and the
non-whitespace sequence is identical to the old extractor's, **828 of 828
files**, character ratio **1.000000**. `_SKIP_TAGS` stays `{script, style}`.
Exposure elsewhere measured rather than argued: **0 of 86** K2 corpus files and
**0 of 5** smoke-folder files are HTML, and the smoke bundle is byte-identical
before and after. `_EVIDENCE` gains a `.html` row at `measured` -- and a `.pdf` row at
`measured` since 2026-09-10, the row with the most measurement behind it and
no entry in the table at all -- with the limit
that travels with it -- one product, one format, one publisher, and a
generator's cut, not 828 documents anyone wrote.
**`.xml` became a CORE type 2026-09-11** and it is the first row whose
ceiling is structural rather than recovered. A NISO-STS zip from a
publisher's own viewer was **110 of 110 unreadable, 0 plans, exit 2** --
`.xml` was in neither registry -- and the one xml file in it IS the whole
product: R761 Prosesskoden:2025, the document round 12 met as a 701-page
PDF, carrying 7 715 `<sec>`, **2 761 with a `<title>`**, 4 954 with a
`<label>` and no title, 10 `<table-wrap>`, root `<standard>`, 0 `<!DOCTYPE`.
Its `<sec>`-nesting depths over the titled sections are
28/118/500/1141/868/97/9, row for row the fasit's own. The reader is stdlib
(`xml.etree.ElementTree`) and adds NO dependency -- `defusedxml` and `lxml`
are 0 occurrences in `uv.lock` -- so it is core beside `.html` rather than
behind `[extract]`, which would make a pure-stdlib type binary-dependent.
**The output grammar is MARKDOWN**, the same the office rows reach the
proposer through: `<sec>` with a `<title>`
becomes one ATX line carrying `<label>` + space + `<title>` at its own
nesting depth, `<sec>` with only a `<label>` becomes a body line with the
label in front (never a heading -- 4 954 of 7 715 are lettered points and one
heading each buries the document's own 2 761), and `<table-wrap>` becomes its
label plus one markdown table through `render_table`. **`<label>` carries the
number and `<title>` carries the text** -- 2 of 2 761 titles begin with a
digit -- so emitting `<title>` alone scores 0 of 2 761 with nothing in the
code looking wrong. Inline is an ALLOWLIST and block is the default, the
inverse of the HTML reader, because block-versus-inline is a property of HTML
and XML has no such universal; the allowlist is load-bearing at 1 701
`<italic>` and 1 396 `<bold>` inside that document's prose. The ATX ceiling is
6 and STS nesting reaches 7, so the depth is CLIPPED and not dropped: 9 of
2 761 sit at depth 7 and `#######` matches nothing. Since K3-21 the clip is
the HEADING's alone: the `OutlineMark` beside it carries the declared depth,
so the plan reads those 9 at 7. A `<!DOCTYPE` is REFUSED
unparsed with its own code, a guarantee about the code rather than about the
machine -- measured on 3.14.0 with pyexpat 2.7.3, an external SYSTEM entity
is refused by the stdlib but the billion-laughs limit comes from libexpat
>= 2.4.0 and not from Python, while `pyproject.toml` requires only `>=3.10`.
XML that is not STS keeps its text in document order and gets NO invented
structure, and `.xml` never routes through the converter -- a second parser
that would never see that refusal. **XML that is not STS also gets 0 plans
and a FAILED build (exit 2), and that is NOT an `.xml` defect**: a folder
holding one `.txt` of prose with no headings gives the same three lines and
the same exit, so it is general `okf build` behaviour for any structureless
document. The gate stays -- a run replaying zero plans would emit a flat
bundle and call it success -- because separating "0 plans, 0 unreadable" from
"0 plans because nothing could be read" changes the outcome on **0 of the 4**
reference corpora.
**THE READER REACHED ITS CEILING IN ROUND 13 AND THE BUILD DID NOT, AND
ROUND 14 CLOSED IT AT THE SHIPPED DEFAULTS.** The reader emitted 2 761 of
2 761 heading lines while the build delivered **23 concepts and 15 of 2 761
boundaries** -- two steps after the reader, each measured: the orphan check
took **710 of 2 761** (710 of 710 removed headings are followed immediately
by another heading, **0 of 2 051** delivered ones are -- they are container
sections) and Arm F took **2 066** more, 2 089 -> 23. `find_candidates`
already skipped both for `outline_marks`, which is why the PDF bookmark arm
reaches 2 762; an STS `<sec><title>` is the same class of declaration and
only arrived as `rule:heading`. The fix is ONE new rule constant reached from
ONE row: `extract.xml_outline` reports the marks the reader WROTE ITSELF --
**no bridge, no tolerance constant, no `unresolved` bucket**, the difference
from `pdf_outline` whose naive nearest-line rule was wrong on 1 840 of 2 762
-- `propose.RULE_XML_SECTION` (`rule:xml-section`) is its own name in
`RULE_NAMES` and `_ORPHAN_EXEMPT`, and `build_plan` chooses the route by the
**ROW** (`DECLARED_STRUCTURE_IDS`), never by the text: the same markdown from
a `.md` file is still a guess and still carries `rule:heading`. At shipped
defaults, no flag: **2 761 concepts**, **2 761 of 2 761** declared sections
became a concept with the source's own directory AND title, **0** concepts
matching no declaration, `a)`-points **0 of 4 954**, table blocks **10 of
10**, hit@1/8/50 **3/6 / 5/6 / 6/6** from 0/6 / 0/6 / 0/6 with the
known-positive at rank 1, and **2 761 shared concept ids** with the PDF arm
(100 % of this bundle, 2 761 of 2 762 of that one) against round 13's 2 022.
**NO other file type changes one byte and it is MEASURED on the bytes**: the
whole 43-document reference corpus rebuilt is `diff -r`-identical to the
pinned bundle (865 md), the five-document folder is `diff -r`-identical,
`okf project` stays byte-equal to `okf build`, and the PDF arm still proposes
2 762. Two directories of 2 738 still hold two concepts (`11`, `12`) -- the
publisher reuses a section number, the same 2 the PDF arm has, and 0 is not
reachable without inventing an id; round 13's 14 such directories were false
positives of the TEXT route reading the document's own contents listing and
are gone. Report:
`docs/2026-09-10-k3-runde14-deklarert-struktur-tar-ruten.md`.
**Since K3-19 an STS document's own identity names its directory**
(`extract.declared_identity`, read by `cli._document_prefixes` and the door):
the directory was the delivery file's stem, a UUID occurring **0 times** in
the document, while its one `<std-ident>` carried `<doc-number>`. Only the
stem is replaced, and a declared name two documents in one run claim is
used by NEITHER -- the `slug_owners` gate would refuse both with "rename
one", which a name read from inside a document cannot obey. The `sources`
title is `<doc-number>` + `<year>`, then `<title-wrap>`, then the file name:
R761's `<full>` carries a COMMA, a flow terminator, so it is never written
and never cleaned up. A titled section's `description` is its own FIRST
spec point (first `<p>` of the first DIRECT-child `sec-type="spec"`, whole),
carried by the plan entry, screened by the gate, and written only where a
YAML reader reads it verbatim (`inbox._yaml_plain`): 2 026 of 2 761 titled
sections on R761 carry a point, **1 807** are written (2 have no `<p>`, 217
carry `: ` and PyYAML refused exactly those frontmatters), none invented.
SS 4.1 sets no length, so the one-paragraph limit is ours. **The directory
name reached the RANKING, and K3-20 closed it in `consume`:** signal 1 read
every id segment, so on a one-document bundle every concept carried the
document's own name, and a question naming the document matched all of them
-- measured, the known-positive went rank 1 -> not delivered at the default
`k` (13 at k=50) with S1-S6 unmoved. `consume.shared_id_prefix` now keeps
the leading directories EVERY id shares out of that signal: KP rank 1 at
both `k`, S1-S6 6/6, and K2 (12 payloads), N100/N200/N500 (15) and the
five-document folder (5) byte-identical, because ids that share no prefix
read exactly as before. Dropping each concept's OWN document directory
instead was measured and felled -- a K2 hit@8 row went rank 5 -> not
delivered. Reports:
`docs/2026-09-11-k3-runde19-dokumentidentitet-og-frontmatter.md` and
`docs/2026-09-11-k3-runde20-delt-katalog-og-arvet-kontekst.md`. The
registries are COUPLED: a row in
`_CORE_EXTRACTORS` and not in `segmentation._STDLIB_EXTRACTOR_IDS` refuses
every proposal for the type, two layers away from the extractor.
`pdf`/`docx`/`xlsx` only via
the optional `[extract]` extra; without it those types are rejected
fail-fast. The extra ships `pdfplumber` for `pdf` (chosen on ONE measured
property: it keeps a requirement table's label and value on the same line
where three alternatives do not); `docx`/`xlsx` still ship no parser.
`pptx` and `md` were measured end to end for the first time 2026-09-10
(`docs/2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md` § 4) on two
hand-built documents, which is more than zero and is not a fasit: `md`
recovers 3 of 4 declared headings, and `pptx` segments per slide only where
the deck's slides carry title placeholders the converter recognises — a deck
whose slides do not lands as ONE concept. A converter attribute also leaks
into concept titles (`{#slide-N}`, `{#sheet-1}`), reaching 2 of 810 files on
the K2 default bundle and 1 of 30 on the operator's test folder; because a
filename is reduced from its title, fixing it RENAMES concept ids a consumer
has already cited, so it is an operator question and not a patch.
Structured table recovery is **out of scope** — two independent parsers
return the same wrong shape, so the breakage is document geometry, not a
library choice. PDFs enter as prose, and drawn content (figures) does not
survive extraction at all, which every `pdf` extraction warns about.
Under the `STRUCTURED_V1` profile Door B additionally DERIVES structure —
title (leading heading → `title` key → `path.stem`), document number,
hierarchy, and cross-references — writes it into the concept frontmatter, and
projects it into a faceted index entry. Every inferred field is named in a
`derived` list; an unmarked heuristic is worse than none.
Under the SEGMENTED v0.2 profile a concept additionally POINTS BACK at the
original: `sources: [{ resource, title }]` in SPEC §5.1's form (`resource` is
the inbox-relative path), plus a locator per format — `source_pages`,
`source_sheet`+`source_rows`, else `source_lines`. The locator keys are OURS
and must stay top-level: §5.1 has no field for a place within a resource, and
the pinned guard rejects every route to putting one inside a `sources` entry
(non-allowlisted key, nested flow list, quoted scalar), so a locator in the
entry would emit bundles Door C could never read back. The unit table is
built AT EXTRACTION — a page number cannot be recovered from joined text —
and `source_offset` stays. `source_lines` indexes the EXTRACTED text, never
the original's paragraphs: measured, docx `<w:p>` counts and converted-line
counts do not agree on a single one of five documents. Record:
`docs/2026-09-08-proveniens-k2.md`. The index is a
PROJECTION recomputed from the whole bundle each round, which is what makes
rebuild-from-scratch equal an incremental update byte for byte. `DEFAULT` is
untouched and byte-identical. Record: `docs/plan/structure-derivation.md`.
**SINCE 0.10.0 DOOR B CARRIES THE IMAGES ITS SOURCES DECLARE.** Until then no
reader here fetched, named or copied one -- `<img>`'s `attrs` were never
read, an STS `<graphic>` was walked past, a PDF was opened for text alone,
the converter's markdown writer dropped every picture, and the only writer
into a bundle was `materialize.write_bytes(..., content: str)`. Measured on
R761 Prosesskoden:2025: the process text is carried in full while **12
`Tabell N-N` and 9 `Figur N-N`** captions stand over nothing, because that
publisher ships those tables as raster pictures in BOTH the PDF and the
NISO-STS delivery -- process 84's "toleranseklasse ... er gitt i tabell 84-2"
points at empty space. **Five readers PLACE and one module DECIDES:**
`assets.py` owns what an image is (sniffed from the bytes, never from the
claimed extension), what it is called (`<sha256[:12]>-<the source's own
BASENAME>`, so one image reached by two paths is one file) and how it is
pointed at (one two-line block, one regex, `IMAGE_POINTER`, which is what
`okf describe` will find its work with). The pointer is a markdown image at
`/assets/<name>` -- bundle-absolute, because a segmented bundle puts concepts
at different depths -- followed by one line carrying the source's own file
name and the size in px. **`.xlsx` is deliberately NOT a row**: its converter
writes one pipe table per sheet and a two-line block inside one would break
the `source_rows` locator read back out of it; 0 of 4 K2 workbooks hold any
media, so it is a stated limit and not a loss taken. **A PDF stream that is
already a file is carried VERBATIM** (`DCTDecode`, `JPXDecode` -- 29 of
R761's 50 objects), and raw samples are encoded to PNG with stdlib `zlib`.
Rendering the page region was the alternative and was FELLED on
determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the installed
rasteriser -- the one property `OCR_DPI`'s docstring already admits OCR text
cannot have. What the encoder cannot express EXACTLY (stencil mask, `Decode`
array, CMYK, anything but 8-bit samples) is refused with a code and counted,
never approximated. **NO SIZE FLOOR, and that is a measurement:** over the
4 828 image objects of the K2 corpus the distribution is 149 / 162 / 92 /
406 / 498 / 590 / 2 931 across the size buckets -- a broad spread with no
gap, unlike `OCR_CID_SHARE`'s bimodal one, so a threshold would be a number
this package chose. **ON by default; `--no-assets` reproduces the pre-0.10.0
bytes.** Cost measured on the 43-document reference corpus, two builds of one
commit: **453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets (2 964
carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ.** The ONE new
concept has a measured cause and not a guessed one: the pointers are body
text, so a section holding 146 of that document's images grew from **19.0 %
to 30.6 %** of the extracted text and crossed `--outline-gate`'s 0.20 share
clause. **THE PLAN AND THE RUN MUST AGREE**: a plan records `text_sha256` of
the exact string it was proposed against, so `propose` and the door take the
same `assets` value and each computes the SAME resolver root independently --
the document's own directory, containment by `connectors.safe_resolve`. A
reference above it is refused (`asset_unresolved`), a remote one is never
fetched (`asset_remote`, extraction opens no socket) and both leave a line in
the concept saying what was there. **THE IMAGE BYTES ARE NOT SCREENED** --
the guard is text-only, the pointer block passes the gate as body text, the
picture beside it passes nothing -- and `log.md` says so on every run.
- **Door C — external bundle import:** third-party OKF bundles are assessed
per concept via the guard's `okf.import_bundle`; only concepts clearing the
guard's non-blocking floor are merged/indexed here. Two invariants, both
load-bearing: a merged concept is written **verbatim** (this library's
line-oriented frontmatter parser cannot round-trip the block lists the
guard's parser accepts, so stamping an external concept would destroy sender
data and persist bytes the guard never screened), and ownership is therefore
proven by **content identity** — an occupied target name is re-used only
when the bytes there are already identical, never overwritten otherwise.
**Since 0.10.0 it also carries the ASSETS its merged concepts point at**, by
that same content-identity rule. Measured before the repair: a bundle built
with `--assets` imported as **6 of 6** concepts and **no `assets/` at all**,
so every pointer in the imported bundle named a missing file — the same
"complete and not" defect one door over. POINTED AT, never every file in the
sender's `assets/`: an asset belonging to a concept the gate refused must not
ride in on the back of one it cleared, and an asset nothing names is a file
no retirement pass reaches. A pointer whose asset the sender did not ship is
left alone, because SPEC §6.1 requires a consumer to tolerate a broken link
and a pointer recording a figure nobody holds is information.
**`okf build` RUNS a real guard and NAMES it in the bundle (F1, 2026-09-15).**
From the day the command was packaged until then, `corpus.measure` wired an
unconditional approve-everything stub into `process_inbox` and **0 of 90**
`add_argument` calls in the package named a gate — so the one path most people
use screened nothing, while `pyproject.toml` made the guard a MANDATORY runtime
dependency and the README recommended a composition the command line could not
reach. Reported from outside by `claude-code-llm-wiki`, reproduced here first.
`--gate` takes `guard-trusted-source` (default), `guard-user-upload` or `none`,
`corpus.resolve_gate` is the ONE name→callable map (guard imported lazily, so
importing the package still does not pull the dependency in), and an unknown
name RAISES (`gate_invalid`) rather than falling back — a fallback reproduces
the defect with an extra step. **The default was chosen on a measurement:** over
the 453 concept bodies of the pinned reference bundle, `PRESET_TRUSTED_SOURCE`
persists **453 of 453** and `PRESET_USER_UPLOAD` holds **1**, costing that
concept's whole source document (1 of 39) — and neither tier waves anything
through, an invisible carrier and a CRITICAL finding are `fail_secure` at
BOTH. Door B's library default is UNCHANGED at `PRESET_USER_UPLOAD`: an inbox
drop is an untrusted upload, an operator pointing this command at their own
folder is not. The second tier is
`guard_adapter.inbox_gate_trusted_source`, the three-line adapter that module's
docstring already described — never a preset parameter. **The gate's NAME is
written into the §9 `log.md`**, because a stub is only dangerous when nothing
downstream can see it; `--gate none` renders `NOTHING WAS SCREENED`. The corpus
harness carries the same flag and the SAME default (a test holds the two paths
byte-equal); `okf project` takes none, owning no flag that moves bytes. The
composition `process_inbox(segmentations=..., gate=inbox_gate)` now has a test —
before this, `grep -rl inbox_gate tests/` gave **1** file with **0** occurrences
of `segment`, which is how the defect survived.
**AND THAT GATE WAS SILENT ABOUT WHAT IT DROPPED, UNTIL v1 (2026-09-20).**
Naming the gate in `log.md` said which screen ran; it did not say what the
screen took. Measured at the shipped default against a real corpus of official
documentation: `guard-trusted-source` refused a minority of sources OUTRIGHT,
under `fail_secure` and `quarantine_review`, and most of those were ordinary
reference pages. Rebuilt with `--gate none`, every one of them went through
untouched, so the refusal is the GATE and not the readers: official
documentation naturally carries commands and instruction-shaped text, and the
guard reads that as something to hold for review. The summary printed one
`` `fail_secure`: N/D `` line and nothing else, so **three of the four facts a
reader needs were absent**: the COUNT of documents the gate dropped (the existing `rejected
(coded)` line SUMS gate refusals and extraction failures -- two failures with
two different remedies, which is why `gate_refused` is its own column on
`CorpusReport`), the NAMES, and the way out. `corpus.REFUSED_HEADING` now opens
a section carrying all four, directly under the denominator, and
`refusal_banner()` says the same thing in one line on stderr because `okf build
> report.txt` is ordinary; both read the SAME field, so the loud line and the
summary cannot come to disagree about a number. The per-name list is capped at
`REFUSED_NAME_CAP` = 10 and `log.md` gains one bullet naming **every** refused
document, uncapped -- a cap is only honest if the rest is somewhere a reader can
go, and §9 already holds the one fact about a run the bundle cannot otherwise
recover. **THE EXIT CODE DOES NOT MOVE**: the build is valid, every refusal is
coded, the conservation identity holds, and the bundle is a true record of what
the gate allowed; what was wrong was the silence, not the status. A run the gate
refused nothing from is byte-identical in both places -- the known-negative in
`tests/test_gate_refusal_is_loud.py`, and the reason this cannot have moved a
byte measured here: **0 of the bundles this repository ships** was built with a
gate refusal. The gate's SEMANTICS are untouched and the guard is not touched
from here (another repo, another release, its own security assessment); that a
trusted-source tier should CARRY a `quarantine_review` document marked rather
than throw it is a candidate for the round after v1, not work.
**v1.0.0 (2026-09-20) IS THE SCOPE THIS TOOL IS FINISHED AT.** Operator
20.09.2026: «Vi må lukke OKF-jobben så snart vi praktisk kan. Vi gjør bare det
vi MÅ for at den skal være på v1. 3D visualisering er ikke med der.» It adds NO
capability over `v0.10.1`; what it adds is that the tool says what it does not
do -- the refusal section above, and a `Known limitations` section on the front
page carrying only numbers already measured (the gate's refusals and the way
out; the absent ceiling on what ONE RUN pays for images, where each decode link
is bounded at `MAX_FILTER_DECODE_BYTES` = 512 MiB but a 70 KB PDF with 16 images
each under the declared limit reached **851 MB peak RSS** and `RLIMIT_AS` is not
enforceable on Darwin; the three gates of this repository that are RED today --
retrieval **5, 7, 8, 9**, MCP **2**, accounting **2, 3, 6**, all three re-run on
the closing commit; what the content accounting does not count; and that nothing
here draws a bundle). The two `pip install` lines under "Install in detail" now
install `[extract]` -- the first screen did and those two did not, so the two
recipes produced different installations and the detailed one reported
`extractor_extra_missing`. After this tag the library is touched for defects
found in its OWN use; the next planned round is Google OKF v0.3, triggered by
v0.3 shipping and not by a date. Destination, frames and the candidate list live
in `PLAN.md` (local-only: `origin` is the public `open/` repo).
**A FENCED CODE BLOCK DECLARES NO STRUCTURE (F2, 2026-09-15).** The proposer
read every line with the same grammars, so `# Use the opus[1m] alias` inside a
```` ```bash ```` fence became a level-1 ATX heading. Two effects and the
SMALLER one was visible: the document was REFUSED entirely when the line carried
`[` or `]` (**5 of 191** pages of the reporter's corpus, `inbox_title_invalid`),
and the concept TITLE was silently taken from somebody's shell session on
**62 of 191 (32.5 %)**. The fix is in the PROPOSER and **never** in Door B's
title rule — that rule is right, and a heading that was never a heading is what
has to stop being proposed. `propose.fenced_lines` is computed once per text and
NO rule reads a fenced line: not `_ATX`, not the numbered grammar, not a table
row, not `--bold-title`, and not Arm D's outline RUN, which selects from the
whole line list (filtering only at admission would let a fenced install listing
decide which run wins). Four CommonMark § 4.5 details are load-bearing, each a
way to remove REAL boundaries: three leading spaces still open a fence; a
backtick fence's info string may not contain a backtick (or a line holding only
`` `okf build` `` silences the document); a closing fence must be at least as
long as its opener; an unclosed fence runs to the end. **It lands
unconditionally, not as an eleventh flag, and the exposure is measured on the
bytes:** **0 of 865** concept files in the pinned default bundle and **0** of the
shipped fixtures and goldens reaching the proposer carry a fence of either kind,
so a rule that can only fire INSIDE one cannot have moved anything measured
here. It is a defect, not a default move.
**BOTH CHANGES TOGETHER MOVE ONE LINE, AND IT IS MEASURED ON THE BYTES:** the
43-document reference corpus built at `b6da09c` (from `git archive`, never the
editable tree) and rebuilt at the shipped defaults differ in `log.md` alone, by
the added `**Gate**:` bullet -- **865 concept files on both sides, every
concept byte-identical**. The same run found something this work did NOT cause:
the pinned artifact `K2-bundle-default-20260912` was written **2026-09-09
21:38**, two days before `ed0418f` (K3-22) changed `title:` quoting, so it
differs from what HEAD produces on **42 concept files** -- and
`tests/test_default_bundle_pin.py` stayed green because it pinned the count and
the hit@8 ranks, not the bytes. (That pin was REMOVED 2026-09-21 with the
retired K2 test track, v1.1 K; the artifact is no longer read by anything.) Report:
`docs/2026-09-15-f1-f2-gaten-og-kodefencen.md`.
**Boundary rule (non-negotiable, zero overlap):** `llm-ingestion-guard`
(pinned `>=1.2,<2.0`) answers "is this content safe to persist?" —
scan/sanitize/quarantine/fail-secure/provenance-stamp. This library is
plumbing: connect source → materialize deterministic OKF bundle → generate
index. Never reimplement security; call the guard at persist gates
(`prepare_input`/`screen_output`, `okf.import_bundle`). When in doubt which
side of the boundary something belongs on: ask the operator.
**Implementation baseline:** the stricter behaviors from
`portfolio-optimiser` (streaming row caps, utf-8-sig, in-memory staging with
pre-mutation collision gate, validated `ingested_at`, typed `IngestError`)
are the library baseline. First consumer: `portfolio-optimiser-claude`.
### Roadmap (phases 1–3 shipped; what follows is demand-driven)
1. **Phase 1 — Door A (Python).** ingest-spec implementation + the §11
golden fixtures. Consumers: `portfolio-optimiser-claude` first, then
`portfolio-optimiser`.
2. **Phase 2 — Doors B/C (Python).** Bundle inbox and external-bundle
import, guard-gated.
3. **Phase 3 — Configurable bundle contract.** Types, layers, frontmatter
sets, index shape, and reserved-file policy become config instead of
constants; proving consumer is `claude-code-llm-wiki` (`strict-v1`
profile). Two consumers hold opposite postures on whether an index is
authored or directory-derived, so neither is a library invariant and
nothing here enumerates a directory unless the profile says derived.
4. **Phase 4 — Node half (`node/`).** Zero-dependency Node/ESM package
(importable *and* CLI-invokable, vendorable per plugin — matching the
marketplace precedent) for the second-brain world: bundle check, index
generation, inbox split/frontmatter/write, and doc conversion
(docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect,
and the marketplace catalog.
5. **Phase 5 — MCP as a way to populate a bundle. NOT COMMITTED; needs-based
(operator 2026-08-02, superseding the 2026-07-27 commitment.)** No MCP work,
and no data-lake or database source types, are undertaken without a stated
need. `docs/plan/mcp-bundle-population.md` stays as a design record, not a
queue. Its open fork — whether we are the MCP **server** (an agent calls our
doors as tools) or an MCP **client** (a manifest source type pulling from
someone else's server) — no longer blocks anything, because nothing waits
behind it. It is a question to answer *if* a need arrives, not before. This
is also why `sql` staying sqlite-only is not a gap: a Postgres driver would
be runtime dependency number two, bought for no asked-for use.
The two halves share the OKF contract and fixture suite, **not code**.
**Standing posture (operator 2026-08-02).** Phases 1–3 shipped; the library now
runs on what it has. Work is defect fixes, improvements, and features that a
consumer has actually asked for or that measured feedback shows are needed —
not roadmap completion for its own sake. The upstream version policy below is
the one exception, and it is not a counterexample: "always latest" is a promise
already made to consumers, so an upstream release *is* the stated need.
Phase 4 keeps four named consumers with working implementations to lift, so its
need is real but untriggered — it starts when one of them asks, not on a date.
### Upstream version policy (standing, non-negotiable)
**The library always supports the current latest version of Google OKF.** Set by
the operator 2026-07-26. Phases 1–3 were built against v0.1; v0.2 shipped
2026-07-25, so v0.2 support is committed work — not contingent on a consumer
asking for it. Plan: `docs/plan/okf-v0.2-alignment.md`.
Support is **additive, expressed as a new profile**, never a migration of
existing ones. This is what makes the policy sustainable instead of a recurring
crisis, and it is bounded by three facts that do not yield to it:
- `DEFAULT` states commons' ingest-spec §5 layer — its `generated` shape is
commons' call, raised there, never patched locally. **This fired 2026-08-09:**
commons ratified and executed the O2 form, so `DEFAULT` now stamps
`generated: { by: process:okf-ingest, at: <ingested_at> }` and four goldens
moved with it. It is not a counterexample to "additive, never a migration" —
that rule governs *upstream* versions, and commons' spec is a separate axis
`DEFAULT` tracks by definition. `DEFAULT` stays v0.1 on everything upstream
owns. Ownership recognition is one-way, so the cost to a consumer stays a
re-run: a profile carrying an actor still owns the older literal stamp.
- `STRICT_V1` mirrors the proving consumer's ratified contract — changing another
repo's contract from here violates O2.
- `okf_version`'s *value* belongs to catalog (decision E1).
**Rollout is pilot-first.** A new upstream version reaches a small pilot set on a
pre-release tag and is revised on their feedback before general availability —
consumers testing real data find what fixtures cannot. `OKF_LATEST` means the
latest version supported as *stable*, so flipping that alias is the GA event, not
a merge side effect.
Two invariants fall out: no profile hard-codes an upstream version, and no bundle
declares a version its shape has not earned. The first has a mechanism, not just
an intention: **a profile names a key, a caller owns its value.** `okf_version`
is declared through `materialize_bundle(..., root_frontmatter_values=...)`
because its value tracks the upstream Google version and belongs to catalog
(decision E1) — a constant here would claim a decision we do not own, and would
be the one thing to chase on every upstream release. Where upstream itself defers a
contract — v0.2's attestation receipt and verdict wire formats — the format is
supported and the unspecified runtime is not; it re-enters scope when upstream
specifies it. Because "always latest" decays silently, the release checklist
carries an upstream-version re-check.
**Structured frontmatter values are emitted in YAML *flow* form, never block.**
Both can be valid YAML -- within the flow-scalar limit below -- and an upstream
reader then recovers the same structure from either, but this library's parser
is line-oriented: it round-trips a flow
mapping as an opaque value and cannot read the block form at all — two block
mappings sharing an inner key (§10.2's `executor` and `attester`, both carrying
`resource`) collapse into one namespace and the first is lost silently.
Emitting block would produce bundles we cannot read back. Reading it needs the
structured reader (D1b); until then the constraint binds what we write.
**Every value is written so a YAML reader reads it back the same (K3-22).**
SPEC § 11 point 1 requires "a parseable YAML frontmatter block", and before
K3-22 the pinned K2 default bundle failed PyYAML on 41 of 455 blocks and each
R761 build on 1 -- block scalars written verbatim. A block scalar that is not
plain-safe (`profiles.yaml_block_plain`, K3-19's rule) is now written
double-quoted with `\` and `"` escaped; every other value keeps its bytes. A
FLOW leaf has no quoted form -- the pinned guard refuses any quote inside a
flow mapping -- so a leaf PyYAML would refuse or misread (`?`, `": "`, `" #"`,
a quote, a leading indicator) is refused by `profiles.yaml_flow_plain` with the
door's existing code, never written. Readers unquote a `"`-wrapped value only:
0 such values existed in any measured bundle, while 11 193 `'`-wrapped ones do
and stay untouched. PyYAML is a dev dependency that validates the rules in
`tests/test_yaml_frontmatter.py`; `src/` imports no yaml. Report:
`docs/2026-09-11-k3-runde22-yaml-lesbar-frontmatter.md`.
**Every upstream release runs `docs/upstream-okf-upgrade-runbook.md`.** Pin the
commit, enumerate the whole `okf/` tree, **read the shipped example bundles and not
only `SPEC.md`**, classify the diff, measure our exposure and each consumer's, plan
additively, pilot before GA, then inform every OKF-consuming repo. The runbook is
not optional and not a summary of good intentions: each step names the concrete
failure it prevents, and all of them are failures that happened during v0.1 → v0.2.
**This repo is a black box for its consumers.** The target cost of an upstream
release to a consuming repo is **a re-run, nothing more**: support is additive
(a new profile, never a migration), existing profiles stay byte-stable, new public
parameters are keyword-only with defaults so positional call sites stay
source-compatible, and consumer golden fixtures must not churn. The boundary is
stated every time rather than glossed — the library absorbs *shape* changes, not
upstream changes to content a consumer authored (v0.2's `timestamp` and
`# Citations` supersessions). For that class the deliverable is a measured exposure
report per consumer, sent before they ask.
Phase 4 preconditions (coordination, not unilateral moves):
- Lifts okr's reference implementations (`okf-check.mjs`, `okf-index.mjs`,
innboks libs) in agreement with okr and the marketplace catalog; the
catalog remains the convention owner and re-pins its shared gate here.
- linkedin-studio's `ingest/published/` provenance-record grammar stays
plugin-local by design (different lifecycle) — do not normalize it.
- The Node-side persist gate remains security territory: guard-as-contract
(per okr's adoption doc) until a Node guard exists in the security repo.
No security reimplementation here, in either runtime.
### Non-goals (all phases)
- Verdict/feedback machinery (method-spec) — stays in consumer repos.
- Embedding/RAG/retrieval layers.
- Security functionality — always the guard's domain.
## Stack
Python 3.10+. Package `llm_ingestion_okf` (src layout, hatchling).
**Exactly one runtime dependency, ever:** `llm-ingestion-guard>=1.2,<2.0`
(itself zero-dep), landed with the Door B/C persist gates. Everything else is
stdlib, and a packaging test enforces it. Only `guard_adapter.py` imports the
guard; importing the package does not. Install channel until the package
index exists (a direct reference is a channel, not the pin):
`pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.2.0"`.
Binary extraction parsers live behind the `[extract]` extra only — today
`pdfplumber>=0.11.10,<0.12` for `pdf`. Extracted PDF text is pinned to an
exact transitive parser version (`pdfminer.six==20260107`), so widening that
range is a fixture migration, guarded by a frozen literal in
`tests/test_extract.py`; see `tests/fixtures/README.md`.
Phase 4 adds a `node/` half: Node/ESM with zero npm dependencies
(`node:` builtins only), both importable and CLI-invokable, consumed by
vendoring per plugin rather than npm publishing. The halves share contract
and fixtures, never code.
## Conventions
- Type hints everywhere; `mypy --strict` target.
- Determinism is bit-exact: `ingested_at` is an explicit required argument
(no wall-clock defaults); LF-only output; golden fixtures compared
byte-for-byte.
- Filenames and titles are normalized to Unicode NFC before use
(`materialize.reduce_to_id_grammar`, `inbox.process_inbox`): macOS/APFS
hands filenames over in decomposed form, so an `é` arrives as `e` +
combining acute. Without normalizing first, the same visual name (e.g. a
Norwegian slugger title like "linkedin-studio") reduces differently
depending on which form it arrived in, splitting one title into two
generated filenames.
- No model calls anywhere in the run path.
- Credentials only as env-var *references* resolved at runtime; never in
manifests, logs, or frontmatter.
- Network access requires an explicit per-run opt-in flag; refuse fail-fast
otherwise.
- Conventional Commits: `type(scope): description`.
- English for all code, docs, and commit messages (public repo).
## Commands
- Test: `pytest`
- Lint: `ruff check .` + `ruff format --check .`
- Type check: `mypy --strict src/`
- Folder to questionable bundle in ONE command: `okf project <folder>` —
`okf build` with the package default into `<out>/.okf/<id>/` plus `okf skill`
into `<out>/.claude/skills/okf-consume-any/`, `<out>` defaulting to cwd and
`<id>` to the folder name reduced to `[a-z0-9-]`. **Since v1.1 ordre A
(2026-09-21) the skill directory does NOT carry the bundle id**, because the
skill it writes is the GENERIC one: a second bundle in the same project
reuses it instead of installing a second document saying the same thing about
different numbers. It owns exactly ONE flag that moves a bundle's bytes,
`--gate`, added the same day — the gate is a screen about whether a document
may be persisted at all, not a rule about how one is cut, and a command that
could not reach it screened by a default nothing said was a choice. Every
other `okf build` flag stays absent and a test holds the two byte-equal at
the shipped defaults; two build paths would leave every measurement report
pinned to a bundle nobody produces. **That invariant was FALSE from the day those two
flags became defaults until O6 measured it, and the test could not see it:** `cli.build`'s Python SIGNATURE defaulted
`keep_table_heading` and `sheet_section_rows` to `False` while argparse
defaulted both to `True`, and `project.create` calls `build()` as a function,
so it read the signature. Measured on a five-document folder, `okf project`
wrote **15 concepts / 30 files** against `okf build`'s **26 / 52**, the whole
difference in the priced spreadsheet -- the document a question about price
has to reach. The byte-equality test compared `project.create` against the
same `build()`, so both sides carried the same wrong value, and its two
fixture documents had neither a table nor a sheet: **a test and the code
agreeing over a set where the difference cannot appear.** Two tests now hold
it -- one comparing the signature's defaults against argparse's for every
same-typed parameter, one building a document whose concept count actually
moves with the two flags. `skills/okf-prosjekt/` is the Claude Code skill
over it.
- **The generated consumption skill states THREE modes and RELATIVE paths**
(O6, 2026-09-09). Question (the default), hypothesis (decomposed into
premises and answered PER PREMISE as `confirmed` / `refuted` /
`undecidable-from-bundle` -- three literals, no fourth; a weak source is
`[sourced-not-sufficient]` on that PREMISE, because four premises and one
weak source is three answers and one gap), and a task producing a document
(source per claim IN the artefact, an ungrounded paragraph written and marked
rather than dropped, the cut declared inside the document because the
document travels without the chat). The five markings are untouched -- the
modes add no sixth. In the layout `okf project` writes, the commands are
`okf consume .okf/<id>` and `okf check --skill
.claude/skills/<id>-consume/SKILL.md`, runnable from the project root, which
is where `okf project`'s own closing line tells the reader to start `claude`;
a path OUTSIDE that root stays absolute on purpose, since `../../..` is not
more portable, only harder to read. Two absolute paths to zero -- and
**O5's published "4 absolute paths -> 0" was measured with `grep -c "^/"`
against paths indented by two spaces**, a query that could not have found one
either way, so the zero was never a measurement. Every path assertion here
runs its pattern against a known-positive first.
- Build a bundle: `okf build <folder> --bundle <dir> --bundle-id <id>
--okf-version <v>` — the installed console script (`[project.scripts]`),
the packaged form of what used to be a shell loop over two `tools/`
scripts. It is orchestration only: the proposer and the corpus harness
live in `llm_ingestion_okf.propose` and `llm_ingestion_okf.corpus`, and
the `tools/` scripts are thin entry points to the same functions so the
published reproduction blocks still run. Path scope for a document's
proposals is its RELATIVE path minus the extension (the door walks
recursively, and two same-named documents in different folders must not
collide); `--ingested-at` and `--proposed-at` default to one shared epoch
constant rather than the clock, because a wall-clock default takes
rebuild-equals-incremental away from anyone who omits them.
**TEN segmentation rules are REACHABLE here, and since 2026-09-09 ALL
TEN are ON by default** -- the tenth is `--contents-name`, round 9's repair
of clause 1, which admits a title into a contents run only when a NAME
survives stripping its page number. Measured: clause 1 discarded 68
candidates over 11 of 39 readable documents, 19 of them over 5 documents
rows of a drawing's dimension chain, a schematic's labels, a door schedule,
a coordinate column and a soil-layer table. The threshold is SWEPT
(`propose.CONTENTS_NAME_RUN = 2`) and collapses at both ends: at 1 it
rescues 13 of 19, at 3 the two-letter section name `VA` stops being a name
and takes four REAL contents entries with it. At 2 it rescues 16 of 19 and 0
of 49. Corpus 429 -> 447 candidates, K2 436 -> 453 concepts / 832 -> 865 md
(`21af4a1aa98315cf...`), the 12-position reference label-identical in BOTH
readings and hit@8 `[1,1,1,1,1,None]` on the new bundle AND Arm B. Opt-out
`--no-contents-name`.
**An ELEVENTH flag, `--bold-title`, is OFF** (round 10, 2026-09-09): the rule
for the type whose container declares nothing. `rtf` measured 0 of 0 declared
headings, 0 concepts, 1368 of 1368 characters in no segment. The grammar is
MARKDOWN, not `rtf` -- the converter already writes the author's bold title
as `**...**` in the same output every office row produces, so no `rtf`-only
heading grammar exists, the same shape of decision as the PDF font reader's
ATX form. Three parameters swept over 47 readable documents and ONE carried
(refusing a line that ends in terminal punctuation: false-positive lines
9-12 -> 1-2); a maximum title length and a stand-between-blank-lines clause
are both FLAT and neither is in the rule. The last false positive is closed
by G1, `_gate_outline`'s own principle, so false positives are **0 of the 31
declaring documents**. Reach **2 of 39** corpus documents, both `docx`, **0
of 33 `pdf`**. **BOTH alternatives the order named were measured and
FELLED**: a hand-laid fixture that DECLARES a heading style has it discarded
by the converter, and `rtf` -> `docx` -> markdown yields 0 ATX headings on
that same document, because the loss is in the `rtf` READER before any
writer.
The nine below are unchanged -- `--outline-run 3`, `--table-grid` and
`--unit-fold` since 2026-09-08, `--drop-wrapped-outline` and
`--outline-gate` since 2026-09-09, `--sheet-section-rows`,
`--keep-table-heading` and `--first-span-from-zero` since 2026-09-10, and
`--close-span-gaps` since 2026-09-11, each
with an explicit opt-out (`--outline-run 0`, `--no-table-grid`,
`--no-unit-fold`, `--keep-wrapped-outline`, `--no-outline-gate`,
`--no-sheet-section-rows`, `--no-keep-table-heading`,
`--no-first-span-from-zero`, `--no-close-span-gaps`) that together reproduce
the pre-move bytes -- measured, `diff -rq` 0 differences, not asserted. **The 2026-09-09 pair is one
decision and cannot be split**: the gate takes `pdf` from 2 of 8 to 5 of 8
and the pair takes it to 7 of 8 (the sheet 5 of 12 -> 10 of 12, `docx`
unchanged at 3 of 3). **The gate is G1+G2:** Arm D's RECOVERED headings are
admitted only where the document DECLARES none of its own -- which is
`fold_units` clause 2's principle moved from voting to admission -- plus any
one recovered heading covering `propose.OUTLINE_SHARE` (0.20, swept flat
from 0.10 to 0.30 and collapsing at both ends) of the text. It filters at
ADMISSION, before spans close, so the text a removed mark opened is carried
by the mark above; the post-filter form scores identically and loses that
text, which is why only one of them shipped. **The bar it had to clear is
now the bar**: reference cells up AND hit@8 holding rank 1 on every row on
every bundle. `--sheet-section-rows --keep-table-heading` reaches 11 of 12
and SHIPPED 2026-09-10, after two rounds off. It was held back because on a
K2 bundle built with it row 1 fell rank 1 -> 2 (the gold document goes 1
concept -> 12), under both prior exponents. **That was never these rules'
defect and it is not a segmentation question**: RRF emits a distinct rank
for every concept in a signal that scored them all EQUALLY, so the gold
document's own twelve concepts fill the document-prior tie group and the one
leading the body signal takes position 11 instead of 1. The repair is the
reading side's `consume.DEFAULT_TIE_SHARED_RANK`, and with it every
acceptance condition holds at once. **The fusion was punishing fine-graining
for being fine-grained**, which put the segmentation side and the retrieval
side in competition over one number for two rounds. Arm E joined a session after the other two, on a number measured
AFTER the first move: without it Arm F's table clause has no joined table to
fold, and the shipped D+F default scored 2 of 12 with `docx` 0 of 3 against
the 5 of 12 the fold was published with. **The proposer's own defaults did NOT move** (`propose.py`'s rules stay
off): the goldens and every published reproduction block are pinned to them,
so the two layers disagree on purpose and `cli.DEFAULT_OUTLINE_RUN` /
`cli.DEFAULT_UNIT_FOLD` say where. The cost to a consumer is a re-run and it
is not small: the 43-document reference corpus goes 629 concepts / 1108 files
(the delivered 2026-09-03 tree) to 492 / 944 after the 2026-09-08 move and to
425 / 810 after the 2026-09-09 one (`bdf4977ca5a443c4...`) and to
**436 / 832** after the 2026-09-10 one (`8dff8a8e6c15d2f7...`, default flags,
default epoch stamp, measured on `38104b7` + this round). On the operator's
own five-document folder the last move is 15 concepts / 30 files -> 26 / 52. Digests published
before 2026-09-09 were computed with a path-DEPENDENT command and are not
comparable to this one; the reproducible form is `find . -type f | sort |
xargs shasum -a 256 | shasum -a 256` from inside the bundle, under which the
previous default is `862116da16e422f6...`. The pinned artifact lives at
`~/corpora/okf-telling-20260829/K2-bundle-default-20260910` and
`tests/test_default_bundle_pin.py` (removed 2026-09-21, v1.1 K) held its
concept count AND its per-row hit@8 ranks -- the count alone survived a configuration that lost a rank,
which is how a previous round's regression hid. Since 2026-09-10 it also
holds the KNOWN-NEGATIVE on the same bytes: read with
`--no-tie-shared-rank`, the shipped default bundle reproduces the very fall
the rules were held back for, so the pin names its own cause instead of
being green for an unstated reason. **And the number the
decision cites belongs to another configuration:** Arm F's 5 of 12 was
measured with `--table-grid` ON; without it the same sample scores 2 of 12
and `docx` 0 of 3, because the fold's table clause has no joined table to
fold. The ten: `--contents-name` (round 9), `--outline-run N` (Arm D),
`--table-grid` (Arm E),
`--unit-fold` (Arm F), `--keep-table-heading` (D1), `--sheet-section-rows`
and `--drop-wrapped-outline` (both D3), `--outline-gate` (G1+G2),
`--first-span-from-zero` and `--close-span-gaps`, each passed to the
proposer unchanged. **The last two are the same defect at two ends and
NEITHER is a segmentation rule**: a mark removed after its neighbour's span
was closed takes that text out of the plan. Round 8 measured the whole
remainder -- 43 631 characters, 2.51 %, over 8 of 32 documents with a plan --
down to **0**, with the concept count identical at 436 and every hit@8 row
holding rank 1 on both K2 bundles and the reference sheet label-identical at
11 of 12. Three steps leak: the orphan check (18 527 characters over 15 of
39 documents), `fold_units` clause 1 between entries (7 514) and the same
clause on the last run (all 17 590 tail characters; with `unit_fold=False`
the corpus tail gap is 0). **Round 7's own § 5 does not reproduce**: it
reports `md` at 3 of 4 declared headings and a `rule:table-block` displacing
`## 3 Prising`, but D1 -- the repair for exactly that -- became the default
in the same commit, so the number describes the configuration that existed
before the move. On `a364ef4` the default recovers **4 of 4**, and
`tests/test_md_declared_headings.py` now holds the cell with its cause as a
known-negative. Report:
`docs/2026-09-11-k3-runde8-tabellblokk-og-siste-spenn.md`. That last
one is ON since 2026-09-10 and is not a segmentation rule at all -- it adds
no boundary, and the K2 concept count is identical with and without it
(425 = 425 on the 2026-09-09 default). It repairs a measured loss: **32 of
the 32** documents that get a plan left the text above their first concept
in NO segment. **The hole is bigger than that rule, and this is the number
to carry:** measured 2026-09-10, the pre-move default left **207 435
characters, 11.92 %** of the corpus in no segment -- 163 804 above the first
entry, 26 041 BETWEEN entries, 17 590 after the last. The rule closes the
first part entirely and 79 % of the whole; **43 631 characters, 2.51 %, over
8 of 32 documents remain**, and the between-part has a named mechanism (a
`rule:table-block` candidate displacing a DECLARED heading and opening below
it). Neither remainder is a ceiling; both are in STATE with their numbers. Until that day the build path called the proposer with no
arm flag at all, so a tender PDF that Arm D splits into nine concepts landed
as one -- a build path a full arm behind the proposer. Exposing them was not
the same decision as moving one, and the two were taken a session apart:
**which arm ships as the default is the operator's**, answered 2026-09-08 as
above. Adding a flag still leaves the default byte-identical (measured by
digest before and after, and by Arm E over all 43 corpus documents); MOVING
the default is the one thing that does not, which is why it took an operator
decision and carries an opt-out. Arm C
(`--max-segment-chars`) stays unexposed: no reference has ever been measured
for its cap. The two D3 rules read grammars nothing else here reads: a table
row's FIRST CELL (a run of bare numeric labels cuts the block that holds
them, which is the only way to reach a sheet whose units are rows and the
opposite direction from Arm E), and whether a RECOVERED heading's line is a
wrapped sentence (a heading is a complete line; quoted regulation and a
recovered table row are not). Both remove or add nothing anywhere else: over
the 43-document corpus they change 1 and 5 of 39 readable documents, and
**0 of 5 `docx` either way**. Reports:
`docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`,
`docs/2026-09-08-k3-runde2-per-filtype.md` and
`docs/2026-09-08-k3-runde3-per-filtype.md`.
- **`--assets` / `--no-assets` (0.10.0) is not a segmentation flag either**,
and it is the first flag here that writes a NON-MARKDOWN file. ON by default.
It adds no boundary rule; it changes what the extracted text SAYS, so it sits
with the three PDF reader flags rather than with the twelve arms — and like
them it must be given the same value on both sides of a plan. The full
measurement, the layout and the refusal codes are in the Door B paragraph
above; the spec proposal for the layout is
`docs/plan/okf-assets-section-6-4.md`. `okf project` does not take it: it
owns no flag that moves a bundle's bytes, so it gets the default. The corpus
harness takes it with the SAME default, for the reason `--gate` does — a test
holds the two paths byte-equal, and two defaults would make that equality
depend on which command you ran.
- **`--frontmatter KEY=VALUE` (K3-19, repeatable) is not a segmentation flag**
and moves no byte unless given: it stamps a key on every concept of the run,
split on the FIRST `=` and written on ONE line -- a block-form
`sources` is invisible to `parse_frontmatter`, so the flow form is the only
one that survives our own readers. Since K3-22 a scalar goes out
double-quoted where a YAML reader would not read it plain, and a flow value
goes out as given but is REFUSED (exit 2) when a leaf has no flow form both
PyYAML and the guard read -- so a `sources` URL with a query string, the form
K3-19's own flagged build wrote on 2 761 of 2 761 concepts, fails the build. It adds any key and REPLACES only
`sources` and `description`, the two with a derived layer below them:
precedence flag > what the document declares > file name. Every other key
the door writes (`inbox._door_keys`, including Door A's `ingest_manifest`,
which would make that door claim a Door B file) is refused before anything
is read. `okf project` does not take it -- it owns no flag that moves a
bundle's bytes.
- **`--shell-parent` (K3-20) is OFF and is not a segmentation flag either**:
a plan entry whose span holds its heading alone gets `parent_id` naming the
nearest PRECEDING entry at a smaller level whose own span holds text
(`propose._link_shells`), passing over an empty ancestor; the door writes
the existing `parent:` key. Nothing is copied -- a consumer's own build of
the same standard copied the inherited text in and took hit@1 6/6 -> 2/6.
The route reads the PLAN, never the row: on R761 it names the ancestor the
`<sec>` nesting names on **710 of 710** shells since K3-21 D (708 before: the
2 misses sit at depth 7, and the reader clipped the outline mark to 6 along
with the heading), where reading section numbers gets 686 (`12` begins with
`1`). 35 of 710 have no ancestor holding text and get none. **Since K3-21
`okf consume` reads the key** (`consume.link_parents`, resolved among the
concepts of the concept's OWN `source_file`, because `p1` exists in every
document): an excerpt carries `parent: { concept_id, title }`, conditional
like `req_number`, or `parent_unresolved: true` where the pointer lands
nowhere, and a heading-only body gains ONE line `Enclosing section:
[title](/path)` (SPEC SS 5.1 lineage through links, SS 6.1 the recommended
absolute form), appended AFTER structure derivation -- read as body text it
became a second, unresolved `references` edge -- and screened on its own.
`okf check`'s seventeenth rule, `parent_unfollowable`, holds the form.
**Since K3-25 the RANKING does not score that line**
(`consume.DEFAULT_LINK_IN_SIGNAL = False` on `searchable_text`,
`concept_scores` and `build_payload`; no CLI flag, `True` still reachable).
The line reaches the file and the excerpt exactly as before, so what moved is
ORDER and never an excerpt's bytes. Measured: of the newcomers it ever added
a question token to, **39 of 39** gained it from the bundle-absolute PATH and
**0 of 39** from the link's TITLE, every such token being a segment of the
document's own directory -- the saturation `shared_id_prefix` took OUT of the
id signal, back in through the body. Under this reading a `--shell-parent`
bundle delivers what the unflagged build delivers on **16 of 16** rows (list,
order and `spent`), hit@1/8/50 **6/6** at both `k` with the known-positive at
rank 1. Exposure today is zero: **0 of 5** shipped bundles carry the line, so
**5 of 5** payloads are byte-identical across the move, and the day a bundle
carries it is the day the path would have started costing rank instead.
`okf consume --follow-parent` (K3-21 B) is the second form: `parent` also
carries the enclosing concept's `text` with that concept's own `sha256`,
placed AFTER the cut from the room it left, in rank order, so the delivered
set is the same with it as without it; a text that does not fit is cut to
the longest prefix that does and marked `truncated`. Whether either default
moves is K3-21 B's measurement, not this line's.
**Since K3-21 C the index RESOLVES such a parent**: it rendered every one
unresolved (`parent: p1?`, 675 of 675) because `structure` read `parent` as
a document NUMBER and a segment id answers to none. `structure._segment_lookup`
keys `(source_file, segment_id)` off the concept's own frontmatter and is
asked first, inside the pointing concept's document; a value no segment
answers to is a number, looked up as before, and a pointer naming nothing
keeps `?`. The one key keeps its two meanings (`inbox.py`). Report:
`docs/2026-09-11-k3-runde20-delt-katalog-og-arvet-kontekst.md`.
- **A TWELFTH flag, `--pdf-outline`, is OFF** (round 12, 2026-09-10) and it is
the only one here that does not read the extracted text at all: it cuts a PDF
at the boundaries its own `/Outlines` bookmark tree declares. It is NOT Arm D
-- `--outline-run`/`--outline-gate` are a TEXT heuristic over numbered lines
in the extracted text, and this opens a structure index the file already
carries. Measured on ONE 701-page process code whose publisher also ships a
NISO-STS structure for it, so the fasit is the publisher's own 2 761 titled
sections: the shipped default finds **1 967 of 2 761**, **0 of its 28**
chapters, and **794 of 794** misses have their heading text PRESENT in the
extracted text -- the line is read, the boundary is never opened. With the
arm: **2 759 of 2 761 (99.9 %)**, chapter level **28 of 28**, concept titles
identical to the source after normalisation **2 761 of 2 761** (the bookmark
title is complete because it does not come from the page), false positives
**163 of 2 182 -> 3 of 2 762**, directories carrying two concept files
**132 -> 2** with the 65 contents-copy pairs at **0**, front-matter concepts
**72 -> 2**. Consumption: fasit present in the bundle **4 of 7 -> 7 of 7**,
hit@1/8/50 **1/6 - 2/6 - 4/6 -> 3/6 - 5/6 - 6/6**; the known-positive is a
real concept now and ranks 13, so it is delivered at k=50 and not at k=8 --
the segmentation half of that row is closed and the ranking half is not.
**The bridge from (page, `/XYZ` top) to a line index is the whole risk and
BOTH routes are measured:** `extract_text_lines` splits lines identically to
`extract_text` on **701 of 701** pages and that check SHIPS per page, the y
route and the title route disagree on **0 of 2 762**, flat from 0 to 8 pt and
collapsing at 12, so the rule carries **no tolerance constant**; the naive
nearest-line rule was wrong on **1 840 of 2 762**, one line early every time.
The orphan check is NOT applied to a bookmark mark -- it asks whether anything
stands under a candidate's first line, the right question for a guess and the
wrong one for a publisher's declaration; **683 of 2 762** marks are container
sections and applying it scores 2 079. An unresolvable `/Dest` is dropped and
COUNTED (R761 has 0 of 2 763; one of the eight reference PDFs has 2 of 2). NO
new dependency: `pdfminer.six` already ships under `pdfplumber` in
`[extract]`, so `uv.lock` is untouched and `pypdf` stays out. Cost 119.22 s ->
183.31 s wall, peak RSS 3 252 -> 3 251 MiB, pages parsed 1 -> 1. **The default
did not move**, and the reach is why: **1 of 8** reference PDFs carries a
usable tree, and a bookmark tree is the publisher's CLAIM about its own
structure. On the folder where no PDF has one, `diff -r` is empty against both
the arm off and the pre-change tree. Report:
`docs/2026-09-10-k3-runde12-pdf-outlines.md`.
- **Three PDF READER flags, all off, and they sit BEFORE every segmentation
flag** -- an arm changes how the proposer cuts a text, these change what the
text says. `--pdf-headings font` infers a heading from typography (dominant
font size above the document's character-weighted body median AND a bold font
name -- the CONJUNCTION measured at recall 1.000 / precision 0.846, where
adding weight as a disjunct took precision 0.786 -> 0.524) and emits it as an
ATX heading in the SAME markdown the office path produces, so `_ATX` applies
unchanged and **no PDF-only heading grammar exists**. It is off **by
measurement, not by caution**: against the operator's unit worksheet it takes
`pdf` from **2 of 8 to 0 of 8**, losing two exact matches, because on those
documents the outline rule already recovers the document's own numbered
chapters and a second heading source can only add. The cost of the ATX form is
named rather than hidden: a font-inferred heading carries `rule:heading` and is
indistinguishable in the artifact from one the document declared, which is why
`RULE_POPPLER_SIZE_AND_BOLD` was deliberately NOT assigned to it -- that name
records a poppler measurement on a path that cannot ship. `--ocr` reads a page
as an IMAGE when its own text never arrived (empty, or `(cid:N)` codes at or
above `OCR_CID_SHARE = 0.10`, a threshold READ OFF the measured per-page
distribution: 834 pages over 32 files, 818 at exactly 0.0 and 16 at 0.93 or
above, nothing between). Its engine is the optional `ocr` group
(`rapidocr`/`onnxruntime`/`pypdfium2`) and **never** a runtime dependency; a
packaging test pins both halves, and without it every affected file is a coded
rejection (`extractor_ocr_group_missing`), never a crash. `--ocr` can never
become a default -- an optional dependency in the default path would make an
ordinary install fail on the first scanned page. Report:
`docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md`.
**`--pdf-headings font-reserve`** is the third value on that same option
(`none`, `font`, `font-reserve` — three answers to one question, so no caller
can ask for two at once): the same typographic rule applied ONLY where Arm D's
outline gate admits no run at all, typography as a second heading source where
there is no first one. The condition lives in ONE function
(`propose.heading_reserve_applies`) that the proposer and the door both
consult, the door receiving it as a callable the way it already receives
`gate` — a plan indexes the exact string it was proposed against, so a reserve
firing on one side only would turn every document it touches into a coded
rejection. It reads the gate AS CONFIGURED, so at `--outline-run 0` it is
unconditional and equals round 4's "font instead of Arm D". **Off, and the
measurement is that it changes nothing measurable:** on the twelve-position
reference it alters **not one cell** — the five positions where it fires are
one PDF whose glyphs carry no ToUnicode mapping and four office documents the
PDF reader never touches — and the position it was built for has **three**
outline runs, so the reserve is silent there by construction. Its reach is
real but unrated: **4 of 39** readable corpus documents, none in the sample.
Report: `docs/2026-09-08-k3-runde5-hitat8-og-skriftakse.md`, which also
corrects two of this repository's own published figures — the S7 candidate
ranks (96 of 629 / 159 of 492 were measured with the cost vocabulary reaching
only half the ranker; consistently scored they are **10 of 629** and **19 of
492**) and round 4's attribution of that concept's non-delivery to the default
move (it is not delivered on the Arm B bundle either, by a different
mechanism). hit@8 over the six published questions holds at **5 of 6 on both
K2 bundles**, so the default move cost the retrieval side nothing.
- **The image path has a size CEILING and no live links (0.10.1).** TWO
independent reviews, a day apart. The first found two MAJOR defects in
`v0.10.0`, both with the shipped defaults, both new in 0.10.0. The second
read the fix for them and found that ONE of the two was not closed --
which is why the version below is 0.10.1 and not 0.10.0 with a patch note.
- A remote `<img src>` / `xlink:href` became a LIVE markdown image link in
the persisted concept. This package opens no socket; a consumer that
renders the bundle does, and the guard refuses such a line at
`user-upload` while the build's default tier persisted it. Now inert text
with the address in a code span, pinned by a PROPERTY over the readers
(`FOREIGN_IMAGE_LINK`), never one string.
- **THE FIRST FIX FOR IT WROTE THE ADDRESS TWICE**, once in a code span and
once BARE, and a GFM/linkify renderer autolinks a bare URL. Weaker than an
image link -- it takes a click -- but "inert" was half true. It also
dropped the figure's CAPTION: `label` stayed in the signature and no
branch read it, so an image the bundle does not carry lost the one fact
that lets a reader weigh the absence. Both closed; the caption is written
in the same `-- <label>` form a carried pointer uses.
- **NOTHING BOUNDED WHAT THE RUN PAYS, only what the document CLAIMS.**
`MAX_IMAGE_PIXELS = 40 000 000` and `MAX_IMAGE_BYTES = 256 MiB` were
checked against `/Width` x `/Height`, which is an untrusted document's
claim, while `get_data()` pays for the actual stream -- two independent
numbers, since `/Length` is the COMPRESSED length. Measured by the second
review: a **408 516-byte** PDF declaring **1x1** and carrying 400 MB of
deflated zeros was CARRIED, no rejection, **892 MB peak RSS**, and 1,2 GB
of zeros cost **2 436 MB** -- about 2 100x the file size, linear. The
first fix's four mutations all died, because not one of them separated
declared from actual.
- **THREE NUMBERS ARE BOUNDED NOW.** What the container DECLARES (as
before), what a carried FILE measures (`read_image`, so a 49 MP PNG of
47 705 bytes is not passed on to the consumer with `7000x7000 px` beside
it), and what a PDF image's STREAM decompresses to -- measured a chunk at
a time and DISCARDED, before `get_data()`. With it: **0 carried,
`asset_too_large`, 54 MB** at 400 MB and **62 MB** at 1,2 GB, so the cost
no longer scales with the bomb.
- **THE CHAIN, NOT ITS FIRST LINK.** That fix read `filters[0]`, and a PDF
decodes a stream through a LIST of filters: measured 18.09 on `0f308c1`,
`/Filter [/FlateDecode /FlateDecode]` cost **886 554 624 B peak RSS from
1 636 B of file** and three links the same from 1 070 B -- about
542 000x -- with the picture still refused at the end, by the backstop,
after the memory was spent. It also left the **16** corpus objects behind
an `[/ASCII85Decode /FlateDecode]` chain unmeasured, since `filters[0]` is
not `FlateDecode` there. Bounded (idle machine, paired subprocesses):
**52 367 360 B** at two links, **61 390 848 B** at three, **60 403 712 B**
where the old path cost 2 567 204 864 B, and the single-link control
unmoved at 59 232 256 -> 62 017 536. Everything
else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`,
anything written later -- is refused UNREAD with its own code
`asset_pdf_unbounded`, the same decision `corpus.resolve_gate` takes for
an unknown gate name. **The three-class split this round shipped with --
`ASCII85Decode`/`ASCIIHexDecode` "bounded by their own input because they
SHRINK" -- was FALSE and round 3 below replaced it.** Cost measured over the **5 142** image objects of
78 PDFs: the refused class is **4** `CCITTFaxDecode` objects, all 1-bit
stencil masks already refused one step later by the encoder, and **0**
objects in an encrypted document. **NOT ONE PICTURE CHANGES HANDS AND IT
IS MEASURED BY NAME**: over the same 78 PDFs, carried images are
**9 356 -> 9 356**, **0 of 78** documents lose one and 0 gain one; the
only movement is **8** objects going `asset_pdf_unsupported` ->
`asset_pdf_unbounded` (the 4 masks, counted twice because `trinn1` and
`trinn2` hold the same document). An ENCRYPTED stream is
now deciphered and then measured (deciphering does not change a length),
where `stream.decipher is not None` used to return unmeasured.
- **THE COST OF A LINK, NOT THE SIZE OF ITS OUTPUT (round 3).** Three
rounds each bound an OUTPUT and the bomb stepped one link along: the
declared size, then the first `FlateDecode`, then every `FlateDecode`.
Round 2's third class was FALSE -- `ASCII85Decode`'s `z` is the shorthand
for four zero bytes, so it QUADRUPLES its input, and `base64.a85decode`
appends one 4-byte object per group to a list, costing about **a hundred
bytes of memory per byte of INPUT** (measured on CPython 3.14: **101.4x**
at 1 MiB, **96.1x** at 4 MiB, **94.5x** at 16 MiB). Paired subprocesses,
idle machine, both sides from pinned trees, the document built once and
read from a FILE because `ru_maxrss` never falls and `b"z" * 64 MiB`
alone costs 171 MB: `[/Fl /A85]` **33 475 B of file -> 3 261 599 744 B
peak and the picture CARRIED**, now **42 070 016 B** and
`asset_too_large`; at twice the `z` run **6 461 558 784 -> 40 280 064**,
so the cost no longer follows the bomb; `[/A85]` alone **933 085 184 ->
62 484 480**; `[/Fl /A85 /Fl]` **3 519 180 800 -> 43 438 080** and from
`asset_samples_invalid` to a bound's own code. **The rule is a BUDGET per
link** (`assets.MAX_FILTER_DECODE_BYTES`, 512 MiB, a separate number from
`MAX_IMAGE_BYTES`): `FlateDecode` is measured as it is paid, every other
permitted filter has a MEASURED cost ratio
(`assets.PDF_FILTER_COST_RATIO`) checked against its input BEFORE its
decoder is called, and a filter with no ratio is refused unread. **The
budget TRAVELS**: a deflate link is inflated under the smaller of the
picture's bound and what the next link's decoder may be handed, or
`[/Fl /A85]` pays 256 MiB for a refusal. The ASCII85 cap (**~5.0 MB**) is
READ OFF the corpora: of **9 668** image objects over **77** PDFs, **16**
decode through such a link and the largest input is **450 739 B**, ten
times under it. **A PROPERTY TEST replaces the list of known shapes**:
every chain of length 1-3 over the ten filters pdfminer decodes, **1 110
of 1 110**, both payload fills, each delivered under the bound or refused
with a published code and never paid for on the way (`tracemalloc`, which
counts allocations and is not disturbed by load). Known-positive: **258 of
258** chains over the permitted filters still carry a small image.
**10 of 10 mutants killed**, control green. **NOT A ROW'S DIFFERENCE ON
REAL DOCUMENTS**: paired `_pdf_images` over every PDF on this machine from two PINNED trees: **9 306 -> 9 306** bårne over 77 filer plus **50 -> 50** on R761 (**9 356 -> 9 356** together), **0 of 78** files moving a count and **0** moving a code. R761 also settles the `[/Fl /A85]` question: **50 objects, 29 `[/DCTDecode]`, 21 `[/FlateDecode]`, 0 ASCII85 links** -- round 2's count of **580** is reproducible from nothing on this machine, which changes no decision but should not stay unqualified. **`resource.setrlimit(RLIMIT_AS)` was
MEASURED before anything was built on it and is NOT usable here** --
Darwin 26.6.2 raises `ValueError: current limit exceeds maximum limit` and
does not enforce it -- so no child-process cap exists and the per-link
budget is the whole bound. Report:
`docs/2026-09-18-utgangsbudsjett-per-ledd.md`.
- **THE LIMIT IS STATED RATHER THAN IMPLIED, and it is now ONE case**: a
stream something else has ALREADY decoded, where the memory is spent
before this package is asked. `check_payload(len(data))` after
`get_data()` COUNTS it -- a counted refusal, not a bounded one. Deleting
exactly that line passed all **2 132** tests on `0f308c1`; it has a test
now.
- **A non-positive declared dimension is `asset_size_invalid`**, its own
code, raised before the stream is read. `-1 x 40 000 000 000` is a
NEGATIVE pixel count, under which every `>` bound reads as satisfied: the
check returned silently, 400 MB was decompressed, and the refusal came
from the PNG encoder as `asset_samples_invalid`. Its own code because a
publisher shipping a picture bigger than we carry and a dictionary written
to be read wrong are different facts about a document.
- **An inline PDF image is named from its position, not from `id()`** —
pdfminer's name broke the bit-exact rebuild on two K2 concepts.
- Documented rather than changed: `images: N` counts POINTER BLOCKS, and a
pointer-only concept is substantive.
- Reports: `docs/2026-09-17-bildestien-0-10-1.md` and
`docs/2026-09-18-bildestien-holder-0-10-1.md`.
- **EVERY CARRIED IMAGE IS ONE A MODEL CAN BE SHOWN (0.10.1, 2026-09-19).**
Until this round the path carried whatever a publisher shipped. Measured over
the frozen R761 delivery's own `assets/`, denominator 50: 29 JPEG, 2 PNG and
**19 "PC bitmap, Windows 3.x, 8-bit, compression 1"** -- byte-correct RLE8
BMP files that nothing reads, so 19 figures were present and invisible at
once while `images: N` said they had arrived. `assets.VIEWABLE_MEDIA_TYPES`
(`png`, `jpeg`, `gif`, `webp`) is tested against every asset's SNIFFED type,
so it is a PROPERTY and not a list of formats we met; WebP is on it and
`sniff` does not recognise one, which is stated rather than implied. BMP is
converted losslessly to PNG -- 8-bit uncompressed, 8-bit RLE8 and 24-bit
uncompressed -- and everything else outside the set is refused with
`asset_not_viewable`, a BMP variant this reader does not express with
`asset_bmp_unsupported`, both leaving the concept's own "not carried" line.
**THE READER IS STDLIB AND PILLOW WAS MEASURED AND FELLED**, though the tree
already carries it transitively under `pdfplumber`: `read_image` is on the
CORE path (`.html` and `.xml` carry images with no `[extract]`), and an
asset's name is its content digest, so encoding through an installed library
would make a bundle's identity move with that library's version -- 0.10.0's
own reason for felling page rasterisation. Pillow is the INDEPENDENT decoder
in the tests instead, and against it **19 of 19** real RLE8 assets convert
with identical RGB, **2 366 365 pixels** compared. The ceiling is paid on the
DECLARATION before a row is allocated, and an RLE run is one CLIPPED slice --
painting pixel by pixel leaves the memory bounded and the CPU unbounded, a
megabyte of `FF` runs being a hundred million steps against a 32-pixel frame.
Traceability is on the pointer's second line where the rest of the asset
metadata already lives (original media type, original sha256 in full, new
sha256), and a converted asset is ONE asset: one file, one pointer, one
accounting row. Exposure measured with a committed script
(`tools/okf_asset_census.py`, one row per image, run from two pinned trees
over 18 403 files and 67 PDFs, **9 714 image rows**): exactly **35 rows
moved**, **9 321 of 9 321** JPEG and PNG rows byte-identical, so not one
already-viewable picture changed hands. **16 of the 35 are the finding the
round did not go looking for**: JPEG 2000 objects out of PDF streams, a
format no model decodes and no stdlib route converts, so they go from
carried-and-invisible to `asset_not_viewable` with a line in the concept --
a consumer loses 16 pictures and gains 16 statements that a picture was
there. **The judge had to move with it and it caught itself**: the gate's
`asset_holds` proved a carry by hashing the SOURCE file, which a converted
image's bundle cannot satisfy, so R761 went 0 -> **19 claimed and not
found** (exactly its RLE8 count) before the judge learned to read the two
digests the bundle states and then hash the asset itself; after, R761 is
u = 0, d = 0, 0 claimed and not found, 50 asset files, 31 081 carried
elements, and the gate's verdict is the same `RED: rows 2, 3, 6` as before
the round. Report:
`docs/2026-09-19-hvert-baaret-bilde-kan-vises.md`.
**THAT ROUND SHIPPED TWO DEFECTS AND 0.10.1 CLOSES THEM.** A TRUNCATED RLE8
stream was carried as a partly blank PNG: the decoder painted what the
stream held and left the frame at index 0, which is what the format says
about a SKIPPED pixel, so no decoder disagreed -- measured on a real R761
asset of 192 896 pixels, a stream cut to 90 % was carried with 13 923 pixels
wrong, to 50 % with 95 890, to 10 % with 166 525, no code and no row. The
decode may now end at an explicit **end-of-bitmap escape and nowhere else**
(`asset_samples_invalid`, the code the uncompressed path already uses), and
the rule is the terminator rather than `biSizeImage` -- a claim by the same
untrusted header -- or a coverage count, which would refuse the delta the
format defines: over the 19 real RLE8 assets **19 of 19** end at an explicit
EOB, on **19 of 19** it is the last two bytes, and on **19 of 19**
`biSizeImage` equals the available bytes, so the same 19 still convert
losslessly, 2 366 365 pixels, now with stdlib on BOTH sides. And the JUDGE's
new conversion route could be **FORGED BY THE DOCUMENT**: it read the two
digests out of the WHOLE bundle text, so a BMP refused `asset_too_large` and
absent from `assets/` was reported as held, through an image's `alt` text
and through ordinary body text alike. The claim now counts only inside a
POINTER BLOCK this code wrote and only where it names that block's own
asset, and `assets._inline` disarms a checksum field in anything that came
from the document -- **the boundary is that everything `_inline` returns
came from the document and everything `render_block` appends after it came
from the run**; neither half is sufficient alone. The gate proves CARRIAGE
and not FIDELITY, which its docstring now states: a converter writing a
blank PNG is accepted, the suite fells that mutant and the judge cannot.
Report: `docs/2026-09-19-bmp-resten-terminator-og-pekerblokk.md`.
**AND THAT ROUND NARROWED BOTH DEFECTS WITHOUT CLOSING EITHER.** A POINTER
BLOCK IS A FORM AND A FORM IS NOT A SIGNATURE: measured through the real
`okf build`, one ordinary HTML file with **two `<p>` elements** writes the
two lines `_POINTER` matches, and the same refused BMP read as held again --
also with `<p>`+`<br>`, and with a markdown note beside the HTML carrier,
**3 of 3**. So the pair is read from the RUN's own ledger (`--accounting`
now books `conversions: [{from, to}]` per document, off `assets.conversion`)
and the bundle text only CONFIRMS it: the confirmation can be forged and the
ledger cannot. **Chosen over neutralising pointer-shaped text at
extraction**, which would change what every document SAYS to defend a tool
outside the build. A build with no accounting door reports a converted image
claimed-and-not-found rather than believing it. Measured: 3 of 3 forged ->
refused with the known-positive True in all three, the text-level regression
guard 3 arms -> 13, R761 rebuilt `diff -r`-identical, 50 assets, **19 of 19**
conversions confirmed against 19 declared, SHY 71, u = 0, d = 0. And the
TERMINATOR asks only that a stream say it is finished, which it may do
anywhere: a stream whose FIRST two bytes are the EOB escape was carried with
**32 of 32** pixels never decoded while Pillow refuses the same file. The
cursor must now also stand at or past the end of the last row. **The line is
the CURSOR and not the pixels** -- a pixel-coverage count would refuse the
delta the format defines. The corpus cannot choose (over the **25** RLE8
BMPs the delivery ships, 25 of 25 paint every pixel, 25 of 25 reach the end,
**0 of 25** use a delta) and an independent decoder can. The earlier
`19 of 19` figures above are the BUNDLE's carried assets; 25 is what the
delivery ships. Report:
`docs/2026-09-19-pekerblokken-bindes-til-kjoringen.md`.
**AND THAT ROUND'S OWN SENTENCE WAS TRUE OF THE DELTA AND FALSE OF THE
END-OF-LINE.** It said both escapes state a skip every decoder agrees on.
A delta states a DISTANCE; an end-of-line states only "the rest of THIS
row", so an end-of-line at column 0 closes nothing and claims a row the
stream never wrote -- four of them and an EOB carried an 8x4 frame with
**32 of 32** pixels never decoded, and Pillow refuses those bytes. The class
is wider than that construction and this round SWEPT it instead of patching
it: over every opcode sequence of length 1-4 on a 4x3 frame, **22 620**
streams, this reader carried **703** streams Pillow refuses and drew
**1 492** more differently. PM's recommendation alone (refuse a stream that
painted nothing) leaves **512** and **1 171** -- it would have narrowed the
class a third time. Two clauses ship instead, both `asset_samples_invalid`:
an end-of-line at column 0, and a delta whose horizontal offset would leave
the row (the format puts it inside the line; this reader keeps the cursor
past the row end where a flat decoder rolls it into the next row). After:
**0** carried-here-refused-there and **32** drawn differently. **WHAT IS NOT
CLOSED IS STATED**: all 32 are a run or absolute block that OVERRUNS its
row, which this reader clips at the row end and Pillow spills into the next;
refusing those gives 0 and 0 and costs **15 of the 25** real files, so it is
a limit and not a loss taken. Cost measured on the corpus FIRST: over
**11 441** files scanned across the four raw deliveries and the K2 reference
corpus, the only **25** BMPs on this machine use an end-of-line at column 0
in **0 of 25** and a delta in **0 of 25**, and 25 of 25 still decode to
Pillow's pixels exactly (**3 117 220** compared). R761 rebuilt from a pinned
pre-change tree and from this one is `diff -r`-IDENTICAL, 5 551 files, 50 of
50 viewable, 19 of 19 lossless. `CURSOR_CASES` goes 8 arms to 12 (one for
the cursor rule's ROW clause, PM's `P8`, which survived 51 tests) and the
mutant runner 45 to 47. Report:
`docs/2026-09-19-eol-stroemmen-som-ikke-maler.md`.
- **Account for content: `okf build --accounting PATH` and its judge
`python3 tools/okf_accounting_gate.py` (2026-09-17).**
- The gate was written RED at `0b00de4` and turned GREEN on all six rows by
`accounting.py`. Per README file type it asks how many of the elements a
SOURCE holds are booked as carried / pointer / coded rejection, and both
u (unaccounted) and d (double-booked) must be 0.
- **The fasit is `tools/okf_witness.py`.** It imports no package module
(tested on the live import graph) and is committed as
`tests/fixtures/accounting/*inventory.json`. The package's inventory
implements the SAME definitions a second time: agreement is evidence that
the definitions can be implemented as written, not an independent
measurement of the source.
- **`carried` is checked against the written concept bodies**
(alphanumerics, casefolded), so a gate that drops a line is FOUND.
Measured limit: short elements (labels, one-word titles) are often
present elsewhere in the document. With R761 cut to half, 4 823
paragraphs were reported lost but only 3 titles.
- **Two changes hold without the flag.**
- A build that extracted documents and persisted none exits 1. Door B and
`corpus.measure` are unchanged.
- An image file carried through a persisted document is its own K1b
column, never also `extractor_unknown`. The files come from
`ExtractedDocument.files` (what the resolver actually resolved), never
from byte equality, which would credit R761's 7 unpointed duplicates.
- **R761 WAS refused whole by guard 1.4.0 because of its 71 U+00AD soft
hyphens, and since 2026-09-19 okf removes them itself.** Operator decision
2026-09-18, the chosen alternative verbatim: **one documented
normalisation door BEFORE the guard**, `extract.normalise_extracted`,
removing U+00AD and BOOKING the count as `normalised_soft_hyphen` per
document and per run, in the accounting JSON and in a `**Normalisation**`
bullet in `log.md`. The guard is untouched -- the two alternatives
(weaken the rule, ship a cleaned corpus) were felled as a weaker screen
for every consumer and a fix for one document. `output:zero-width-present`
is an ANY-TIER carrier, so the verdict was formally right at every trust
level and materially a false positive: the 71 are Norwegian hyphenation
points inside words (`ar[SHY]beider`, `bitu[SHY]men`), and the four real
zero-width characters (U+200B, U+200C, U+200D, U+FEFF) plus U+2060 are
**0 of 0** in both deliveries. Those four are NOT removed and U+00A0 NBSP
(6 633 in R761) is not either -- it is not in the guard's set at all.
Applied ONCE, at the end of `extract_document`, so `propose` and Door B
cannot disagree about the text a plan's `text_sha256` indexes; `_pdf_units`
measures its page offsets through the same door, because that table is
CHARACTER offsets rebuilt from the pages while the text it indexes comes
back normalised. Removing a character removes no newline, so every
line-indexed rule downstream is unmoved by construction. **Reach measured
rather than argued: 0 of the 78** readable documents of the K2 reference
corpus carry any of the six characters, 0 in the pinned bundle's concept
bodies, and U+00AD is **0 of the 230** readable tracked files across
`tests/fixtures`, `examples`, `skills`, `docs`, `src`, README and
CHANGELOG -- the door cannot have moved a byte measured here. That last
figure said "0" of ALL SIX until 2026-09-19 and was wrong: **2 of the
230** carry U+200B (`tests/fixtures/accounting/rejected/avvist.html` and
`rejected-inventory.json`), this repo's own known-negative, which the door
is built not to touch.
- **A DOCUMENT REFUSED WHOLE IS NEVER CLEAN, AND A PARTIAL REFUSAL IS NEVER
SILENT (H1, 2026-09-19).** `refused_whole` asked its question only when a
corpus persisted NOTHING, and a refused document's own unit balances by
construction -- every element booked as a coded rejection is u = 0 and
d = 0. Measured by PM: one refused source beside one accepted one gave
`clean = 4 of 4`, `refused_whole = None`, `okf build` exit 0 and three
elements gone unseen, which is the ORDINARY case on a heterogeneous
corpus. `Unit.refused` is a fifth column and the only one that is not a
defect in the report; the build writes `refused` into the accounting and
`R of D document(s) refused whole` into `log.md`. **The exit code does not
move**: it belongs to the whole run, and a corpus holding one unreadable
file among many is ordinary -- so the order's other half, state it in the
accounting, is the one taken.
- **THE JSON ROLE MAP READS THE PUBLISHER'S OWN TAGS (H3, 2026-09-19).**
`count_sts_json` compared the RAW tag string where `count_sts_xml` has
always used `_local`, so N200 Vegbygging:2024's **74 `mml:math`** (2 811
characters) counted as 0; and the publisher's JSON writes a figure's
caption as `figcaption` under the `graphic`, not as the `fig/caption`
NISO-STS writes, so **49 `figcaption`** counted as 0. The `figcaption`
clause is in the JSON map ALONE -- the two maps are written twice on
purpose so row 5 can see a hole in either. NOT ONE OTHER COUNT MOVES,
measured role by role over the eight JSON deliveries on this machine and
the three committed twins: `figure_caption` 0 -> 123/127/62/49/43/27 and
`math` 0 -> 74/3/88, R761 and N601 unchanged, the two existing twins
unchanged. The known-negative for stripping a prefix is N101's **568
`tbx:`** tags, not one of whose local names is a role.
- **The mutation harness is a GATE since 2026-09-19 (H4)**: a surviving
mutant exits 1, where `2 if errors else 0` made `killed 0 of 1` an exit 0.
PM's **X2** -- "a report may declare a document rejected while the bundle
holds it" -- is in the set and killed by a test driven from both sides;
it survived 98 green tests when it was written. The skipped-row guard
measures the DEFAULT sources and not the arguments (H5: row 6 is SKIPPED
exactly when the arguments' corpora are absent, so the branch was
unreachable), and row 6 now SAYS when a corpus measures no element class
at all (H6: on N200 the build proposes 0 plans and exits 2 before the
accounting door, so 16 549 elements land with no declared fate -- a
finding about the run that read as a finding about the build; what it
would take is an STS reader for the JSON delivery, and `standardContent`
occurs 0 times in `src/`).
- **THE PUBLISHED DOOR CONTRACT NAMES WHAT THE GATE READS (2026-09-19).**
The JSON sketch at the top of `okf_accounting_gate.py` is what a consumer
implements `--accounting` from, and it did not name `conversions` -- which
`asset_holds`' conversion route DEPENDS on -- nor `normalised_soft_hyphen`,
`unaccounted` or `double_booked`. A door built from it writes a ledger the
gate reads as "nothing was converted", and every converted image comes out
claimed-and-not-found: **19 of 50** on R761. Two tests hold the sketch
against both sides -- what the gate LOOKS UP (measured with a ledger that
records its own lookups, never by grep) and what the door SERIALISES -- so
"publish everything" is not a fix either.
- **A DOORLESS BUNDLE SAYS WHY (2026-09-19).** Without the flag there is no
ledger, so a converted image cannot be proved carried and is counted
claimed-and-not-found -- honest, and until now silent: the count was
printed and its cause was not. `_tally` names the missing ledger when, and
only when, something was claimed and not found. The gate always passes the
flag, so **no row moves** (RED 2, 3, 6, exit 1, unchanged).
- **Operator 2026-09-17:** the door stays OPT-IN until the reference
corpus's losses are fixed, and of the three proposed exceptions only the
PDF one (heading, paragraph, table) is APPROVED. It moves no number --
no witness counts a heading in a PDF -- so what it changes is that the
gap is a stated limit. An image in xlsx, md, txt, csv, json, odt or rtf
stays unaccounted, and therefore red.
- Reports: `docs/2026-09-17-innholdsregnskapet-rod-gate.md`,
`docs/2026-09-17-innholdsregnskapet-bygget.md`.
- **Judge the RETRIEVAL: `python3 tools/okf_retrieval_gate.py` (2026-09-19).**
A separate question from `okf quality` and a separate command: quality asks
what a bundle looks like, this asks whether the payload for a question
carries the fasit. **Written RED, no capability**: eight rows, exit 0 only
when all are green, 1 otherwise, 2 on wrong input, `--json`. Rows 1-4, 6, 7
run against a synthetic corpus this repo GENERATES and eight sets it ships
pinned by sha256 -- no network, no private corpus, no clock; a set whose
bytes moved is exit 2, never a quiet continuation, because **a gold set is
an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and
this repo is public). **THE CORPUS IS PINNED THE SAME WAY** since
2026-09-19 (`SPECS_SHA256`). Today (measured on the default run
2026-09-22): **1 (14/14) - 2 (7/7) - 3 (5/5) - 4 (6/6) - 5 (0/1) -
6 (14/14) - 7 (15/15) - 8 (0 of 1 set, NOT RUN without `--real`)**,
`GATE RED: rows 5, 8`. **Row 7 fells 14 of 14 since 2026-09-22**: the three
that survived v1.1 (M06 passage body, M07 title/path weight, M10
`bm25.RRF_K`) each got a fixture that makes its mechanism decide a delivery
(`PASSAGE`, `PATH`, `FUSION`, set `set-mechanisms.json`; the fusion one keeps
its lead through K = 180 and loses it at 190), with `src/` untouched.
**THE JUDGE READS A PASSAGE AS A PASSAGE (2026-09-22).** It predated v1.1
C1 and required a delivered text to be the whole concept's bytes, so every
excerpt `as_passage` cut was class (e) even when its span carried the
citation -- the dominant miss class when row 8 was run on the real set.
`passage_span` now accepts a passage only when the delivered text IS its
exact reconstruction: the span is `body[start:end]` byte for byte, `of` is
the body's length, a `[...]` stands exactly where text is left out, and the
one other line allowed is a heading line of the body above the span (or a
prefix of one); WHICH heading is the product's choice and is not re-derived.
The citation is read in the SPAN alone, never in the heading, the markers or
across the seam. Two traps are pinned by tests: the offsets count in the
DELIVERED body (read in the concept file they land a frontmatter's length
off), and "the span occurs somewhere in the text" accepts an invented
sentence beside it. **M15** (a passage carrying one sentence the file does
not, M14's shape for passages) is row 7's fifteenth mutant, felled through a
new fixture that delivers a passage (`DELIVERY`, set `set-passage.json`),
because no synthetic concept was long enough to be cut. `src/` untouched;
`MUTANT_BAR` unchanged.
**THE K2 TEST TRACK IS RETIRED (operator 2026-09-21, v1.1 K)**: material
tied to the operator's employer is not re-measured and not frozen, and
public tests and gates run on invented material. Row 9 (K2) and the `r761`
and `vegnormal` adapters of row 8 are gone (row 8 requires `wiki-20` alone),
and so are the tests that read the K2 bundles, the road-standard builds or
the R761/N101/N200 deliveries; `okf_accounting_gate.py` has no default real
corpus. The synthetic corpus was re-measured for BM25 (`MISS`, `LOOKUP`,
`QUOTA`), so rows 2 and 3 are green through their own fixtures again, and
row 7's mutants patch `bm25` where the default runs there; the three that
survived with 0 ranks moved are felled since 2026-09-22 (above). What
the text below says about rows 8/9, K2 and the three sets is history.
**FOUR OF EIGHT CHEATING ATTACKS WENT THROUGH IT AND ARE CLOSED
(2026-09-19).** PM's checkpoint on `2c8296b` found rows 3, 5, 7 and 8 could
go GREEN without one label becoming true or one concept ranking better, and
an eval written before the capability has exactly that one job beyond being
red today. **Row 8** counted whatever `--real` gave it, so one set of three
read `6 of 6 GREEN` -- and this repo's OWN test asserted `(1, 1, GREEN)` for
a single set; it now requires all three by name and is NOT RUN otherwise,
the measured sets' numbers still printed, and its headline is at QUESTION
granularity because the three sets share no unit. **Rows 2 and 3** took
their denominator from the run, so `k = 32` shrank it to the cases that were
already honest (rows 1, 2, 3 and 6 all green at once); the denominator is
now the PINNED set's, and a forced fixture that stops producing its declared
class is a BROKEN PREMISE counted against its row. **Row 3 also carries a
known-positive**: with the quota off every printed reason is true, which is
not a lie but an empty measurement, so a set may declare
`source_quota_in_force` and the row is NOT RUN when the default cut and the
quota-off cut deliver the same concepts. **The control's own premise was
measured first and was FALSE where it was first put** -- over the five
existing sets the two cuts deliver the SAME concepts (the quota is topped
back up) and 52 labels move `source_quota_exceeded` -> `below_k` with 0
deliveries changing -- so `set-quota.json` is the one set where the quota
genuinely decides. **Row 5** was seven checks the registration made about
itself; three now read GIT (committed and unmodified, its commit is not
itself a ranking change, a ranking change landed AFTER it), the last being
the one that cannot be self-attested. What git cannot prove -- that nobody
read the number first -- is stated, not implied. **Row 7's bar** was 90 % of
a list in the file a capability session edits, so seven duplicate `k = 1`
mutants read `18 of 20 GREEN`; `MUTANT_ROSTER` and `MUTANT_COUNT` are pinned
apart from `MUTANTS`, duplicates are refused, and the bar is the roster's.
**PM's G9 is decided**: `hit = bool(hit_ids) and bool(confirmed)` is
observable and was merely unobserved -- `M14` (a delivery that carries the
citation and is not the concept file's bytes) is the shape that reaches it,
it is FELLED, and no production line changed. The judge's independence is
measured rather than asserted: the index is warmed BEFORE the first
mutation, so patching `delivered_text` moves the payload and not the judge
(index built UNDER the patch: every unit a hit; warmed first: every unit a
miss). Row 9 takes `--k2 SET SHA BUNDLE` and stays RED without one, because
ITS denominator is known -- a deliberate divergence from the order, stated.
Report: `docs/2026-09-19-gjenfinningsgaten-motstand.md`. **ROWS 3 AND 4 WERE THE TWO FINDINGS AND BOTH ARE
CLOSED** (`f81683e`, `05cb190`, 2026-09-20). Row 3: in a bundle built from
ONE source document every concept past the first two carries that document's
`source_file`, so a concept the RANK had already lost came back withheld as
`source_quota_exceeded` -- PM measured the same lie on **13 of 25** real
misses 17.09. A drop now keeps the rule the SAME cut without the quota would
have given it, decided by the code the run itself uses: **2 of 5 -> 5 of
5**. Row 4: no payload key said "this bundle does not answer that", so the
only honest case was `delivered == 0`. `coverage` states the terms the
pre-pass read, the terms no concept in the bundle answers and the terms no
delivered excerpt answers -- facts and no verdict, both candidate verdicts
falsified over 81 questions -- with the gate's own `UNANSWERED_BAR = 2/3`:
**3 of 6 -> 6 of 6**, 0 of 65 real questions marked. **A miss has exactly ONE class** (a fasit not in the bundle, b
below k, c cut by quota/budget, d delivered without the citation, e other),
each FORCED by its own fixture, and an unplaced miss takes the row to 0 of
N. **The judge opens the bundle**: `BundleIndex` reads every concept off
disk before a payload is built, a delivered excerpt whose text is not the
concept file's bytes is not a hit, and the title rules are imported from
`quality` rather than written a second time. **Felled means WORSE, never
merely different** -- removing the quota makes row 3 green, and calling that
a kill would credit the gate with a check it does not have; the two
survivors (document prior dead, `RRF_K` flattened) are reported with what
they measured: **0 ranks and 0 deliveries moved**, and a mechanism for each.
Rows 8 and 9 are never green by leaving something out: the three real sets
are read from path + sha256 (`--real wiki|r761|vegnormal`, one Question per
standard where a question cites two), and K2 is **0 of 6 because the gold
set exists nowhere** while the bundles do. Granularity is stated and the two
forms are NEVER summed: a citation-level set and a concept-level set are two
numbers. Nevnerne re-measured against the sources 2026-09-19 and they hold:
R761 **8** (S1-S6 + KP + KN), vegnormal **32** questions / **43**
`must_cite` / 32 `must_contain`, K2 **6**. The fourth set's own figures are
a consumer's and are not restated here; the gate reads it by path and
sha256 like the others. PM's mixed sum over two sets at two granularities
is carried labelled as recorded, never as the gate's own.
- **Judge a bundle: `okf quality <bundle>` (G37, 2026-09-12).** A per-file-type
verdict, with the denominator on every line, and it is a SEPARATE command
from `okf check` on purpose: `check` is the contract check, and a green one
is not a quality gate -- measured 10.09 by `vegnormal-okf`, three arms over
one corpus all gave 0 findings and exit 0 while hit@k ranged 6 of 6 to 0 of
6. Three verdicts and no fourth (`PASS` / `FAIL` / `UNMEASURED`), a type with
no measured threshold is **never** `PASS`, and exit **3** exists for
"nothing could be judged" so exit 0 over a table of unmeasured rows cannot
be a silent pass (0 clean, 1 a `FAIL`, 2 did not run). Two bars today, both
`structure_null_share` off the pinned 43-document reference -- `.pdf` 8/32,
`.docx` 2/5 -- plus one definitional bar for every type (0 empty bodies,
measured **0 of 8 602** concepts over four bundles). A bar needs **five
documents on BOTH sides**, its own and the judged bundle's, which was found
by RUNNING the gate: a one-PDF bundle scored 0 of 1 against the 32-document
reference and read as `PASS`. **The bars are regression bars against a
pinned artifact, never a quality claim**, and the defect that started G37 --
the HTML arm's 1 148 of 2 761 boundaries -- is `UNMEASURED` here, because no
bundle-only metric reaches it: three candidates were measured over the same
four bundles and **two order the known-bad and known-good arms the wrong way
round** (duplicate titles within a document 0/3 206 against 349/2 761; short
concepts 5.6 % against 14.6 %), while the third (duplicate titles across the
whole bundle, 37.8 / 16.3 / 12.6 / 5.7 %) orders them correctly and ships
anyway WITHOUT a bar, since any bar separating them sits between the two
bundles that define it. **The gate's denominator is the bundle's, never the
corpus's**: a rejected document leaves no row at all (the pinned corpus holds
33 PDFs, the bundle shows 32), so the run log is printed beside the counts
and a bundle without one says so. Three of the four evidence corpora
(`n100`/`n200`/`n500`) carry `source_file` on **0 of 446 / 0 of 1 133 / 0 of
270** concepts, so they name no file type and every row is `UNMEASURED` --
the order expected them to PASS. Thresholds, the nine bundles and the
premises re-measured: `docs/2026-09-12-g37-terskler.md`. README publishes the
bars behind `<!-- quality-thresholds: ... -->`, pinned to the code AND the
document by `tests/test_docs_promises.py`.
**`--fasit <json>` REACHES that defect (G37b, 2026-09-13)** and is the only
input this gate takes: one whole-bundle row, `boundary_share` = declared
boundaries that became a concept over declared boundaries. It is whole-bundle
and never per file type, because a fasit names ONE document's sections and a
bundle can spread them over 828 source files -- which the known-bad arm does.
**The normalisation was derived before the metric was built**, not guessed:
strip all whitespace, lowercase, reproduces the fasit's own `norm` from its
own `title` on **2 761 of 2 761** rows (alphanumerics-only scores 58 -- it
eats the `.` in `2.1Hovedprosesser`). **A boundary is recovered in EITHER of
two forms and neither is a fallback**: the concept's normalised title equal to
`norm`, or the pair `(concept's own directory, residual title)` -- the literal
form wants the declared title WITH its numbering token, the pair form WITHOUT,
and no bundle can offer both, because okf's default route moves that token
into the concept id. Measured on the known-good arm: literal **22 of 2 761**,
paired **2 737**, either **2 759 (99.9 %)**; on `r761-2025-d1` the split is
exactly inverted (2 727 literal, 0 paired), so a gate scoring one form alone
reports a 99.9 % arm as 0.8 % and calls it a segmentation defect. The two
forms are `vegnormal-okf`'s M8 correction, which they took verbatim from THIS
repository's round-14 report -- the instrument reproduces both so the two
repos cannot silently measure different things. One bar, at the pinned
artifact's own value: **2 759/2 761 with `corpora = 1`**, and P2 is in the
OUTPUT and not only in the document (`N = 1 corpus` on every boundary row).
The known-bad arm is **1 148 of 2 761 (41.6 %)**, now `FAIL` + exit 1 where
the bundle-only gate gave exit 3. **`--fasit` is an ASSERTION** (the posture
`okf consume --ref` has) that this bundle is a build of the document the fasit
describes: the K2 reference and `n100-2023` both score **0 of 2 761** and read
`FAIL`, which is the assertion being wrong and not the bundle -- a gate
telling those apart would need a bar read off the bundles it judges. The bar
is TIGHT and the cost is published: 2 of 4 R761 builds fall under it (2 752
and 2 727 of 2 761), while **any bar between 41.6 % and 98.8 % separates the
known-bad arm from every R761 build measured** -- the shipped one is the only
point in that interval read off a pinned artifact. An unreadable fasit exits
**2** with its reason, never a quiet `UNMEASURED`, and a fasit under five rows
is `UNMEASURED` (`MIN_DECLARED_FOR_A_THRESHOLD`, the document floor in the
fasit's unit). **Without `--fasit` the command is byte-for-byte what it was**,
held by a test. README publishes this bar behind
`<!-- quality-boundary-threshold: ... -->`; SS 7 of the threshold document
carries the seven bundles and the honesty limits.
- **Serve a bundle over MCP: `okf mcp --bundle <dir>` (one bundle) or
`okf mcp --root <dir>` (every bundle under the roots, none known by name).**
Four tools -- `okf_list`, `okf_describe`, `okf_ask`, `okf_fetch` -- each
carrying its REASON in the description a client reads, and every answer
carrying the bundle id and concept id a claim must be attributed to. The
JSON-RPC is written narrowly with stdlib only: an MCP SDK would be this
package's SECOND runtime dependency on the default install path, and
`test_the_only_runtime_dependency_is_the_security_boundary` pins that list
literally; `uv.lock` is untouched. **NOTHING IS CACHED ACROSS CALLS** -- every
call re-walks the roots and recomputes `bundle_ref`, so a bundle added,
removed or rebuilt while the process runs is seen by the next call with no
restart, no config edit and no code change (measured, 9 of 9 discovery checks
over three bundles written while serving). The cost is paid per call and is
published: **0.75 s** for the identity of a 2 756-concept bundle, **5.6 s**
for one ask. Containment is TWO independent checks -- the bundle's own index
must name the concept AND `connectors.safe_resolve` must place it inside the
bundle -- and a mutant removing either one alone still refuses, with a
different code. A concept above `MAX_CONCEPT_BYTES` is refused whole rather
than truncated, and a directory that cannot be read as a bundle is REPORTED
in `okf_list`'s `unreadable` rather than skipped: an absence with no
denominator is not a boundary. The eval is
`tools/okf_mcp_gate.py`, written RED at `5f1772e` before any server existed;
it speaks real stdio to a SUBPROCESS and never imports the server. Today:
**1 (7/7) - 2 (83/181) - 3 (4/4) - 4 (9/9) - 5 (3/3) - 6 (6/6)**,
`GATE RED: rows 2`. **Row 2 decomposes into three numbers and the middle one
is the finding**: 99 of 181 (bundle, anchor) pairs are present in the bundles
at all, 83 of those 99 were reached, and **0 of 83 were met by `okf_fetch` on
the anchor as a concept id** -- the set's anchors and this library's concept
ids are different vocabularies, so every pair met was met through the ranker
and 83 is a FLOOR on the ceiling, never the ceiling. **The architecture choice
between the two shapes is the OPERATOR's**; the rows are its input. Report:
`docs/2026-09-20-mcp-to-varianter.md`.
- **`okf skill --out <dir>` WRITES THE GENERIC SKILL, and that is the default
since v1.1 ordre A (2026-09-21); `--for-bundle` is the opt-in for the
instantiated form.** The candidate below won, on the cost the per-bundle form
actually carries: its numbers go stale the moment its bundle is rebuilt, one
copy per consuming project, and `okf card` derives the same numbers in under
a second so the generic document neither invents them nor states another
bundle's. The instantiated copy still refuses out loud (`bundle_mismatch`)
rather than answering from stale numbers — safe to keep, not enough to keep
default. Both forms now carry `## Working method` and `## Answer form`,
required by `okf check` and by the contract's SS 2.5/2.6.
- **`okf card <folder>` SEES EVERY BUNDLE UNDER A FOLDER (v1.1 F1,
2026-09-21).** Until then only the server (`okf mcp --root`) could; the
generic skill read the one bundle it was pointed at. A path that carries no
`index.md` is a folder (`mcp_server.is_bundle`, discovery's own rule), and the
command prints `mcp_server.overview`: `okf_list` and `okf_describe` with no
bundle named, joined, computing nothing of its own -- one source, two doors,
held by a test comparing the printed bytes against the two functions. A
bundle path prints its card exactly as before. Tests over two invented
bundles: `tests/test_folder_of_bundles.py`.
- **`okf consume <folder>` ASKS EVERY BUNDLE UNDER IT IN ONE CALL (v1.1 F2).**
The reply is `mcp_server.call_ask` with no bundle named (or `--bundle-id` as
its `bundle_id`), serialised by the pre-pass's own `serialise` -- no ranking
of its own, held by a test comparing the bytes. `--question` repeats as
before. Every other flag acts on ONE bundle's cut and the server takes none
of them, so over a folder it is REFUSED by name with exit 2
(`consume.FOLDER_FLAGS` is the allowlist), never dropped; `--bundle-id` on a
bundle path is refused the same way. A bundle path reads exactly as before.
- **`okf check` READS A FOLDER'S REPLY AND THE SKILL TEXT SAYS SO (v1.1 F4).**
`contract_check.check_reply`: a reply carrying `answers` and no `bundle` is
one payload per bundle, each held to all 19 rules on its own; a finding is
named `[bundle_id]`, one every payload carries alike is reported once
unnamed (it is the SKILL's), an answer whose label is not its payload's
bundle is `answer_misattributed`, no answer at all is `payload_invalid`. No
rule was added to `RULES` -- the count stays 19 and a single payload's report
is byte-for-byte as before; a folder's report says `over N payloads`.
Contract SS 2.5.4 names the folder run, SS 8.11 fixes the reply. The
template's step 1 and 4 name the folder (`<FOLDER>`: both generators fill
a lower-case instruction, never a path -- the bundle's parent written
absolute named a checkout, and `test_the_generated_commands_name_this_repository_nowhere` fell on it); the generic header says the server comes first and the skill
is the supplement, and that `--ref` belongs to one bundle.
- **THE SERVER IS THE STANDARD WAY IN AND THE SKILL THE SUPPLEMENT (v1.1 F3,
operator 2026-09-21).** `okf project`'s closing text and README's first
screen say it in that order: register `okf mcp --root` once (every project,
subagents too); the skill beside the bundle is for someone who would rather
register nothing; neither is made again when a bundle is rebuilt. Two tests
hold the ORDER, not just the presence, in both places.
- **`okf card <bundle>` and the generic skill are the one-to-many form.** The card is one bundle's identity, concept count,
conditional-field counts and whole-bundle cost as JSON, **DERIVED on every run
and never written into the bundle** -- storing it would move the bytes of all
six `examples/*/expected-bundle` trees (23 files compared byte-for-byte) and
of the pinned reference bundle, to keep something recomputable in under a
second, and a stored card is one more artefact that can disagree with the
bytes beside it. `okf skill --generic` writes ONE installable consumption
skill for ANY bundle: it carries no bundle's identity and no bundle's numbers,
and the property that makes that checkable rather than asserted is that
`skill.render_generic()` **takes no argument** -- there is no bundle it could
have read. Measured 2026-09-20: two per-bundle skills are identical on **281
of 313** and **311** lines, and the 62 that differ are exactly identity,
concept count, the conditional-field table, the whole-bundle cost and the
breaking point -- the five things that go stale on a rebuild. The update
drill, four artefact classes: MCP one-to-one **0 artefacts / 0 steps**, MCP
one-to-many **0 / 0**, today's per-bundle skill **1 / 1 per consuming
project** (it refuses out loud through `bundle_mismatch`, so its cost is not
silence), generic skill **0 / 0**.
- **THE DEFAULT RANKING IS BM25 SINCE v1.1 ORDER C (C1+C3), 2026-09-21**
(`bm25.py`, `consume.DEFAULT_RANKING`): the best 500-character passage and a
title/path-weighted field, fused by rank; the document prior and the rarity
weight are out of the default. Three rules came over from the fusion because
the suite showed what BM25 alone lost: a directory every id shares is not
read, an identifier (`4.2`) stays one token, and a question word the bundle
lacks is read as the bundle's words it shares a leading WORD with
(`tokens_match`). A concept over `PASSAGE_CHARS` (4 000) is delivered as its
answering passage (`as_passage`, `passage: {start, end, of}`).
`--ranking fusion` keeps everything below reachable; the fusion-mechanism
tests and the reference-bundle pins ask for it by name. The retrieval gate
still measures the DEFAULT, so four of its fusion-built premises are
`xfail(strict=True)` until re-measured. **C4:** `coverage.absent_terms`
(question words held in NO form, after the bridge) and `coverage.weak`
(one such word, or nothing delivered) -- a reading, not a verdict; the
retrieval gate's `marked` reads it beside its own bar. Words that only frame
a question are stopwords in both languages. **C2:** `build_multi_payload`
(`okf consume --question A --question B`, `okf_ask` `questions`) reads the
bundle once (`bm25.prepare`), cuts each sub-question as alone and
interleaves the deliveries round-robin under the same `k`/`limit`; one
question is `build_payload`'s bytes. The search gate's (e)/(f) go through it.
**C3 title inheritance:** a concept titled `Tabell linje N` (the proposer's
mechanical table-block name, `consume.MECHANICAL_TITLE`) is read under the
nearest concept above it in its document (`inherit_table_titles`, ordered by
`source_offset` else `source_lines`), in ranking, excerpt and near misses;
the excerpt carries `own_title`. A reading only -- no bundle bytes move.
**C5 the map:** `bundlemap.build_map` -- one line per source document with
its concept titles in document order, a series (names differing only in
numbers, `SERIES_MIN` = 5) as one line with count and span, at most
`TITLES_PER_LINE` = 24 titles a line, lines capped at `MAP_MAX_BYTES` =
48 000 (`lines_truncated`). The card (`okf card`, `okf_describe`) carries it
as `map` and no longer carries `source_files`. The skill and the server
instructions say: read the map, 2-4 sub-questions in its words, ONE call.
**C6:** the contract's SS 8 example and points 1, 7-10 state the current
form (`okf-consumption/2`'s `withheld` mapping, `absent_terms`/`weak`,
`passage`, `questions`/`subquestions`, `own_title`); `okf check` gains
`passage_malformed` and `subquestions_unindexed` (19 rules). Editing the
contract moved the known-positive to 23 672 / delta 580, and v1.1 F4's
SS 2.5.4 / SS 8.11 edit to **24 620 / delta 592** (`wc -c` 24 028).
What follows describes the fusion.
- Consume a bundle: `okf consume <bundle> --question "<q>"
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
`docs/consumption-contract.md` § 1 defines, and the only reading direction
this library has. **It moved into the package 2026-09-08 (O5)** and the move
it was written for is the one that happened: `build_payload(...)` was always
the entry point with the CLI a thin `main()`, so it was a move and not a
rewrite. What forced it was the generated skill — from `tools/` it emitted
`python3 <absolute path>/tools/okf_consume.py`, so the skill could not be
moved, shared or run by anyone without that clone. `tools/okf_consume.py`
remains as an ALIAS (`sys.modules[__name__] = _impl`, never a re-export: a
re-export binds copies, and a caller patching one patches a binding the
implementation never reads). Deterministic and offline by construction: no model call, no socket,
no clock, stdlib plus this package only. It **walks the index tree, never a
directory** — § 9.2 forbids enumerating one unless the named profile says the
index is derived, and measured, `entries_match_directory` is `True` for
`STRICT_V1` alone; the walk loses nothing (629 = 629 on the K2 bundle,
controlled in a test against the very method § 9.2 forbids). `--ref` is an
**assertion**, never an override: the emitted identity is always the computed
one, because § 3.3 exists to stop a payload being labelled with an identity
its bytes do not have. Three exit codes: 0 written, 1 refused, 2 did not run.
**Every excerpt carries the concept's `title`**, plus `req_number`, the § 5.1
address `sources`, and **every top-level `source_*` key by PREFIX** — never an
allowlist, because a list names the producers its author thought of and one
bundle locates by `source_element_id` on 269 of 274 concepts. A prefix, never
a substring (`resource_owner` is not a locator). An absent key stays absent
and an undecodable address is named (`sources_unreadable`). Since K3-21 an
excerpt also carries `parent`, the enclosing concept's `concept_id` and
`title` -- never the raw `segment_id` -- or `parent_unresolved`, and the
contract's SS 8 point 6 says the checker reads it. `sources` is READ
in both YAML forms because the two real bundles disagree (flow 629/629 on one,
block 270/270 on the other) — reading block is not a licence to write it, the
emission rule is unchanged. Contract § 8 makes `title` a MUST (checker code
`excerpt_unnamed`) and the rest SHOULD, because they are conditional on the
producer. The measurement behind it: rank 1 of 8 on 3 of 3 bundles, correct
answer on 1 of 3.
- Connect a bundle to Claude Code: `okf skill <bundle> --out
<dir>` instantiates `skills/okf-consume-template/` for THAT bundle — its id,
ref, concept count, conditional-field denominators, whole-bundle cost and
breaking point, all measured, plus a reference payload the checker accepts.
It was kept in `tools/` until 2026-09-08 because a wheel-installed
`okf skill` would emit a command pointing at a file the wheel does not carry.
That objection was about what the GENERATED skill NAMES, and O5 answered it
by changing that: the emitted commands are `okf consume` and `okf check`,
names on PATH. The template and `docs/consumption-contract.md` (the § 7.4
known-positive) are force-included into the wheel from the file they are
authored in — one authored copy, no committed duplicate.
The form was chosen on a measurement: the contract checker passed the
UNFILLED template and passed a skill built for another bundle, so it could not
tell the two apart — the choice rests on § 5/§ 6.4/§ 7.6 being per-bundle
numbers a generic skill can only leave as holes or state falsely, and that
argument NEVER rested on conformance, so closing the measurement leaves it
standing. **That half is CLOSED 2026-09-10 by `bundle_mismatch`, the checker's
SIXTEENTH rule** (seventeen since K3-21 added `parent_unfollowable`;
`RULES` is a tuple and `Report.rules_evaluated` is
`len(RULES)`, so every published «15 rules» line is now «16 rules» — a
contract change for anyone quoting it). It compares the identity a skill
DECLARES against the identity its payload declares, **both `bundle_id` AND
`ref`**: three distinct builds on this machine carry the one `bundle_id`
`k2-trinn1-20260903` at three refs, so an id-only rule would pass a stale
skill, which is the case the generated skill warns about in its own words.
SS 3.3 is the ground — «a version is the producer's assertion; a ref is a fact
about bytes» — and SS 3.1 for the same question one level down, an excerpt
naming a bundle its payload does not. Reproduced on `113b3f8` before any code
moved, three pairs at `conformant: 15 rules over 8 excerpts and 438 withheld
entries, 0 findings` and rc 0; after, all three at rc 1 with **one** finding
over 16 rules — a foreign corpus's payload, the unfilled template, and (the
arm that separates a whole rule from half of one) a payload sharing the id at
a foreign ref. **An identity the rule cannot READ is a finding, never a silent
pass**, or the template passes again. The right pair is untouched at rc 0 / 0
findings, and `{}` is unchanged at **9** findings because a payload declaring
no identity stays `ref_missing`'s defect — no rule restates another. NO new
field was needed: the identity already lived in the generated skill's prose,
now one authored copy in `skill.identity_line` read back by
`contract_check.skill_identity`, generated skill bytes byte-identical before
and after on both tracked bundles. **The rule compares DECLARED against
DECLARED and never opens the bundle**, so a payload lying about its own `ref`
still passes — that is `okf consume --ref`'s job and needs a bundle path this
command deliberately does not take; «closed» means the three measured forms
now fell, not that no fourth exists. Report:
`docs/2026-09-10-k3-runde15-bundle-mismatch.md`.
The first instantiated consumption skill is `skills/okf-consume/`; the
measurement behind the form, including the control that FAILED, is
`docs/2026-09-07-okf-konsumskill-maaling.md`. **That skill was the new rule's
first REAL find**: hand-filled for K2 before `okf skill` existed, it declared
no bundle identity a reader can act on, so `okf check` refused it against its
own shipped example payload (rc 1, 1 finding) — **1 of 1** shipped hand-made
instantiated skill. **Since 2026-09-11 (K3-18) it is GENERATED** by `okf
skill` from `examples/ingest-golden-segmented-okf-v0-2/expected-bundle`, the
bundle its payload always came from, with the command in its
`references/README.md`: `--force` and `--example-question "Hva sier
veiledningen om krav?"` are both required (the payload test asserts bytes for
that question), and the checkout prefix is then stripped, because `okf skill`
writes the bundle root and the skill path ABSOLUTE when `--out` is not under
`.claude/skills/`. The pair is rc 0, 19 rules since v1.1 C6 (17 before, 16
before K3-21), 0 findings,
and a test holds
the shipped bytes to the generator's. Its frontmatter `name` is now
`b-golden-segmented-okf-v0-2-consume`: Claude Code takes a project skill's
COMMAND from the directory name and uses `name` only as a display label, and
nothing here named the skill `okf-consume`. Report:
`docs/2026-09-11-k3-runde18-konsumskillen-regenerert.md`. **The ranking is this
repository's own choice** — the contract binds a payload, not a retrieval
algorithm (§ 10) — and it has FOUR optional widenings. Three are **off by
default** and keep the default payload byte-identical; the fourth
(`--tie-shared-rank`) became the default 2026-09-10 and is the one change in
this repository that alters a payload with NO bundle changing, so a consumer
pinned to the old excerpt order needs `--no-tie-shared-rank`. `--cost-vocabulary`: a
declared cost/price/quantity vocabulary family that bridges a question and a
document naming money with different words, gated on the QUESTION carrying
such a term, so a question without one is byte-identical either way. It moves
a measured case from candidate rank 249 to 10 and does **not** deliver it —
the budget is a second, independent lock. Measured, with two rules falsified
before building and the `k`-sweep that showed a higher `k` can EVICT a gold
concept, in `docs/2026-09-08-blindsone-below-k-k2.md`.
`--reserve-top-rank` is that second lock: the pack is an exact knapsack over a
SUM, so it has no opinion about rank and out-sums a top-ranked excerpt costing
a large share of the budget. The flag gives rank one its bytes first, AFTER
the `over_budget_alone` pre-exclusion and never before, and declares
`budget.reserved` in the payload. It fixes the eviction and does **not** close
the mandate-shaped blind spot (that concept ranks 10, not 1); the budget stays
the caller's decision, because deriving a limit from the corpus was measured
and falsified — two defensible derivations, 49x apart, one of them breaking
the known-positive. `docs/2026-09-08-blindsone-laas2-budsjett-k2.md`.
**A SIXTH flag, `--stem-prefix`, is ON since 2026-09-09** and is the second
change here that alters a payload with NO bundle changing (opt-out
`--no-stem-prefix`). `MIN_SHARED_PREFIX = 4` exists for Norwegian
compounding and also matches four characters that are not a stem: on the
pinned 453-concept bundle, control first, `under` occurs 79 times by equality
and matches 172 by prefix, `bilateral` occurs **0** and matched **400 of
453** through `bilag`, `standhaftig` 0 and 219 through `standard`. The two
extra known-negatives were FOUND, not chosen -- every 4-character prefix
ranked by document frequency, then a real word taken from the widest. **Three
candidates were measured and all three failed on the SAME row**: a longer
floor (5-8), a coverage share (0.5-0.8) and a long-words-only floor (>= 8).
Decomposed, row 1's token `prisene` reaches its gold document through
`pris|sammenstilling` on four characters -- 0.57 of one word and 0.22 of the
other -- so **the over-match and the wanted match are one mechanism** and no
threshold on length or coverage separates them. The fourth candidate does:
the shared prefix must be a WORD the bundle uses. `bilateral` 400 -> 0 and
512 -> 0, `standhaftig` 219 -> 56 and 235 -> 33, every hit@8 row keeping rank
1 on BOTH bundles. `undersjoisk` stops at **162** because `under` IS a word
here -- a genuine Norwegian morpheme, so that residual is a different answer,
never a ceiling. The vocabulary is the BUNDLE's own, so the rule makes a
payload corpus-dependent the way `rarity_weights` already is.
**A SEVENTH flag, `--source-quota N`, is ON at 2 since 2026-09-10** (opt-out
`--no-source-quota`) and is the third change here that alters a payload with
NO bundle changing. It caps how many DELIVERED places one `source_file` may
take, cutting where `shortlist = candidates[:k]` cuts, so the freed place goes
to the next candidate and `k` is still delivered in full. The defect was
measured OUTSIDE this repo on a 3206-concept bundle of a published handbook:
the code's own process overview is **28 of 3206 concepts (0.87 %)** and **8.0 %
of the source characters** yet took **8 of 8** delivered places on one question
and **7 of 8** on the known-positive, which was not delivered at all --
identical at 343 and 1651 concepts, so it is the corpus's COMPOSITION (it holds
its own table of contents) and not its size, and a split would move it rather
than remove it. Swept over {2, 3, 4, off} on three bundles with the fasit
prefixes validated against the bundle FIRST (that control caught a defect in
the measuring query itself): at 2 and 3 hit@8 goes **5 of 6 to 6 of 6 on BOTH
K2 bundles** with all five standing rank-1 rows unmoved -- the recovered row
had missed on every bundle and every configuration measured until now -- and at
4 and off it stays 5 of 6. On the handbook bundle hit@8 goes **2 of 6 to 4 of
6** and the dominant document's share **8 of 8 to 2 of 8**. 2 rather than 3 on
rank. **What the gain is NOT:** hit@8 asks whether the gold DOCUMENT was
delivered and a document quota raises how many distinct documents a payload
holds, so that metric is not neutral with respect to this rule; the five rows
already at rank 1 are, and did not move. **The adverse case is named:** a
one-document bundle has one `source_file` on every concept, so the quota would
deliver 2 where `k` were asked -- the shortlist is topped back up from the
best-ranked over-quota candidates, making such a bundle byte-identical to the
quota being off. The `WITHHOLDING_RULES` vocabulary goes six to seven
(`source_quota_exceeded`, a DIVERSITY drop and not a relevance one) and is
published in the contract SS 5.3 and in the generated SKILL.md, verified by
reading the generated file. Editing the contract moved the SS 7.4
known-positive (12 563 -> 13 238 encoded, delta 336 -> 345), which is that
coupling working. `--rarity-weight` was measured against the same defect and
does NOT repair it -- it leaves the dominant document at 8 of 8 places on the
question it floods -- and stays off.
`--rarity-weight` is the third: each lexical hit weighs `log(N/df)` over the
bundle's own concepts instead of 1, so an identifier is not worth what a
common verb is worth. It enters the RANKING and never the GATE — `lexical`
stays a count, because a word every concept carries weighs exactly 0 and a
weighted gate is what `54a0bc2` falsified. Off by default BY MEASUREMENT: it
delivers one of three requirement lookups and takes a priced sheet from
candidate rank 10 to 2, leaves one gold unmoved and costs another seven rank
positions. Two limits are decomposed rather than guessed, and both are
someone else's mechanism: `MIN_SHARED_PREFIX = 4` makes a unique identifier
read as 135-of-446 common, and RRF consumes RANKS, so no weighting inside a
signal can move a gold that already leads it.
`docs/2026-09-08-sjeldenhetsvekt.md`.
`--tie-shared-rank` is the fourth and **the only one that is now ON**
(2026-09-10, opt-out `--no-tie-shared-rank`). It is a correction to the
TIE-BREAK
rather than a weight: RRF ranks every concept in every signal, including a
signal that scored them all the same, and the declared `(-score, concept_id)`
tie-break then orders that group by id. Measured on N500, whose document
prior has **two** distinct values over 270 concepts, that signal contributed
alphabetical UUID order and put a concept answering 7 of 7 question tokens at
fused rank 14 — outside the cut — behind concepts sharing only `tunnel` and
`vann`. Under shared ranks it is rank 3 and 2 of the 16 covering concepts are
delivered. **It shipped OFF on a measurement that was CONDITIONAL and stopped
being true in a commit reported as changing nothing.** The published cost —
hit@8 falling 5 of 6 to 4 of 6 — is real only at `DOCUMENT_PRIOR_EXPONENT`
1.0. Round 6 moved that exponent to 0.5 for an unrelated reason and correctly
reported it moved no hit@8 row; nobody measured the PAIR. Swept 2026-09-10
over 2 exponents x 3 bundles x 6 rows: at 0.5 the rule holds
`[1,1,1,1,1,–]` on all three bundles and FIXES the split bundle's row 1
(2 -> 1), which is what let `--sheet-section-rows --keep-table-heading`
become a build default. **A flag's "off by measurement" is a measurement of a
CONFIGURATION, not a property of the flag** — when a constant it interacts
with moves, its default is unmeasured again, and nothing in the tree says so
because the two decisions live in different files. The adverse case is
recorded rather than hidden: on a synthetic 30-concept fixture where one
signal separates and two do not, shared ranks move a gold from rank 18 to 30
(`tests/test_okf_consume.py`). Note also that
`docs/2026-09-08-sjeldenhetsvekt.md`'s figures were measured under the older
tie-break and are NOT re-measured — on one fixture the change takes the
weight's gold from fused rank 18 to 1.
`docs/2026-09-08-rangeringsbom-sammensatte-ord.md` and
`docs/2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md`.
The other three stay off.
**`--withheld-titles` IS RETIRED (v1.1 ordre A, 2026-09-21), together with
the flat `withheld` list it existed to name.** Measured on a large real
bundle: the list came to **65.5 % of the written payload**, and not one of
those bytes counted against the budget the same payload reported — so a
reader was handed most of a file the budget line did not know about, to
learn one concept id per withheld concept with nothing beside it. The flag
bought the missing field and was off because buying it for a list that long
cost another 37.9 %. (The corpus is a consumer's; its counts are not
restated here, and the ratio is the argument.)
**`withheld` is now a MAPPING** (`CONTRACT_REVISION` = `okf-consumption/2`,
bumped so a consumer indexing the old key as a list breaks loudly):
`total` (equal to `denominators.withheld`, so SS 5.2's identity closes on the
NUMBERS), `by_rule` (the same total over the closed rule set), `nearest`
(the best-ranked drops BY NAME with title and source document, read off
`ranked` and not off `cut`'s id-sorted output, capped at
`WITHHELD_NEAREST_DEFAULT` = 20) and `complete`. Same question, same bundle,
after: **18.4 % of the old file**. `--withheld-full` brings the
whole list back and is what an instrument classifying EVERY miss asks for —
the retrieval gate and `okf_consume_measure` both pass it and assert
`complete`. `--withheld-nearest N` sets the cap. Three checker rules moved
with the shape, and one is the interesting case: `parent_unfollowable` used
`excerpts` + `withheld` as the bundle's own denominator, which a truncated
block is not, so that clause runs only where the payload SAYS it is complete
— stated in SS 8.6 rather than left as a silence.
## Workflow
- TDD: no production code without a failing test first.
- This repo is published PUBLICLY (`open/` namespace on Forgejo). `STATE.md`
and `docs/oppstartsprompt.md` are LOCAL-ONLY (gitignored) — never commit
session state or internal briefs. No secrets, sober English prose, no
marketing language.
- **Consumer content stays at form level in public files.** Some consumers we
read are private (`claude-code-llm-wiki` is, pending an Anthropic ToS
assessment). Key names, counts, gate names, profile fields and contract shapes
are publishable; page bodies, full title or path lists from a consumer's
bundle, and Anthropic-derived prose are not. Findings about a private
consumer's data go back to them through coord, never as a file here. This
costs nothing — every question this library asks of a consumer is about shapes
and key sets — and it is not reversible once pushed.
- After `git commit`: push to Forgejo (`git push origin`) immediately.
Never GitHub.
## Communication patterns
### Linking to local files
When pointing to local files in responses, always use markdown link syntax
with a descriptive name:
- Use `[Human-friendly name](file:///absolute/path)` — never bare
`file:///...` URLs or autolinks `<file://...>`.
- Always use absolute paths. Never `~/` or relative paths.
- For multiple files, render as a bullet list of named markdown links.
Why: bare `file://` URLs only render the first as clickable across multiple
lines. Named markdown links make each entry independently clickable and look
cleaner.
Reference in a new issue View git blame Copy permalink