llm-ingestion-okf/docs
Kjell Tore Guttormsen e3169ec50c feat(consume): the withheld set is counts plus names, not one entry per concept
Measured 2026-09-20 on a 2313-concept bundle of one project's own
documentation: `withheld` held 2 305 entries = 186 440 B of compact JSON =
**65.5 % of the 284 850-byte payload**, and not one of those bytes counted
against the budget the same payload reports (`spent` was 45 192). A reader was
handed 239 658 bytes the budget line did not know about, to learn 2 305 concept
ids with nothing beside them -- the title being exactly what `--withheld-titles`
existed to buy, and which was off because buying it for 2 305 entries cost
another 37.9 %.

`withheld` is now a mapping: `total` (equal to `denominators.withheld`, so
SS 5.2's identity is unmoved and closes on the NUMBERS), `by_rule` (the same
total decomposed over the closed rule set, so "what kind of drop" is answerable
without the list), `nearest` (the best-ranked drops BY NAME, with title and
source document, so a reader who sees a near miss can ask for it) and
`complete`. The near misses are read off the ranking, not off `cut`'s output:
`cut` sorts by id so the partition is comparable, and that order says nothing
about which concept a reader might want next.

Same question, same bundle, after: **52 421 bytes, 18.4 % of the old file**.
The whole list stays reachable behind `--withheld-full`, and the two
instruments that classify EVERY miss by its rule -- the retrieval gate and
`okf_consume_measure` -- now ask for it explicitly and assert `complete`
rather than assuming it. `--withheld-nearest N` sets the cap (default 20,
which is `k` plus the next twelve). `--withheld-titles` is retired: a flag
whose only remaining effect would be to STRIP the title from a list the caller
asked for in full names no decision worth two shapes for one list.

`CONTRACT_REVISION` moves to `okf-consumption/2`, because a consumer indexing
the old key as a list would otherwise break silently. Three checker rules move
with it, and one of them is the interesting case: `parent_unfollowable` used
`excerpts` + `withheld` as the bundle's own denominator, which a truncated
block is not -- so that clause now runs only where the payload SAYS it is
complete, stated in SS 8.6 rather than left as a silence, with the other two
clauses (shape, self-reference) running either way. `Report` carries both
denominators, because a report claiming it examined 2 305 entries it never saw
is the same defect one level up.

The generated skill's "breaking point" section goes with it: it extrapolated a
concept count from the cost of ONE withheld entry, and there is no such slope
any more. It now states what this bundle's bookkeeping cost and that the block
is bounded by the cap rather than by the bundle -- an extrapolation from a
slope the code no longer has would be a measurement of the previous revision.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 23:32:22 +02:00
..
plan feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
2026-08-21-g2-pdf-extraction-measurement.md docs(extract): measure one Vegnormalene PDF page against the extraction registry 2026-08-21 19:16:15 +02:00
2026-08-29-konverteringsarmen-arm-a.md docs(measure): run the conversion arm on K2 - pandoc wins both office formats 2026-08-30 00:41:48 +02:00
2026-08-29-telling-offentlig-prosjektkorpus.md docs(measure): count the public project corpus - K2 measured, K1 blocked 2026-08-29 07:03:50 +02:00
2026-09-02-k3-k4-k5-baseline.md chore(tools): remove okf_adjudicate -- nothing here starts Claude Code 2026-09-20 15:41:33 +02:00
2026-09-02-k3-k4-k5-metode.md chore(tools): remove okf_adjudicate -- nothing here starts Claude Code 2026-09-20 15:41:33 +02:00
2026-09-02-korpuskjoring-k1-k2.md docs(measurement): K1 and K2 over the K2 corpus 2026-09-02 15:43:45 +02:00
2026-09-03-k2-bundle-rebuild.md test(corpus): a rebuild with log.md present reproduces the bundle exactly 2026-09-03 05:16:25 +02:00
2026-09-04-cid-andel-k2.md docs(measure): CID-glyph share across K2 -- Bilag 9.1 alone, not a K1 arm 2026-09-04 20:23:41 +02:00
2026-09-04-k2-pptx-odt-rtf.md docs(measure): K2 on pptx/odt/rtf -- denominator 0 in both named corpora 2026-09-04 21:12:41 +02:00
2026-09-04-k3-arm-c.md docs(measure): K3 with Arm C beside the baseline, two numbers, no threshold 2026-09-04 17:58:50 +02:00
2026-09-07-k2-pptx-odt-rtf-fixtures.md test(fidelity): K2 on pptx, odt and rtf -- 59/59 and 20/20, on a set we wrote 2026-09-07 05:22:12 +02:00
2026-09-07-k3-arm-d.md docs(measure): K3 Arm D beside a re-rated Arm B -- reach and no threshold 2026-09-07 03:04:01 +02:00
2026-09-07-k3-arm-e.md docs(measure): K3 Arm E beside Arm B and Arm D -- three rows, no threshold 2026-09-07 11:48:58 +02:00
2026-09-07-k3-enhetsark-metode.md docs(measure): K3 unit worksheet delivered for operator, no code changed 2026-09-07 16:36:16 +02:00
2026-09-07-okf-konsumskill-maaling.md docs(consume): refutation sweep — smoothed-IDF rarity weighting cannot zero both known-negatives 2026-09-07 09:56:36 +02:00
2026-09-08-blindsone-below-k-k2.md feat(consume): measure the below-k blind spot, add one flag-gated vocabulary bridge 2026-09-08 04:55:45 +02:00
2026-09-08-blindsone-laas2-budsjett-k2.md feat(consume): measure the budget lock, add one flag-gated top-rank reservation 2026-09-08 05:18:30 +02:00
2026-09-08-claude-code-skill-vilkaarlig-bundle.md docs(consume): the Claude Code recipe, measured end to end on two bundles 2026-09-08 15:32:32 +02:00
2026-09-08-eksakt-oppslag.md fix(consume): match an identifier by equality, deliver the concept a question names 2026-09-08 13:27:59 +02:00
2026-09-08-k3-arm-f-mot-enhetsarket.md feat(propose): Arm F, one unit fold behind a flag, measured against the operator's worksheet [skip-docs] 2026-09-08 16:28:41 +02:00
2026-09-08-k3-runde2-per-filtype.md feat(cli,propose): reach the arms from okf build, keep a sheet heading behind a flag, fix two PDF contents mechanisms 2026-09-08 18:21:25 +02:00
2026-09-08-k3-runde3-per-filtype.md docs(cli): the new default's K2 digest, measured on the bundle it produces 2026-09-08 21:14:51 +02:00
2026-09-08-k3-runde4-pdf-skrift-og-ocr.md feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off 2026-09-08 23:10:47 +02:00
2026-09-08-k3-runde5-hitat8-og-skriftakse.md feat(propose,cli): typography as a reserve, and the two of our own numbers it took to measure it 2026-09-09 00:25:51 +02:00
2026-09-08-kravnummer-tokenisering.md feat(propose,cli): the sheet's rows, the wrapped sentence, and a default that moved 2026-09-08 20:34:18 +02:00
2026-09-08-o5-okf-project.md feat(cli): okf project/consume/check/skill, and a generated skill with no path into a checkout 2026-09-08 21:47:38 +02:00
2026-09-08-prisform-og-loggen-k2.md fix(extract,build): write a spreadsheet as pipe tables, stop linking the run log from the index 2026-09-08 10:06:58 +02:00
2026-09-08-proveniens-k2.md feat(inbox): point every concept at the document it came from, with a locator per format 2026-09-08 14:39:24 +02:00
2026-09-08-rangeringsbom-sammensatte-ord.md fix(consume): the compound-word miss is a degenerate signal's tie-break, behind a flag 2026-09-08 18:53:18 +02:00
2026-09-08-sjeldenhetsvekt.md feat(propose,cli): the sheet's rows, the wrapped sentence, and a default that moved 2026-09-08 20:34:18 +02:00
2026-09-09-k3-runde6-outline-gaten-og-prioren.md feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment 2026-09-09 14:17:00 +02:00
2026-09-09-k3-runde9-restene.md feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
2026-09-09-k3-runde10-rtf-prefiks-shim-ruff.md docs(k3): round 10, and the fasit that held a working rule off the default [skip-docs] 2026-09-09 23:27:03 +02:00
2026-09-09-o6-forsteskjerm-og-skillmodi.md feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id 2026-09-09 21:45:08 +02:00
2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md feat(cli,consume): the first span, and the fusion that punished fine-graining 2026-09-09 15:40:01 +02:00
2026-09-10-k3-runde11-html-og-dominans.md docs(k3): round 11, and the corpus that rebuilt byte-identical [skip-docs] 2026-09-10 01:10:59 +02:00
2026-09-10-k3-runde12-pdf-outlines.md feat(propose): the PDF shipped a structure index and the build discarded it unopened 2026-09-10 02:27:41 +02:00
2026-09-10-k3-runde14-deklarert-struktur-tar-ruten.md docs(k3): round 14, the declared-structure route at the shipped defaults 2026-09-10 07:53:22 +02:00
2026-09-10-k3-runde15-bundle-mismatch.md feat(check): a skill and a payload naming different bundles is a finding 2026-09-10 23:42:06 +02:00
2026-09-10-k3-runde16-hele-tittelen-tar-ruten.md docs(k3): round 16, the whole title takes the route precision could not 2026-09-10 22:26:57 +02:00
2026-09-10-r1-release-v080.md docs(release): R1, v0.8.0 tagged locally and installed from the tag 2026-09-10 21:14:19 +02:00
2026-09-10-r3-release-v081.md docs(release): R3, v0.8.1 tagged locally on 3daf983 2026-09-10 23:10:30 +02:00
2026-09-11-k3-runde8-tabellblokk-og-siste-spenn.md feat(propose,cli): the coverage gap had one cause, and round 7's own decomposition did not reproduce 2026-09-09 17:43:56 +02:00
2026-09-11-k3-runde13-xml-sts.md docs(k3): round 13, and the two proposer rules between a reader and its ceiling [skip-docs] 2026-09-10 04:13:56 +02:00
2026-09-11-k3-runde17-dekningen-stopper-ved-en-bredere-tittel.md docs(k3): round 17, a covered title stops beneath a title that answers more 2026-09-11 01:45:53 +02:00
2026-09-11-k3-runde18-konsumskillen-regenerert.md fix(skill): regenerate skills/okf-consume from the golden bundle it ships a payload for 2026-09-11 02:21:35 +02:00
2026-09-11-k3-runde19-dokumentidentitet-og-frontmatter.md docs(k3): round 19, a document's own identity, per-run frontmatter, and a description from the source 2026-09-11 03:59:34 +02:00
2026-09-11-k3-runde20-delt-katalog-og-arvet-kontekst.md docs(k3): round 20, a directory every concept shares and a section whose body is its heading 2026-09-11 05:38:35 +02:00
2026-09-11-k3-runde21-parent-til-leseren.md docs(k3): round 21, parent all the way to the reader 2026-09-11 14:28:42 +02:00
2026-09-11-k3-runde22-yaml-lesbar-frontmatter.md docs(k3): round 22, frontmatter a YAML reader reads back the same 2026-09-11 11:47:40 +02:00
2026-09-11-r4-release-v082.md docs(release): R4, v0.8.2 installed and measured from the tag 2026-09-11 02:40:31 +02:00
2026-09-11-r5-release-v083.md docs(release): R5, v0.8.3 installed and measured from the tag 2026-09-11 06:21:31 +02:00
2026-09-11-r6-release-v084.md docs(release): R6, v0.8.4 installed and measured from the tag 2026-09-11 15:04:21 +02:00
2026-09-12-g37-terskler.md feat(quality): okf quality --fasit, boundary recall against a declared structure 2026-09-13 07:27:33 +02:00
2026-09-12-k3-runde23-stien-i-kroppssignalet.md docs(k3): round 25, the default reading drops the door's link line 2026-09-12 19:07:55 +02:00
2026-09-12-k3-runde24-blokkform-til-de-flate-leserne.md docs(k3): round 24, a reason that became false is struck and the emitter stays flow 2026-09-12 16:50:26 +02:00
2026-09-12-k3-runde25-standardlesingen-uten-lenkelinja.md docs(release): R7, v0.8.5 tagged locally and measured from the tag 2026-09-12 19:47:30 +02:00
2026-09-12-r7-release-v085.md docs(release): R7, v0.8.5 tagged locally and measured from the tag 2026-09-12 19:47:30 +02:00
2026-09-15-f1-f2-gaten-og-kodefencen.md fix(gate,propose): okf build runs a real guard; a code fence declares no structure 2026-09-16 00:19:50 +02:00
2026-09-17-bilder-i-bundlen-trinn1.md feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
2026-09-17-bildestien-0-10-1.md docs(privacy): no private path and no coordination id in what this span added 2026-09-20 15:40:38 +02:00
2026-09-17-innholdsregnskapet-bygget.md feat(accounting): okf build accounts for every source element 2026-09-17 18:35:20 +02:00
2026-09-17-innholdsregnskapet-rod-gate.md test(accounting): content-accounting gate for okf build, written red 2026-09-17 15:40:37 +02:00
2026-09-18-bildestien-holder-0-10-1.md fix(assets): bound every link of the filter chain, and cover the backstop 2026-09-18 15:34:07 +02:00
2026-09-18-filterkjeden-og-backstoppen.md fix(assets): budget every link by what its decoder COSTS (0.10.1) 2026-09-18 19:00:38 +02:00
2026-09-18-regnskapsgaten-herdet.md docs(privacy): no private path and no coordination id in what this span added 2026-09-20 15:40:38 +02:00
2026-09-18-utgangsbudsjett-per-ledd.md fix(assets): budget every link by what its decoder COSTS (0.10.1) 2026-09-18 19:00:38 +02:00
2026-09-19-bmp-resten-terminator-og-pekerblokk.md docs: the BMP rest, the terminator and the pointer block 2026-09-19 09:51:39 +02:00
2026-09-19-eol-stroemmen-som-ikke-maler.md docs(privacy): no private path and no coordination id in what this span added 2026-09-20 15:40:38 +02:00
2026-09-19-gjenfinningsgaten-motstand.md docs(retrieval-gate): the round the gate had to resist the work it judges 2026-09-19 22:13:25 +02:00
2026-09-19-hvert-baaret-bilde-kan-vises.md docs(privacy): no private path and no coordination id in what this span added 2026-09-20 15:40:38 +02:00
2026-09-19-pekerblokken-bindes-til-kjoringen.md test(mutants,assets): a mutant is judged by the suite that owns it, 45 of 45 2026-09-19 19:33:23 +02:00
2026-09-19-regnskapsgaten-rest-og-normaliseringsdoren.md docs(privacy): no private path and no coordination id in what this span added 2026-09-20 15:40:38 +02:00
2026-09-20-mcp-to-varianter.md docs(front-page): the gate numbers the gate actually prints, and a breaking point that was measured 2026-09-20 15:40:20 +02:00
consumption-contract.md feat(consume): the withheld set is counts plus names, not one entry per concept 2026-09-20 23:32:22 +02:00
okf-nokkelinventar.md build(deps): pin llm-ingestion-guard v1.3.0 so the gate reads our own goldens 2026-09-03 20:41:03 +02:00
phase-3-split-table.md docs(phase-3): keep the index-entry grammar on one line 2026-08-04 12:37:02 +02:00
upstream-okf-upgrade-runbook.md feat(watch): weekly OKF upstream watch that can prove it found nothing 2026-08-23 20:38:37 +02:00