llm-ingestion-okf/src/llm_ingestion_okf
Kjell Tore Guttormsen d300338e4d
docs(front-page): the gate numbers the gate actually prints, and a breaking point that was measured
Four claims on the front page were false on this commit, and one of them was a
number no division ever produced.

**The retrieval gate.** README reported it RED on rows 3, 4, 5, 7, 8 and 9,
with row 3 at 2 of 5 and row 4 at 3 of 6. Run on this commit it is RED on rows
5, 7, 8 and 9, with row 3 at 5 of 5 and row 4 at 6 of 6: `f81683e` made a
withheld concept carry the rule that actually decided it, and `05cb190` gave
the payload a `coverage` block, and neither updated the table. Row 8 is `0 of 3
| NOT RUN` on the default run and was published as `44 of 64 questions`, which
is what it scores the day all three private sets are handed to it -- now
labelled with the day and the machine rather than printed as a row. The same
four figures were stale in `CLAUDE.md`.

**The breaking point in a generated skill.** `int(LIMIT / per_withheld) if
per_withheld else 0` printed `At roughly 0 concepts the bookkeeping alone
reaches the 120000-byte limit` whenever the generation run withheld nothing --
the absence of a measurement, rendered as one, and read as a bundle that breaks
before it holds anything. A run with no withheld entry has no slope to
extrapolate from, so the sentence is withheld with its reason. The shipped
`skills/okf-consume/SKILL.md` is generated with the question its
`references/README.md` names, withholds nothing, and carried exactly that `0`;
it is regenerated. Two arms in the test, because one would pass on an empty
set: the bundles that withhold something must still state a positive figure.

The sentence for that arm also stopped saying `**4 bytes** for 3 concepts`
where the 4 bytes were the cost of 0 withheld entries. It is now `for N of M
concepts`, which moves two generated skills' line counts and therefore the
published comparison: 280 of 312 and 310 -> 281 of 313 and 311, re-measured,
with the 62 differing lines unchanged.

**Four tools.** A single-bundle server exposes three: `okf_list` is absent
where there is nothing to list. README's table already said so in a cell; the
heading and the CHANGELOG did not.

**What `--accounting` accounts for.** The account is over the element classes
each format's vocabulary names, verified against `accounting._READERS` rather
than against the report: a file whose suffix has no reader is accounted at file
level only, `.docx` reads `document.xml` and `footnotes.xml` (so headers,
footers, endnotes and comments are outside), `.pptx` reads the slides (so
speaker notes are outside), `.xlsx` reads the worksheets (so cell comments are
outside and a cell contributes its cached value, never its formula), and `.rtf`
skips its header and footer groups. A hidden slide or sheet IS counted -- it
lives in the same part as a visible one. Nothing is built for this; the list is
what `0 unaccounted` does not claim.

Gates re-run on the commit: retrieval `GATE RED: rows 5, 7, 8, 9` (exit 1),
MCP `GATE RED: rows 2` (exit 1), both matching what is now written.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 15:40:20 +02:00
..
__init__.py fix(assets): bound what the run pays, not what the document claims (0.10.1) 2026-09-18 13:41:18 +02:00
_pandoc.py feat(extract): resolve the vendored converter and refuse an unpinned version 2026-09-02 14:07:20 +02:00
accounting.py fix(accounting,gate): the conversion claim comes from the run's ledger 2026-09-19 19:12:20 +02:00
assets.py fix(assets): an end-of-line escape at column 0 states no skip, and neither does a delta out of its row 2026-09-19 22:47:29 +02:00
cli.py feat(mcp): serve OKF bundles over MCP in two shapes, plus the generic skill 2026-09-20 10:25:55 +02:00
connectors.py fix(errors): type the NUL-byte path failure as SourceError 2026-07-20 10:30:42 +02:00
consume.py fix(consume): every read path into a bundle is contained, not just okf_fetch 2026-09-20 15:20:38 +02:00
contract_check.py feat(consume): parent reaches the reader -- excerpt field, body link, checker rule 2026-09-11 12:36:23 +02:00
corpus.py feat(accounting): okf build accounts for every source element 2026-09-17 18:35:20 +02:00
errors.py feat(assets): every carried image is one a model can be shown 2026-09-19 08:07:04 +02:00
extract.py feat(extract): one normalisation door removes U+00AD before the guard 2026-09-19 06:10:53 +02:00
guard_adapter.py fix(gate,propose): okf build runs a real guard; a code fence declares no structure 2026-09-16 00:19:50 +02:00
importer.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
inbox.py fix(accounting,gate): the conversion claim comes from the run's ledger 2026-09-19 19:12:20 +02:00
manifest.py feat(profiles): materialize_bundle takes a keyword-only profile (req 6) 2026-07-31 15:37:17 +02:00
materialize.py docs(k3): round 24, a reason that became false is struck and the emitter stays flow 2026-09-12 16:50:26 +02:00
mcp_server.py fix(consume): every read path into a bundle is contained, not just okf_fetch 2026-09-20 15:20:38 +02:00
profiles.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
project.py fix(consume): every read path into a bundle is contained, not just okf_fetch 2026-09-20 15:20:38 +02:00
propose.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00
py.typed feat(typing): ship the PEP 561 py.typed marker 2026-07-17 03:46:51 +02:00
quality.py fix(consume): every read path into a bundle is contained, not just okf_fetch 2026-09-20 15:20:38 +02:00
render.py feat(errors): stable machine-readable codes on IngestError 2026-07-17 04:01:04 +02:00
segmentation.py feat(consume): parent reaches the reader -- excerpt field, body link, checker rule 2026-09-11 12:36:23 +02:00
skill.py docs(front-page): the gate numbers the gate actually prints, and a breaking point that was measured 2026-09-20 15:40:20 +02:00
structure.py feat(assets): a bundle carries the images its sources declare (0.10.0) 2026-09-17 10:01:31 +02:00