open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity
4 releases 16 tags
  • v1.1.0 43499c0ca5

    v1.1.0 Stable

    ktg released this 2026-09-21 10:33:05 +00:00 | 18 commits to main since this release

    Signed by ktg
    SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q

    The reading side. Nothing here changes how a bundle is built from a folder;
    what changes is how a question finds its way into one, and how many bundles
    one question can reach.

    Changed

    • A new default ranking: BM25 over passages and titles. A concept is scored
      on its best 500-character passage and on a title/path-weighted field, and the
      two are fused by rank. A concept longer than 4 000 characters is delivered as
      its answering passage, with passage: {start, end, of} saying where in the
      concept it sits. The previous fusion stays reachable as --ranking fusion.
      A reader will see different excerpts, in a different order, for the same
      question on the same bundle.
    • BREAKING for anyone who reads the payload: the contract is
      okf-consumption/2.       withheld is no longer a list with one entry per
      concept left out; it is a mapping with total, by_rule, nearest (the
      best-ranked drops by name, title and source document, 20 by default) and
      complete. Measured on a large real bundle, the old list was about two
      thirds of the written payload and none of it counted against the budget the
      payload reported. --withheld-full brings the whole list back. The revision
      was bumped so that code indexing withheld as a list fails loudly instead of
      reading the wrong thing. --withheld-titles is retired.
    • The server and the generic skill are the standard way in, over a whole
      folder.       okf project and the front page say it in that order: register
      okf mcp --root once, on user scope, and every project can ask every bundle
      under it; the skill beside a bundle is the supplement for someone who would
      rather register nothing. Neither is made again when a bundle is rebuilt.
    • okf skill writes ONE generic skill by default, carrying a working
      method and an answer form and no bundle's numbers, so it does not go stale
      when a bundle is rebuilt. --for-bundle keeps the per-bundle form.
      okf project puts it at .claude/skills/okf-consume-any/.

    Added

    • Several sub-questions in one call. okf consume --question A --question B (and questions on the server's okf_ask) reads the bundle once, cuts
      each sub-question as if asked alone and interleaves the deliveries under the
      same k and byte limit. One question gives exactly the payload it gave
      before.
    • A map per bundle. okf card and the server's okf_describe carry
      map: one line per source document with its concept titles in document
      order, a numbered series folded into one line. The working method is to read
      the map first and ask two to four sub-questions in the map's own words.
    • A signal for weak coverage. The payload's coverage carries
      absent_terms (question words the bundle holds in no form) and weak (one
      such word, or nothing delivered). It is a reading, not a verdict.
    • A folder of bundles in one command. okf card <folder> shows every
      bundle under it, okf consume <folder> asks every bundle in one call
      (--bundle-id narrows it to one), and okf check reads that reply, holding
      each payload to all 19 rules on its own. These are the same functions the
      server runs.
    • A table fragment is read under its heading. A concept carrying the
      proposer's mechanical table title is ranked and shown under the nearest
      concept above it in its document; the excerpt keeps own_title. No bundle
      byte moves.
    • okf project --gate: the gate can be chosen when building a project,
      and the bundle's log.md names which one ran.
    • okf check gains passage_malformed and subquestions_unindexed
      (19 rules).

    Removed

    • The old test track tied to one reference corpus is retired. Its gate
      row, its adapters and the tests that read those bundles are gone; public
      tests and gates run on invented material. The retrieval gate's premises were
      re-measured for BM25.

    Known limitations added

    See Known limitations: a question worded
    differently from the bundle reaches it first through the map's words, the
    retrieval gate is red on rows 5, 7 and 8, and speed was not measured.

    Downloads