open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity
4 releases 15 tags
  • v1.1.0 43499c0ca5

    v1.1.0 Stable

    ktg released this 2026-09-21 10:33:05 +00:00 | 0 commits to main since this release

    Signed by ktg
    SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q

    The reading side. Nothing here changes how a bundle is built from a folder;
    what changes is how a question finds its way into one, and how many bundles
    one question can reach.

    Changed

    • A new default ranking: BM25 over passages and titles. A concept is scored
      on its best 500-character passage and on a title/path-weighted field, and the
      two are fused by rank. A concept longer than 4 000 characters is delivered as
      its answering passage, with passage: {start, end, of} saying where in the
      concept it sits. The previous fusion stays reachable as --ranking fusion.
      A reader will see different excerpts, in a different order, for the same
      question on the same bundle.
    • BREAKING for anyone who reads the payload: the contract is
      okf-consumption/2.       withheld is no longer a list with one entry per
      concept left out; it is a mapping with total, by_rule, nearest (the
      best-ranked drops by name, title and source document, 20 by default) and
      complete. Measured on a large real bundle, the old list was about two
      thirds of the written payload and none of it counted against the budget the
      payload reported. --withheld-full brings the whole list back. The revision
      was bumped so that code indexing withheld as a list fails loudly instead of
      reading the wrong thing. --withheld-titles is retired.
    • The server and the generic skill are the standard way in, over a whole
      folder.       okf project and the front page say it in that order: register
      okf mcp --root once, on user scope, and every project can ask every bundle
      under it; the skill beside a bundle is the supplement for someone who would
      rather register nothing. Neither is made again when a bundle is rebuilt.
    • okf skill writes ONE generic skill by default, carrying a working
      method and an answer form and no bundle's numbers, so it does not go stale
      when a bundle is rebuilt. --for-bundle keeps the per-bundle form.
      okf project puts it at .claude/skills/okf-consume-any/.

    Added

    • Several sub-questions in one call. okf consume --question A --question B (and questions on the server's okf_ask) reads the bundle once, cuts
      each sub-question as if asked alone and interleaves the deliveries under the
      same k and byte limit. One question gives exactly the payload it gave
      before.
    • A map per bundle. okf card and the server's okf_describe carry
      map: one line per source document with its concept titles in document
      order, a numbered series folded into one line. The working method is to read
      the map first and ask two to four sub-questions in the map's own words.
    • A signal for weak coverage. The payload's coverage carries
      absent_terms (question words the bundle holds in no form) and weak (one
      such word, or nothing delivered). It is a reading, not a verdict.
    • A folder of bundles in one command. okf card <folder> shows every
      bundle under it, okf consume <folder> asks every bundle in one call
      (--bundle-id narrows it to one), and okf check reads that reply, holding
      each payload to all 19 rules on its own. These are the same functions the
      server runs.
    • A table fragment is read under its heading. A concept carrying the
      proposer's mechanical table title is ranked and shown under the nearest
      concept above it in its document; the excerpt keeps own_title. No bundle
      byte moves.
    • okf project --gate: the gate can be chosen when building a project,
      and the bundle's log.md names which one ran.
    • okf check gains passage_malformed and subquestions_unindexed
      (19 rules).

    Removed

    • The old test track tied to one reference corpus is retired. Its gate
      row, its adapters and the tests that read those bundles are gone; public
      tests and gates run on invented material. The retrieval gate's premises were
      re-measured for BM25.

    Known limitations added

    See Known limitations: a question worded
    differently from the bundle reaches it first through the map's words, the
    retrieval gate is red on rows 5, 7 and 8, and speed was not measured.

    Downloads
  • v1.0.0 404bed26e2

    v1.0.0 Stable

    ktg released this 2026-09-20 19:57:02 +00:00 | 21 commits to main since this release

    Signed by ktg
    SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q

    Added

    • A document the gate refuses WHOLE is named in the run's own summary.
      Measured 2026-09-20 against a real corpus of official documentation built
      with the shipped default gate: 17 sources were refused outright, 16 of them
      among its ordinary reference pages, and the summary said only
      one fail_secure line and one quarantine_review line. The count of
      documents the gate dropped was not there (rejected (coded) sums gate
      refusals and extraction failures, which have different remedies), the names
      were not there, and neither was the way out. okf build now prints a
      Documents the gate refused WHOLE section carrying all four — the count with
      its denominator, the names (capped at ten, with the rest in the bundle's
      log.md), the codes, and --gate none for a source you vouch for yourself —
      and repeats it in one line on stderr, where a redirected stdout cannot hide
      it. log.md gains a bullet naming every refused document, uncapped. The
      exit code does not move      : the build is valid, every refusal is coded and the
      bundle is a true record of what the gate allowed; what was wrong was the
      silence. A run the gate refused nothing from is byte-identical, in the
      summary and in log.md.
    • A Known limitations section on the front page, high up and before the
      install detail: the gate's measured refusals and the way out, the absent
      ceiling on what one run pays for images, the three gates of this repository
      that are RED and what each red row means for a user, what the content
      accounting does not count, and the rough edges nothing is planned for. No new
      measurement — every number was already taken.
    • The payload says what of the question it reached — a new top-level
      coverage member carrying three lists: the terms the pre-pass read the
      question as, the terms no concept in the bundle answers, and the terms no
      delivered excerpt answers. Without it a reader holding eight excerpts cannot
      tell a bundle that ANSWERED its question from one that merely ranked
      something; the two payloads have the same shape. Documented as SS 8 point 7
      of docs/consumption-contract.md, and the generated consumption skill is
      told to read it.
      • Facts, and no verdict, which is a measurement rather than caution. Two
        readings were built and both falsified over 81 questions (16 synthetic,
        65 across three real gold sets, 2026-09-20): the share of a question's
        terms a delivered excerpt answers separates the synthetic controls at 0.33
        against 0.50 and then REVERSES on real data, where covered questions run
        down to 0.27 while one genuinely uncovered question sits at 0.71; and the
        share of a bundle tying the best lexical match is ~0.00 for every question
        in a large bundle, covered or not. Question style dominates the first and
        corpus size the second, so a pre-pass emitting a verdict would assert
        across corpora what was measured on one.
      • Contract change, and the cost to a consumer is a re-run. Every payload
        grows the member; the checker does not read it, so a third-party pre-pass
        that omits it stays conformant. The SS 7.4 known-positive moves with the
        document it is measured on (14 721 / 375 → 16 389 / 417).
    • The retrieval gate is measurable where it was assertable
      (tools/okf_retrieval_gate.py, not shipped in the wheel):
      • Row 8 prints the identity of every bundle it measured — path,
        bundle_id and content ref — beside the set's sha256. Measured the same
        day: two builds of one consumer's corpus carrying the SAME bundle_id at
        different refs score differently on the same pinned set, which is why the
        ref and not the id is what a row is attributed to.
      • REAL_SET_PINS states what each of the three real sets IS — questions,
        fasit entries, controls and sha256 — so a self-written file in the right
        shape is refused instead of reading 1 of 1 | 3 of 3 | GREEN.
      • Row 5 reads the hold-out threshold as a number in [0, 1] and RUNS the
        registered set against the registered bundle, printing
        answered of asked = share against threshold. bool(threshold) was the
        whole check, so report-only; any number is acceptable for v1 passed it.
      • Row 4's marking reads the payload's coverage: UNANSWERED_BAR = 2/3,
        swept and collapsing at both ends (at 0.50 eleven real covered questions
        are marked; at 0.70 the row falls to 5 of 6). The margin is thin — 0.6087
        against 0.6667 — and what it does not catch is published with it.

    Changed

    • The two pip install lines under "Install in detail" install
      [extract].       The first screen installs llm-ingestion-okf[extract] and
      those two omitted it, so a reader following the detailed instructions got a
      build that reports resolved converter path: unresolved (extractor_extra_missing) and reads no binary format. Two recipes, two
      different installations.
    • Version 1.0.0. The scope this tool is finished at. It adds no
      capability over v0.10.1; what it adds is that the tool says what it does
      not do. After this tag the library is touched for defects found in its own
      use, and the next round is Google OKF v0.3.
    • A withheld concept now carries the rule that actually decided it. The
      source quota filters the WHOLE ranked candidate list rather than the top
      k, so every over-quota candidate came back source_quota_exceeded —
      including the ones the RANK had already put outside k, which the quota
      only reached because it ran first. Measured on 25 real misses 2026-09-17:
      13 of them were labelled by the quota and decided by the rank.
      consume._fates_without_quota asks the same cut what would have become of
      each candidate with no quota in force, and the drop keeps THAT rule; only a
      candidate the quota-off cut would have delivered is named as the quota's.
      The budget step is lifted into consume._pack and used by both, so the
      quota-off fate is decided by the code the run itself uses. This moves the
      rule string a consumer reads for some withheld entries; no delivery, no
      rank and no excerpt byte moves, and no committed payload in this repository
      changed. The retrieval gate's row 3 goes 2 of 5 RED to 5 of 5 GREEN.
    Downloads
  • v0.10.0 bc39e8091f

    v0.10.0 Stable

    ktg released this 2026-09-17 08:01:38 +00:00 | 115 commits to main since this release

    Signed by ktg
    SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q

    Added

    • A bundle carries the images its sources declare (0.10.0). Until now no
      reader in this package fetched, named, described or copied a single image:
      <img>'s attributes were never read, a NISO-STS <graphic> was walked past,
      a PDF was opened for its text alone, the converter's markdown writer dropped
      every picture, and the only writer into a bundle took content: str. The two
      lossiness warnings said so on every run, which made the loss honest and did
      not make it smaller. Measured on R761 Prosesskoden:2025: the process text is
      carried in full while 12 Tabell N-N and 9 Figur N-N captions stand over
      nothing, so process 84's "toleranseklasse ... er gitt i tabell 84-2" points
      at empty space.
      Five readers place, one module decides. pdf (embedded image XObjects),
      docx/pptx/odt/rtf (the converter's media, through --extract-media),
      html/htm (<img src alt>, local paths and inline data URIs) and xml
      (<graphic xlink:href>, resolved against the href and then against a sibling
      graphics/). llm_ingestion_okf.assets decides what an image IS, what it is
      called and how it is pointed at, so "carried N of M" means one thing across
      all five. .xlsx is deliberately excluded: a two-line block inside its pipe
      tables would break the row locator read back out of them.
      The bytes go to assets/ at the bundle root under
      <sha256[:12]>-<the source's own base name>, and the concept carries a
      two-line pointer where the picture stood -- a markdown image, then the
      source's own file name and the size in pixels. A PDF stream that is already a
      file (DCTDecode, JPXDecode) is carried VERBATIM; raw samples are encoded
      to PNG with zlib from the stdlib, so no new dependency and no rasteriser
      version enters an asset's bytes or its content-addressed name. What this
      encoder cannot express exactly -- a stencil mask, a Decode array, CMYK,
      anything but 8-bit samples -- is refused with a code and counted, never
      approximated.
      ON by default, with --no-assets reproducing the pre-0.10.0 bytes.
      Measured over the 43-document reference corpus, two builds of one commit:
      453 -> 454 concepts, 865 -> 867 markdown files, 0 -> 2 964 assets (2 964
      carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
      3 088 s, peak RSS 6.26 -> 8.74 GB, and 422 of 865 markdown files differ. The
      one new concept has a measured cause: the pointers are body text, so a
      section holding 146 of that document's images grew from 19.0 % to 30.6 % of
      the extracted text and crossed --outline-gate's 0.20 share clause.
      log.md states it either way -- "N carried of M found", or NOT CARRIED
      under --no-assets, so a bundle nobody looked for figures in cannot be
      mistaken for a bundle of documents that had none. A concept on this
      repository's own profiles also carries images: N, conditional, counted out
      of the concept's own text.
      The image bytes are NOT screened, and the log says so: the guard is
      text-only, the pointer block passes the gate as body text, and the picture
      beside it passes nothing.
      Door C carries them too. Measured before the repair: importing a bundle
      built with --assets merged 6 of 6 concepts and wrote no assets/ at all,
      so every pointer in the imported bundle named a missing file. Only the assets
      a MERGED concept points at are carried -- an asset belonging to a refused
      concept must not ride in on the back of a cleared one.
      A proposed SPEC section 6.4 for the layout is in
      docs/plan/okf-assets-section-6-4.md; _okf-canonical is not edited from
      here.

    Fixed

    • A markdown image is no longer read as a cross-reference.
      structure._LINK reads [...](target) and never looked at the character in
      front of the bracket, so an asset pointer would have arrived in the index as
      a references edge to a concept that cannot exist -- and the digits in an
      asset's file name would have been read as a document number. The link's span
      is still masked, so the number scan cannot see it either.

    • okf build now runs a real guard, and the bundle says which one (F1).
      From the day the command was packaged until 2026-09-15, corpus.measure
      wired an unconditional approve-everything stub into process_inbox and no
      add_argument call anywhere in the package named a gate -- so the only path
      most people use screened nothing, while pyproject.toml made the guard a
      MANDATORY runtime dependency and the README recommended a composition the
      command line could not reach. Reported from outside by claude-code-llm-wiki
      and reproduced here before anything moved.
      --gate takes guard-trusted-source (the new default),
      guard-user-upload or none      , and the name is written into the bundle's
      section 9 log.md either way, so a consumer holding a bundle can tell a
      screened one from an unscreened one. An unknown name is refused rather than
      resolved to the stub: falling back would reproduce the defect with an extra
      step. okf project owns no flag that moves a bundle's bytes and takes the
      default; the corpus harness carries the same flag and the same default,
      because a test holds the two paths byte-equal.
      The default was chosen on a measurement, not on caution. Over the 453
      concept bodies of the pinned reference bundle, PRESET_TRUSTED_SOURCE
      returns the persist disposition on 453 of 453 while PRESET_USER_UPLOAD
      holds 1, taking one of the 39 source documents out. Neither tier waves
      anything through: an invisible carrier and a CRITICAL finding fail secure at
      both, measured against guard 1.4.0. Door B's own library default is
      unchanged at PRESET_USER_UPLOAD -- an inbox drop is an untrusted upload,
      an operator pointing okf build at their own folder is not. The second tier
      ships as guard_adapter.inbox_gate_trusted_source, the three-line adapter
      that module's own docstring describes, rather than as a preset parameter.
      The composition the README recommends is now tested. Before this change
      grep -rl inbox_gate tests/ gave ONE file with 0 occurrences of segment,
      while the nine files passing segmentation= all injected a local warn-stub:
      no test ran a real guard verdict and a segmentation plan in the same call.

    • A fenced code block no longer declares structure (F2). The proposer read
      every line of the extracted text with the same grammars, so # Use the opus[1m] alias inside a ```bash fence became a level-1 ATX heading. Two
      effects, and the smaller one was the visible one: the document was REFUSED
      entirely when the line carried [ or ] (Door B validates a title fail-fast
      and never repairs one) -- 5 of 191 pages of the reporter's corpus -- and the
      concept TITLE was silently taken from somebody's shell session everywhere
      else, on 62 of 191 pages (32.5 %).
      No rule in find_candidates reads a fenced line now: not ATX, not the
      numbered grammar, not a table row, not a bold title, and not Arm D's outline
      run, which selects from the whole line list and would otherwise let a fenced
      install listing decide which run wins. Backtick and tilde fences, up to three
      leading spaces, a closing fence at least as long as its opener, and
      CommonMark's rule that a backtick fence's info string may not contain a
      backtick -- that last one is what keeps a line holding only `okf build`
      from silencing the rest of a document.
      It lands unconditionally rather than behind a flag, and the exposure is
      measured on the bytes      : 0 of 865 concept files in the pinned default bundle
      and 0 of the shipped fixtures and goldens that reach the proposer carry a
      fence of either kind, so a rule that can only fire INSIDE a fence cannot have
      moved anything this repository has measured. It is a defect, not a default
      move.

    Added

    • okf quality <bundle> --fasit <json> -- boundary recall against the
      structure the source itself declares (G37b).       The bundle-only gate returned
      UNMEASURED and exit 3 on the very arm it was built for, because no
      bundle-only metric reaches it; boundary_share -- declared boundaries that
      became a concept, over declared boundaries -- is the one metric measured that
      orders the arms correctly, and it needs the publisher's own structure, so it
      arrives as an input rather than as a constant. The fasit is a JSON list whose
      rows carry title and norm, validated at the door: a file that is not a
      list, a row missing either key, or anything that is not JSON exits 2 with
      the reason, never a quiet UNMEASURED.
      The normalisation was measured before the metric was built: stripping all
      whitespace and lowercasing reproduces the fasit's own norm from its own
      title on 2 761 of 2 761 rows (keeping only alphanumerics scores 58).
      A boundary is recovered in either of two forms, and both are load-bearing:
      a concept whose normalised title equals norm, or the pair of the concept's
      own directory and its residual title -- because the numbering token a
      publisher glues into a heading lands in the concept id on one route and in the
      title on another. Measured on the known-good arm, the literal form alone
      reaches 22 of 2 761 where the two together reach 2 759; on another
      build of the same product the split is the exact opposite (2 727 literal, 0
      paired). One bar, at the value measured on the pinned artifact: 2 759/2 761,
      corpora = 1. It separates the known-bad arm at 1 148 of 2 761 (41.6 %),
      which is now a FAIL and exit 1 instead of exit 3. --fasit is an
      assertion      , like okf consume --ref: a bundle of another product scores 0 of
      2 761 (measured on two of them) and reads FAIL -- the assertion being wrong,
      not the bundle. The bar rests on one product, and the run says so on every
      boundary row. docs/2026-09-12-g37-terskler.md SS 7 carries the premises
      re-measured, the seven bundles, the interval any bar could sit in, and the two
      R761 builds this one fails.

    Unchanged

    • Without --fasit the command is exactly what it was, held by a test: no
      boundary row, and 860019-mdb-100 still exits 3. No version bump, no tag,
      okf check untouched.
    Downloads
  • v0.5.0a2 805a2d50d9

    v0.5.0a2 Pre-release

    ktg released this 2026-07-31 18:50:39 +00:00 | 382 commits to main since this release

    This is the pre-release the pilots pin. v0.5.0a1 was tagged and abandoned
    unused — do not pin it.     It carried a generated.by actor that the spec owner
    had already excluded, and it was caught before any pilot was notified.

    Fixed

    • OKF_V0_2's generated.by actor is process:okf-ingest, not
      process:llm-ingestion-okf. Commons decided the fixed id on this repo's own
      proposal 2026-07-31, superseding option (d) chosen here 2026-07-27. The
      exclusion is ingest-spec.md:7-8, frozen on the spec being framework-neutral:
      normalising our repo name into the id would force every other conformant
      implementation to write it into its own output.

      Nothing in the wild carried the excluded value — OKF_V0_2 did not exist at
      v0.4.0, and no pilot had been notified — so this costs a tag rather than a
      migration. It is recorded rather than quietly folded in because the failure it
      avoids is specific: actor is both the stamp written and the value owned back
      (OwnershipPolicy), and recognition is one-way, so a pilot holding bundles
      stamped with the excluded id would have hit collision_unstamped on its own
      files the moment the id was corrected.

    Downloads