open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): carry every source_* key by prefix, and generate a skill per bundle

Browse source 
Two changes, one theme: what a reader needs in order to cite is a property of
the PRODUCER, so neither the excerpt nor the skill may hard-code a list of the
producers someone thought of.

The pass-through rule is now the `source_` PREFIX, not the five keys this
library writes. Measured on the N500 bundle currently on disk: 269 of 274
concepts carry `source_element_id`, a locator that repository chose under this
chain's own rule ("the key says what it indexes") and that this library never
writes. The allowlist dropped it, and an excerpt that names a document without
naming the place in it is the defect this work exists to close. A prefix and
never a substring - `resource_owner` contains the literal and is not a locator,
and promoting it would be fabricated provenance produced by a matching bug. The
known-negative is tested: `bundle_id`, `type` and `ingested_at` do not travel.
Contract 8.5 states the rule as a prefix rather than a list.

K2 control, re-measured against the frozen tool at b6a8c8b, same question and
same k: the RANKING is untouched - same 8 ids in the same order, identical
`text_sha256`, identical `withheld`, denominators 629 = 621 + 8. The FIELD moved:
payload 108 877 -> 113 143 B (+3.92 %), spent 18 606 -> 22 210 (+450.5 B per
excerpt), excerpt members 9 -> 17, 99 changed lines. Known-positive follows the
contract document's bytes again: 12 049 -> 12 563 measured, 11 719 -> 12 227
raw, delta 330 -> 336.

`tools/okf_skill.py` instantiates the template for one bundle: id, ref, concept
count, the conditional-field table with a denominator per field (the `source_*`
rows DISCOVERED from the bundle, not listed), the whole-bundle cost by the gate's
own instrument, the share one measured answer spent, the concept count at which
the withheld bookkeeping alone reaches the limit, and the index-walk-against-
directory control - run once at generation time, never on the question path.

The form was chosen on a measurement that came out against the obvious gate:
the contract checker passes the UNFILLED template against a real payload, and
passes a skill built for a different bundle against this one's. It cannot tell
the two forms apart, so conformance could not decide it. What decides it is that
5's denominators, 6.4's conditional fields and 7.6's breaking point are
per-bundle numbers - a generic skill either leaves them as holes (the template's
own definition of unfinished) or states another corpus's numbers, which is worse
than a gap. Every gate the checker lacks is therefore a test here: no placeholder
survives, the skill names its own bundle's id and ref and not another's, its
commands are absolute and point at files that exist, and it refuses a directory
with no index (exit 1, `bundle_unreadable`), an index with no `bundle_id`
(`bundle_id_missing`), an empty bundle, and an occupied target without --force.

It lives in `tools/` for the reason `okf_consume.py` and `okf_contract_check.py`
state for themselves - outside `src/`, so no consumer's install surface changes -
and because a wheel-installed `okf skill` would emit a command pointing at
`tools/okf_consume.py`, which the wheel does not contain.

Suite 1372 (1347 before), ruff clean, mypy src clean.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 15:13:22 +02:00
commit c95d18905a
9 changed files with 957 additions and 44 deletions
Download patch file Download diff file Expand all files Collapse all files

59
tools/okf_consume.py
View file

@ -330,7 +330,9 @@ def read_concept(path: Path, *, bundle_root: Path, root_bundle_id: str) -> Conce
sources=entries,
sources_present=sources_present,
locators={
key: frontmatter[key] for key in LOCATOR_KEYS if frontmatter.get(key, "").strip()
key: value
for key, value in frontmatter.items()
if key.startswith(SOURCE_KEY_PREFIX) and value.strip()
},
frontmatter=frontmatter,
body=_body(path),
@ -464,6 +466,17 @@ LOCATOR_KEYS = (
"source_offset",
)
#: What a locator key looks like to a reader that does not know the producer.
#: The pass-through rule is this PREFIX and not `LOCATOR_KEYS`, which is a list
#: of the producers someone thought of: measured 2026-09-08, 269 of 274 concepts
#: in the N500 bundle carry `source_element_id`, a locator that repository chose
#: under the same rule ("the key says what it indexes") and this library never
#: writes. An allowlist drops it, and the excerpt then names a document without
#: naming the place in it. A PREFIX, never a substring -- `resource_owner`
#: contains the literal and is not a locator, and promoting it would be
#: fabricated provenance produced by a matching bug.
SOURCE_KEY_PREFIX = "source_"
def _frontmatter_lines(path: Path) -> list[str]:
"""The raw lines between the two `---` fences, indentation intact.
@ -571,14 +584,14 @@ KNOWN_POSITIVE_CASE = "docs/consumption-contract.md, encoded as a JSON string"
#: `measure()`'s own answer for that file. Vacuous ALONE -- which is why the
#: delta below exists.
KNOWN_POSITIVE_EXPECTED = 12_049
KNOWN_POSITIVE_EXPECTED = 12_563
#: The second, independent route. `wc -c` reports 11 719 raw bytes for the same
#: The second, independent route. `wc -c` reports 12 227 raw bytes for the same
#: file; the difference is this file's JSON quoting and escaping overhead. A
#: reader can derive it without running `measure()` at all, and it moves the
#: moment `measure()` changes what it counts -- which is what stops
#: `expected == measured` from proving nothing.
KNOWN_POSITIVE_ENCODING_DELTA = 330
KNOWN_POSITIVE_ENCODING_DELTA = 336
_KNOWN_POSITIVE_PATH = Path(__file__).resolve().parents[1] / "docs" / "consumption-contract.md"
@ -1311,6 +1324,28 @@ CONTRACT_REVISION = "okf-consumption/1"
DEFAULT_K = 8
def root_bundle_id_of(bundle_root: Path, *, profile: BundleProfile = DEFAULT_PROFILE) -> str:
"""The `bundle_id` the root index declares, or a refusal naming which half
of SS 3.1's identity tuple is missing. Shared with the skill generator, so
the two agree on what makes a directory a readable bundle."""
root_index = bundle_root / profile.index.name
if not root_index.is_file():
raise ConsumeError(
f"{bundle_root} carries no {profile.index.name}, so there is no index "
"tree to walk and no way to read the bundle without enumerating a "
"directory, which SS 9.2 forbids",
code="bundle_unreadable",
)
declared = parse_frontmatter(root_index).get("bundle_id", "")
if not declared:
raise ConsumeError(
f"{root_index} declares no `bundle_id`; identity across bundles is "
"the (bundle_id, concept_id) tuple (SS 3.1) and half of it is missing",
code="bundle_id_missing",
)
return declared
def build_payload(
bundle_root: Path,
*,
@ -1340,21 +1375,7 @@ def build_payload(
"two agree",
code="instrument_unvalidated",
)
root_index = bundle_root / profile.index.name
if not root_index.is_file():
raise ConsumeError(
f"{bundle_root} carries no {profile.index.name}, so there is no index "
"tree to walk and no way to read the bundle without enumerating a "
"directory, which SS 9.2 forbids",
code="bundle_unreadable",
)
root_bundle_id = parse_frontmatter(root_index).get("bundle_id", "")
if not root_bundle_id:
raise ConsumeError(
f"{root_index} declares no `bundle_id`; identity across bundles is "
"the (bundle_id, concept_id) tuple (SS 3.1) and half of it is missing",
code="bundle_id_missing",
)
root_bundle_id = root_bundle_id_of(bundle_root, profile=profile)
concept_ids = enumerate_concepts(bundle_root, profile=profile)
concepts = [
read_concept(