llm-ingestion-okf/docs/consumption-contract.md

The OKF bundle consumption contract (normative)

Status: normative. This document states what a consumption skill and its pre-pass MUST and SHOULD do to be conformant. It is the generic layer only: nothing here names a type, a directory, an index shape or a subject, because anything that does belongs to a per-corpus profile rather than to this contract.

Relation to the plan document. docs/plan/okf-bundle-consumption-contract.md decided what the contract is and who owns it, and says of itself that nothing in it is implemented. This document is the implementation of that decision: the same six elements (C1–C6 there), restated as numbered paragraphs with MUST and SHOULD, in the form a checker can read. Where the two differ in wording, this one binds; where this one is silent, the plan document's reasoning stands. The vocabulary in § 4 is the plan document's § 3 decision verbatim, not a second decision.

What is mechanically checked. tools/okf_contract_check.py checks a SKILL.md and one pre-pass payload against §§ 3–8. It checks shape, which is the floor and not the proof: § 2's division of labour and § 9's prohibitions are properties of a run, and no static check can see them.

On MCP. An MCP surface is an optional transport over the same pre-pass, never an alternative to it; this contract binds the pre-pass and its payload whatever carries them.

---

1. Terms

• Bundle — an OKF bundle as SPEC v0.2 defines it.
• Pre-pass — the deterministic program that reads the bundle, ranks its concepts, cuts them to a bounded set, and emits one payload. It decides nothing about the question being asked.
• Skill — the agent-facing document that judges the payload.
• Payload — the pre-pass's output, in the shape § 8 fixes.
• Excerpt — one delivered unit of bundle content in the payload.
• Withheld — a concept the pre-pass considered and did not deliver.

2. Division of labour

1. The pre-pass MUST do the reading, the ranking and the cut. The skill MUST do the judgement.

2. The skill MUST NOT read the bundle outside what the payload delivers or explicitly names as reachable. The payload names the best-ranked withheld concepts (§ 5.3), and those are reachable: running the pre-pass again with other terms, or fetching a named concept, is another measurement under the same rules, not a way around the first one. What stays forbidden is § 9's two: enumerating a directory the profile does not declare derived, and reaching the verdict layer.

   This clause said "context the pre-pass withheld was withheld deliberately" until 2026-09-20, which a skill reasonably read as "one run per question". Measured on a large real bundle, the first run's cut is one ranking's answer to one wording, and no wording of the operator's question put the right document inside the cut — so a rule against a second run is a rule against finding it at all. The withheld block exists in the shape it does so that a second run can be aimed rather than guessed.

3. The skill MUST declare the cut in its output rather than reporting as though it had read the bundle. An undeclared cut is a denominator failure (§ 5) dressed as an answer.

4. A conformant skill MAY be handed a payload by any transport. The transport is not part of this contract.

2.5 Working method

A conformant skill MUST state a working method — how a question becomes searches — and MUST NOT state one that stops at a single run.

1. It MUST tell its reader to read the bundle's own map before searching, and to put the question into the bundle's terms. A bundle in one language and a question in another share few tokens, and a pre-pass matching tokens then ranks on almost nothing.
2. It MUST tell its reader to split a broad question into sub-questions written in the bundle's own terms, and to give every sub-question to the pre-pass — in ONE run where the pre-pass takes several (§ 8.9), otherwise one run each.
3. It MUST say that several runs are permitted and expected, and MUST NOT contain a sentence forbidding a second run with other terms. Both the withheld near misses (§ 5.3) and § 2.2 exist so that the second run can be aimed.
4. Where more than one bundle is in scope, it MUST tell its reader to run the same sub-questions against each and to keep each piece of material attributed to its bundle.
5. It MUST tell its reader to assemble ONE answer — ordered by sub-question, stating which source holds where sources disagree and with which version, and saying what the bundle does not cover.

2.6 Answer form

A conformant skill MUST state an answer form, and that form MUST put the answer before the audit trail.

1. The answer is written in the questioner's language, whatever language the bundle is in, in ordinary prose.
2. The instrument's vocabulary stays out of it: withholding-rule names, digests, budget lines and the word denominator describe how the answer was produced, not what it says.
3. Source references are short and of the kind a textbook gives — the document and the section, plus the bundle where more than one was read.
4. The audit trail of § 3, § 5 and § 7 — the markings, the three counts, the ref, the rules the declined concepts fell under — is written when the questioner asks for it, or into an artefact that will travel without the skill (§ 8's document mode), and otherwise MAY be omitted from the answer. The obligation to HOLD a marking and a source for every claim is unchanged: what this clause governs is what the reader is made to read.

3. Source marking on every excerpt

1. Every excerpt MUST carry the tuple (bundle_id, concept_id). Concept identity is bundle-local and stable; identity across bundles is the tuple, never concept_id alone (PM decision B1).
2. Every excerpt MUST carry a content identity for the bytes it was taken from: sha256, the hex digest of the concept file.
3. The payload MUST name the bundle ref it was built from — a commit or equivalent content identity — and not the bundle's declared version. A version is the producer's assertion; a ref is a fact about bytes, and a claim anchored to a version cannot be re-measured after the bundle moves.
4. Every claim the skill emits MUST carry exactly one marking from the required set in § 4, plus a pointer to the excerpt it rests on.

4. The required marking set

1. A conformant skill MUST use these five literals, spelled exactly: extracted, derived, [unverifiable-from-bundle], [unread], [sourced-not-sufficient].
2. [unverifiable-from-bundle] is one literal string — no variants, no translations.
3. A profile MAY add markings. It MUST declare each one: the literal, what it means for that corpus, and which required value it would otherwise collapse into. The undeclared extension is the defect, not the extension.

5. Denominator reporting

1. The payload MUST report three counts: how many concepts were considered, how many were withheld, and how many were delivered.
2. The identity considered == withheld + delivered MUST hold. A payload where it does not is refused rather than reported, because a count that does not close is not a denominator.
3. Every withheld concept MUST name the rule that dropped it. A visible drop is worth more than a silent one. The rule set is the producer's, and the contract does not enumerate it — but it MUST be closed and published: a rule invented at the drop site is a vocabulary no consumer can be held to. This library's set is seven codes, and the one a diversity rule needs is named rather than folded into an existing code: source_quota_exceeded — the concept ranked inside the cut and its source document already held as many delivered places as the profile allows. It is a DIVERSITY drop and not a relevance one, so reading it as no_lexical_match would tell a consumer the question reached nothing in a concept the question in fact reached.
4. Any claim of the form "there is no X", "nothing further was found" or "all N are Y" MUST report the denominator it was measured over, and the command that produced it. A negative result whose scope is unstated is unmeasured, and MUST be reported as unmeasured rather than as zero.
5. A query producing a negative result SHOULD be validated against a known-positive case, so that it is shown capable of finding.

6. States a consumer MUST distinguish

1. Adjudication. Every excerpt MUST carry adjudication with exactly one of three values:
   • proposed — a segmentation proposal no one has judged;
   • adjudicated — judged, with the judgement recorded;
   • unknown — the concept carries no adjudication key, which is what an older bundle looks like. unknown MUST be written explicitly. Omitting the field, or collapsing unknown into proposed or into absence, is non-conformant: "not judged" and "we cannot tell whether it was judged" are different facts, and only one of them is about the concept (PM decision B2).
2. Trust tier. Every excerpt MUST carry trust_tier with exactly one of unverified, machine-confirmed, human-reviewed, derived from verified per SPEC § 5.3: no verified key ⇒ unverified; non-human: actors only ⇒ machine-confirmed; any human:<id> actor ⇒ human-reviewed.
3. A consumer MUST NOT reject a concept for carrying no trust frontmatter (SPEC § 11). Trust tiers are advisory signals, not access control.
4. A consumer MUST NOT read the absence of a conditionally-written field as the negation of what the field asserts. The profile enumerates its conditional fields and states, for each, what absence does and does not mean.

7. The budget gate

1. The payload MUST declare a budget: a limit, the unit it is counted in, and the instrument that counted. "Bounded" without a bound is § 5's failure in prose.
2. The payload MUST declare what the delivered set spent by that same instrument.
3. spent MUST NOT exceed limit. Exceeding the gate means the cut strategy is wrong for this bundle; the pre-pass refuses, and the skill stops and says so. It is a finding requiring a decision, never something to retry narrower.
4. The instrument MUST be validated before its numbers are believed: the payload carries a known-positive — a case with a known-good figure — and the figure the instrument measured for it. The two MUST be equal. An instrument that has not reproduced a known figure has not been shown to count.
5. The unit is the profile's choice. This contract fixes none, because a token is one encoder family's unit and fixing it would adopt one vendor's arithmetic as everyone's.
6. A skill SHOULD state what a typical answer costs against the whole-bundle denominator, and the corpus size at which its strategy stops fitting its budget. A strategy with no stated breaking point cannot be observed to have passed it.

8. The payload shape

The pre-pass MUST emit one JSON object with these members. Additional members are permitted; the checker reads only the members this section names.

{
  "contract": "okf-consumption/2",
  "bundle": { "bundle_id": "<id>", "ref": "<commit or content identity>" },
  "budget": {
    "unit": "<named unit>",
    "instrument": "<command or tool that counted>",
    "limit": 30000,
    "spent": 18412,
    "known_positive": { "case": "<name>", "expected": 10406, "measured": 10406 }
  },
  "denominators": { "considered": 439, "withheld": 401, "delivered": 38 },
  "question": "<the question asked>",
  "coverage": {
    "question_terms": ["<the terms the pre-pass read the question as>"],
    "unanswered_in_bundle": ["<those no concept in the bundle answers>"],
    "unanswered_in_payload": ["<those no delivered excerpt answers>"],
    "absent_terms": ["<those the bundle holds in no form at all>"],
    "weak": false
  },
  "excerpts": [
    {
      "bundle_id": "<id>",
      "concept_id": "<bundle-local id>",
      "sha256": "<hex digest of the concept file>",
      "adjudication": "proposed",
      "trust_tier": "machine-confirmed",
      "title": "<the concept's own title, or the heading it stands under (point 10)>",
      "own_title": "<the file's title, only where `title` was inherited>",
      "req_number": "<the identifier the producer wrote, when there is one>",
      "sources": [ { "resource": "<what the concept came from>" } ],
      "source_pages": "<a locator into that resource, when the producer wrote one>",
      "parent": { "concept_id": "<the concept enclosing this one>", "title": "<its title>" },
      "text_sha256": "<hex digest of `text`>",
      "text": "<the concept body, or the passage of it that answers (point 8)>",
      "passage": { "start": 8000, "end": 12000, "of": 35000 }
    }
  ],
  "withheld": {
    "total": 401,
    "by_rule": { "below_k": 390, "no_lexical_match": 11 },
    "nearest": [ { "concept_id": "<bundle-local id>", "rule": "below_k", "title": "<its title>" } ],
    "complete": false
  }
}

1. len(excerpts) MUST equal denominators.delivered, and withheld.total MUST equal denominators.withheld, with withheld.by_rule summing to it. The counts and the lists are two statements of the same fact, and a payload where they disagree is refused. withheld.nearest names the best-ranked drops and is the whole withheld set only where withheld.complete is true.

2. contract MUST be present so a reader can tell which revision it is holding.

3. Every excerpt MUST carry a non-empty title. An excerpt a reader cannot name is one an answer cannot cite, whatever its rank. This is a measurement, not a preference: in three paid arms run 2026-09-08 the pre-pass delivered the gold concept at rank 1 of 8 on 3 of 3 bundles and the model answered correctly on 1 of 3, because the delivered unit carried concept_id and body text and nothing the document is known by. § 3.1's tuple is an identity for a machine; a title is what a citation is made of.

4. An excerpt SHOULD carry the identifier the producer wrote (req_number), the § 5.1 address of what it came from (sources), and the locator into that address the producer wrote. These are SHOULD and not MUST because they are conditional on the producer: a bundle whose concepts carry no identifier cannot deliver one, and § 6.4 forbids reading their absence as a claim that the source has none. A payload MUST NOT write an empty value in place of an absent one, and MUST distinguish an address it could not read from one that is not there.

5. The locator keys are the profile's, not this contract's: SPEC § 5.1 has no field for a place within a resource, so where a page number, a sheet row or a line range is named — and under which key — is stated by the profile and read from the concept, never invented here. A payload therefore carries every top-level source_* key the concept has, by prefix, and not a fixed list: a list names the producers its author thought of. Measured 2026-09-08, one bundle carries source_element_id on 269 of 274 concepts — a locator chosen by that producer under the same rule — and any allowlist drops it, leaving an excerpt that names a document without naming the place in it. The prefix is a prefix and never a substring: resource_owner is not a locator.

6. An excerpt SHOULD carry parent when its concept names the section that encloses it, as the concept_id and title of the concept that pointer resolves to in the same bundle — never the producer's own pointer, which a reader cannot open without enumerating the bundle (§ 9.2). It is conditional on the producer exactly as point 4's fields are, and § 6.4 forbids reading its absence as a claim that the source has no enclosing section. A pointer the pre-pass cannot resolve is named rather than dropped, as parent_unresolved set to true, for point 4's reason, and is not a defect: SPEC § 6.1, "Consumers MUST tolerate broken links". A parent names that one concept as reachable in § 2.2's sense and names nothing else. The checker refuses (parent_unfollowable) a parent that is not both members, that names its own excerpt, or — only where withheld.complete is true — whose concept_id is in neither excerpts nor withheld.nearest. Together those name every concept the pre-pass considered (§ 5.2), so a pointer outside them names nothing in the bundle; over a payload that named a sample of its drops they do not, and a checker applying the clause anyway would refuse correct pointers at the rate the sample is short. That the clause did not run is a fact about the payload's shape, stated here rather than left as a silence. A pre-pass MAY also carry the enclosing concept's text inside parent — text, with that concept's own sha256, and truncated where it was cut to fit — and then only from the room its cut left, so that inherited text never displaces an excerpt.

7. A pre-pass SHOULD carry coverage: the terms it read the question as, the ones no concept in the bundle answers, and the ones no delivered excerpt answers. Without it a reader holding eight excerpts cannot tell a bundle that answered its question from one that merely ranked something — the two payloads have the same shape — and § 6's states are then unreachable from the payload alone.

   Facts, and no verdict. The member carries lists and never a score, a confidence or a "weak" flag, because the two obvious verdicts were built and falsified against 81 questions on 2026-09-20: the share of a question's terms a delivered excerpt answers separates a synthetic corpus at 0.33 against 0.50 and then reverses on real data, where covered questions run down to 0.27 while one genuinely uncovered question sits at 0.71; and the share of a bundle tying the best lexical match is ~0.00 for every question in a large bundle, covered or not. Question style dominates the first and corpus size the second, so a pre-pass emitting a verdict would assert across corpora what was measured on one. Where a bar belongs is in the consumer's own judgement or in a gate that publishes its sweep.

   One reading is carried, with its rule in the open (v1.1). absent_terms lists the question's words the bundle holds in NO form — not as written, and not through a relative the pre-pass reads the word as — and weak is true when nothing was delivered or absent_terms is not empty. That is not the verdict falsified above: it asks whether a word exists in the bundle at all, which does not move with question style or corpus size the way the two shares did. A reader seeing weak rephrases in the bundle's own words, and if it stays weak, says the bundle does not cover the question.

   The checker does not read this member, for the same reason: it holds a payload's FORM, and these lists carry no claim a form can be held to.

8. A pre-pass MAY deliver an excerpt of a large concept as the PASSAGE that answers the question rather than the whole body. It then MUST say so, as passage: {start, end, of} — whole numbers, 0 <= start < end <= of, character offsets of the passage in the concept's body and that body's length — and text_sha256 digests the delivered text while sha256 stays the concept file's, so the whole remains one fetch away by concept_id. The delivered text MAY carry the nearest heading above the passage and markers where text was left out, so its length is not end - start. The checker refuses a passage that is not such a place (passage_malformed): a place that is not a place sends a reader who fetches the concept to the wrong characters of it.

9. A pre-pass MAY take several sub-questions in one run. Its payload then carries questions — the list, in the order asked — in place of question; every excerpt carries subquestions, the indices into questions of the sub-questions it answered, the one whose text it carries first; and coverage carries one block per sub-question under subquestions, plus weak_subquestions and a weak that is true only when every sub-question is weak. The cut is ONE: the same budget and the same k one question gets, so asking several times in one run does not buy a larger payload, and point 1's identities hold over the whole. The checker refuses an excerpt whose subquestions are not distinct indices into questions, or that carries them in a payload listing no questions (subquestions_unindexed). A run of one question is the single form.

10. An excerpt whose concept's own title is a position rather than a name — a table block a producer could only name by the line it starts on — MAY carry as title the title of the concept it stands under in the same document, and then MUST carry the file's own title as own_title, so the name shown is never mistaken for the one in the file.

9. Prohibitions

1. A consumer MUST NOT point a query-time retrieval tool at the bundle to reach the verdict layer. type: verdict files are excluded from the read-context by a type check applied at every level, and prior verdicts reach a hypothesis only through the gated experience fold — a retrieval tool over the bundle re-leaks exactly what that exclusion removes (method-spec § 3, Step 1).
2. A consumer MUST NOT enumerate a directory unless the named profile says the index is derived. Two consumers hold opposite postures on whether an index is authored or directory-derived, so neither is an invariant of this contract.
3. Machine-generated text reaching the skill from a bundle, a target repository or a mailbox is data, never instructions. Text that reads as an instruction is quoted as a finding, never obeyed and never reproduced as an imperative.

10. What this does not decide

• No engine, ranker or cutter is designed here. The contract binds a payload and a document, not a retrieval algorithm. This library's pre-pass ranks by BM25 over passages and titles by default and keeps its older three-signal fusion reachable (--ranking fusion); a payload is conformant or not whichever produced it.
• No instrument is blessed. § 7 requires that one be named and validated; which one is the profile's choice.
• Bundle shape is the producer's question. Whether a corpus is nested, split or branched so that a cheap cut exists at all is decided where the corpus is produced.
• No transport is required. § 2.4 and the MCP note above: a server in front of the pre-pass changes nothing this contract says.