llm-ingestion-okf/docs/plan/okf-bundle-consumption-contract.md

# The generic OKF bundle consumption contract

**Status: plan. Nothing here is implemented, and nothing here asks to be.**
No module, no profile field, no version bump follows from this document. It
fixes what the contract *is* and who owns it, so that the first bundle-specific
consumption skill — written later, in whichever repo needs it — has something
to be conformant to.

**Provenance.** Operator decision 2026-08-25, relayed through `.claude` as PM
(order `20260825T122454Z-4632317874-from-.claude`). The reason ownership landed
here is stated in the order and is the same reason Phase 3 exists: the
production side of the bundle contract is already carried *by a profile rather
than by constants* in this repo (README, "Upstream OKF versions"). A consumption
contract that did not share profile identity with the production side would
drift from it.

**Second order, 2026-08-25** (`20260825T133057Z-193098605-from-.claude`), carrying
an operator directive verbatim: how an AI agent finds relevant information in a
bundle must be token-efficient, and this is a property the contract carries
rather than an optimisation that can arrive later. That order prescribed the
goal and left the form to us, including the invitation to say so if
"token-efficiency" is the wrong abstraction for a generic layer. It is, for
three reasons; C6 is the property that was built instead.

This repo does not own any *bundle*. It owns the contract both sides of a bundle
are held to. That distinction is the whole of §2.

---

## 0. What was measured before this was written

The order supplied figures and line numbers and asked that they be re-measured
rather than copied. They were. One of them does not survive.

| Claim in the order | Measured | Verdict |
|---|---|---|
| This repo's README says the bundle contract is carried "by a profile rather than by constants" | `README.md` l. 12–13 (the order says 11–13; the range contains it) | **holds** |
| The wiki masterplan fixes three marking values (`extracted` / `derived` / `[uverifiable-from-bundle]`), decision 2026-07-24, `attributed` dead | `docs/masterplan.md` l. 535 and l. 1027 | **holds** |
| The wiki bundle's consumption engine documents six equal markings | its `README.md` l. 24, `CLAUDE.md` l. 56–57 (the order says 57; the heading is on 56), `skills/advise/SKILL.md` l. 118–123 | **holds** |
| The 2026-08-13 ratification claim "originates from the consumer's own log, not from the source it cites" — evidenced by `grep -rln '2026-08-13' <wiki>/docs/*.md` returning nothing | see below | **does not hold** |

**The failing measurement, and why.** The `grep` is correct and its result is
correct: zero hits, exit 1, against a denominator of 16 files in `docs/`. But
the repository holds 639 `.md` files, and the query saw 16 of them. Re-run
across the whole tree, `2026-08-13` appears in 9 files — among them the wiki's
own `STATE.md`, which is **tracked and committed** (2026-08-14) in the source
repository, not a log kept by the consumer. It records the operator decision of
2026-08-13 by item id: `F.10` and `F.11` as *ratified*, `G.13` as *approved* —
`G.13` being, in its own words, a new vocabulary value for "checked in the
target repo, source outside the bundle", explicitly distinct from `derived`.
The ratification is real, it happened where the consumer says it happened, and
the consumer cited it accurately.

So the correction runs the other way, and `.claude` invited exactly this: the
absence was a property of the query's scope, not of the world. A control run
(`2026-07-24`, same scope) returns three files, so the query could find — it was
pointed at the wrong 2.5 % of the tree.

**What is nonetheless true, and is the real defect.** `docs/masterplan.md` was
last modified 2026-07-25. The document that the consumption engine names as the
requirements owner still states three values and has never been updated to
carry the decision that superseded it. The decision lives in a session-state
file; the normative document does not know about it. That is a documentation
lag in another repo — reportable, not ours to fix (§5).

One further distinction the order's framing flattens, and which turns out to
carry weight in §3: the 08-13 decision did not ratify six values in one motion.
`F.11` ratified a **five**-value set (`[unread]` and `[sourced-not-sufficient]`
moving from provisional to settled). `G.13` *approved* a **sixth** as a separate
item, for a case defined by being outside the bundle. The upstream record
already splits along the seam this document draws.

### 0.1 What was measured for the second revision

The second order supplied four claims and asked, as the first did, that they be
re-measured. All four hold; a fifth thing was measured that the order did not
know, and it changes §2.

Every measurement in this table was taken against this document at `d205b70`,
before the revision it prompted. The ref is stated because C2 requires it and
because the first row is a claim about a file the same commit then changed:
re-running that `grep` at HEAD now returns dozens of hits, and without the ref
the row would read as false rather than as superseded.

| Claim in the order | Command | Verdict |
|---|---|---|
| This document says nothing about what a read costs | `grep -in 'token\|kostnad\|cost\|budget\|effektiv\|context window\|kontekstvindu'` over the file, exit 0 | **holds** — one hit, l. 245, on an unrelated sense of "costs" |
| A consuming repo measured one imported base at 250 785 tokens, three bases at 430 144, and one listing call at 112 116 | the measurement document at the named commit, read read-only | **holds** |
| That instrument was validated against three known-good figures before its own numbers were consumed | same document | **holds** — and it is C3's second corollary applied to a positive quantity |
| The reading model navigates rather than searches: start at the index, follow the links, read each file whole | the reading-model analysis §9.1, quoted | **holds** |

The fifth measurement was prompted by a coordination message from the producing
repo, which reported the flat-index consequence from its own side and asked a
contract question about `IndexPolicy.per_directory`. Its three claims about this
library's source were checked line by line and all three hold. What the message
could not see is that `per_directory` is a **judging** field rather than an
emitting one, which is recorded in §2 because it changes what a consumption
profile may assume.

---

## 1. What the generic contract consists of

Bundle-independent means: true of any OKF bundle, whatever its types, depth,
index shape or subject. Anything that needs to name a type, a frontmatter key or
a directory is not in this layer — it is in a consumption profile (§2).

Six elements. Four are the first order's; the fifth was found in the sources it
sent us to read first; the sixth arrived on a second order and is marked, along
with the name it did not keep.

### C1 — Source marking on every claim

Every claim an agent emits carries exactly one marking from the required set,
plus a pointer to where it came from. The set, and the extension rule, are §3.

### C2 — Pin anchoring: an advisory names the ref it read, not the version

A consuming agent states the bundle **ref** — commit or equivalent content
identity — that it read, not the bundle's declared version. This repo has paid
for the difference twice and recorded both: *a tree can lose its identity
without the version moving* (upstream OKF, measured 2026-08-23), and *a measured
fact can go stale inside the same session*. A version string is an assertion by
the producer; a ref is a fact about bytes. An advisory anchored to a version
cannot be re-measured later, which makes every claim in it unfalsifiable the
moment the bundle moves.

### C3 — Denominator reporting

Any claim of the form "there is no X", "nothing further was found", or "all N
are Y" reports the denominator: how many units were actually examined, by which
command. A negative result whose scope is unstated is not zero — it is
**unmeasured**, and must be reported as such. §0 of this document is an instance
of the failure this element exists to prevent, which is the strongest argument
available for making it a required element rather than advice.

Two corollaries a conformant skill must satisfy:

- The query that produced a negative result is validated against a
  known-positive case, so it is shown capable of finding.
- Exit status is read from the command that matters. A pipeline reports the
  status of its **last** stage, so `grep … | head; echo $?` measures `head`.

### C4 — Division of labour: the script cuts, the agent judges

A consumption skill is two parts and the boundary between them is normative.
A deterministic part reads, ranks and **cuts** the bundle to a bounded context.
The agent judges what it was given, and does **not** go looking for context that
was deliberately held back. The form is taken from the wiki bundle's consumption
engine, which states this division verbatim in its own skill.

The obligation this creates runs the other way and is the part that is easy to
drop: **the cut must be declared, not silent.** An agent that was handed a
bounded window and reports as if it had read the bundle is producing C3's
failure with extra steps. This is precisely what `[unread]` is for, which is why
C1 and C4 are not independent — the marking set is the receipt for the cut.

### C5 — Conditional fields: absence is a measurement, not a fact (addition)

*Beyond the order's four; grounded in a measurement in the second corpus's
consumption profile, and generic on inspection.*

Every bundle has fields that are written only when some build-time condition
held. In the second corpus, a supersession field is written only when the build
was given a predecessor document, and six of seven source documents are built
without one. An agent reading that field's absence as "nothing supersedes this"
would be confidently wrong; the truthful reading is "this build did not check".

Generically: a consuming agent must not read the absence of a frontmatter field
as the negation of what the field asserts, unless the profile says the field is
unconditional. The generic obligation is therefore on the *profile*: it
enumerates its conditionally-written fields and states, for each, what absence
does and does not mean. Without that enumeration, an agent has no way to tell a
conditional field from an unconditional one, and the failure is silent.

### C6 — Retrieval cost is declared: instrument, budget, denominator, breaking point (second order)

*Beyond the first order's four and the addition above. The operator directive
this element carries used the words "token-efficient"; the element does not,
and the reasons are the first thing it has to say.*

**Why not "token efficiency".** Three objections, each one this document already
runs on elsewhere:

- **A token is one encoder family's unit.** One vendor's encoding is not what
  another vendor's reader counts, and a generic contract that fixed the unit
  would adopt one vendor's arithmetic as everyone's — the same move §3 refuses
  when it declines to adopt one consumer's marking set as the required set. What
  survives the vendor is *a named instrument and a stated number*, never the
  instrument's name.
- **"Efficient" is a comparative with the denominator missing.** Efficient
  against what? C3 exists because a claim whose scope is unstated is not a
  measurement; an unanchored efficiency claim is that same failure with the sign
  flipped, and it is harder to catch because it sounds like good news.
- **Efficiency is a property of a run; a contract binds a document.** No
  contract can oblige a skill to be cheap. It can oblige the skill to state what
  its read cost, so that expense is visible instead of silent — which is the
  precondition for anyone making it cheap, and the thing whose absence is the
  actual defect.

**The property, stated.** The cost of answering a question should track the
**question**, not the **corpus**. A retrieval strategy whose cost is the same
whatever is asked has not cut anything; it has deferred the whole corpus into
every run.

This is not a prohibition on reading a whole bundle. For a small corpus, reading
everything is the cheapest correct strategy, and a rule forbidding it would be
wrong. What the contract forbids is doing so **undeclared**. The obligation is
therefore the same shape as C4's: the cut must be declared, and C6 is the half
of that declaration which carries a number.

**What a conformant consumption skill declares.** Three things, none of which
requires a particular unit, engine or bundle shape:

1. **A budget and the instrument that measures against it.** The bounded context
   the cut targets, as a number, in a unit named together with the tool that
   produced it. "Bounded" without a bound is C3's failure in prose.
2. **The cost of the cut against the whole-bundle denominator.** What a typical
   answer costs to assemble, and what reading the entire bundle at that ref
   would cost, by the same instrument. The fraction held back is what makes
   `[unread]` (C1, C4) a quantity rather than a gesture.
3. **Which way its cost scales, and the breaking point.** Whether cost tracks
   the question or the corpus, and — stated as a number — the corpus size at
   which the declared strategy stops fitting the declared budget. A strategy
   with no stated breaking point cannot be observed to have passed it.

**The instrument is validated before its numbers are believed.** C3's second
corollary — a query is shown capable of finding before its negative result is
consumed — transfers unchanged to a positive quantity: a cost instrument
reproduces known-good figures before its own are used. This is not a
hypothetical discipline. The measurement that prompted this element did exactly
that, reproducing three published reference figures exactly before any new
number was consumed, and it is cited here as **form**, never as authority.

**The failure this exists to prevent, measured.** A corpus of three imported
bases was measured at 430 144 tokens in one encoding, one base alone at 250 785
— which is more than a 128k-context reader can hold at all — and a single
listing call at 112 116. None of those numbers was wrong, hidden or anyone's
bug. They were simply **never stated**, because nothing in any contract required
them to be, and the reading model in play reads every linked file whole, so a
flat index of several hundred links is several hundred whole files in every run.
The cost was a fact about the corpus long before it was a finding, and it became
a finding only because somebody happened to measure it.

**What this element does not reach.** C6 binds the **reading** side: what an
agent must be able to say about its own retrieval. Whether a bundle is *shaped*
so that a cheap cut is available — nested indexes, branch granularity, section
splitting — is the producer's question, is being decided under a separate order
in the repo that owns that corpus, and is deliberately not decided here. The two
halves meet at the ref (C2): the producer changes the shape, and the consumption
profile's cost figures are re-measured against the new ref rather than carried
forward.

---

## 2. How this relates to the profile mechanism we already own

**Decision: a parallel document, bound to the production profile by shared
identity and a pin. Not a field on `BundleProfile`.**

The pull toward one object is real and is the order's stated reason for placing
ownership here — two documents drift. It is nonetheless the wrong shape, for
three measured reasons:

1. **`BundleProfile` is a construction-time gate in one runtime.** It refuses
   illegal profiles at construction (`TypePolicy.__post_init__` refuses the
   reserved layer; the schema refuses the `timestamp`/`generated` pair). A
   consumption rule cannot be enforced that way — it is judged by an agent, at
   read time, possibly with no Python present at all. The wiki bundle's engine
   is Node; Phase 4 of this repo is Node. A contract that only exists where our
   Python package is installed is not the generic layer.
2. **The profile deliberately excludes judgement vocabulary.** `profiles.py`
   states that security is the guard's and that no disposition, origin or
   channel vocabulary belongs on a profile. Marking vocabulary is the same
   category of thing: it describes an agent's epistemic position, not a bundle's
   shape.
3. **The production profile is about bytes; the consumption contract is about
   reading.** Nothing in C1–C5 can be expressed as "which keys are emitted, in
   which order".

**How drift is prevented instead — identity plus pin, not colocation.** A
consumption profile:

- **names the production profile it reads** (`DEFAULT`, `STRICT_V1`, or the
  named profile a corpus builds under), so the two halves share one identity;
- **names the bundle ref** it was measured against (C2), so every structural
  claim it makes has a stated as-of;
- **re-derives its structural claims by command**, never by restating the
  production profile's fields in prose. A consumption profile that says
  "concepts live at the root, one index" must have measured it against a bundle
  built by the named production profile. Prose copied from the other half is the
  drift; a command is not copyable and goes red when it stops being true.

**One binding already exists and must be honoured.** `IndexPolicy` carries
`entries_match_directory`: whether the index is derived from the directory or
authored. The two consumers hold opposite postures on this and neither is a
library invariant. The consumption side inherits it unchanged: **a consuming
agent enumerates a directory only where the named profile says the index is
derived.** Where the index is authored, the directory is not the index's
denominator and a listing is not a check.

That rule was written as a correctness rule — a listing is not a check where the
index is authored. It is **also** a cost rule, because it bounds what the agent
reads, and naming both readings is what keeps it from being discarded as
pedantry by someone optimising (C6).

**A second binding, measured for this revision, and it cuts against the
optimistic reading.** `IndexPolicy` also carries `per_directory` — "an index at
every level, or only at the bundle root" — off by default. It is a **judging**
field: it is read by `required_indexes` and `violations`, which tell a caller
what to require and what to flag. No code path in this library *emits* a nested
index. The materializer refuses `STRICT_V1` for exactly this reason, in its own
words: that profile "sets `per_directory` and `entries_match_directory`, neither
of which this materializer honours". The external-bundle door writes one root
index and flattens each merged concept into the root namespace.

Two consequences for a consumption profile:

- **A permitted shape is not an emitted shape.** That a named profile allows
  nesting says nothing about whether the bundle in front of the agent has any.
  "This bundle is nested" is a structural claim and must be re-derived by
  command against the ref, never read off the profile's field list — which is
  the third bullet above, in the one case where the temptation is strongest.
- **A cost figure inherits the ref, not the profile.** Because shape can change
  without the profile changing, a consumption profile's C6 numbers are anchored
  to the ref they were measured at, and go stale when it moves.

This is the same rule Phase 3 already runs on, one level up: **a profile names
the key, a caller owns the value.** Here: the generic contract names the
required marking set; a consumption profile declares its extensions.

---

## 3. The source-marking vocabulary, decided

**Decision: five required values, plus a declared-extension mechanism.
`[verified-in-target]` is an extension, not a required value.**

Required in every conformant consumption skill, whatever the bundle:

| Marking | Meaning |
|---|---|
| `extracted` | the bundle states it directly |
| `derived` | inferred from the bundle; the reasoning is shown |
| `[unverifiable-from-bundle]` | outside what the bundle covers |
| `[unread]` | the source exists in the bundle and was not read |
| `[sourced-not-sufficient]` | the quote is real but does not carry the conclusion |

`[unverifiable-from-bundle]` is one literal string — no variants, no
translations — and that English spelling is the ratified wire literal
(item `F.10`). A repo may keep a translated *concept name* in its own prose; the
wire form does not translate.

### Which document is normative — and why the question dissolves

Neither. The order asks us to pick between the wiki masterplan (three values)
and the wiki bundle's consumption engine (six). Both are artefacts of **one**
consumer: the masterplan owns the vocabulary *for that bundle*, the engine
implements it *for that bundle*. Deciding that either is normative for the
generic layer would be adopting one consumer's contract as everyone's — the
mistake this repo has a name for (O2: we do not change, or inherit, another
repo's contract from here).

Taking ownership of the generic layer means the generic set is ours, and both of
those documents become **conformant profiles** of it. Neither has to change.
That is also why the masterplan's documentation lag (§0) does not block this
decision: it is a defect in how one consumer records its own contract, and the
generic layer does not depend on which of its two documents wins.

### Why exactly these five, and why the sixth is not among them

The line is drawn by a property, not by counting: **a required value must be
meaningful for every bundle.** `[verified-in-target]` means "read at its source
in the target repo the advisory is about" — it presupposes a target outside the
bundle that the agent can read. The second corpus measured a case where that
presupposition fails: its consumers advise about a physical installation or an
exemption application, not about a codebase, so there is no target to verify in.
Its profile also rejects the nearest-looking substitute — fields verified by a
human curator at **build** time are not verified by the agent at **consume**
time; those are two different acts and collapsing them would launder one into
the other.

A generic contract that required all six would force a marking onto bundles
where it can only be misapplied. Requiring five and letting the sixth be
declared costs nothing and is exactly how the upstream record already reads:
`F.11` settled five; `G.13` approved the sixth separately, for a case defined by
being outside the bundle.

### The extension rule

A consumption profile may add markings. It must **declare** them: the literal,
what it means for that bundle, and which required value it would otherwise
collapse into. An undeclared extension is the defect, not the extension —
`[verified-in-target]` itself exists because a fact read at its source was
falling to `derived`, reporting a checked fact as an inferred one.

Under this rule, both existing sets are conformant as they stand: the wiki
engine is five plus a declared `[verified-in-target]`; the second corpus is five
with no extension and an explicit statement of why the sixth does not transfer.

---

## 4. What a bundle-specific consumption skill must deliver to be conformant

A checklist, not a design. Each item is checkable by reading the skill — and
items 2, 7 and 8 are checkable by re-running the command the skill states, which
is the point of requiring a command rather than a claim.

1. **Names its production profile and bundle ref.** Which profile the bundle is
   built under, and the ref the profile's structural claims were measured
   against (C2, §2).
2. **States its structural claims as commands, with denominators.** File counts,
   type breakdown, index shape and depth, each with the command that produced it
   and the total it is a fraction of (C3, §2).
3. **States the retrieval unit and what would falsify it.** Which unit is
   fetched (one concept, one section, one release), and the concrete observation
   that would show the choice wrong. A retrieval hypothesis with no falsifier is
   a preference.
4. **Enumerates conditionally-written fields** and, for each, what absence does
   and does not mean (C5).
5. **Declares its marking set**: the required five verbatim, plus any extension
   with its meaning and the required value it would otherwise collapse into
   (C1, §3).
6. **Draws the cut boundary explicitly**: what the deterministic part reads,
   ranks and cuts; what the agent is forbidden to go fetch; and how the cut is
   surfaced in output as `[unread]` rather than passing silently (C4).
7. **Declares a retrieval budget and names the instrument.** The bound the cut
   targets, as a number in a stated unit, with the tool that measures it and the
   known-good figures that tool was shown to reproduce first (C6).
8. **States its cost against the whole-bundle denominator, and its breaking
   point.** What a typical answer costs to assemble, what the whole bundle at
   that ref would cost by the same instrument, whether cost tracks the question
   or the corpus, and the corpus size at which the strategy stops fitting the
   budget (C6). A skill that cannot state these has not measured its own cut.
9. **Enumerates a directory only where the named profile says the index is
   derived** (§2) — a correctness rule and a cost rule at once.
10. **Carries its own quality claim honestly.** If no run has been scored against
   its checklist, it says so. The one implemented precedent we have is
   explicitly unmeasured by its own admission, and was used here as a model for
   *form*, never as evidence that the form works.

---

## 5. What this does not decide, and what is owed elsewhere

- **No skill is designed or built here**, generic or bundle-specific. This
  document has no implementation follow-on queued.
- **Where the first generic implementation would live** — this repo's Python
  half, the Node half, or a skill — is untouched. Ownership of the *contract*
  does not carry ownership of an *engine*, and no consumer has asked for one.
- **Not ours to fix, to be reported by coord:** the wiki masterplan states three
  markings and has not been updated with the 2026-08-13 decision that settled
  five and approved a sixth. The decision is recorded in that repo's tracked
  state file; the document its own consumption engine names as requirements
  owner does not carry it. Reported, not edited (O2).
- **`.claude`'s premise correction** (§0) is owed back as a coord message: the
  ratification exists in the source repo; the query that found it absent was
  scoped to 16 of 639 files. *(Sent; the correction is accepted and recorded
  upstream.)*
- **Bundle shape is not decided here.** C6 binds what a reading agent must be
  able to say about its retrieval. Whether a corpus is nested, split or
  branched so that a cheap cut exists at all belongs to the repo that produces
  it, and is under a separate order there. Deciding it from here would be O2.
- **No instrument is named, shipped or blessed.** C6 requires that a skill name
  the instrument it used and show it reproducing known-good figures. Which tool
  that is stays the skill's choice, because the unit is vendor-specific and
  fixing one would be the mistake C6's first paragraph refuses.
- **No cost engine, ranker or cutter is designed here**, in either runtime. C6
  is a disclosure obligation on a document, not a retrieval algorithm, and
  nothing about how to *achieve* a cheap cut follows from it.