open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(consume): the withheld set is counts plus names, not one entry per concept

Browse source 
Measured 2026-09-20 on a 2313-concept bundle of one project's own
documentation: `withheld` held 2 305 entries = 186 440 B of compact JSON =
**65.5 % of the 284 850-byte payload**, and not one of those bytes counted
against the budget the same payload reports (`spent` was 45 192). A reader was
handed 239 658 bytes the budget line did not know about, to learn 2 305 concept
ids with nothing beside them -- the title being exactly what `--withheld-titles`
existed to buy, and which was off because buying it for 2 305 entries cost
another 37.9 %.

`withheld` is now a mapping: `total` (equal to `denominators.withheld`, so
SS 5.2's identity is unmoved and closes on the NUMBERS), `by_rule` (the same
total decomposed over the closed rule set, so "what kind of drop" is answerable
without the list), `nearest` (the best-ranked drops BY NAME, with title and
source document, so a reader who sees a near miss can ask for it) and
`complete`. The near misses are read off the ranking, not off `cut`'s output:
`cut` sorts by id so the partition is comparable, and that order says nothing
about which concept a reader might want next.

Same question, same bundle, after: **52 421 bytes, 18.4 % of the old file**.
The whole list stays reachable behind `--withheld-full`, and the two
instruments that classify EVERY miss by its rule -- the retrieval gate and
`okf_consume_measure` -- now ask for it explicitly and assert `complete`
rather than assuming it. `--withheld-nearest N` sets the cap (default 20,
which is `k` plus the next twelve). `--withheld-titles` is retired: a flag
whose only remaining effect would be to STRIP the title from a list the caller
asked for in full names no decision worth two shapes for one list.

`CONTRACT_REVISION` moves to `okf-consumption/2`, because a consumer indexing
the old key as a list would otherwise break silently. Three checker rules move
with it, and one of them is the interesting case: `parent_unfollowable` used
`excerpts` + `withheld` as the bundle's own denominator, which a truncated
block is not -- so that clause now runs only where the payload SAYS it is
complete, stated in SS 8.6 rather than left as a silence, with the other two
clauses (shape, self-reference) running either way. `Report` carries both
denominators, because a report claiming it examined 2 305 entries it never saw
is the same defect one level up.

The generated skill's "breaking point" section goes with it: it extrapolated a
concept count from the cost of ONE withheld entry, and there is no such slope
any more. It now states what this bundle's bookkeeping cost and that the block
is bounded by the cap rather than by the bundle -- an extrapolation from a
slope the code no longer has would be a measurement of the previous revision.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-20 23:32:22 +02:00
commit e3169ec50c
15 changed files with 708 additions and 251 deletions
Download patch file Download diff file Expand all files Collapse all files

21
README.md
View file

@ -86,11 +86,7 @@ here was measured; none of it is a plan.
(`--accounting PATH`) for that reason. The full list is under
[Build](#build).
- **A few rough edges, named rather than fixed.** `okf skill` requires `--out`
and exits 2 without it. Over MCP, `okf_describe` requires `bundle_id` where
`okf_ask` treats it as optional. One `okf_ask` answer measured over 300 KB on
a 2 002-concept bundle, because the payload's `withheld` list carries one
entry per concept that was not delivered — the payload itself was well inside
its budget.
and exits 2 without it.
- **There is no context graph and no visualisation.** Nothing in this package
draws a bundle.
@ -965,7 +961,20 @@ taken. 0.8.1's unbounded order is reproducible by no flag;
It emits the § 8 shape — `contract`, `bundle` (`bundle_id` plus a
`sha256-tree:` content identity), `budget` (unit, instrument, limit, spent and a
validated known-positive), `denominators`, `excerpts` and `withheld` — and every
withheld concept names the rule that dropped it, from a closed set of seven.
withheld concept is accounted for by the rule that dropped it, from a
closed set of seven.
**`withheld` is counts plus names, not one entry per concept** (revision
`okf-consumption/2`). It carries the `total`, the same total decomposed
`by_rule`, the best-ranked drops by name — with title and source document, so a
reader who sees a near miss can ask for it — and `complete`, which says whether
those names ARE the whole set. `--withheld-nearest N` sets how many are named
(default 20) and `--withheld-full` names every one, which is what an instrument
classifying every miss should ask for. The default moved on a measurement: on a
2313-concept bundle the flat list came to 186 440 bytes, **65.5 % of the written
payload**, none of it counted against the budget the same payload reported, and
none of it anything a reader could act on. `--withheld-titles` is retired by
that change — it existed to buy the one field the near misses now carry.
Every excerpt carries the concept's `title`, and — when the producer wrote them
— `req_number`, the SPEC § 5.1 address `sources`, and **every top-level

11
docs/consumption-contract.md
View file

@ -223,9 +223,14 @@ are permitted; the checker reads only the members this section names.
SPEC § 6.1, "Consumers MUST tolerate broken links". A `parent` names that one concept as reachable
in § 2.2's sense and names nothing else. The checker refuses
(`parent_unfollowable`) a `parent` that is not both members, that names its
own excerpt, or whose `concept_id` is in neither `excerpts` nor `withheld` —
together those name every concept the pre-pass considered (§ 5.2), so a
pointer outside them names nothing in the bundle. A pre-pass MAY also carry
own excerpt, or — **only where `withheld.complete` is true** — whose
`concept_id` is in neither `excerpts` nor `withheld.nearest`. Together those
name every concept the pre-pass considered (§ 5.2), so a pointer outside
them names nothing in the bundle; over a payload that named a sample of its
drops they do not, and a checker applying the clause anyway would refuse
correct pointers at the rate the sample is short. That the clause did not
run is a fact about the payload's shape, stated here rather than left as a
silence. A pre-pass MAY also carry
the enclosing concept's text inside `parent` — `text`, with that concept's
own `sha256`, and `truncated` where it was cut to fit — and then only from
the room its cut left, so that inherited text never displaces an excerpt.

32
skills/okf-consume-template/references/example-payload.json
View file

@ -1,5 +1,5 @@
{
"contract": "okf-consumption/1",
"contract": "okf-consumption/2",
"bundle": {
"bundle_id": "example-corpus",
"ref": "9dd86b1f0c2e4a7b8d3f5061a2c4e6809bdf1357"
@ -57,14 +57,26 @@
"title": "Scope"
}
],
"withheld": [
{
"concept_id": "appendix-a-tables",
"rule": "over budget after knapsack"
"withheld": {
"total": 2,
"by_rule": {
"over budget after knapsack": 1,
"type not in question scope": 1
},
{
"concept_id": "revision-log",
"rule": "type not in question scope"
}
]
"nearest": [
{
"concept_id": "appendix-a-tables",
"rule": "over budget after knapsack",
"title": "Appendix A — tables",
"source_file": "handbook.pdf"
},
{
"concept_id": "revision-log",
"rule": "type not in question scope",
"title": "Revision log",
"source_file": "handbook.pdf"
}
],
"complete": true
}
}

20
skills/okf-consume/SKILL.md
View file

@ -210,7 +210,7 @@ carries its denominator.
| Limit | `120000` |
| Unit | `utf-8 bytes of emitted JSON` |
| Instrument | `okf_consume.measure (len of the ensure_ascii=False JSON encoding, utf-8)` |
| Known-positive | `docs/consumption-contract.md, encoded as a JSON string` at `16389` |
| Known-positive | `docs/consumption-contract.md, encoded as a JSON string` at `16738` |
The instrument reproduces the known-positive figure before any of its own
numbers are believed. Report what the run actually spent.
@ -227,15 +227,15 @@ one answer delivered all 3 concepts — so that answer was about
**101.5 %** of the corpus. One question is one measurement: a
different question moves `spent` and this figure with it.
**The breaking point could not be measured on this bundle.** The
`withheld` list carries one entry per considered concept, and on this
bundle at generation time nothing was withheld: all 3 concepts
were delivered. There is therefore no per-entry cost to extrapolate
from, and no concept count is stated here — a bundle large enough to
withhold something states one. What does hold either way: the
bookkeeping is not counted against `spent`, and the pre-pass reads
every concept body on every run, so growth is a wall-clock cost with
no precomputed index behind it.
**What the payload costs beyond its excerpts.** The `withheld` block is
counts plus names, not one entry per concept: here it is
**70 bytes** — 0 withheld of 3 concepts,
of which **0** are named. Its size is bounded by that cap
and by the number of distinct withholding rules, not by the bundle's size,
so it does not overtake the excerpts as a bundle grows; the whole list is
still reachable with `--withheld-full`, and then it does. None of it counts
against `spent`. The pre-pass reads every concept body on every run, so
growth is a wall-clock cost with no precomputed index behind it.
## Denominators

17
skills/okf-consume/references/example-payload.json
View file

@ -1,5 +1,5 @@
{
"contract": "okf-consumption/1",
"contract": "okf-consumption/2",
"bundle": {
"bundle_id": "b-golden-segmented-okf-v0-2",
"ref": "sha256-tree:cce7a02c769793cdb6e3afda45c955461b57373deab13a986d8bf7843d6e436f"
@ -11,10 +11,10 @@
"spent": 2289,
"known_positive": {
"case": "docs/consumption-contract.md, encoded as a JSON string",
"expected": 16389,
"measured": 16389,
"raw_bytes": 15972,
"encoding_delta": 417
"expected": 16738,
"measured": 16738,
"raw_bytes": 16316,
"encoding_delta": 422
}
},
"denominators": {
@ -115,5 +115,10 @@
"rank": 3
}
],
"withheld": []
"withheld": {
"total": 0,
"by_rule": {},
"nearest": [],
"complete": true
}
}

174
src/llm_ingestion_okf/consume.py
View file

@ -717,14 +717,14 @@ KNOWN_POSITIVE_CASE = "docs/consumption-contract.md, encoded as a JSON string"
#: `measure()`'s own answer for that file. Vacuous ALONE -- which is why the
#: delta below exists.
KNOWN_POSITIVE_EXPECTED = 16_389
KNOWN_POSITIVE_EXPECTED = 16_738
#: The second, independent route. `wc -c` reports 15 972 raw bytes for the same
#: The second, independent route. `wc -c` reports 16 316 raw bytes for the same
#: file; the difference is this file's JSON quoting and escaping overhead. A
#: reader can derive it without running `measure()` at all, and it moves the
#: moment `measure()` changes what it counts -- which is what stops
#: `expected == measured` from proving nothing.
KNOWN_POSITIVE_ENCODING_DELTA = 417
KNOWN_POSITIVE_ENCODING_DELTA = 422
#: The two places that file can be, resolved in this order.
#:
@ -2120,7 +2120,31 @@ def cut(
# --- The payload (SS 8) -------------------------------------------------------
#: SS 8.2: present so a reader can tell which revision it is holding.
CONTRACT_REVISION = "okf-consumption/1"
#:
#: **`/2` since 2026-09-20**, and the bump is the point of the field. `/1`'s
#: `withheld` was a FLAT LIST of every concept the cut dropped; `/2`'s is a
#: mapping carrying the total, the count per rule and the best-ranked near
#: misses by name. A consumer indexing the old key as a list would break, and
#: a revision string that did not move would let it break silently.
CONTRACT_REVISION = "okf-consumption/2"
#: How many near misses the payload names by default. The rest of the withheld
#: set is reported as COUNTS -- total and per rule -- which is what a
#: denominator is for; the near misses are what a reader can act on.
#:
#: **20, and the number is read off a measurement rather than chosen.**
#: Measured 2026-09-20 on a 2313-concept bundle of one project's own
#: documentation: the flat list held 2 305 entries = 186 440 B of compact JSON
#: = **65.5 % of the 284 850-byte file**, and not one of those bytes counted
#: against the budget the same payload reports (`spent` was 45 192). So a
#: reader was handed 239 658 bytes the budget line did not know about, in
#: order to learn 2 305 concept ids with nothing beside them -- the field
#: `--withheld-titles` existed to buy, and which was off because buying it for
#: 2 305 entries cost another 37.9 %. At twenty entries the title and the
#: document are free, and the list becomes the one thing it never was: a set
#: of names a reader can ask for. `k` is 8, so twenty is the cut plus the next
#: twelve; the whole list stays reachable behind one switch.
WITHHELD_NEAREST_DEFAULT = 20
#: `--k` caps the DELIVERED set. The budget is the gate; this is a second,
#: cheaper bound so a question matching half the corpus does not run a
@ -2202,6 +2226,60 @@ def unanswered_terms(
return [term for term in terms if term in remaining]
def withheld_block(
withheld: Sequence[tuple[str, str]],
ranked: Sequence[tuple[Concept, float, int]],
*,
titles_by_id: Mapping[str, str],
sources_by_id: Mapping[str, str],
nearest: int = WITHHELD_NEAREST_DEFAULT,
) -> dict[str, object]:
"""The withheld set as a reader can use it: counts, then names.
**The near misses are read off `ranked`, not off `withheld`.** `cut` sorts
its drops by id so the partition is comparable between runs; that order is
alphabetical and says nothing about which concept a reader might want
next. `ranked` is in fused-rank order, so walking it and keeping the
dropped ones gives the drops in the order the instrument itself put them.
`complete` is a fact about THIS block and not about the run: true exactly
when `nearest` holds every withheld concept. A checker cannot use
`excerpts` + `withheld` as the bundle's own denominator unless it is true,
and the field is how it tells.
"""
rules: dict[str, int] = {}
for _, rule in withheld:
rules[rule] = rules.get(rule, 0) + 1
by_rule = {rule: rules[rule] for rule in WITHHOLDING_RULES if rule in rules}
# A rule outside the closed set would be a vocabulary no consumer can be
# held to (SS 5.3). It is carried rather than dropped -- a count that
# vanished would break the identity below without saying why.
by_rule.update({rule: count for rule, count in sorted(rules.items()) if rule not in by_rule})
rule_of = dict(withheld)
dropped = set(rule_of)
entries: list[dict[str, object]] = []
for concept, _, _ in ranked:
if len(entries) >= nearest:
break
if concept.concept_id not in dropped:
continue
entry: dict[str, object] = {
"concept_id": concept.concept_id,
"rule": rule_of[concept.concept_id],
}
if titles_by_id.get(concept.concept_id):
entry["title"] = titles_by_id[concept.concept_id]
if sources_by_id.get(concept.concept_id):
entry["source_file"] = sources_by_id[concept.concept_id]
entries.append(entry)
return {
"total": len(withheld),
"by_rule": by_rule,
"nearest": entries,
"complete": len(entries) == len(withheld),
}
def build_payload(
bundle_root: Path,
*,
@ -2214,7 +2292,8 @@ def build_payload(
rarity_weight: bool = False,
tie_shared_rank: bool = DEFAULT_TIE_SHARED_RANK,
title_covered: bool = DEFAULT_TITLE_COVERED,
withheld_titles: bool = False,
withheld_nearest: int = WITHHELD_NEAREST_DEFAULT,
withheld_full: bool = False,
stem_prefix: bool = DEFAULT_STEM_PREFIX,
source_quota: int | None = DEFAULT_SOURCE_QUOTA,
follow_parent: bool = DEFAULT_FOLLOW_PARENT,
@ -2224,24 +2303,31 @@ def build_payload(
Pure with respect to the clock and the network: the same
`(bundle_root, question, k, limit, cost_vocabulary, reserve_top_rank,
rarity_weight, tie_shared_rank, title_covered, withheld_titles)` at the
same bytes returns
rarity_weight, tie_shared_rank, title_covered, withheld_nearest,
withheld_full)` at the same bytes returns
the same object, every time.
**`withheld_titles` (default off) names what was dropped.** A `withheld`
entry carries `concept_id` and `rule` and no title, so a reader told that
262 concepts were withheld cannot tell WHAT was withheld without reading
the bundle -- which SS 2.2 forbids. The title closes that.
**`withheld` is COUNTS plus NAMES, never the whole list.** See
`WITHHELD_NEAREST_DEFAULT` for the measurement: the flat list was 65.5 %
of one real payload and carried nothing a reader could act on. The mapping
states `total` (equal to `denominators.withheld`, so SS 5.2's identity is
unmoved), `by_rule` (the same total decomposed over the closed rule set,
so "what kind of drop" is answerable without the list), `nearest` (the
best-ranked drops, by name), and `complete` (whether `nearest` IS the
whole set -- so a reader can tell a truncated list from a short one).
OFF BY DEFAULT BY MEASUREMENT, not by taste. Measured 2026-09-08: on N500
the payload grows 41 364 -> 57 023 bytes (+37.9 %), and on the 629-concept
K2 bundle the bookkeeping -- everything that is not an excerpt -- grows to
**122 704 bytes, past the 120 000-byte limit itself**. The instantiated
skill publishes that breaking point as "~75 KB at 629 concepts, reached at
roughly 8 000 concepts"; on by default would make that sentence false and
would move every consumer's bytes for a field none of them asked for.
Whether the naming is worth the bookkeeping is the caller's call, and the
flag is how it stays one.
**`withheld_full` is the explicit switch that brings the whole list
back**, and `withheld_nearest` is the cap when it is not given. Nothing is
deleted: the bookkeeping is off the default path, not gone, and the
per-rule counts mean the default still answers every question the flat
list answered in aggregate.
`--withheld-titles` is retired by this change rather than kept beside it.
It existed to buy the one field the near misses now carry by default, and
it was off because buying that field for 2 305 entries cost another 37.9 %.
A flag whose only remaining effect would be to STRIP the title from a list
the caller explicitly asked for in full names no decision worth two shapes
for one list.
**`link_in_signal` defaults to `False` since K3-25 and carries no CLI
flag** -- see :func:`concept_scores`. It keeps the door's `Enclosing
@ -2339,6 +2425,7 @@ def build_payload(
# payload rather than about the algorithm that produced it.
raise ConsumeError(f"spent ({spent}) exceeds limit ({limit})", code="budget_exceeded")
question_terms = list(dict.fromkeys(normalise(question)))
nearest_cap = len(withheld) if withheld_full else max(withheld_nearest, 0)
return {
"contract": CONTRACT_REVISION,
"bundle": {
@ -2389,19 +2476,16 @@ def build_payload(
),
},
"excerpts": list(delivered),
# Emitted only under the flag, and then only where the concept carries
# a title, so a bundle whose concepts have none produces the same bytes
# either way. See this function's docstring for the measurement that
# keeps the default off.
"withheld": [
{"concept_id": concept_id, "rule": rule}
| (
{"title": titles_by_id[concept_id]}
if withheld_titles and titles_by_id.get(concept_id)
else {}
)
for concept_id, rule in withheld
],
# SS 5.1/5.2 on the NUMBERS and SS 5.3 on the names. See this
# function's docstring and `WITHHELD_NEAREST_DEFAULT` for why the flat
# list is not here.
"withheld": withheld_block(
withheld,
ranked,
titles_by_id=titles_by_id,
sources_by_id={concept.concept_id: concept.source_file for concept in concepts},
nearest=nearest_cap,
),
}
@ -2564,13 +2648,24 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
help="The rule's explicit opt-out: `parent` is the pointer alone",
)
parser.add_argument(
"--withheld-titles",
"--withheld-nearest",
type=int,
default=WITHHELD_NEAREST_DEFAULT,
metavar="N",
help=(
"how many of the concepts that missed the cut the payload names, "
f"best-ranked first, with title and document. Default {WITHHELD_NEAREST_DEFAULT}. "
"The rest are reported as counts per rule; 0 names none"
),
)
parser.add_argument(
"--withheld-full",
action="store_true",
help=(
"give each withheld entry the concept's title, so a reader can see "
"WHAT was withheld without reading the bundle. OFF by default: it "
"grew a 270-concept payload by 37.9 %% and pushed a 629-concept "
"bundle's bookkeeping past the budget limit itself"
"name EVERY withheld concept instead of the nearest N. Measured "
"2026-09-20 on a 2313-concept bundle: the whole list is 186 440 B "
"= 65.5 %% of the written file, and none of it counts against the "
"budget the payload reports"
),
)
parser.add_argument("--out", type=Path, default=None, help="write here instead of stdout")
@ -2611,7 +2706,8 @@ def main(argv: list[str] | None = None) -> int:
title_covered=args.title_covered,
stem_prefix=args.stem_prefix,
source_quota=args.source_quota,
withheld_titles=args.withheld_titles,
withheld_nearest=args.withheld_nearest,
withheld_full=args.withheld_full,
follow_parent=args.follow_parent,
)
except ConsumeError as error:

166
src/llm_ingestion_okf/contract_check.py
View file

@ -109,12 +109,24 @@ class Report:
findings: tuple[Finding, ...]
rules_evaluated: int
excerpts_examined: int
#: The withheld entries this report READ, which since `okf-consumption/2`
#: is the sample the payload names and not the whole withheld set. The
#: total is in the payload; this is the denominator of what was checked,
#: and conflating the two would let a report claim it examined 2 305
#: entries it never saw.
withheld_examined: int
#: What the payload says its withheld set holds. `None` when it states no
#: total -- unmeasured, never zero.
withheld_total: int | None = None
def render(self) -> str:
named = (
f"{self.withheld_examined} withheld entries"
if self.withheld_total is None or self.withheld_total == self.withheld_examined
else f"{self.withheld_examined} of {self.withheld_total} withheld entries"
)
denominator = (
f"{self.rules_evaluated} rules over {self.excerpts_examined} excerpts "
f"and {self.withheld_examined} withheld entries"
f"{self.rules_evaluated} rules over {self.excerpts_examined} excerpts and {named}"
)
if not self.findings:
return f"conformant: {denominator}, 0 findings"
@ -311,12 +323,22 @@ def rule_excerpt_parent(ctx: Context) -> list[Finding]:
the resolved form: a `concept_id` and a `title`, naming a concept other
than the excerpt itself.
**The payload is its own denominator.** `excerpts` and `withheld` together
name every concept the pre-pass considered, which is every concept of the
bundle (SS 5.2), so a `parent.concept_id` in neither names nothing in the
bundle -- and the rule sees that without opening the bundle, the boundary
`rule_bundle_identity` keeps too. A payload lying about both lists at once
passes here and fails `denominator_identity` instead.
**The payload is its own denominator ONLY WHERE IT SAYS IT IS.** `excerpts`
and `withheld` together name every concept the pre-pass considered, which
is every concept of the bundle (SS 5.2) -- so a `parent.concept_id` in
neither names nothing in the bundle, and the rule sees that without
opening the bundle, the boundary `rule_bundle_identity` keeps too.
Since `okf-consumption/2` that premise is CONDITIONAL and the payload
states which it is: `withheld.nearest` is a sample of the drops unless
`withheld.complete` is true. Over a truncated block the clause is not run,
because every real parent pointing at a drop the sample did not name would
be refused -- a rule firing on the correct answer. It is stated rather than
silent: SS 8.6 says so, and the known-positive in the suite fires the
clause over a complete block on the same payload it is skipped on. The
other two clauses -- the shape, and a parent naming its own excerpt -- do
not read the denominator and run either way. A payload lying about both
lists at once passes here and fails `denominator_identity` instead.
**Conditional, like SS 8.4's fields.** An excerpt with no `parent` meets
this rule as it always did. `parent_unresolved` is not a finding: SPEC SS
@ -326,10 +348,14 @@ def rule_excerpt_parent(ctx: Context) -> list[Finding]:
if not ctx.payload_is_mapping:
return []
excerpts = [_mapping(raw) for raw in _sequence(ctx.payload.get("excerpts"))]
considered = {_text(excerpt.get("concept_id")) for excerpt in excerpts} | {
_text(_mapping(raw).get("concept_id")) for raw in _sequence(ctx.payload.get("withheld"))
}
considered.discard("")
block = _mapping(ctx.payload.get("withheld"))
complete = block.get("complete") is True
considered: set[str] | None = None
if complete:
considered = {_text(excerpt.get("concept_id")) for excerpt in excerpts} | {
_text(_mapping(raw).get("concept_id")) for raw in _sequence(block.get("nearest"))
}
considered.discard("")
findings = []
for position, excerpt in enumerate(excerpts):
if "parent" not in excerpt:
@ -340,7 +366,7 @@ def rule_excerpt_parent(ctx: Context) -> list[Finding]:
reason = "is not a `concept_id` and a `title`, so a reader can neither open nor cite it"
elif target == _text(excerpt.get("concept_id")):
reason = f"names the excerpt itself ({target!r})"
elif target not in considered:
elif considered is not None and target not in considered:
reason = (
f"names {target!r}, which is in neither `excerpts` nor `withheld` and so is "
"no concept of this bundle"
@ -420,41 +446,132 @@ def rule_denominator_identity(ctx: Context) -> list[Finding]:
def rule_denominator_lists(ctx: Context) -> list[Finding]:
"""SS 8.1: `delivered` against the excerpts, `withheld` against its TOTAL.
The two sides are not symmetric, and since `okf-consumption/2` they say so.
`excerpts` is the payload's product and its length is the count. `withheld`
is bookkeeping, and the flat list of it was 65.5 % of one measured payload
while answering nothing a reader could act on -- so the count it is held
against is the one the block states, not the length of the sample of near
misses it names. What keeps that honest is the next rule: the total is
decomposed over the closed rule set and the decomposition must add up.
"""
if not ctx.payload_is_mapping:
return []
counts = _mapping(ctx.payload.get("denominators"))
findings = []
for key, member in (("delivered", "excerpts"), ("withheld", "withheld")):
declared = _whole(counts.get(key))
if declared is None:
continue
actual = len(_sequence(ctx.payload.get(member)))
declared = _whole(counts.get("delivered"))
if declared is not None:
actual = len(_sequence(ctx.payload.get("excerpts")))
if declared != actual:
findings.append(
Finding(
"denominator_list_mismatch",
f"denominators.{key} is {declared} but {member} holds {actual}; "
f"denominators.delivered is {declared} but excerpts holds {actual}; "
"the count and the list are two statements of one fact (SS 8.1)",
)
)
declared = _whole(counts.get("withheld"))
block = _mapping(ctx.payload.get("withheld"))
total = _whole(block.get("total"))
if declared is not None and total is None:
findings.append(
Finding(
"denominator_list_mismatch",
"withheld reports no whole-number `total`; a bookkeeping block that "
"does not state its own count cannot be held against the "
"denominator (SS 8.1)",
)
)
elif declared is not None and total != declared:
findings.append(
Finding(
"denominator_list_mismatch",
f"denominators.withheld is {declared} but withheld.total is {total}; "
"the count and the block are two statements of one fact (SS 8.1)",
)
)
return findings
def rule_withheld_rules(ctx: Context) -> list[Finding]:
if not ctx.payload_is_mapping:
"""SS 5.3: every drop names the rule that dropped it -- in aggregate, and
by name for the ones the payload names.
Three statements have to hold together, or the truncation stops being a
sample and becomes a silence: `by_rule` adds up to `total` (so a reader
knows what KIND of drop the unnamed ones were), `nearest` is no longer
than `total`, and `complete` is true exactly when `nearest` IS the whole
set -- which is what `rule_excerpt_parent` reads before it treats the
payload as its own denominator.
"""
if not ctx.payload_is_mapping or "withheld" not in ctx.payload:
# A payload carrying no `withheld` at all is `denominator_identity`'s
# defect and `denominator_lists`' -- no rule here restates another.
# A `withheld` that is PRESENT and is not this block (the flat list of
# `okf-consumption/1`, say) falls through and is named below.
return []
block = _mapping(ctx.payload.get("withheld"))
findings = []
for position, raw in enumerate(_sequence(ctx.payload.get("withheld"))):
total = _whole(block.get("total"))
rules = _mapping(block.get("by_rule"))
counted = [_whole(value) for value in rules.values()]
if any(value is None for value in counted):
findings.append(
Finding(
"cut_undeclared",
"withheld.by_rule carries a count that is not a whole number; a "
"count that is not reported is unmeasured, not zero (SS 5.3)",
)
)
elif total is not None and sum(value for value in counted if value is not None) != total:
findings.append(
Finding(
"cut_undeclared",
f"withheld.by_rule sums to "
f"{sum(value for value in counted if value is not None)} but "
f"withheld.total is {total}; a decomposition that does not close "
"leaves drops with no rule at all (SS 5.3)",
)
)
nearest = _sequence(block.get("nearest"))
for position, raw in enumerate(nearest):
entry = _mapping(raw)
for key in ("concept_id", "rule"):
if not _text(entry.get(key)):
findings.append(
Finding(
"cut_undeclared",
f"withheld entry {position} carries no {key!r}; a visible "
"drop is worth more than a silent one (SS 5.3)",
f"withheld.nearest entry {position} carries no {key!r}; a "
"visible drop is worth more than a silent one (SS 5.3)",
)
)
if total is not None and len(nearest) > total:
findings.append(
Finding(
"cut_undeclared",
f"withheld.nearest names {len(nearest)} concepts but withheld.total "
f"is {total}; the sample cannot be larger than the set (SS 5.3)",
)
)
complete = block.get("complete")
if not isinstance(complete, bool):
findings.append(
Finding(
"cut_undeclared",
"withheld carries no boolean `complete`; without it a reader cannot "
"tell a truncated list from a short one (SS 5.3)",
)
)
elif total is not None and complete is not (len(nearest) == total):
findings.append(
Finding(
"cut_undeclared",
f"withheld.complete is {complete} while nearest holds {len(nearest)} "
f"of {total}; `complete` is a fact about the block, not a claim "
"about the run (SS 5.3)",
)
)
return findings
@ -608,7 +725,8 @@ def check(skill_text: str, payload: object) -> Report:
findings=tuple(findings),
rules_evaluated=len(RULES),
excerpts_examined=len(_sequence(ctx.payload.get("excerpts"))),
withheld_examined=len(_sequence(ctx.payload.get("withheld"))),
withheld_examined=len(_sequence(_mapping(ctx.payload.get("withheld")).get("nearest"))),
withheld_total=_whole(_mapping(ctx.payload.get("withheld")).get("total")),
)

83
src/llm_ingestion_okf/skill.py
View file

@ -373,15 +373,12 @@ def render(
budget = payload["budget"]
assert isinstance(budget, dict)
withheld = payload["withheld"]
assert isinstance(withheld, list)
assert isinstance(withheld, dict)
bookkeeping = okf_consume.measure(json.dumps(withheld, ensure_ascii=False))
per_withheld = bookkeeping / len(withheld) if withheld else 0.0
# `0` is what this was until 2026-09-20, and it was not a small number: it
# was the absence of a measurement printed as one. A bundle small enough to
# deliver everything withholds nothing, so there is no per-entry cost to
# extrapolate from -- and `int(LIMIT / 0.0)` was written as `0 concepts`,
# which reads as a bundle that breaks before it holds anything.
breaking = int(okf_consume.DEFAULT_LIMIT / per_withheld) if per_withheld else 0
withheld_total = withheld["total"]
assert isinstance(withheld_total, int)
named = withheld["nearest"]
assert isinstance(named, list)
name = f"{slug(bundle_id)}-consume"
text = template_path().read_text(encoding="utf-8")
@ -403,8 +400,8 @@ def render(
spent=int(budget["spent"]),
delivered=int(denominators["delivered"]),
bookkeeping=bookkeeping,
breaking=breaking,
withheld_count=len(withheld),
withheld_count=withheld_total,
withheld_named=len(named),
)
# Claude Code reads this header with a YAML reader, and `description`
# carries the root index's `bundle_id` raw -- a bundle this library did not
@ -454,8 +451,8 @@ def _rewrite(
spent: int,
delivered: int,
bookkeeping: int,
breaking: int,
withheld_count: int,
withheld_named: int,
) -> str:
replacements: list[tuple[str, str]] = [
(
@ -514,8 +511,8 @@ def _rewrite(
spent=spent,
delivered=delivered,
bookkeeping=bookkeeping,
breaking=breaking,
withheld_count=withheld_count,
withheld_named=withheld_named,
),
),
(
@ -580,8 +577,8 @@ def _scaling(
spent: int,
delivered: int,
bookkeeping: int,
breaking: int,
withheld_count: int,
withheld_named: int,
) -> str:
share = (spent / cost * 100) if cost else 0.0
return (
@ -593,43 +590,43 @@ def _scaling(
f"**{share:.1f} %** of the corpus. One question is one measurement: a\n"
"different question moves `spent` and this figure with it.\n\n"
+ _breaking_point(
total=total, bookkeeping=bookkeeping, breaking=breaking, withheld_count=withheld_count
total=total,
bookkeeping=bookkeeping,
withheld_count=withheld_count,
withheld_named=withheld_named,
)
)
def _breaking_point(*, total: int, bookkeeping: int, breaking: int, withheld_count: int) -> str:
"""The concept count at which the bookkeeping alone fills the budget.
def _breaking_point(
*, total: int, bookkeeping: int, withheld_count: int, withheld_named: int
) -> str:
"""What the payload costs beyond its excerpts, and whether it grows.
It is EXTRAPOLATED from the cost of one `withheld` entry, so a generation
run that withheld nothing has no slope to extrapolate from. That case used
to print `0 concepts` -- a division that never happened, rendered as a
measurement, and the one number in this document that said the bundle
breaks before it holds anything. The sentence is withheld instead, with the
reason and the two facts that are measured.
Until `okf-consumption/2` this section extrapolated a concept count at
which the bookkeeping alone would fill the budget, because `withheld`
carried one entry per considered concept and grew linearly. Measured
2026-09-20 on a 2313-concept bundle, that growth had arrived: the list was
186 440 B = 65.5 % of the written file, none of it counted against
`spent`.
It does not grow that way any more, so this section no longer states a
concept count -- a number extrapolated from a slope the code no longer has
would be a measurement of the previous revision. What it states instead is
the two facts that are measured on THIS bundle: what the bookkeeping cost
here, and that it is bounded by the near-miss cap rather than by the
bundle's size.
"""
if withheld_count == 0:
return (
"**The breaking point could not be measured on this bundle.** The\n"
"`withheld` list carries one entry per considered concept, and on this\n"
f"bundle at generation time nothing was withheld: all {total} concepts\n"
"were delivered. There is therefore no per-entry cost to extrapolate\n"
"from, and no concept count is stated here — a bundle large enough to\n"
"withhold something states one. What does hold either way: the\n"
"bookkeeping is not counted against `spent`, and the pre-pass reads\n"
"every concept body on every run, so growth is a wall-clock cost with\n"
"no precomputed index behind it."
)
return (
"**The breaking point, stated so it can be observed to have been passed.**\n"
"The `withheld` list carries one entry per considered concept and grows\n"
f"linearly: here it is **{bookkeeping} bytes** for {withheld_count} of\n"
f"{total} concepts. At roughly **{breaking} concepts** the bookkeeping alone\n"
f"reaches the {okf_consume.DEFAULT_LIMIT}-byte\n"
"limit, and although it is not counted against `spent`, a payload whose\n"
"bookkeeping dwarfs its content has stopped being a cut. The pre-pass also\n"
"reads every concept body on every run, so the same growth is a wall-clock\n"
"cost with no precomputed index behind it."
"**What the payload costs beyond its excerpts.** The `withheld` block is\n"
"counts plus names, not one entry per concept: here it is\n"
f"**{bookkeeping} bytes** — {withheld_count} withheld of {total} concepts,\n"
f"of which **{withheld_named}** are named. Its size is bounded by that cap\n"
"and by the number of distinct withholding rules, not by the bundle's size,\n"
"so it does not overtake the excerpts as a bundle grows; the whole list is\n"
"still reachable with `--withheld-full`, and then it does. None of it counts\n"
"against `spent`. The pre-pass reads every concept body on every run, so\n"
"growth is a wall-clock cost with no precomputed index behind it."
)

3
tests/test_contract_check.py
View file

@ -96,6 +96,7 @@ def test_the_report_carries_its_denominators() -> None:
assert report.rules_evaluated > 0
assert report.excerpts_examined == 3
assert report.withheld_examined == 2
assert report.withheld_total == 2
rendered = report.render()
assert str(report.rules_evaluated) in rendered
assert "3" in rendered and "2" in rendered
@ -150,7 +151,7 @@ def test_a_bundle_without_a_ref_is_named() -> None:
def test_a_withheld_concept_without_a_rule_is_an_undeclared_cut() -> None:
payload = load_example()
del payload["withheld"][0]["rule"]
del payload["withheld"]["nearest"][0]["rule"]
assert "cut_undeclared" in codes(instantiated(), payload)

319
tests/test_okf_consume.py
View file

@ -511,6 +511,22 @@ def test_document_scores_are_identical_across_two_calls() -> None:
# --- Step 6: stage-two concept ranking, fused by RRF --------------------------
def _withheld_entries(payload: Mapping[str, Any]) -> list[Any]:
"""Every withheld entry of a payload whose block says it names them all.
Since `okf-consumption/2` `withheld` is counts plus a CAPPED sample, so a
test reading entries has to say which it is reading. `complete` is
asserted rather than assumed: over a truncated block these tests would be
measuring the first twenty of something and reporting it as the set.
"""
block = payload["withheld"]
assert isinstance(block, Mapping)
assert block["complete"] is True, "the block is a sample here, not the set"
nearest = block["nearest"]
assert isinstance(nearest, list)
return nearest
def _fixture_concepts() -> list[okf_consume.Concept]:
return [
okf_consume.read_concept(
@ -638,51 +654,6 @@ def test_a_concept_whose_verified_cannot_be_read_is_withheld_by_name() -> None:
assert dict(withheld)["dyp/nivaa/blokkform-verifisert"] == "verified_unreadable"
def test_a_withheld_entry_names_what_was_dropped_under_the_flag() -> None:
# A reader who is told 262 concepts were withheld, by id and rule alone,
# cannot tell WHAT was withheld without reading the bundle -- which SS 2.2
# forbids. The title closes that, and it is emitted only where the concept
# carries one.
payload = okf_consume.build_payload(
FIXTURE, question="Hvordan skal prisene fylles ut?", withheld_titles=True
)
entries = payload["withheld"]
assert isinstance(entries, list) and entries
titled = [entry for entry in entries if "title" in entry]
assert titled, "no withheld entry carried a title, so the rule measures nothing"
concepts = {concept.concept_id: concept for concept in _fixture_concepts()}
for entry in entries:
concept = concepts[str(entry["concept_id"])]
if concept.title:
assert entry["title"] == concept.title
else:
assert "title" not in entry
def test_no_withheld_entry_names_anything_without_the_flag() -> None:
# The default is what every consumer already runs, and this is the
# measurement that keeps it theirs: a title on every withheld entry grew a
# 270-concept payload by 37.9 % and pushed a 629-concept bundle's
# bookkeeping past the budget limit itself.
payload = okf_consume.build_payload(FIXTURE, question="Hvordan skal prisene fylles ut?")
entries = payload["withheld"]
assert isinstance(entries, list) and entries
assert all(set(entry) == {"concept_id", "rule"} for entry in entries)
def test_the_withheld_title_flag_costs_bytes_and_the_default_pays_none() -> None:
question = "Hvordan skal prisene fylles ut?"
off = okf_consume.serialise(okf_consume.build_payload(FIXTURE, question=question))
explicit_off = okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, withheld_titles=False)
)
on = okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, withheld_titles=True)
)
assert off == explicit_off
assert len(on.encode("utf-8")) > len(off.encode("utf-8"))
def test_delivered_and_withheld_partition_the_considered_set() -> None:
delivered, withheld, considered = _cut_fixture()
delivered_ids = {excerpt["concept_id"] for excerpt in delivered}
@ -757,7 +728,7 @@ def test_the_payload_passes_the_checker_against_a_skill_for_its_own_bundle() ->
def test_the_payload_carries_every_section_eight_member() -> None:
payload = _payload()
assert payload["contract"] == "okf-consumption/1"
assert payload["contract"] == "okf-consumption/2"
assert set(payload) >= {
"contract",
"bundle",
@ -801,8 +772,9 @@ def test_spent_moves_when_an_excerpt_moves_and_holds_when_withheld_grows() -> No
def test_the_counts_and_the_lists_are_two_statements_of_one_fact() -> None:
payload = _payload()
counts, excerpts, withheld = payload["denominators"], payload["excerpts"], payload["withheld"]
assert isinstance(counts, dict) and isinstance(excerpts, list) and isinstance(withheld, list)
counts, excerpts = payload["denominators"], payload["excerpts"]
withheld = _withheld_entries(payload)
assert isinstance(counts, dict) and isinstance(excerpts, list)
assert counts["delivered"] == len(excerpts)
assert counts["withheld"] == len(withheld)
assert counts["considered"] == counts["delivered"] + counts["withheld"]
@ -858,14 +830,17 @@ def test_a_question_with_no_answer_returns_a_measured_empty_set_not_a_guess() ->
# still closes and the skill can say "measured, nothing cleared the bar"
# rather than "nothing was found".
payload = _payload(question="Hva er reglene for sveising av titan i vakuum?")
counts, excerpts, withheld = payload["denominators"], payload["excerpts"], payload["withheld"]
assert isinstance(counts, dict) and isinstance(excerpts, list) and isinstance(withheld, list)
counts, excerpts = payload["denominators"], payload["excerpts"]
assert isinstance(counts, dict) and isinstance(excerpts, list)
assert excerpts == []
assert counts["delivered"] == 0
assert (
counts["withheld"] == counts["considered"] == len(okf_consume.enumerate_concepts(FIXTURE))
)
assert {entry["rule"] for entry in withheld} == {"no_lexical_match", "verdict_layer_excluded"}
assert {entry["rule"] for entry in _withheld_entries(payload)} == {
"no_lexical_match",
"verdict_layer_excluded",
}
# And the control: the SAME payload builder returns a non-empty set for a
# question this bundle does answer, so the zero is a measurement.
answered = _payload()
@ -930,7 +905,16 @@ def test_spent_is_the_delivered_set_where_the_whole_payload_reading_would_refuse
# from the plan: the plan predicted 101 576 B for this excerpt and 188 758 B
# for the payload, both taken before per-line trailing-whitespace stripping
# landed. What this build actually produces is recorded instead.
payload = okf_consume.build_payload(K2_BUNDLE, question="Hvordan skal prisene fylles ut?")
# Run with `--withheld-full`, because the DEFAULT no longer reaches this
# case and that is the point of `okf-consumption/2`: the whole payload of
# this bundle came to 117 561 B under the default -- inside the limit --
# where under the flat list it did not. The distinction between the two
# readings of SS 7.2 is still real and still reachable, so it is measured
# where it is reachable rather than deleted with the shape that produced
# it. The control below fires if that stops being true.
payload = okf_consume.build_payload(
K2_BUNDLE, question="Hvordan skal prisene fylles ut?", withheld_full=True
)
budget, excerpts = payload["budget"], payload["excerpts"]
assert isinstance(budget, dict) and isinstance(excerpts, list)
whole_payload = len(okf_consume.serialise(payload).encode("utf-8"))
@ -940,6 +924,11 @@ def test_spent_is_the_delivered_set_where_the_whole_payload_reading_would_refuse
)
assert int(budget["spent"]) <= int(budget["limit"])
# And the default is now the other side of that line, on the same bundle
# and the same question: the bookkeeping stopped dwarfing the content.
default = okf_consume.build_payload(K2_BUNDLE, question="Hvordan skal prisene fylles ut?")
assert len(okf_consume.serialise(default).encode("utf-8")) < whole_payload
#: The gold set is LOCAL-ONLY: it names corpus documents, which never reach a
#: tracked file here. The test reads it rather than restating it, so this file
@ -1481,8 +1470,9 @@ def test_a_cost_question_reaches_no_price_concept_without_the_flag() -> None:
# The known-negative this whole step is measured against. Without it, the
# flag's effect below would have no denominator.
payload = _payload(question="Hvor kan vi kutte kostnader?")
counts, withheld = payload["denominators"], payload["withheld"]
assert isinstance(counts, dict) and isinstance(withheld, list)
counts = payload["denominators"]
withheld = _withheld_entries(payload)
assert isinstance(counts, dict)
assert counts["delivered"] == 0
# By the two fields this test is about, not by the whole entry: the entry
# also carries the concept's title, and pinning the exact dict here would
@ -1683,10 +1673,9 @@ def test_the_knapsack_evicts_the_top_ranked_candidate_that_costs_half_the_budget
}
assert EVICTION_TOP not in weights
assert (
dict(
(entry["concept_id"], entry["rule"])
for entry in payload["withheld"] # type: ignore[union-attr]
)[EVICTION_TOP]
dict((entry["concept_id"], entry["rule"]) for entry in _withheld_entries(payload))[
EVICTION_TOP
]
== "over_budget_after_knapsack"
)
# The shape itself, stated as numbers rather than assumed: the top candidate
@ -1714,10 +1703,7 @@ def test_reserving_the_top_rank_delivers_the_candidate_the_knapsack_evicted(
assert isinstance(excerpts, list)
assert excerpts[0]["concept_id"] == EVICTION_TOP
assert excerpts[0]["rank"] == 1
assert EVICTION_TOP not in {
entry["concept_id"]
for entry in payload["withheld"] # type: ignore[union-attr]
}
assert EVICTION_TOP not in {entry["concept_id"] for entry in _withheld_entries(payload)}
def test_the_reservation_is_off_by_default_and_the_default_payload_is_byte_identical(
@ -1747,10 +1733,7 @@ def test_a_top_candidate_that_alone_exceeds_the_budget_is_still_refused_by_name(
root = _eviction_bundle(tmp_path / "bundle")
limit = _eviction_bundle_top_weight(root) - 1
payload = _eviction_payload(root, limit=limit, reserve_top_rank=True)
rules = dict(
(entry["concept_id"], entry["rule"])
for entry in payload["withheld"] # type: ignore[union-attr]
)
rules = dict((entry["concept_id"], entry["rule"]) for entry in _withheld_entries(payload))
assert rules[EVICTION_TOP] == "over_budget_alone"
spent = payload["budget"]["spent"] # type: ignore[index]
assert isinstance(spent, int)
@ -1778,7 +1761,7 @@ def test_the_reservation_displaces_lower_ranked_excerpts_under_the_rule_that_exi
assert len(with_reservation["excerpts"]) < len(without["excerpts"]) # type: ignore[arg-type]
displaced = {
entry["concept_id"]
for entry in with_reservation["withheld"] # type: ignore[union-attr]
for entry in _withheld_entries(with_reservation)
if entry["rule"] == "over_budget_after_knapsack"
}
delivered_before = {
@ -1786,10 +1769,9 @@ def test_the_reservation_displaces_lower_ranked_excerpts_under_the_rule_that_exi
for excerpt in without["excerpts"] # type: ignore[union-attr]
}
assert displaced & delivered_before
assert {
entry["rule"]
for entry in with_reservation["withheld"] # type: ignore[union-attr]
} <= set(okf_consume.WITHHOLDING_RULES)
assert {entry["rule"] for entry in _withheld_entries(with_reservation)} <= set(
okf_consume.WITHHOLDING_RULES
)
def test_the_payload_declares_which_concept_the_reservation_took_and_what_it_cost(
@ -2718,3 +2700,198 @@ def test_a_question_the_bundle_answers_leaves_the_unanswered_lists_short() -> No
]
assert in_payload, "the delivered excerpts answer nothing of a question they were cut for"
assert len(coverage["unanswered_in_bundle"]) < len(terms) # type: ignore[arg-type]
# --- A2: the withheld list is bookkeeping, not the payload --------------------
def _mapping_of(value: object) -> Mapping[str, Any]:
assert isinstance(value, Mapping)
return value
def _sequence_of(value: object) -> list[Any]:
assert isinstance(value, list)
return value
def _ranked_fixture(question: str) -> list[tuple[okf_consume.Concept, float, int]]:
"""The same ranking `build_payload` runs on, with the same defaults.
Rebuilt here rather than exported: a helper that took the payload's own
order would be comparing the order against itself.
"""
concepts = _fixture_concepts()
texts = okf_consume.searchable_text(concepts, link_in_signal=okf_consume.DEFAULT_LINK_IN_SIGNAL)
stems = (
frozenset(token for text in texts for token in okf_consume.normalise(text))
if okf_consume.DEFAULT_STEM_PREFIX
else None
)
return list(
okf_consume.concept_scores(
concepts,
question,
okf_consume.document_scores(FIXTURE, question, stems=stems),
tie_shared_rank=okf_consume.DEFAULT_TIE_SHARED_RANK,
title_covered=okf_consume.DEFAULT_TITLE_COVERED,
stems=stems,
link_in_signal=okf_consume.DEFAULT_LINK_IN_SIGNAL,
)
)
def test_the_default_payload_reports_the_withheld_as_counts_and_near_misses() -> None:
"""Measured 2026-09-20 on a 2313-concept bundle: `withheld` held 2 305
entries = 186 440 B = 65.5 % of the written file, and NONE of it counted
against the budget the payload reports. A reader was handed 239 658 bytes
the budget line did not know about, to learn 2 305 ids they could do
nothing with.
The replacement states the same facts in the units a reader can act on:
the total, the count per rule, and the near misses BY NAME.
"""
payload = okf_consume.build_payload(FIXTURE, question="Hvordan skal prisene fylles ut?")
withheld = payload["withheld"]
assert isinstance(withheld, Mapping)
assert set(withheld) == {"total", "by_rule", "nearest", "complete"}
counts = payload["denominators"]
assert isinstance(counts, Mapping)
assert withheld["total"] == counts["withheld"]
by_rule = withheld["by_rule"]
assert isinstance(by_rule, Mapping)
assert sum(by_rule.values()) == withheld["total"]
assert set(by_rule) <= set(okf_consume.WITHHOLDING_RULES)
nearest = withheld["nearest"]
assert isinstance(nearest, list)
assert len(nearest) <= okf_consume.WITHHELD_NEAREST_DEFAULT
def test_a_near_miss_is_named_so_a_reader_can_ask_for_it() -> None:
"""Every entry carries the title and the document, not just the id.
`--withheld-titles` bought the title for all 2 305 entries and was off by
measurement (+37.9 % on N500). At twenty entries the same field costs
nothing and is the one thing that makes the list usable: a reader who sees
a near miss by name can ask for it.
"""
payload = okf_consume.build_payload(FIXTURE, question="Hvordan skal prisene fylles ut?")
withheld = payload["withheld"]
assert isinstance(withheld, Mapping)
nearest = withheld["nearest"]
assert isinstance(nearest, list) and nearest
concepts = {concept.concept_id: concept for concept in _fixture_concepts()}
named = 0
for entry in nearest:
assert isinstance(entry, Mapping)
assert set(entry) <= {"concept_id", "rule", "title", "source_file"}
assert entry["concept_id"] and entry["rule"]
concept = concepts[str(entry["concept_id"])]
if concept.title:
assert entry["title"] == concept.title
named += 1
else:
assert "title" not in entry
assert named, "no near miss carried a title, so the rule measures nothing"
def test_the_near_misses_are_the_best_ranked_of_the_withheld() -> None:
"""Rank order, not id order.
`cut` sorts its drops by id so the partition is comparable between runs;
that order is alphabetical and says nothing about which concept a reader
might want next. Driven directly rather than through `build_payload`,
because the synthetic fixture's six drops come back in the SAME order
either way -- a bundle where the two orders agree cannot tell them apart,
which the control below asserts before the rest is believed.
"""
concepts = _fixture_concepts()
assert len(concepts) >= 4
# Rank order deliberately the reverse of id order.
ranked = [(concept, 1.0, 1) for concept in sorted(concepts, key=lambda c: c.concept_id)[::-1]]
withheld = sorted((concept.concept_id, "below_k") for concept, _, _ in ranked)
rank_order = [concept.concept_id for concept, _, _ in ranked]
assert rank_order != [concept_id for concept_id, _ in withheld], (
"the two orders agree here, so the assertion below would measure nothing"
)
block = okf_consume.withheld_block(
withheld,
ranked,
titles_by_id={c.concept_id: c.title for c in concepts},
sources_by_id={c.concept_id: c.source_file for c in concepts},
nearest=3,
)
assert [str(entry["concept_id"]) for entry in _sequence_of(block["nearest"])] == rank_order[:3]
assert block["total"] == len(withheld)
assert block["complete"] is False
def test_every_withheld_rule_is_counted_even_when_it_is_not_named() -> None:
"""`by_rule` is what makes the truncation honest: the near misses are a
sample, the counts are the whole set."""
payload = okf_consume.build_payload(
FIXTURE, question="Hvordan skal prisene fylles ut?", withheld_nearest=0
)
block = _mapping_of(payload["withheld"])
by_rule = _mapping_of(block["by_rule"])
assert block["nearest"] == []
assert (
sum(by_rule.values()) == block["total"] == _mapping_of(payload["denominators"])["withheld"]
)
assert len(by_rule) > 1, "one rule only, so the decomposition measures nothing"
def test_the_whole_list_is_reachable_behind_one_explicit_switch() -> None:
"""The bookkeeping is not deleted; it is moved off the default path."""
question = "Hvordan skal prisene fylles ut?"
full = okf_consume.build_payload(FIXTURE, question=question, withheld_full=True)
block = full["withheld"]
assert isinstance(block, Mapping)
assert block["complete"] is True
nearest = block["nearest"]
assert isinstance(nearest, list)
assert len(nearest) == block["total"]
# The bytes the switch buys, measured against the same payload with no
# near misses named at all -- this fixture holds 7 concepts, so its
# DEFAULT is already complete and cannot show the difference.
none_named = okf_consume.build_payload(FIXTURE, question=question, withheld_nearest=0)
assert _mapping_of(none_named["withheld"])["complete"] is False
assert len(okf_consume.serialise(full)) > len(okf_consume.serialise(none_named))
def test_the_near_miss_cap_is_a_number_the_caller_may_set() -> None:
question = "Hvordan skal prisene fylles ut?"
payload = okf_consume.build_payload(FIXTURE, question=question, withheld_nearest=1)
block = payload["withheld"]
assert isinstance(block, Mapping)
nearest = block["nearest"]
assert isinstance(nearest, list)
assert len(nearest) == min(1, int(str(block["total"])))
none_at_all = okf_consume.build_payload(FIXTURE, question=question, withheld_nearest=0)
empty = none_at_all["withheld"]
assert isinstance(empty, Mapping)
assert empty["nearest"] == []
def test_the_cli_carries_both_switches() -> None:
parsed = okf_consume.parse_args(["b", "--question", "q", "--withheld-full"])
assert parsed.withheld_full is True
assert parsed.withheld_nearest == okf_consume.WITHHELD_NEAREST_DEFAULT
assert okf_consume.parse_args(["b", "--question", "q"]).withheld_full is False
assert (
okf_consume.parse_args(["b", "--question", "q", "--withheld-nearest", "3"]).withheld_nearest
== 3
)
def test_the_payload_declares_the_revision_whose_shape_it_has() -> None:
"""`withheld` went from a list to a mapping. A consumer reading the old
revision string and indexing it as a list would break silently, so the
string moves with the shape.
"""
assert okf_consume.CONTRACT_REVISION == "okf-consumption/2"
payload = okf_consume.build_payload(FIXTURE, question="Hvordan skal prisene fylles ut?")
assert payload["contract"] == "okf-consumption/2"

50
tests/test_okf_skill.py
View file

@ -366,42 +366,46 @@ def test_a_project_skill_still_passes_the_contract_checker(tmp_path: Path) -> No
@pytest.mark.parametrize("bundle", BUNDLES, ids=lambda path: path.name)
def test_the_breaking_point_is_a_measurement_or_it_is_withheld(
def test_the_payload_cost_section_is_measured_on_the_bundle_it_names(
bundle: Path, tmp_path: Path
) -> None:
"""`0 concepts` was a division that never happened, printed as a number.
"""The section that used to extrapolate a breaking point now states cost.
The figure is EXTRAPOLATED from what one `withheld` entry costs, so a
generation run that withheld nothing has no slope to extrapolate from:
`per_withheld` was `0.0`, the guard returned the literal `0`, and the
document told its reader the bundle's bookkeeping fills a 120000-byte
budget at zero concepts -- before the bundle holds anything at all.
`0 concepts` was a division that never happened, printed as a number: the
figure was extrapolated from what ONE `withheld` entry cost, and a run
that withheld nothing had no slope. Since `okf-consumption/2` there is no
slope at all -- the block is counts plus a capped sample -- so the section
states what this bundle's bookkeeping cost and no extrapolated count, and
a generator that printed one would be describing the previous revision.
Driven from both sides so a generator that simply stopped stating the
figure would fail: the bundle that withholds nothing must say it could not
measure it, and a bundle that withholds something must still print a
positive count.
Driven from both sides: the numbers must be the payload's own, and the
retired figure must not come back.
"""
written = _generate(bundle, tmp_path / "out")
text = written.read_text(encoding="utf-8")
payload = json.loads((tmp_path / "out" / "references" / "example-payload.json").read_text())
assert payload["withheld"], "the known-positive arm withheld nothing to extrapolate from"
assert "**0 concepts**" not in text
stated = re.search(r"At roughly\s+\*\*(\d+) concepts\*\*", text)
assert stated is not None, "a bundle that withheld something states no figure"
block = payload["withheld"]
assert block["total"], "the known-positive arm withheld nothing to report"
assert "At roughly" not in text and "**0 concepts**" not in text
stated = re.search(r"\*\*(\d+) bytes\*\* — (\d+) withheld of (\d+) concepts", text)
assert stated is not None, "the section states no measured cost"
assert int(stated.group(2)) == block["total"]
assert int(stated.group(1)) > 0
named = re.search(r"of which \*\*(\d+)\*\* are named", text)
assert named is not None and int(named.group(1)) == len(block["nearest"])
def test_a_generation_that_withheld_nothing_says_so_instead_of_printing_zero(
def test_a_generation_that_withheld_nothing_states_a_zero_it_measured(
tmp_path: Path,
) -> None:
"""The arm the SHIPPED skill is on, and the one that was wrong.
"""The arm the SHIPPED skill is on.
`okf skill --example-question "Hva sier veiledningen om krav?"` delivers
all three concepts of the golden bundle, so `withheld` is empty and there
is no per-entry cost. The question is part of what the shipped file is
(`skills/okf-consume/references/README.md`), which is why the defect was
in the repository rather than only reachable in theory.
all three concepts of the golden bundle, so nothing is withheld. Under the
flat list that left no per-entry cost and the guard printed `0 concepts`;
under counts-plus-names the zero is a count the run actually made, so it
is stated rather than withheld -- and the retired figure must still be
absent.
"""
written = okf_skill.generate(
GOLDEN,
@ -411,7 +415,7 @@ def test_a_generation_that_withheld_nothing_says_so_instead_of_printing_zero(
)
text = written.read_text(encoding="utf-8")
payload = json.loads((tmp_path / "out" / "references" / "example-payload.json").read_text())
assert payload["withheld"] == [], "the premise of this arm no longer holds"
assert payload["withheld"]["total"] == 0, "the premise of this arm no longer holds"
assert "**0 concepts**" not in text
assert "breaking point could not be measured" in text
assert "At roughly" not in text
assert "0 withheld of 3 concepts" in text

10
tests/test_shared_id_prefix.py
View file

@ -83,9 +83,15 @@ def _leaf(concept_id: str) -> str:
def _read(bundle: Path) -> tuple[list[str], dict[str, str]]:
payload = okf_consume.build_payload(bundle, question=QUESTION, k=10)
# `withheld_full`: this test names the rule for EVERY concept, so it asks
# for the whole set rather than the nearest N a reader is handed.
payload = okf_consume.build_payload(bundle, question=QUESTION, k=10, withheld_full=True)
delivered = [_leaf(e["concept_id"]) for e in payload["excerpts"] if isinstance(e, dict)]
withheld = {_leaf(w["concept_id"]): w["rule"] for w in payload["withheld"]}
block = payload["withheld"]
assert isinstance(block, dict) and block["complete"] is True
entries = block["nearest"]
assert isinstance(entries, list)
withheld = {_leaf(w["concept_id"]): w["rule"] for w in entries}
return delivered, withheld

9
tests/test_tie_shared_rank.py
View file

@ -183,7 +183,8 @@ def test_the_cli_exposes_the_flag_and_defaults_it_on(tmp_path: Path) -> None:
assert parsed.tie_shared_rank is True
parsed_off = okf_consume.parse_args([str(root), "--question", QUESTION, "--no-tie-shared-rank"])
assert parsed_off.tie_shared_rank is False
# `--withheld-titles` did NOT move with it, asserted here so the two are
# one measurement rather than two files' worth of trust: it is off for a
# reason of BYTES, which nothing this round touched.
assert parsed.withheld_titles is False
# The withheld cap did NOT move with it, asserted here so the two are one
# measurement rather than two files' worth of trust: it is a number chosen
# for reasons of BYTES, which nothing this round touched.
assert parsed.withheld_nearest == okf_consume.WITHHELD_NEAREST_DEFAULT
assert parsed.withheld_full is False

12
tools/okf_consume_measure.py
View file

@ -244,14 +244,20 @@ def main(argv: list[str] | None = None) -> int:
for entry in spec.get("negatives", []):
reach = token_reach(args.bundle, entry["question"])
started = time.perf_counter()
payload = okf_consume.build_payload(args.bundle, question=entry["question"], k=args.k)
# `withheld_full`: the row below reports the SET of rules a question
# fell under, so it needs every drop and not the nearest N.
payload = okf_consume.build_payload(
args.bundle, question=entry["question"], k=args.k, withheld_full=True
)
elapsed = time.perf_counter() - started
counts, budget, withheld = (
counts, budget, block = (
payload["denominators"],
payload["budget"],
payload["withheld"],
)
assert isinstance(counts, dict) and isinstance(budget, dict) and isinstance(withheld, list)
assert isinstance(counts, dict) and isinstance(budget, dict) and isinstance(block, dict)
withheld = block["nearest"]
assert isinstance(withheld, list)
negatives.append(
{
"question": entry["question"],

24
tools/okf_retrieval_gate.py
View file

@ -777,7 +777,22 @@ class Unit:
def _withheld_rules(payload: Mapping[str, object]) -> dict[str, str]:
entries = payload.get("withheld")
"""The rule for EVERY withheld concept, which is why the runs below ask
for the whole list.
Since `okf-consumption/2` a payload names only the nearest N drops by
default -- the right shape for a reader and the wrong one for an
instrument that classifies every miss by the rule it fell under. The block
states `complete`, so the demand is checked rather than assumed: a
truncated block here would silently classify most misses as unfound.
"""
block = payload.get("withheld")
assert isinstance(block, Mapping)
assert block.get("complete") is True, (
"the payload names a sample of the withheld set, so a rule map built "
"from it would be missing the concepts it was asked about"
)
entries = block.get("nearest")
assert isinstance(entries, list)
return {
str(entry["concept_id"]): str(entry["rule"]) for entry in entries if isinstance(entry, dict)
@ -805,7 +820,11 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
"""
index = bundle_index(bundle)
default = consume.build_payload(
bundle, question=question.question, k=question.k, limit=question.limit
bundle,
question=question.question,
k=question.k,
limit=question.limit,
withheld_full=True,
)
truth_run = consume.build_payload(
bundle,
@ -813,6 +832,7 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
k=question.k,
limit=question.limit,
source_quota=None,
withheld_full=True,
)
delivered = _delivered(default)
withheld = _withheld_rules(default)