feat(consume): the withheld set is counts plus names, not one entry per concept

Measured 2026-09-20 on a 2313-concept bundle of one project's own
documentation: `withheld` held 2 305 entries = 186 440 B of compact JSON =
**65.5 % of the 284 850-byte payload**, and not one of those bytes counted
against the budget the same payload reports (`spent` was 45 192). A reader was
handed 239 658 bytes the budget line did not know about, to learn 2 305 concept
ids with nothing beside them -- the title being exactly what `--withheld-titles`
existed to buy, and which was off because buying it for 2 305 entries cost
another 37.9 %.

`withheld` is now a mapping: `total` (equal to `denominators.withheld`, so
SS 5.2's identity is unmoved and closes on the NUMBERS), `by_rule` (the same
total decomposed over the closed rule set, so "what kind of drop" is answerable
without the list), `nearest` (the best-ranked drops BY NAME, with title and
source document, so a reader who sees a near miss can ask for it) and
`complete`. The near misses are read off the ranking, not off `cut`'s output:
`cut` sorts by id so the partition is comparable, and that order says nothing
about which concept a reader might want next.

Same question, same bundle, after: **52 421 bytes, 18.4 % of the old file**.
The whole list stays reachable behind `--withheld-full`, and the two
instruments that classify EVERY miss by its rule -- the retrieval gate and
`okf_consume_measure` -- now ask for it explicitly and assert `complete`
rather than assuming it. `--withheld-nearest N` sets the cap (default 20,
which is `k` plus the next twelve). `--withheld-titles` is retired: a flag
whose only remaining effect would be to STRIP the title from a list the caller
asked for in full names no decision worth two shapes for one list.

`CONTRACT_REVISION` moves to `okf-consumption/2`, because a consumer indexing
the old key as a list would otherwise break silently. Three checker rules move
with it, and one of them is the interesting case: `parent_unfollowable` used
`excerpts` + `withheld` as the bundle's own denominator, which a truncated
block is not -- so that clause now runs only where the payload SAYS it is
complete, stated in SS 8.6 rather than left as a silence, with the other two
clauses (shape, self-reference) running either way. `Report` carries both
denominators, because a report claiming it examined 2 305 entries it never saw
is the same defect one level up.

The generated skill's "breaking point" section goes with it: it extrapolated a
concept count from the cost of ONE withheld entry, and there is no such slope
any more. It now states what this bundle's bookkeeping cost and that the block
is bounded by the cap rather than by the bundle -- an extrapolation from a
slope the code no longer has would be a measurement of the previous revision.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

tools/okf_consume_measure.py

@@ -244,14 +244,20 @@ def main(argv: list[str] | None = None) -> int:
     for entry in spec.get("negatives", []):
         reach = token_reach(args.bundle, entry["question"])
         started = time.perf_counter()
-        payload = okf_consume.build_payload(args.bundle, question=entry["question"], k=args.k)
+        # `withheld_full`: the row below reports the SET of rules a question
+        # fell under, so it needs every drop and not the nearest N.
+        payload = okf_consume.build_payload(
+            args.bundle, question=entry["question"], k=args.k, withheld_full=True
+        )
         elapsed = time.perf_counter() - started
-        counts, budget, withheld = (
+        counts, budget, block = (
             payload["denominators"],
             payload["budget"],
             payload["withheld"],
         )
-        assert isinstance(counts, dict) and isinstance(budget, dict) and isinstance(withheld, list)
+        assert isinstance(counts, dict) and isinstance(budget, dict) and isinstance(block, dict)
+        withheld = block["nearest"]
+        assert isinstance(withheld, list)
         negatives.append(
             {
                 "question": entry["question"],

tools/okf_retrieval_gate.py

@@ -777,7 +777,22 @@ class Unit:
 
 
 def _withheld_rules(payload: Mapping[str, object]) -> dict[str, str]:
-    entries = payload.get("withheld")
+    """The rule for EVERY withheld concept, which is why the runs below ask
+    for the whole list.
+
+    Since `okf-consumption/2` a payload names only the nearest N drops by
+    default -- the right shape for a reader and the wrong one for an
+    instrument that classifies every miss by the rule it fell under. The block
+    states `complete`, so the demand is checked rather than assumed: a
+    truncated block here would silently classify most misses as unfound.
+    """
+    block = payload.get("withheld")
+    assert isinstance(block, Mapping)
+    assert block.get("complete") is True, (
+        "the payload names a sample of the withheld set, so a rule map built "
+        "from it would be missing the concepts it was asked about"
+    )
+    entries = block.get("nearest")
     assert isinstance(entries, list)
     return {
         str(entry["concept_id"]): str(entry["rule"]) for entry in entries if isinstance(entry, dict)
@@ -805,7 +820,11 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
     """
     index = bundle_index(bundle)
     default = consume.build_payload(
-        bundle, question=question.question, k=question.k, limit=question.limit
+        bundle,
+        question=question.question,
+        k=question.k,
+        limit=question.limit,
+        withheld_full=True,
     )
     truth_run = consume.build_payload(
         bundle,
@@ -813,6 +832,7 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
         k=question.k,
         limit=question.limit,
         source_quota=None,
+        withheld_full=True,
     )
     delivered = _delivered(default)
     withheld = _withheld_rules(default)