open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): measure the budget lock, add one flag-gated top-rank reservation

Browse source 
The prior measurement (docs/2026-09-08-blindsone-below-k-k2.md SS 3) found that
the budget, not the ranking, is the second lock on a mandate-shaped cost
question -- and that the same mechanism was a REGRESSION on the question that
works: raising `--k` to 16 evicted the gold concept, because the exact knapsack
maximises a SUM of fused scores and has no opinion about rank, so twenty small
excerpts out-value one that costs 56.5 % of the budget.

Measured here on the same 629-concept bundle, with the three known-positive
figures from `4c699fd` reproduced first:

- Corpus distribution, denominator 629: median excerpt 857 B, max 223 391 B,
  3 concepts over the limit alone.
- Candidate rule (b), a corpus-derived budget, is FALSIFIED by two numbers: two
  defensible derivations are 49x apart on the same corpus, the small one turns
  the gold concept into `over_budget_alone` (13 refusals against 2), the large
  one changes nothing at the default k. A budget is the consumer's constraint,
  not a property of the corpus; `--limit` already belongs to the caller.
- Built instead, behind `--reserve-top-rank` (default OFF): the top-ranked
  candidate gets its bytes before the pack runs, AFTER the `over_budget_alone`
  pre-exclusion and never before, and the payload declares `budget.reserved`.
- It fixes the eviction: k=16 and k=24 deliver the gold concept at rank 1,
  costing one and two excerpts, and 20.4 % / 27.3 % FEWER o200k tokens.
- It changes the delivered list in 2 of 24 measured combinations -- both of them
  that eviction. In the other 22 the list, its order and `spent` are identical.
- It does NOT close the mandate-shaped blind spot: that concept ranks 10, not 1.
  The one delivering command is `--cost-vocabulary --k 12 --limit 160000`
  (62 149 tokens against 58 401), and that is a consumer's decision.

11 new tests (RED first), 7 mutations 7 red with an unmutated negative control
green before and after; two of the seven survived the first test set and the
tests were strengthened. Default payload byte-identical, both goldens unchanged.

Report: docs/2026-09-08-blindsone-laas2-budsjett-k2.md
Suite 1279 green, mypy --strict clean over 28 files, ruff clean.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 05:18:30 +02:00
commit 6776c37d23
5 changed files with 746 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

64
tools/okf_consume.py
View file

@ -846,9 +846,14 @@ def knapsack(items: Sequence[tuple[float, int]], *, capacity: int) -> tuple[int,
def cut(
ranked: Sequence[tuple[Concept, float, int]], *, k: int, limit: int
) -> tuple[tuple[dict[str, object], ...], tuple[tuple[str, str], ...]]:
"""The ranked concepts split into delivered excerpts and named drops.
ranked: Sequence[tuple[Concept, float, int]],
*,
k: int,
limit: int,
reserve_top_rank: bool = False,
) -> tuple[tuple[dict[str, object], ...], tuple[tuple[str, str], ...], tuple[str, int] | None]:
"""The ranked concepts split into delivered excerpts, named drops, and the
reservation that was made, if any.
**The partition is the invariant, not a consequence.** Every considered
concept lands in exactly one of the two, so `considered == withheld +
@ -862,6 +867,17 @@ def cut(
bundle still returns eight excerpts -- a confident guess wearing a
denominator -- and hit@k over such a ranker measures the corpus's size
rather than the ranker.
**`reserve_top_rank` (default off) buys the highest-ranked candidate its
bytes before the pack runs.** The DP maximises a SUM of fused scores, so a
single candidate costing a large share of the budget loses to enough small
ones no matter how far ahead it ranks -- measured, a top-ranked excerpt
worth 56.5 % of the budget is evicted as soon as the shortlist holds enough
alternatives, which makes `k` a dial that can remove the one concept a
question was asked about. The reservation makes rank one a floor rather
than a bid, and the pack fills what is left. It runs AFTER the
`over_budget_alone` pre-exclusion, never before: a reservation for an
excerpt the budget can never hold would deliver bytes the gate refuses.
"""
withheld: list[tuple[str, str]] = []
candidates: list[tuple[Concept, float, dict[str, object], int]] = []
@ -894,7 +910,16 @@ def cut(
# list would silently turn "position in the payload" into a different number
# from "position in the ranking".
pool = sorted(shortlist, key=lambda entry: entry[0].concept_id)
capacity = limit // WEIGHT_BUCKET
reserved: tuple[str, int] | None = None
room = limit
if reserve_top_rank and shortlist:
# `shortlist` is in fused-rank order, so its first entry IS the
# top-ranked candidate -- not the heaviest, and not the first by id.
top = shortlist[0]
reserved = (top[0].concept_id, top[3])
room = limit - top[3]
pool = [entry for entry in pool if entry[0] is not top[0]]
capacity = room // WEIGHT_BUCKET
packed = {
id(pool[index][0])
for index in knapsack(
@ -902,6 +927,8 @@ def cut(
capacity=capacity,
)
}
if reserved is not None:
packed.add(id(shortlist[0][0]))
delivered: list[dict[str, object]] = []
for concept, _, excerpt, _ in shortlist:
if id(concept) in packed:
@ -909,7 +936,7 @@ def cut(
else:
withheld.append((concept.concept_id, "over_budget_after_knapsack"))
withheld.sort()
return tuple(delivered), tuple(withheld)
return tuple(delivered), tuple(withheld), reserved
# --- The payload (SS 8) -------------------------------------------------------
@ -931,12 +958,13 @@ def build_payload(
limit: int = DEFAULT_LIMIT,
profile: BundleProfile = DEFAULT_PROFILE,
cost_vocabulary: bool = False,
reserve_top_rank: bool = False,
) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload.
Pure with respect to the clock and the network: the same
`(bundle_root, question, k, limit, cost_vocabulary)` at the same bytes
returns the same object, every time.
`(bundle_root, question, k, limit, cost_vocabulary, reserve_top_rank)` at
the same bytes returns the same object, every time.
"""
case, expected, measured = known_positive()
if expected != measured:
@ -981,7 +1009,7 @@ def build_payload(
cost_vocabulary=cost_vocabulary,
)
matched = sum(1 for _, _, lexical in ranked if lexical > 0)
delivered, withheld = cut(ranked, k=k, limit=limit)
delivered, withheld, reserved = cut(ranked, k=k, limit=limit, reserve_top_rank=reserve_top_rank)
spent = sum(excerpt_weight(excerpt) for excerpt in delivered)
if matched and not delivered:
# SS 7.3: a finding requiring a decision, never something to retry
@ -1022,6 +1050,15 @@ def build_payload(
"raw_bytes": len(_KNOWN_POSITIVE_PATH.read_bytes()),
"encoding_delta": KNOWN_POSITIVE_ENCODING_DELTA,
},
# Present only when a reservation was made, because a cut whose
# strategy changed without saying so is the silent cut SS 5.3
# forbids -- and absent otherwise, so the default payload keeps
# every byte it had.
**(
{"reserved": {"concept_id": reserved[0], "bytes": reserved[1]}}
if reserved is not None
else {}
),
},
"denominators": {
"considered": len(concepts),
@ -1072,6 +1109,16 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"naming no term in that vocabulary is unaffected either way"
),
)
parser.add_argument(
"--reserve-top-rank",
action="store_true",
help=(
"give the highest-ranked candidate its bytes before the budget is "
"packed, so a large top-ranked excerpt is not out-summed by small "
"ones. OFF by default; a candidate that alone exceeds the budget is "
"still refused"
),
)
parser.add_argument("--out", type=Path, default=None, help="write here instead of stdout")
parser.add_argument(
"--ref",
@ -1104,6 +1151,7 @@ def main(argv: list[str] | None = None) -> int:
k=args.k,
limit=args.limit,
cost_vocabulary=args.cost_vocabulary,
reserve_top_rank=args.reserve_top_rank,
)
except ConsumeError as error:
print(f"okf_consume: FAILED - {error}", file=sys.stderr)