open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): measure the budget lock, add one flag-gated top-rank reservation

Browse source 
The prior measurement (docs/2026-09-08-blindsone-below-k-k2.md SS 3) found that
the budget, not the ranking, is the second lock on a mandate-shaped cost
question -- and that the same mechanism was a REGRESSION on the question that
works: raising `--k` to 16 evicted the gold concept, because the exact knapsack
maximises a SUM of fused scores and has no opinion about rank, so twenty small
excerpts out-value one that costs 56.5 % of the budget.

Measured here on the same 629-concept bundle, with the three known-positive
figures from `4c699fd` reproduced first:

- Corpus distribution, denominator 629: median excerpt 857 B, max 223 391 B,
  3 concepts over the limit alone.
- Candidate rule (b), a corpus-derived budget, is FALSIFIED by two numbers: two
  defensible derivations are 49x apart on the same corpus, the small one turns
  the gold concept into `over_budget_alone` (13 refusals against 2), the large
  one changes nothing at the default k. A budget is the consumer's constraint,
  not a property of the corpus; `--limit` already belongs to the caller.
- Built instead, behind `--reserve-top-rank` (default OFF): the top-ranked
  candidate gets its bytes before the pack runs, AFTER the `over_budget_alone`
  pre-exclusion and never before, and the payload declares `budget.reserved`.
- It fixes the eviction: k=16 and k=24 deliver the gold concept at rank 1,
  costing one and two excerpts, and 20.4 % / 27.3 % FEWER o200k tokens.
- It changes the delivered list in 2 of 24 measured combinations -- both of them
  that eviction. In the other 22 the list, its order and `spent` are identical.
- It does NOT close the mandate-shaped blind spot: that concept ranks 10, not 1.
  The one delivering command is `--cost-vocabulary --k 12 --limit 160000`
  (62 149 tokens against 58 401), and that is a consumer's decision.

11 new tests (RED first), 7 mutations 7 red with an unmutated negative control
green before and after; two of the seven survived the first test set and the
tests were strengthened. Default payload byte-identical, both goldens unchanged.

Report: docs/2026-09-08-blindsone-laas2-budsjett-k2.md
Suite 1279 green, mypy --strict clean over 28 files, ruff clean.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 05:18:30 +02:00
commit 6776c37d23
5 changed files with 746 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

305
tests/test_okf_consume.py
View file

@ -486,7 +486,7 @@ def _cut_fixture(
concepts = _fixture_concepts()
scores = okf_consume.document_scores(FIXTURE, question)
ranked = okf_consume.concept_scores(concepts, question, scores)
delivered, withheld = okf_consume.cut(
delivered, withheld, _ = okf_consume.cut(
ranked, k=k, limit=okf_consume.DEFAULT_LIMIT if limit is None else limit
)
return list(delivered), list(withheld), len(ranked)
@ -1045,6 +1045,7 @@ def test_no_corpus_document_name_reaches_any_file_this_work_tracks() -> None:
PROJECT_ROOT / "tests" / "test_okf_consume.py",
PROJECT_ROOT / "docs" / "2026-09-07-okf-konsumskill-maaling.md",
PROJECT_ROOT / "docs" / "2026-09-08-blindsone-below-k-k2.md",
PROJECT_ROOT / "docs" / "2026-09-08-blindsone-laas2-budsjett-k2.md",
PROJECT_ROOT / "README.md",
PROJECT_ROOT / "CLAUDE.md",
]
@ -1216,3 +1217,305 @@ def test_the_cli_exposes_the_flag_and_omitting_it_reproduces_the_default_bytes()
assert flagged.stdout == okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, cost_vocabulary=True)
)
# --- Step 13: the budget reserved for the top-ranked candidate, behind a flag -
#: The shape the corpus measurement found (`docs/2026-09-08-blindsone-below-k-k2.md`
#: SS 3): one top-ranked concept costing more than half the budget, and enough
#: small ones that their SUM of fused scores out-values it. Synthetic and
#: general -- no corpus path, no corpus byte constant, no corpus document name.
EVICTION_QUESTION = "Hva staar i den store tabellen om kontroll?"
EVICTION_LIMIT = 12_000
EVICTION_TOP = "stor/tabell"
_EVICTION_FRONTMATTER = (
"---\ntype: reference\ntitle: {title}\nsource_file: {title}.md\n"
"source_sha256: {digest}\ningested_at: 2026-09-01T00:00:00Z\n"
"adjudication: proposed\nbundle_id: eviction-fixture\n"
"verified: [{{ by: process:okf-check, at: 2026-09-01T00:00:00Z }}]\n---\n\n"
)
def _eviction_bundle(root: Path, *, smalls: int = 12, fat_small_lines: int = 9) -> Path:
# `fat_small_lines` makes ONE lower-ranked concept the heaviest in the
# bundle, so "the top-ranked candidate" and "the largest excerpt" can be
# told apart by a test rather than coinciding by accident.
(root / "stor").mkdir(parents=True)
(root / "smaa").mkdir(parents=True)
(root / "index.md").write_text(
"---\nokf_version: 0.2\nbundle_id: eviction-fixture\n---\n\n"
"- [stor (index)](stor/index.md)\n- [smaa (index)](smaa/index.md)\n",
encoding="utf-8",
)
(root / "stor" / "index.md").write_text(
"- [Stor tabell om kontroll](tabell.md) — adjudication: proposed\n", encoding="utf-8"
)
(root / "stor" / "tabell.md").write_text(
_EVICTION_FRONTMATTER.format(title="Stor tabell om kontroll", digest="1" * 64)
+ "## Stor tabell om kontroll\n\n"
+ "Kontrollen av den store tabellen foelger tabellen rad for rad.\n" * 105,
encoding="utf-8",
)
entries = []
for number in range(1, smalls + 1):
name = f"notat-{number:02d}"
entries.append(f"- [Notat om kontroll {number:02d}]({name}.md) — adjudication: proposed\n")
lines = fat_small_lines if number == smalls else 9
(root / "smaa" / f"{name}.md").write_text(
_EVICTION_FRONTMATTER.format(title=f"Notat om kontroll {number:02d}", digest="2" * 64)
+ f"## Notat om kontroll {number:02d}\n\n"
+ "Notatet gjelder kontroll av ett enkelt punkt.\n" * lines,
encoding="utf-8",
)
(root / "smaa" / "index.md").write_text("".join(entries), encoding="utf-8")
return root
def _eviction_payload(
root: Path, *, k: int = 16, limit: int = EVICTION_LIMIT, reserve_top_rank: bool = False
) -> dict[str, object]:
return okf_consume.build_payload(
root,
question=EVICTION_QUESTION,
k=k,
limit=limit,
reserve_top_rank=reserve_top_rank,
)
def _top_candidate(root: Path) -> str:
# Computed the long way -- through the ranker, not read off the payload --
# so "the top-ranked candidate" in the assertions below is not whatever the
# cut happened to deliver first.
profile = okf_consume.DEFAULT_PROFILE
concepts = [
okf_consume.read_concept(
root / f"{concept_id}.md", bundle_root=root, root_bundle_id="eviction-fixture"
)
for concept_id in okf_consume.enumerate_concepts(root, profile=profile)
]
ranked = okf_consume.concept_scores(
concepts, EVICTION_QUESTION, okf_consume.document_scores(root, EVICTION_QUESTION)
)
return next(concept.concept_id for concept, _, lexical in ranked if lexical > 0)
def test_the_knapsack_evicts_the_top_ranked_candidate_that_costs_half_the_budget(
tmp_path: Path,
) -> None:
# The known-positive for everything below: without it, a delivered top rank
# with the flag on would prove nothing, because nothing would have been
# shown to remove it.
root = _eviction_bundle(tmp_path / "bundle")
assert _top_candidate(root) == EVICTION_TOP
# The DEFAULT path, called without the new argument, so this control runs
# and can be believed before the rule exists.
payload = okf_consume.build_payload(
root, question=EVICTION_QUESTION, k=16, limit=EVICTION_LIMIT
)
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
weights = {
excerpt["concept_id"]: okf_consume.excerpt_weight(excerpt)
for excerpt in excerpts
if isinstance(excerpt, dict)
}
assert EVICTION_TOP not in weights
assert (
dict(
(entry["concept_id"], entry["rule"])
for entry in payload["withheld"] # type: ignore[union-attr]
)[EVICTION_TOP]
== "over_budget_after_knapsack"
)
# The shape itself, stated as numbers rather than assumed: the top candidate
# fits ALONE and still loses, which is what makes this a budget question.
top = _eviction_bundle_top_weight(root)
assert top <= EVICTION_LIMIT
assert top > EVICTION_LIMIT // 2
def _eviction_bundle_top_weight(root: Path) -> int:
concept = okf_consume.read_concept(
root / f"{EVICTION_TOP}.md", bundle_root=root, root_bundle_id="eviction-fixture"
)
excerpt = okf_consume.excerpt_for(concept)
assert excerpt is not None
return okf_consume.excerpt_weight(excerpt)
def test_reserving_the_top_rank_delivers_the_candidate_the_knapsack_evicted(
tmp_path: Path,
) -> None:
root = _eviction_bundle(tmp_path / "bundle")
payload = _eviction_payload(root, reserve_top_rank=True)
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
assert excerpts[0]["concept_id"] == EVICTION_TOP
assert excerpts[0]["rank"] == 1
assert EVICTION_TOP not in {
entry["concept_id"]
for entry in payload["withheld"] # type: ignore[union-attr]
}
def test_the_reservation_is_off_by_default_and_the_default_payload_is_byte_identical(
tmp_path: Path,
) -> None:
root = _eviction_bundle(tmp_path / "bundle")
default = okf_consume.serialise(
okf_consume.build_payload(root, question=EVICTION_QUESTION, k=16, limit=EVICTION_LIMIT)
)
explicit_off = okf_consume.serialise(_eviction_payload(root, reserve_top_rank=False))
assert default == explicit_off
assert '"reserved"' not in default
question = "Hvordan skal prisene fylles ut?"
assert okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question)
) == okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, reserve_top_rank=False)
)
def test_a_top_candidate_that_alone_exceeds_the_budget_is_still_refused_by_name(
tmp_path: Path,
) -> None:
# The control the reservation must not break: `over_budget_alone` is a
# PRE-exclusion, and a reservation that ran before it would deliver an
# excerpt the budget can never hold.
root = _eviction_bundle(tmp_path / "bundle")
limit = _eviction_bundle_top_weight(root) - 1
payload = _eviction_payload(root, limit=limit, reserve_top_rank=True)
rules = dict(
(entry["concept_id"], entry["rule"])
for entry in payload["withheld"] # type: ignore[union-attr]
)
assert rules[EVICTION_TOP] == "over_budget_alone"
spent = payload["budget"]["spent"] # type: ignore[index]
assert isinstance(spent, int)
assert spent <= limit
def test_the_reservation_never_spends_more_than_the_budget(tmp_path: Path) -> None:
root = _eviction_bundle(tmp_path / "bundle")
for limit in (EVICTION_LIMIT, EVICTION_LIMIT + 3_000, EVICTION_LIMIT * 2):
payload = _eviction_payload(root, limit=limit, reserve_top_rank=True)
spent = payload["budget"]["spent"] # type: ignore[index]
assert isinstance(spent, int)
assert spent <= limit
def test_the_reservation_displaces_lower_ranked_excerpts_under_the_rule_that_exists(
tmp_path: Path,
) -> None:
# The cost of the rule, asserted rather than described: reserving the top
# rank buys its bytes from the excerpts the knapsack preferred, and they
# leave under a rule already in the closed set.
root = _eviction_bundle(tmp_path / "bundle")
without = _eviction_payload(root)
with_reservation = _eviction_payload(root, reserve_top_rank=True)
assert len(with_reservation["excerpts"]) < len(without["excerpts"]) # type: ignore[arg-type]
displaced = {
entry["concept_id"]
for entry in with_reservation["withheld"] # type: ignore[union-attr]
if entry["rule"] == "over_budget_after_knapsack"
}
delivered_before = {
excerpt["concept_id"]
for excerpt in without["excerpts"] # type: ignore[union-attr]
}
assert displaced & delivered_before
assert {
entry["rule"]
for entry in with_reservation["withheld"] # type: ignore[union-attr]
} <= set(okf_consume.WITHHOLDING_RULES)
def test_the_payload_declares_which_concept_the_reservation_took_and_what_it_cost(
tmp_path: Path,
) -> None:
root = _eviction_bundle(tmp_path / "bundle")
budget = _eviction_payload(root, reserve_top_rank=True)["budget"]
assert isinstance(budget, dict)
assert budget["reserved"] == {
"concept_id": _top_candidate(root),
"bytes": _eviction_bundle_top_weight(root),
}
spent = budget["spent"]
assert isinstance(spent, int)
assert spent >= _eviction_bundle_top_weight(root)
def test_the_reservation_is_paid_once_and_does_not_bid_for_its_own_bytes(
tmp_path: Path,
) -> None:
# A reserved excerpt left in the pack's pool competes for the budget it has
# already been given, and wins it back from the next candidate. Visible
# only where the reserved excerpt would out-value what the remaining room
# can hold: two candidates, and room enough for the reserved one twice.
root = _eviction_bundle(tmp_path / "bundle")
payload = _eviction_payload(root, k=2, limit=15_000, reserve_top_rank=True)
delivered = [
excerpt["concept_id"]
for excerpt in payload["excerpts"] # type: ignore[union-attr]
]
assert delivered == [EVICTION_TOP, "smaa/notat-01"]
def test_the_reservation_names_the_fused_top_and_not_the_heaviest_candidate(
tmp_path: Path,
) -> None:
# A bundle whose HEAVIEST excerpt is a LOWER-ranked one. A reservation
# reading weight where it should read rank reserves the wrong concept, and
# this is the only place the two come apart.
root = _eviction_bundle(tmp_path / "bundle", fat_small_lines=400)
heaviest = max(
okf_consume.enumerate_concepts(root),
key=lambda concept_id: okf_consume.excerpt_weight(
okf_consume.excerpt_for(
okf_consume.read_concept(
root / f"{concept_id}.md",
bundle_root=root,
root_bundle_id="eviction-fixture",
)
)
or {}
),
)
assert heaviest != EVICTION_TOP
budget = _eviction_payload(root, limit=EVICTION_LIMIT * 3, reserve_top_rank=True)["budget"]
assert isinstance(budget, dict)
assert budget["reserved"]["concept_id"] == EVICTION_TOP # type: ignore[index]
def test_a_payload_carrying_a_reservation_still_passes_the_checker(tmp_path: Path) -> None:
# SS 8 permits additional members; a declaration the checker refuses would
# buy honesty at the price of conformance.
root = _eviction_bundle(tmp_path / "bundle")
report = okf_contract_check.check(
TEMPLATE.read_text(encoding="utf-8"), _eviction_payload(root, reserve_top_rank=True)
)
assert report.findings == ()
def test_the_cli_exposes_the_reservation_and_omitting_it_reproduces_the_default_bytes(
tmp_path: Path,
) -> None:
root = _eviction_bundle(tmp_path / "bundle")
common = (
str(root),
"--question",
EVICTION_QUESTION,
"--k",
"16",
"--limit",
str(EVICTION_LIMIT),
)
plain = _run(*common)
assert plain.returncode == 0
reserved = _run(*common, "--reserve-top-rank")
assert reserved.returncode == 0
assert plain.stdout == okf_consume.serialise(_eviction_payload(root))
assert reserved.stdout == okf_consume.serialise(_eviction_payload(root, reserve_top_rank=True))
assert plain.stdout != reserved.stdout