open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): measure the budget lock, add one flag-gated top-rank reservation

Browse source 
The prior measurement (docs/2026-09-08-blindsone-below-k-k2.md SS 3) found that
the budget, not the ranking, is the second lock on a mandate-shaped cost
question -- and that the same mechanism was a REGRESSION on the question that
works: raising `--k` to 16 evicted the gold concept, because the exact knapsack
maximises a SUM of fused scores and has no opinion about rank, so twenty small
excerpts out-value one that costs 56.5 % of the budget.

Measured here on the same 629-concept bundle, with the three known-positive
figures from `4c699fd` reproduced first:

- Corpus distribution, denominator 629: median excerpt 857 B, max 223 391 B,
  3 concepts over the limit alone.
- Candidate rule (b), a corpus-derived budget, is FALSIFIED by two numbers: two
  defensible derivations are 49x apart on the same corpus, the small one turns
  the gold concept into `over_budget_alone` (13 refusals against 2), the large
  one changes nothing at the default k. A budget is the consumer's constraint,
  not a property of the corpus; `--limit` already belongs to the caller.
- Built instead, behind `--reserve-top-rank` (default OFF): the top-ranked
  candidate gets its bytes before the pack runs, AFTER the `over_budget_alone`
  pre-exclusion and never before, and the payload declares `budget.reserved`.
- It fixes the eviction: k=16 and k=24 deliver the gold concept at rank 1,
  costing one and two excerpts, and 20.4 % / 27.3 % FEWER o200k tokens.
- It changes the delivered list in 2 of 24 measured combinations -- both of them
  that eviction. In the other 22 the list, its order and `spent` are identical.
- It does NOT close the mandate-shaped blind spot: that concept ranks 10, not 1.
  The one delivering command is `--cost-vocabulary --k 12 --limit 160000`
  (62 149 tokens against 58 401), and that is a consumer's decision.

11 new tests (RED first), 7 mutations 7 red with an unmutated negative control
green before and after; two of the seven survived the first test set and the
tests were strengthened. Default payload byte-identical, both goldens unchanged.

Report: docs/2026-09-08-blindsone-laas2-budsjett-k2.md
Suite 1279 green, mypy --strict clean over 28 files, ruff clean.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 05:18:30 +02:00
commit 6776c37d23
5 changed files with 746 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

23
CLAUDE.md
View file

@ -268,15 +268,24 @@ and fixtures, never code.
measurement behind it, including the control that FAILED, is
`docs/2026-09-07-okf-konsumskill-maaling.md`. **The ranking is this
repository's own choice** — the contract binds a payload, not a retrieval
algorithm (§ 10) — and it has ONE optional widening, `--cost-vocabulary`,
**off by default**: a declared cost/price/quantity vocabulary family that
bridges a question and a document naming money with different words, gated on
the QUESTION carrying such a term, so a question without one is byte-identical
either way. It moves a measured case from candidate rank 249 to 10 and does
**not** deliver it: the budget is a second, independent lock, and closing that
one is a decision nobody has made. Measured, with the two rules falsified
algorithm (§ 10) — and it has TWO optional widenings, both **off by default**
and both keeping the default payload byte-identical. `--cost-vocabulary`: a
declared cost/price/quantity vocabulary family that bridges a question and a
document naming money with different words, gated on the QUESTION carrying
such a term, so a question without one is byte-identical either way. It moves
a measured case from candidate rank 249 to 10 and does **not** deliver it —
the budget is a second, independent lock. Measured, with two rules falsified
before building and the `k`-sweep that showed a higher `k` can EVICT a gold
concept, in `docs/2026-09-08-blindsone-below-k-k2.md`.
`--reserve-top-rank` is that second lock: the pack is an exact knapsack over a
SUM, so it has no opinion about rank and out-sums a top-ranked excerpt costing
a large share of the budget. The flag gives rank one its bytes first, AFTER
the `over_budget_alone` pre-exclusion and never before, and declares
`budget.reserved` in the payload. It fixes the eviction and does **not** close
the mandate-shaped blind spot (that concept ranks 10, not 1); the budget stays
the caller's decision, because deriving a limit from the corpus was measured
and falsified — two defensible derivations, 49x apart, one of them breaking
the known-positive. `docs/2026-09-08-blindsone-laas2-budsjett-k2.md`.
## Workflow

14
README.md
View file

@ -117,7 +117,8 @@ cuts them to a bounded set and emits one payload. It decides nothing about the
question; the skill that reads the payload does the judgement. It calls no
model, opens no socket, imports nothing outside the standard library and this
package, and takes no clock: the same bundle bytes and the same
`(question, k, limit, cost_vocabulary)` produce byte-identical output.
`(question, k, limit, cost_vocabulary, reserve_top_rank)` produce
byte-identical output.
`--cost-vocabulary` is off by default and widens one question class: it lets a
declared list of cost/price/quantity terms bridge a question and a document that
@ -125,6 +126,17 @@ name money with different words. The gate is the question — one naming no such
term gets byte-identical bytes either way — and what it does and does not close
is measured in `docs/2026-09-08-blindsone-below-k-k2.md`.
`--reserve-top-rank` is off by default and answers a different objection: the
budget is packed by an exact knapsack, which maximises a SUM of scores and
therefore has no opinion about rank, so a top-ranked excerpt costing a large
share of the budget is out-summed by many small ones. Measured, that made `--k`
a dial that could EVICT the concept a question was asked about. The flag gives
rank one its bytes before the pack runs — after the `over_budget_alone`
pre-exclusion, never before — and the payload then declares
`budget.reserved`. On a 629-concept corpus it changed the delivered set in 2 of
24 measured combinations, both of them that eviction:
`docs/2026-09-08-blindsone-laas2-budsjett-k2.md`.
It emits the § 8 shape — `contract`, `bundle` (`bundle_id` plus a
`sha256-tree:` content identity), `budget` (unit, instrument, limit, spent and a
validated known-positive), `denominators`, `excerpts` and `withheld` — and every

357
docs/2026-09-08-blindsone-laas2-budsjett-k2.md Normal file
View file

@ -0,0 +1,357 @@
# The second lock: budget reserved for the top-ranked candidate, measured on a 629-concept corpus
**Date:** 2026-09-08 · **Order:** `20260908T025839Z-9903024761-from-.claude` ·
**Instrument:** `tools/okf_consume.py` at `4c699fd` plus the one flag this
document reports · **Prior finding:** `docs/2026-09-08-blindsone-below-k-k2.md`
§ 3, § 6, § 7 and § 9.
The document this one continues measured a mandate-shaped cost question against
a 629-concept corpus and found two independent locks on the one concept in that
corpus carrying a price. Lock 1 is vocabulary, and `--cost-vocabulary` moves the
concept from candidate rank 249 of 269 to 10 of 278. Lock 2 is the budget: one
excerpt costs 67 838 B, **56.5 % of the 120 000 B budget**, and the knapsack
maximises a SUM, so it prefers many small excerpts. The same sweep found a
regression on the question that WORKS: raising `k` to 16 evicts the gold
concept, which makes `k` a dial that can remove the one document a question was
asked about.
This document measures lock 2, reports the one flag-gated rule built after the
measurement, and prices the two rules the measurement did not justify.
The corpus is external and private to the measurement; no document name, path or
body from it appears here. Documents are named by shape ("the priced table") and
the numbers are counts.
---
## 0. What IS measured, and what is NOT
**Measured.** The excerpt-weight distribution of the whole corpus with its
denominator; today's cut strategy read out of the code rather than summarised;
the one rule that was built, on three questions (mandate-shaped, specific,
control), both settings of `--cost-vocabulary`, and `k` in {8, 12, 16, 24} --
24 combinations, each with delivered count, delivered list, budget spent,
payload bytes and o200k tokens; the exact excerpts the reservation displaced;
and the two candidate rules the order named, priced against the same corpus.
**Known-positive, run first.** Three figures from `4c699fd`, re-measured on this
tree before any change:
| # | case | expected | measured |
|---|---|---|---|
| 1 | specific question, `k=8`, both flags off | 164 987 B / 40 425 o200k tok | **164 987 B / 40 425** |
| 2 | specific question, `k=16`, flags off | priced table `over_budget_after_knapsack` | **`over_budget_after_knapsack`** |
| 3 | mandate question, `--cost-vocabulary`, `k=12` | priced table `over_budget_after_knapsack` | **`over_budget_after_knapsack`** |
The first pair is the figure published 2026-09-07 by the consumer
(`portfolio-optimiser docs/2026-09-07-okf-prepass-i-debatten.md` § 1), byte for
byte and token for token; the tokenizer is `tiktoken` `o200k_base`. An
instrument that has not reproduced a known figure has not been shown to count
(consumption contract § 7.4).
**NOT measured.** That the rule helps any corpus other than this one: one
corpus, three questions and 24 combinations is not a sample. Not measured
either: whether a model ANSWERS better with the top-ranked excerpt present --
that needs a live model and is the consumer's measurement, in their S7 form, not
this one. And not measured: that `DEFAULT_LIMIT = 120 000` is the right budget.
This document recommends; the default is the operator's decision and is
unchanged here.
**Nothing is decided about the default.** The flag ships OFF. With it off every
payload this repository emits is byte-identical to `4c699fd`, the suite's
byte-identity test covers exactly that, and both golden fixtures are unchanged.
---
## 1. Setup
The bundle is the 629-concept build of the corpus produced by `okf build` on
`5a0c879`, identity
`sha256-tree:f14872a01104e47474093611b1960c6c541e4701dc40147a00c8e1b337c8a92a`,
the same tree `4c699fd` measured. It is still a HEAD build: `4c699fd` touched
`tools/`, `tests/`, `docs/`, `README.md` and `CLAUDE.md` and no file under
`src/`, so no code on the build path moved between the two commits. Every
command in this document is offline: no model call, no socket, no clock.
---
## 2. What decides today, verbatim
From `tools/okf_consume.py`, quoted rather than summarised:
- `DEFAULT_LIMIT = 120_000` — "Chosen, not derived, and the reason is a
measurement rather than a taste: at 60 000 the largest realistic gold concept
(101 313 B encoded) falls to the `over_budget_alone` pre-exclusion, so a
CORRECT implementation would fail its own acceptance criteria ... **A starting
point to be moved by measurement.**"
- `knapsack` — "**The exact 0/1 knapsack: indices of the highest-value subset
that fits.** Exact rather than greedy-by-density, which has an unbounded
approximation factor."
- `cut` — "Exclusions run before the pack, each naming its rule, because 'it did
not fit' and 'it could never be delivered' are different facts about the cut."
- `WEIGHT_BUCKET = 500` — "bucketing UP the item and DOWN the capacity keeps the
error one-sided, so the pack may under-deliver by a bucket and can never
over-spend."
- The two budget rules in the closed set: `over_budget_alone` (a
pre-exclusion, before the DP) and `over_budget_after_knapsack` (a drop by the
pack itself).
And the contract, `docs/consumption-contract.md` § 7.6: a skill "SHOULD state
what a typical answer costs against the whole-bundle denominator, and **the
corpus size at which its strategy stops fitting its budget.** A strategy with no
stated breaking point cannot be observed to have passed it."
**In one line: the pack maximises a sum of fused scores under a byte budget, and
a sum has no opinion about rank.** A candidate ranked first is worth one
summand. Twenty candidates ranked 2 to 21 are worth twenty. Nothing in the DP
prefers the first, and nothing in it should -- the defect is that the cut asks
the DP a question whose answer it then reads as an answer about relevance.
### The corpus's own distribution, since the candidates argue about it
Over all 629 concepts (every one of them yields an excerpt; the denominator is
629, not a subset):
| statistic | bytes |
|---|---|
| min | 369 |
| median | 857 |
| mean | 3 131 |
| p90 | 3 440 |
| p99 | 53 970 |
| max | 223 391 |
| exceed `DEFAULT_LIMIT` alone | **3 of 629** |
The distribution is not merely skewed, it is bimodal in practice: a median of
857 B and a maximum of 223 391 B is a factor of 261.
---
## 3. The three candidate rules, priced
**(b) a profile-sized budget, derived from the corpus — FALSIFIED, and by two
numbers rather than one.** Two derivations are equally defensible from the table
above, and on the SAME corpus they are **49 times apart**:
| derivation | limit (B) | specific question, `k=8` |
|---|---|---|
| 8 x median excerpt | 6 856 | delivered 6, **gold concept `over_budget_alone`**, 13 concepts refused outright (2 today) |
| largest excerpt x 1.5 | 335 086 | delivered 8, spent 72 910 — **identical delivered set to today** |
The small derivation breaks the known-positive: the question that works stops
working, because the concept that answers it can no longer fit alone. The large
one changes nothing at the default `k` while tripling a ceiling nobody was
touching. A rule whose answer moves by a factor of 49 with the choice of
statistic is not a rule, it is a parameter wearing a derivation.
There is a second, larger reason, and it is about ownership rather than
arithmetic: **a budget is the CONSUMER's constraint, not a property of the
corpus.** It exists because a context window is finite. A library that sizes the
budget to the data hands a caller a payload their model may not hold, and does
it silently, in the name of fitting a document the caller never asked to be
sized around. `--limit` is already the caller's, and it is the right place for
that decision.
**(c) no rule — FALSIFIED for the specific question, and it is the reason this
order exists.** Doing nothing leaves the § 3 regression in place: at `k=16` and
`k=24` the gold concept of the question that WORKS is evicted. "Keep `k=8`" is
a mitigation that depends on nobody raising `k`, and `--k` is a public flag.
**(a) the rule that was built: the top-ranked candidate is reserved its bytes
before the pack runs, behind `--reserve-top-rank`, default off.** The
measurement in § 2 says the failure is that the DP has no opinion about rank.
So: give rank one a floor rather than a bid, and let the pack fill what is left.
Three properties, each with a test that goes red without it:
- **The reservation runs AFTER the `over_budget_alone` pre-exclusion, never
before.** A candidate that alone exceeds the budget is still refused by name,
and the reservation then applies to the highest-ranked candidate that could be
delivered at all. Reserving for an excerpt the gate can never admit would
deliver bytes the gate refuses.
- **It reserves the fused TOP, not the heaviest and not the first by id.** The
test uses a fixture whose heaviest excerpt is deliberately a lower-ranked one,
because that is the only place the two come apart.
- **The reservation is paid once.** A reserved excerpt left in the pack's pool
competes for the budget it has already been given and wins it back from the
next candidate; the pool excludes it and the capacity is reduced by exactly
its weight, so the one-sided bucket arithmetic still cannot over-spend.
**The payload declares it.** When a reservation is made, `budget` carries
`reserved: {concept_id, bytes}` -- a cut whose strategy changed without saying
so is the silent cut § 5.3 exists to forbid. The member is absent when the flag
is off, which is what keeps the default payload byte-identical, and § 8 permits
additional members, so a payload carrying it still passes the contract checker
with zero findings.
Development order: eleven tests, the failing ones written first. Seven mutations
of the shipped rule were run against them; **all seven are red**, with an
unmutated negative control green before and after the run (reserve the
last-ranked; reserve the heaviest; capacity not reduced; the reserved item left
in the pool; default flipped on; the pre-exclusion bypassed; the declaration
dropped). Two of the seven survived the first version of the tests -- the pool
mutation and the pre-exclusion bypass -- and the tests were strengthened until
they did not.
---
## 4. The rule, measured on three questions
Same bundle, same budget, `--cost-vocabulary` off and on, `k` swept. Only the
rows where the delivered SET moves are given in full; the other 20 are
summarised below them, which is the honest way round because the summary is the
larger claim.
| question | vocab | reserve | k | delivered | payload B | o200k tok | priced table |
|---|---|---|---|---|---|---|---|
| specific | off | off | 16 | 15 | 194 946 | 65 237 | `over_budget_after_knapsack` |
| specific | off | **on** | 16 | **14** | 206 102 | **51 951** | **delivered, rank 1** |
| specific | off | off | 24 | 20 | 197 287 | 66 799 | `over_budget_after_knapsack` |
| specific | off | **on** | 24 | **18** | 191 409 | **48 545** | **delivered, rank 1** |
**Of the 24 measured combinations (3 questions x 2 vocabulary settings x 4 `k`
values), the reservation changes the delivered list in exactly 2 -- and both of
them are the § 3 eviction.** In the other 22 the delivered list is identical,
in the same order, with `spent` equal to the byte; the payload differs only by
the declaration the flag adds (+129 B / +46 o200k tok on the mandate question,
+151 B / +51 on the control). The reason is not a coincidence: where the pack
already delivered the top-ranked candidate, reserving it changes nothing to
compute.
**The control behaves exactly as the summary predicts.** A question naming no
cost term is unaffected by `--cost-vocabulary` at every `k` (byte-identical, the
finding `4c699fd` reported) and its delivered set is unaffected by the
reservation at every `k` -- its top-ranked candidate, 23 900 B, was already
being delivered. Unlike the vocabulary flag, this one is not gated on the
question and does not need to be: it is gated on whether the pack was about to
drop rank one.
---
## 5. What the rule costs, stated as a working question
The reservation is not free, and the cost is visible in the two rows that moved:
| | `k=16` | `k=24` |
|---|---|---|
| delivered | 15 -> 14 | 20 -> 18 |
| reserved for rank 1 | 67 838 B | 67 838 B |
| bought from | 2 excerpts, 56 913 B | 3 excerpts, 74 066 B |
| payload bytes | 194 946 -> 206 102 (+5.7 %) | 197 287 -> 191 409 (-3.0 %) |
| o200k tokens | 65 237 -> **51 951 (-20.4 %)** | 66 799 -> **48 545 (-27.3 %)** |
Two things in that table deserve to be read rather than skimmed.
**The excerpts it displaced left under a rule that already existed.** They are
`over_budget_after_knapsack`, the same rule that was removing the gold concept a
moment earlier. The closed rule set did not grow, and the payload still
partitions: every considered concept is delivered or withheld under a named
rule.
**Bytes and tokens disagree about the direction, and the gate counts bytes.**
The reserved excerpt is a spreadsheet render: 206 102 B costs 51 951 o200k
tokens (4.0 B/token) where the 194 946 B it replaced cost 65 237 (3.0 B/token).
So on this corpus the reservation delivered the gold concept and made the
payload cheaper for the model while making it larger for the gate. That is a
property of this document's shape, not a general one, and it is exactly why the
budget unit is NAMED in the payload (§ 7.1) rather than assumed.
---
## 6. Does this close the blind spot? No -- and the honest line is one command
**For the mandate-shaped question the reservation changes nothing.** The priced
table is candidate rank 10 with `--cost-vocabulary` on, not rank 1, so it is
never the reserved candidate; the delivered set and `spent` are identical at
every `k` measured, with both flags on.
**Both locks open at once only with a raised budget, and that is a caller's
decision, not a default.** Measured:
```
python3 tools/okf_consume.py <bundle> \
--question "<the mandate-shaped question>" \
--cost-vocabulary --k 12 --limit 160000
```
delivers the priced table at rank 10, 12 excerpts, 150 249 B spent, **62 149
o200k tokens** -- against 58 401 for the same run at the default limit that
withholds it. Adding `--reserve-top-rank` changes neither the set nor the byte
count, for the reason above. So the mandate question's answer costs +3 748
tokens (+6.4 %) and one flag plus one explicit limit; nothing in this repository
decides that it should be paid.
**And "delivered" is still not "answered".** Whether a model does better with
those bytes present is unmeasured here and is the consumer's measurement, in
their S7 form, with a live model. This document moves a document from
unaffordable to delivered under a stated command. That is not the same as an
answer, and the distinction is the whole reason both are written down.
---
## 7. Honesty limits
- **One corpus, three questions, 24 combinations.** Generality is NOT
demonstrated. The reservation's effect depends on there being a
large-and-top-ranked excerpt at all; a corpus of uniform small concepts would
never notice the flag.
- **The rule was written with the failing case visible.** The § 3 eviction is
the case that motivated it, and the fixture that tests it is built to that
shape. The fixture is synthetic and carries no corpus constant, but it is not
a blind test.
- **Two of the seven mutations survived the first test set.** They were caught
by re-running, not by design, and the two tests that now kill them were
written afterwards. A reader should weight "seven of seven red" accordingly.
- **The token direction is a property of one document.** The −20 % and −27 %
above come from replacing prose with a spreadsheet render. A corpus whose
large concepts are prose would show the opposite sign, and the byte figure --
the one the gate enforces -- is up in one of the two rows.
- **`--reserve-top-rank` has no consumer.** Nobody asked for it. It exists so
the § 3 regression could be measured against real code rather than argued
about, and so a decision about the default can be made on numbers.
---
## 8. Recommendation
1. **Ship `--reserve-top-rank` OFF by default, and keep `DEFAULT_K = 8`.** At
`k=8` neither measured question needs the flag; the regression it fixes only
appears when a caller raises `k`. What the flag buys is that raising `k` is
no longer able to remove the top-ranked concept -- which is what a caller
raising `k` was trying to avoid in the first place.
2. **Do not derive the budget from the corpus.** § 3 prices both derivations:
one breaks the known-positive, the other changes nothing at the default `k`,
and they are 49x apart. `--limit` already belongs to the caller.
3. **Answer the contract's § 7.6 question with the number that is actually
true for this corpus.** The strategy does not stop fitting its budget at a
corpus SIZE. It stops fitting when a single relevant concept costs a large
share of the budget -- here 56.5 % -- and the answer is either a raised
`--limit` or this flag, depending on whether the caller wants the concept or
the count.
4. **The blind spot for a mandate-shaped question stays open at the default,
and it is now fully priced.** Both locks are measured, both have a command
that removes them, and the remaining decision -- whether a mandate-shaped
question should cost 62 149 tokens instead of 58 401 -- is a consumer's, not
this library's.
---
## 9. Verification log
| # | Claim | Command → result |
|---|---|---|
| 1 | The instrument reproduces a published figure | specific question, `k=8`, flags off → **164 987 B / 40 425 o200k**, equal to the published pair |
| 2 | The § 3 eviction is still there before the change | specific question, `k=16`, flags off → **`over_budget_after_knapsack`** |
| 3 | The § 6 mandate row is still there | mandate, `--cost-vocabulary`, `k=12` → **`over_budget_after_knapsack`** |
| 4 | The corpus distribution | 629 excerpts: median **857 B**, max **223 391 B**, 3 over the limit alone |
| 5 | Candidate (b), small derivation | limit 6 856 → gold concept **`over_budget_alone`**, 13 refusals against 2 |
| 6 | Candidate (b), large derivation | limit 335 086 → delivered set **identical** to today at `k=8` |
| 7 | The rule fixes the regression | specific question, `k=16` and `k=24`, `--reserve-top-rank` → **delivered, rank 1** |
| 8 | The rule's cost | `k=16`: 15 → 14 excerpts, 2 displaced (56 913 B), tokens **−20.4 %** |
| 9 | It changes nothing elsewhere | **22 of 24** combinations: identical delivered list, identical `spent` |
| 10 | It does not deliver the mandate's priced table | every `k` in {8, 12, 16, 24}, both flags → withheld |
| 11 | Both locks, one command | `--cost-vocabulary --k 12 --limit 160000` → **rank 10, 62 149 o200k tok** |
| 12 | The default does not move | flag off ⇒ byte-identical payload; both goldens unchanged |
| 13 | Seven mutations, seven red | negative control **green** before and after the mutation run |
| 14 | The declaration stays conformant | payload with `budget.reserved` → contract checker **0 findings** |
| 15 | Suite, types, lint | `pytest -q` **1279 passed**; `mypy --strict src/ tools/` 28 files clean; `ruff check` + `ruff format --check` clean |

305
tests/test_okf_consume.py
View file

@ -486,7 +486,7 @@ def _cut_fixture(
concepts = _fixture_concepts()
scores = okf_consume.document_scores(FIXTURE, question)
ranked = okf_consume.concept_scores(concepts, question, scores)
delivered, withheld = okf_consume.cut(
delivered, withheld, _ = okf_consume.cut(
ranked, k=k, limit=okf_consume.DEFAULT_LIMIT if limit is None else limit
)
return list(delivered), list(withheld), len(ranked)
@ -1045,6 +1045,7 @@ def test_no_corpus_document_name_reaches_any_file_this_work_tracks() -> None:
PROJECT_ROOT / "tests" / "test_okf_consume.py",
PROJECT_ROOT / "docs" / "2026-09-07-okf-konsumskill-maaling.md",
PROJECT_ROOT / "docs" / "2026-09-08-blindsone-below-k-k2.md",
PROJECT_ROOT / "docs" / "2026-09-08-blindsone-laas2-budsjett-k2.md",
PROJECT_ROOT / "README.md",
PROJECT_ROOT / "CLAUDE.md",
]
@ -1216,3 +1217,305 @@ def test_the_cli_exposes_the_flag_and_omitting_it_reproduces_the_default_bytes()
assert flagged.stdout == okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, cost_vocabulary=True)
)
# --- Step 13: the budget reserved for the top-ranked candidate, behind a flag -
#: The shape the corpus measurement found (`docs/2026-09-08-blindsone-below-k-k2.md`
#: SS 3): one top-ranked concept costing more than half the budget, and enough
#: small ones that their SUM of fused scores out-values it. Synthetic and
#: general -- no corpus path, no corpus byte constant, no corpus document name.
EVICTION_QUESTION = "Hva staar i den store tabellen om kontroll?"
EVICTION_LIMIT = 12_000
EVICTION_TOP = "stor/tabell"
_EVICTION_FRONTMATTER = (
"---\ntype: reference\ntitle: {title}\nsource_file: {title}.md\n"
"source_sha256: {digest}\ningested_at: 2026-09-01T00:00:00Z\n"
"adjudication: proposed\nbundle_id: eviction-fixture\n"
"verified: [{{ by: process:okf-check, at: 2026-09-01T00:00:00Z }}]\n---\n\n"
)
def _eviction_bundle(root: Path, *, smalls: int = 12, fat_small_lines: int = 9) -> Path:
# `fat_small_lines` makes ONE lower-ranked concept the heaviest in the
# bundle, so "the top-ranked candidate" and "the largest excerpt" can be
# told apart by a test rather than coinciding by accident.
(root / "stor").mkdir(parents=True)
(root / "smaa").mkdir(parents=True)
(root / "index.md").write_text(
"---\nokf_version: 0.2\nbundle_id: eviction-fixture\n---\n\n"
"- [stor (index)](stor/index.md)\n- [smaa (index)](smaa/index.md)\n",
encoding="utf-8",
)
(root / "stor" / "index.md").write_text(
"- [Stor tabell om kontroll](tabell.md) — adjudication: proposed\n", encoding="utf-8"
)
(root / "stor" / "tabell.md").write_text(
_EVICTION_FRONTMATTER.format(title="Stor tabell om kontroll", digest="1" * 64)
+ "## Stor tabell om kontroll\n\n"
+ "Kontrollen av den store tabellen foelger tabellen rad for rad.\n" * 105,
encoding="utf-8",
)
entries = []
for number in range(1, smalls + 1):
name = f"notat-{number:02d}"
entries.append(f"- [Notat om kontroll {number:02d}]({name}.md) — adjudication: proposed\n")
lines = fat_small_lines if number == smalls else 9
(root / "smaa" / f"{name}.md").write_text(
_EVICTION_FRONTMATTER.format(title=f"Notat om kontroll {number:02d}", digest="2" * 64)
+ f"## Notat om kontroll {number:02d}\n\n"
+ "Notatet gjelder kontroll av ett enkelt punkt.\n" * lines,
encoding="utf-8",
)
(root / "smaa" / "index.md").write_text("".join(entries), encoding="utf-8")
return root
def _eviction_payload(
root: Path, *, k: int = 16, limit: int = EVICTION_LIMIT, reserve_top_rank: bool = False
) -> dict[str, object]:
return okf_consume.build_payload(
root,
question=EVICTION_QUESTION,
k=k,
limit=limit,
reserve_top_rank=reserve_top_rank,
)
def _top_candidate(root: Path) -> str:
# Computed the long way -- through the ranker, not read off the payload --
# so "the top-ranked candidate" in the assertions below is not whatever the
# cut happened to deliver first.
profile = okf_consume.DEFAULT_PROFILE
concepts = [
okf_consume.read_concept(
root / f"{concept_id}.md", bundle_root=root, root_bundle_id="eviction-fixture"
)
for concept_id in okf_consume.enumerate_concepts(root, profile=profile)
]
ranked = okf_consume.concept_scores(
concepts, EVICTION_QUESTION, okf_consume.document_scores(root, EVICTION_QUESTION)
)
return next(concept.concept_id for concept, _, lexical in ranked if lexical > 0)
def test_the_knapsack_evicts_the_top_ranked_candidate_that_costs_half_the_budget(
tmp_path: Path,
) -> None:
# The known-positive for everything below: without it, a delivered top rank
# with the flag on would prove nothing, because nothing would have been
# shown to remove it.
root = _eviction_bundle(tmp_path / "bundle")
assert _top_candidate(root) == EVICTION_TOP
# The DEFAULT path, called without the new argument, so this control runs
# and can be believed before the rule exists.
payload = okf_consume.build_payload(
root, question=EVICTION_QUESTION, k=16, limit=EVICTION_LIMIT
)
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
weights = {
excerpt["concept_id"]: okf_consume.excerpt_weight(excerpt)
for excerpt in excerpts
if isinstance(excerpt, dict)
}
assert EVICTION_TOP not in weights
assert (
dict(
(entry["concept_id"], entry["rule"])
for entry in payload["withheld"] # type: ignore[union-attr]
)[EVICTION_TOP]
== "over_budget_after_knapsack"
)
# The shape itself, stated as numbers rather than assumed: the top candidate
# fits ALONE and still loses, which is what makes this a budget question.
top = _eviction_bundle_top_weight(root)
assert top <= EVICTION_LIMIT
assert top > EVICTION_LIMIT // 2
def _eviction_bundle_top_weight(root: Path) -> int:
concept = okf_consume.read_concept(
root / f"{EVICTION_TOP}.md", bundle_root=root, root_bundle_id="eviction-fixture"
)
excerpt = okf_consume.excerpt_for(concept)
assert excerpt is not None
return okf_consume.excerpt_weight(excerpt)
def test_reserving_the_top_rank_delivers_the_candidate_the_knapsack_evicted(
tmp_path: Path,
) -> None:
root = _eviction_bundle(tmp_path / "bundle")
payload = _eviction_payload(root, reserve_top_rank=True)
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
assert excerpts[0]["concept_id"] == EVICTION_TOP
assert excerpts[0]["rank"] == 1
assert EVICTION_TOP not in {
entry["concept_id"]
for entry in payload["withheld"] # type: ignore[union-attr]
}
def test_the_reservation_is_off_by_default_and_the_default_payload_is_byte_identical(
tmp_path: Path,
) -> None:
root = _eviction_bundle(tmp_path / "bundle")
default = okf_consume.serialise(
okf_consume.build_payload(root, question=EVICTION_QUESTION, k=16, limit=EVICTION_LIMIT)
)
explicit_off = okf_consume.serialise(_eviction_payload(root, reserve_top_rank=False))
assert default == explicit_off
assert '"reserved"' not in default
question = "Hvordan skal prisene fylles ut?"
assert okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question)
) == okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, reserve_top_rank=False)
)
def test_a_top_candidate_that_alone_exceeds_the_budget_is_still_refused_by_name(
tmp_path: Path,
) -> None:
# The control the reservation must not break: `over_budget_alone` is a
# PRE-exclusion, and a reservation that ran before it would deliver an
# excerpt the budget can never hold.
root = _eviction_bundle(tmp_path / "bundle")
limit = _eviction_bundle_top_weight(root) - 1
payload = _eviction_payload(root, limit=limit, reserve_top_rank=True)
rules = dict(
(entry["concept_id"], entry["rule"])
for entry in payload["withheld"] # type: ignore[union-attr]
)
assert rules[EVICTION_TOP] == "over_budget_alone"
spent = payload["budget"]["spent"] # type: ignore[index]
assert isinstance(spent, int)
assert spent <= limit
def test_the_reservation_never_spends_more_than_the_budget(tmp_path: Path) -> None:
root = _eviction_bundle(tmp_path / "bundle")
for limit in (EVICTION_LIMIT, EVICTION_LIMIT + 3_000, EVICTION_LIMIT * 2):
payload = _eviction_payload(root, limit=limit, reserve_top_rank=True)
spent = payload["budget"]["spent"] # type: ignore[index]
assert isinstance(spent, int)
assert spent <= limit
def test_the_reservation_displaces_lower_ranked_excerpts_under_the_rule_that_exists(
tmp_path: Path,
) -> None:
# The cost of the rule, asserted rather than described: reserving the top
# rank buys its bytes from the excerpts the knapsack preferred, and they
# leave under a rule already in the closed set.
root = _eviction_bundle(tmp_path / "bundle")
without = _eviction_payload(root)
with_reservation = _eviction_payload(root, reserve_top_rank=True)
assert len(with_reservation["excerpts"]) < len(without["excerpts"]) # type: ignore[arg-type]
displaced = {
entry["concept_id"]
for entry in with_reservation["withheld"] # type: ignore[union-attr]
if entry["rule"] == "over_budget_after_knapsack"
}
delivered_before = {
excerpt["concept_id"]
for excerpt in without["excerpts"] # type: ignore[union-attr]
}
assert displaced & delivered_before
assert {
entry["rule"]
for entry in with_reservation["withheld"] # type: ignore[union-attr]
} <= set(okf_consume.WITHHOLDING_RULES)
def test_the_payload_declares_which_concept_the_reservation_took_and_what_it_cost(
tmp_path: Path,
) -> None:
root = _eviction_bundle(tmp_path / "bundle")
budget = _eviction_payload(root, reserve_top_rank=True)["budget"]
assert isinstance(budget, dict)
assert budget["reserved"] == {
"concept_id": _top_candidate(root),
"bytes": _eviction_bundle_top_weight(root),
}
spent = budget["spent"]
assert isinstance(spent, int)
assert spent >= _eviction_bundle_top_weight(root)
def test_the_reservation_is_paid_once_and_does_not_bid_for_its_own_bytes(
tmp_path: Path,
) -> None:
# A reserved excerpt left in the pack's pool competes for the budget it has
# already been given, and wins it back from the next candidate. Visible
# only where the reserved excerpt would out-value what the remaining room
# can hold: two candidates, and room enough for the reserved one twice.
root = _eviction_bundle(tmp_path / "bundle")
payload = _eviction_payload(root, k=2, limit=15_000, reserve_top_rank=True)
delivered = [
excerpt["concept_id"]
for excerpt in payload["excerpts"] # type: ignore[union-attr]
]
assert delivered == [EVICTION_TOP, "smaa/notat-01"]
def test_the_reservation_names_the_fused_top_and_not_the_heaviest_candidate(
tmp_path: Path,
) -> None:
# A bundle whose HEAVIEST excerpt is a LOWER-ranked one. A reservation
# reading weight where it should read rank reserves the wrong concept, and
# this is the only place the two come apart.
root = _eviction_bundle(tmp_path / "bundle", fat_small_lines=400)
heaviest = max(
okf_consume.enumerate_concepts(root),
key=lambda concept_id: okf_consume.excerpt_weight(
okf_consume.excerpt_for(
okf_consume.read_concept(
root / f"{concept_id}.md",
bundle_root=root,
root_bundle_id="eviction-fixture",
)
)
or {}
),
)
assert heaviest != EVICTION_TOP
budget = _eviction_payload(root, limit=EVICTION_LIMIT * 3, reserve_top_rank=True)["budget"]
assert isinstance(budget, dict)
assert budget["reserved"]["concept_id"] == EVICTION_TOP # type: ignore[index]
def test_a_payload_carrying_a_reservation_still_passes_the_checker(tmp_path: Path) -> None:
# SS 8 permits additional members; a declaration the checker refuses would
# buy honesty at the price of conformance.
root = _eviction_bundle(tmp_path / "bundle")
report = okf_contract_check.check(
TEMPLATE.read_text(encoding="utf-8"), _eviction_payload(root, reserve_top_rank=True)
)
assert report.findings == ()
def test_the_cli_exposes_the_reservation_and_omitting_it_reproduces_the_default_bytes(
tmp_path: Path,
) -> None:
root = _eviction_bundle(tmp_path / "bundle")
common = (
str(root),
"--question",
EVICTION_QUESTION,
"--k",
"16",
"--limit",
str(EVICTION_LIMIT),
)
plain = _run(*common)
assert plain.returncode == 0
reserved = _run(*common, "--reserve-top-rank")
assert reserved.returncode == 0
assert plain.stdout == okf_consume.serialise(_eviction_payload(root))
assert reserved.stdout == okf_consume.serialise(_eviction_payload(root, reserve_top_rank=True))
assert plain.stdout != reserved.stdout

64
tools/okf_consume.py
View file

@ -846,9 +846,14 @@ def knapsack(items: Sequence[tuple[float, int]], *, capacity: int) -> tuple[int,
def cut(
ranked: Sequence[tuple[Concept, float, int]], *, k: int, limit: int
) -> tuple[tuple[dict[str, object], ...], tuple[tuple[str, str], ...]]:
"""The ranked concepts split into delivered excerpts and named drops.
ranked: Sequence[tuple[Concept, float, int]],
*,
k: int,
limit: int,
reserve_top_rank: bool = False,
) -> tuple[tuple[dict[str, object], ...], tuple[tuple[str, str], ...], tuple[str, int] | None]:
"""The ranked concepts split into delivered excerpts, named drops, and the
reservation that was made, if any.
**The partition is the invariant, not a consequence.** Every considered
concept lands in exactly one of the two, so `considered == withheld +
@ -862,6 +867,17 @@ def cut(
bundle still returns eight excerpts -- a confident guess wearing a
denominator -- and hit@k over such a ranker measures the corpus's size
rather than the ranker.
**`reserve_top_rank` (default off) buys the highest-ranked candidate its
bytes before the pack runs.** The DP maximises a SUM of fused scores, so a
single candidate costing a large share of the budget loses to enough small
ones no matter how far ahead it ranks -- measured, a top-ranked excerpt
worth 56.5 % of the budget is evicted as soon as the shortlist holds enough
alternatives, which makes `k` a dial that can remove the one concept a
question was asked about. The reservation makes rank one a floor rather
than a bid, and the pack fills what is left. It runs AFTER the
`over_budget_alone` pre-exclusion, never before: a reservation for an
excerpt the budget can never hold would deliver bytes the gate refuses.
"""
withheld: list[tuple[str, str]] = []
candidates: list[tuple[Concept, float, dict[str, object], int]] = []
@ -894,7 +910,16 @@ def cut(
# list would silently turn "position in the payload" into a different number
# from "position in the ranking".
pool = sorted(shortlist, key=lambda entry: entry[0].concept_id)
capacity = limit // WEIGHT_BUCKET
reserved: tuple[str, int] | None = None
room = limit
if reserve_top_rank and shortlist:
# `shortlist` is in fused-rank order, so its first entry IS the
# top-ranked candidate -- not the heaviest, and not the first by id.
top = shortlist[0]
reserved = (top[0].concept_id, top[3])
room = limit - top[3]
pool = [entry for entry in pool if entry[0] is not top[0]]
capacity = room // WEIGHT_BUCKET
packed = {
id(pool[index][0])
for index in knapsack(
@ -902,6 +927,8 @@ def cut(
capacity=capacity,
)
}
if reserved is not None:
packed.add(id(shortlist[0][0]))
delivered: list[dict[str, object]] = []
for concept, _, excerpt, _ in shortlist:
if id(concept) in packed:
@ -909,7 +936,7 @@ def cut(
else:
withheld.append((concept.concept_id, "over_budget_after_knapsack"))
withheld.sort()
return tuple(delivered), tuple(withheld)
return tuple(delivered), tuple(withheld), reserved
# --- The payload (SS 8) -------------------------------------------------------
@ -931,12 +958,13 @@ def build_payload(
limit: int = DEFAULT_LIMIT,
profile: BundleProfile = DEFAULT_PROFILE,
cost_vocabulary: bool = False,
reserve_top_rank: bool = False,
) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload.
Pure with respect to the clock and the network: the same
`(bundle_root, question, k, limit, cost_vocabulary)` at the same bytes
returns the same object, every time.
`(bundle_root, question, k, limit, cost_vocabulary, reserve_top_rank)` at
the same bytes returns the same object, every time.
"""
case, expected, measured = known_positive()
if expected != measured:
@ -981,7 +1009,7 @@ def build_payload(
cost_vocabulary=cost_vocabulary,
)
matched = sum(1 for _, _, lexical in ranked if lexical > 0)
delivered, withheld = cut(ranked, k=k, limit=limit)
delivered, withheld, reserved = cut(ranked, k=k, limit=limit, reserve_top_rank=reserve_top_rank)
spent = sum(excerpt_weight(excerpt) for excerpt in delivered)
if matched and not delivered:
# SS 7.3: a finding requiring a decision, never something to retry
@ -1022,6 +1050,15 @@ def build_payload(
"raw_bytes": len(_KNOWN_POSITIVE_PATH.read_bytes()),
"encoding_delta": KNOWN_POSITIVE_ENCODING_DELTA,
},
# Present only when a reservation was made, because a cut whose
# strategy changed without saying so is the silent cut SS 5.3
# forbids -- and absent otherwise, so the default payload keeps
# every byte it had.
**(
{"reserved": {"concept_id": reserved[0], "bytes": reserved[1]}}
if reserved is not None
else {}
),
},
"denominators": {
"considered": len(concepts),
@ -1072,6 +1109,16 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"naming no term in that vocabulary is unaffected either way"
),
)
parser.add_argument(
"--reserve-top-rank",
action="store_true",
help=(
"give the highest-ranked candidate its bytes before the budget is "
"packed, so a large top-ranked excerpt is not out-summed by small "
"ones. OFF by default; a candidate that alone exceeds the budget is "
"still refused"
),
)
parser.add_argument("--out", type=Path, default=None, help="write here instead of stdout")
parser.add_argument(
"--ref",
@ -1104,6 +1151,7 @@ def main(argv: list[str] | None = None) -> int:
k=args.k,
limit=args.limit,
cost_vocabulary=args.cost_vocabulary,
reserve_top_rank=args.reserve_top_rank,
)
except ConsumeError as error:
print(f"okf_consume: FAILED - {error}", file=sys.stderr)