open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): one source document took 8 of 8 delivered places, so cap it

Browse source 
Measured outside this repository on a 3206-concept bundle of a published
handbook: the code's own process overview contributes 28 of 3206 concepts
(0.87 %) and 117 488 of 1 469 225 source characters (8.0 %), and took 8 of 8
delivered places on one question and 7 of 8 on the known-positive -- which was
not delivered at all. Identical at 343 and 1651 concepts, so the cause is the
corpus's COMPOSITION, that it holds its own table of contents, and NOT its size.
Splitting the corpus would move the defect, not remove it: any corpus with a
contents list, a project overview or a summary document has the same property.

`--source-quota N` caps how many DELIVERED places one source document may take.
It cuts where the shortlist is cut -- before the pack, never inside the DP,
which maximises a sum over a set it is handed -- so the freed place goes to the
next candidate and `k` is still delivered in full.

DEFAULT 2 SINCE TODAY, and it is the third change here that alters a payload
with NO bundle changing (after `--tie-shared-rank` and `--stem-prefix`).
Opt-out `--no-source-quota` reproduces the previous excerpt order.

Swept over {2, 3, 4, off} on three bundles, with the fasit prefixes validated
against the bundle FIRST (that control caught a defect in the measuring query
itself -- it read the last id segment where the document is the first):

- K2, both bundles: at 2 and 3, hit@8 goes 5 of 6 to 6 of 6 with all five
  standing rank-1 rows unmoved. The recovered row had missed on every bundle and
  every configuration measured until now. At 4 and off it is 5 of 6.
- The handbook bundle: hit@8 2 of 6 -> 4 of 6, the known-positive from not
  delivered to rank 4, and the dominant document's share of delivered places
  8 of 8 -> 2 of 8 (7 of 8 -> 2 of 8 on the known-positive).
- 2 rather than 3 on rank alone: the recovered rows come in at 5 and 4 rather
  than 7 and 5.

WHAT THE GAIN IS NOT. hit@8 asks whether the gold DOCUMENT appears among the
delivered excerpts, and a document quota directly raises how many distinct
documents a payload holds, so that metric is not neutral with respect to this
rule. The five rows that were already rank 1 are neutral, and they did not move.

THE ADVERSE CASE IS NAMED, not left to a consumer. A bundle built from ONE
document carries the same `source_file` on every concept, so a quota applied
literally would deliver 2 excerpts where `k` were asked for -- a rule against
dominance turned into a rule against small bundles. The shortlist is topped back
up from the best-ranked over-quota candidates, which makes such a bundle
byte-identical to the quota being off, and a test holds it.

`--rarity-weight` was measured against the same defect and does NOT repair it:
it leaves the dominant document at 8 of 8 places on the question it floods,
delivers neither that answer nor the known-positive, and holds 5 of 6 on both K2
bundles. Combined with the quota it is worse than the quota alone (the
known-positive falls back out). It stays off.

The vocabulary stays CLOSED and the new code is published in all three places a
consumer can read it: `WITHHOLDING_RULES` (six -> seven),
`docs/consumption-contract.md` 5.3, and the generated SKILL.md -- verified by
reading the generated file, not the code that writes it. `source_quota_exceeded`
is a DIVERSITY drop and not a relevance one, so folding it into
`no_lexical_match` would tell a consumer the question reached nothing in a
concept the question in fact reached. `okf check --skill --payload` stays
conformant, 0 findings over 15 rules.

Editing the contract moved the 7.4 known-positive, which is the coupling
working as intended: 12 563 -> 13 238 encoded, 12 227 -> 12 893 raw, delta
336 -> 345, updated in the constant, the instantiated skill and the shipped
example payload.

Also adds the O6 guard on the reading side: `build_payload`'s signature defaults
are asserted equal to `okf consume`'s argparse defaults for every same-named
parameter. `okf project` shipped that exact disagreement for two rounds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 00:29:02 +02:00
commit 1e9f38b125
9 changed files with 380 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

57
CLAUDE.md
View file

@ -22,7 +22,29 @@ one boundary rule:
the door's own output reachable as its own input (operator 2026-09-06; the
flat listing was not a boundary, it was an absence with no denominator). All
file-type→text extraction lives HERE (the guard is text-only). v1 core:
`md`, `txt`, `csv`, `json`, `html` (stdlib). `pdf`/`docx`/`xlsx` only via
`md`, `txt`, `csv`, `json`, `html` (stdlib). **`html` got a measured
segmentation row 2026-09-10.** Until then `_HTMLTextExtractor.text()` was
`" ".join("".join(parts).split())`, and `str.split()` with no argument splits
on newlines too, so extraction of ANY HTML file returned unconditionally ONE
line while every boundary grammar in `propose` is line-anchored -- measured
outside this repo, **828 of 828** real sections gave 0 plans and exit 2 at
every sample point, and a coarser 145-document cut gave 145 of 145. Block tags
now open their own lines and `h1`-`h6` carry the ATX marker for their OWN level
(a flat `#` would hand `_ATX` three top-level boundaries where the document
declares one section and two subsections). The output grammar is MARKDOWN, the
same the office rows reach the proposer through, so **no HTML-only heading
grammar exists**; the fix is in the extractor and **never** the converter,
because `.html` stays out of `_PANDOC_FORMATS` on CVE-2025-51591. After:
**828 of 828 plans, exit 0, 3206 concepts / 6015 md -- the markdown path's
count EXACTLY**, and the same at 414 (1651) and 83 (343). Text preservation is
an EXACT invariant and not a percentage: strip the added ATX markers and the
non-whitespace sequence is identical to the old extractor's, **828 of 828
files**, character ratio **1.000000**. `_SKIP_TAGS` stays `{script, style}`.
Exposure elsewhere measured rather than argued: **0 of 86** K2 corpus files and
**0 of 5** smoke-folder files are HTML, and the smoke bundle is byte-identical
before and after. `_EVIDENCE` gains a `.html` row at `measured`, with the limit
that travels with it -- one product, one format, one publisher, and a
generator's cut, not 828 documents anyone wrote. `pdf`/`docx`/`xlsx` only via
the optional `[extract]` extra; without it those types are rejected
fail-fast. The extra ships `pdfplumber` for `pdf` (chosen on ONE measured
property: it keeps a requirement table's label and value on the same line
@ -598,6 +620,39 @@ and fixtures, never code.
here -- a genuine Norwegian morpheme, so that residual is a different answer,
never a ceiling. The vocabulary is the BUNDLE's own, so the rule makes a
payload corpus-dependent the way `rarity_weights` already is.
**A SEVENTH flag, `--source-quota N`, is ON at 2 since 2026-09-10** (opt-out
`--no-source-quota`) and is the third change here that alters a payload with
NO bundle changing. It caps how many DELIVERED places one `source_file` may
take, cutting where `shortlist = candidates[:k]` cuts, so the freed place goes
to the next candidate and `k` is still delivered in full. The defect was
measured OUTSIDE this repo on a 3206-concept bundle of a published handbook:
the code's own process overview is **28 of 3206 concepts (0.87 %)** and **8.0 %
of the source characters** yet took **8 of 8** delivered places on one question
and **7 of 8** on the known-positive, which was not delivered at all --
identical at 343 and 1651 concepts, so it is the corpus's COMPOSITION (it holds
its own table of contents) and not its size, and a split would move it rather
than remove it. Swept over {2, 3, 4, off} on three bundles with the fasit
prefixes validated against the bundle FIRST (that control caught a defect in
the measuring query itself): at 2 and 3 hit@8 goes **5 of 6 to 6 of 6 on BOTH
K2 bundles** with all five standing rank-1 rows unmoved -- the recovered row
had missed on every bundle and every configuration measured until now -- and at
4 and off it stays 5 of 6. On the handbook bundle hit@8 goes **2 of 6 to 4 of
6** and the dominant document's share **8 of 8 to 2 of 8**. 2 rather than 3 on
rank. **What the gain is NOT:** hit@8 asks whether the gold DOCUMENT was
delivered and a document quota raises how many distinct documents a payload
holds, so that metric is not neutral with respect to this rule; the five rows
already at rank 1 are, and did not move. **The adverse case is named:** a
one-document bundle has one `source_file` on every concept, so the quota would
deliver 2 where `k` were asked -- the shortlist is topped back up from the
best-ranked over-quota candidates, making such a bundle byte-identical to the
quota being off. The `WITHHOLDING_RULES` vocabulary goes six to seven
(`source_quota_exceeded`, a DIVERSITY drop and not a relevance one) and is
published in the contract SS 5.3 and in the generated SKILL.md, verified by
reading the generated file. Editing the contract moved the SS 7.4
known-positive (12 563 -> 13 238 encoded, delta 336 -> 345), which is that
coupling working. `--rarity-weight` was measured against the same defect and
does NOT repair it -- it leaves the dominant document at 8 of 8 places on the
question it floods -- and stays off.
`--rarity-weight` is the third: each lexical hit weighs `log(N/df)` over the
bundle's own concepts instead of 1, so an identifier is not worth what a
common verb is worth. It enters the RANKING and never the GATE — `lexical`

31
README.md
View file

@ -293,6 +293,35 @@ pre-exclusion, never before — and the payload then declares
24 measured combinations, both of them that eviction:
`docs/2026-09-08-blindsone-laas2-budsjett-k2.md`.
`--source-quota N` is **on** by default at **2** since 2026-09-10 (opt out with
`--no-source-quota`), and it is the third widening here that alters a payload
with no bundle changing. It caps how many DELIVERED places one source document
may take, cutting where the shortlist is cut so the freed place goes to the next
candidate and `k` is still delivered in full. The defect it repairs was measured
outside this repository on a 3206-concept bundle of a published handbook: the
code's own process overview contributes **28 of 3206 concepts (0.87 %)** and
**8.0 % of the source characters**, and took **8 of 8** delivered places on one
question and **7 of 8** on the known-positive, which was not delivered at all.
Identical at 343 and 1651 concepts, so the cause is the corpus's COMPOSITION —
that it holds its own table of contents — and not its size; any corpus with a
contents list, a project overview or a summary document has the same property.
Swept over {2, 3, 4, off} on three bundles: at 2 and 3 hit@8 goes **5 of 6 to
6 of 6 on both K2 bundles** with all five standing rank-1 rows unmoved, and at 4
and off it stays 5 of 6. On the handbook bundle hit@8 goes **2 of 6 to 4 of 6**
and the dominant document's share of delivered places **8 of 8 to 2 of 8**. 2
rather than 3 on rank: the recovered rows come in at 5 and 4 rather than 7 and
5. What the gain is NOT: hit@8 asks whether the gold DOCUMENT was delivered, and
a document quota directly raises how many distinct documents a payload holds, so
that metric is not neutral with respect to this rule — the five rows that were
already rank 1 are, and they did not move. The adverse case is named rather than
found later: a bundle built from ONE document has one `source_file` on every
concept, so the quota would deliver 2 excerpts instead of `k`; the shortlist is
topped back up from the best-ranked over-quota candidates, which makes such a
bundle byte-identical to the quota being off. `--rarity-weight` was measured
against the same defect in the same session and does **not** repair it: on the
handbook bundle it leaves the dominant document at 8 of 8 places on the
question it floods and delivers neither that answer nor the known-positive.
`--rarity-weight` is off by default and weights each lexical hit by
`log(N/df)` over the bundle's own concepts instead of counting it as one, so a
requirement number is not worth what a common verb is worth. The default being
@ -350,7 +379,7 @@ measurement" is a measurement of a *configuration*, not a property of the flag.
It emits the § 8 shape — `contract`, `bundle` (`bundle_id` plus a
`sha256-tree:` content identity), `budget` (unit, instrument, limit, spent and a
validated known-positive), `denominators`, `excerpts` and `withheld` — and every
withheld concept names the rule that dropped it, from a closed set of six.
withheld concept names the rule that dropped it, from a closed set of seven.
Every excerpt carries the concept's `title`, and — when the producer wrote them
— `req_number`, the SPEC § 5.1 address `sources`, and **every top-level

11
docs/consumption-contract.md
View file

@ -82,7 +82,16 @@ carries them.
it does not is refused rather than reported, because a count that does not
close is not a denominator.
3. Every withheld concept MUST name the **rule** that dropped it. A visible drop
is worth more than a silent one.
is worth more than a silent one. The rule set is the producer's, and the
contract does not enumerate it — but it MUST be **closed and published**: a
rule invented at the drop site is a vocabulary no consumer can be held to.
This library's set is seven codes, and the one a diversity rule needs is
named rather than folded into an existing code:
`source_quota_exceeded` — the concept ranked inside the cut and its source
document already held as many delivered places as the profile allows. It is
a DIVERSITY drop and not a relevance one, so reading it as
`no_lexical_match` would tell a consumer the question reached nothing in a
concept the question in fact reached.
4. Any claim of the form "there is no X", "nothing further was found" or "all N
are Y" MUST report the denominator it was measured over, and the command that
produced it. A negative result whose scope is unstated is **unmeasured**, and

13
skills/okf-consume/SKILL.md
View file

@ -67,12 +67,15 @@ and the cut; it decides nothing about the question.
when you were handed a bounded window, is the denominator failure below with
extra steps.
**The five rules this pre-pass may drop a concept under**, so a `withheld` entry
**The seven rules this pre-pass may drop a concept under**, so a `withheld` entry
can be read without guessing: `verdict_layer_excluded` (§ 9.1, a type check),
`no_lexical_match` (the question reached nothing in this concept),
`verified_unreadable` (a `verified` value present but outside what this
library's line-oriented parser can read, so no tier could be derived honestly),
`over_budget_alone` (larger than the whole budget), `below_k` (ranked outside
`over_budget_alone` (larger than the whole budget), `source_quota_exceeded`
(its source document already holds as many delivered places as
`--source-quota` allows, default 2 — the freed place goes to the next
candidate, so `k` is still delivered in full), `below_k` (ranked outside
the delivered cap), `over_budget_after_knapsack` (it fitted alone but not
alongside the set that was chosen).
@ -213,7 +216,7 @@ other states is a claim about the fixture rather than about this corpus.
| Limit | `120000` |
| Unit | `utf-8 bytes of emitted JSON` |
| Instrument | `okf_consume.measure` — `len(json.dumps(value, ensure_ascii=False).encode("utf-8"))` |
| Known-positive | `docs/consumption-contract.md, encoded as a JSON string` at `12563` |
| Known-positive | `docs/consumption-contract.md, encoded as a JSON string` at `13238` |
The instrument reproduces the known-positive figure before any of its own
numbers are believed. Report what the run actually spent.
@ -222,8 +225,8 @@ The known-positive is a **shipped artefact rather than this bundle**, and the
reason is that a per-bundle one cannot work: it would be either a constant wrong
for every bundle but one, or the instrument's own output, which makes
`expected == measured` true by construction and § 7.4 decorative. It is checked
by a **second, independent route**: `wc -c` reports 12 227 raw bytes for the same
file, and the 336-byte difference is that file's JSON quoting and escaping
by a **second, independent route**: `wc -c` reports 12 893 raw bytes for the same
file, and the 345-byte difference is that file's JSON quoting and escaping
overhead. The delta moves the moment the instrument changes what it counts.
`spent` is the cost of the **delivered set**, per § 7.2 — not of the whole

8
skills/okf-consume/references/example-payload.json
View file

@ -11,10 +11,10 @@
"spent": 2125,
"known_positive": {
"case": "docs/consumption-contract.md, encoded as a JSON string",
"expected": 12563,
"measured": 12563,
"raw_bytes": 12227,
"encoding_delta": 336
"expected": 13238,
"measured": 13238,
"raw_bytes": 12893,
"encoding_delta": 345
}
},
"denominators": {

97
src/llm_ingestion_okf/consume.py
View file

@ -593,14 +593,14 @@ KNOWN_POSITIVE_CASE = "docs/consumption-contract.md, encoded as a JSON string"
#: `measure()`'s own answer for that file. Vacuous ALONE -- which is why the
#: delta below exists.
KNOWN_POSITIVE_EXPECTED = 12_563
KNOWN_POSITIVE_EXPECTED = 13_238
#: The second, independent route. `wc -c` reports 12 227 raw bytes for the same
#: The second, independent route. `wc -c` reports 12 893 raw bytes for the same
#: file; the difference is this file's JSON quoting and escaping overhead. A
#: reader can derive it without running `measure()` at all, and it moves the
#: moment `measure()` changes what it counts -- which is what stops
#: `expected == measured` from proving nothing.
KNOWN_POSITIVE_ENCODING_DELTA = 336
KNOWN_POSITIVE_ENCODING_DELTA = 345
#: The two places that file can be, resolved in this order.
#:
@ -1335,8 +1335,31 @@ WITHHOLDING_RULES = (
"over_budget_alone",
"below_k",
"over_budget_after_knapsack",
"source_quota_exceeded",
)
#: `--source-quota N` caps how many DELIVERED places one source document may
#: take, topping the shortlist back up to `k` when the bundle has no
#: alternatives to offer.
#:
#: **2 SINCE 2026-09-10, and it is the third change here that moves a payload
#: with NO bundle changing** (after `--tie-shared-rank` and `--stem-prefix`);
#: a consumer pinned to the previous excerpt order needs `--no-source-quota`.
#: The defect it repairs was measured on a 3206-concept bundle of a published
#: handbook: the code's own process overview contributes 28 of 3206 concepts
#: (0.87 %) and 8.0 % of the source characters, and took 8 of 8 delivered
#: places on one question and 7 of 8 on the known-positive -- identical at 343
#: and 1651 concepts, so the cause is the corpus's COMPOSITION (it holds its
#: own table of contents) and not its size.
#:
#: SWEPT over {2, 3, 4, off} on three bundles. At 2 and 3 hit@8 goes 5 of 6 to
#: **6 of 6 on BOTH K2 bundles** with all five standing rank-1 rows unmoved; at
#: 4 and off it is 5 of 6. On the handbook bundle hit@8 goes 2 of 6 to 4 of 6
#: and the dominant document's share of delivered places 8 of 8 to 2 of 8. 2
#: rather than 3 on rank: the recovered rows come in at 5 and 4 rather than 7
#: and 5.
DEFAULT_SOURCE_QUOTA: int | None = 2
#: The knapsack's weight granularity, in bytes. Bucketing keeps the DP table
#: small; bucketing UP the item and DOWN the capacity keeps the error one-sided,
#: so the pack may under-deliver by a bucket and can never over-spend.
@ -1437,6 +1460,7 @@ def cut(
k: int,
limit: int,
reserve_top_rank: bool = False,
source_quota: int | None = DEFAULT_SOURCE_QUOTA,
) -> tuple[tuple[dict[str, object], ...], tuple[tuple[str, str], ...], tuple[str, int] | None]:
"""The ranked concepts split into delivered excerpts, named drops, and the
reservation that was made, if any.
@ -1486,6 +1510,40 @@ def cut(
withheld.append((concept.concept_id, "over_budget_alone"))
continue
candidates.append((concept, score, excerpt, weight))
if source_quota is not None:
# THE QUOTA CUTS WHERE THE SHORTLIST IS CUT, never inside the pack. The
# DP maximises a sum over a set it is handed; a quota expressed there
# would be a constraint on the sum, which is a different problem and a
# slower one. Here it is a filter on the ranked candidate list, so `k`
# is still delivered in full and the freed place goes to the next
# candidate rather than being lost.
seen: dict[str, int] = {}
keep = [True] * len(candidates)
over: list[int] = []
for index, entry in enumerate(candidates):
document = entry[0].source_file
taken = seen.get(document, 0)
if taken >= source_quota:
keep[index] = False
over.append(index)
continue
seen[document] = taken + 1
# THE QUOTA NEVER SHORTENS THE PAYLOAD, and that is not a nicety. A
# bundle built from ONE document carries the same `source_file` on
# every concept, so a quota applied without this would deliver
# `source_quota` excerpts instead of `k` -- a rule against dominance
# turned into a rule against small bundles. `over` is in fused-rank
# order, so the top-up takes the BEST-ranked over-quota candidates
# back, and a bundle with no alternatives to offer is byte-identical
# to the quota being off.
for index in over[: max(k - sum(keep), 0)]:
keep[index] = True
withheld.extend(
(candidates[index][0].concept_id, "source_quota_exceeded")
for index in range(len(candidates))
if not keep[index]
)
candidates = [entry for index, entry in enumerate(candidates) if keep[index]]
for concept, _, _, _ in candidates[k:]:
withheld.append((concept.concept_id, "below_k"))
shortlist = candidates[:k]
@ -1571,6 +1629,7 @@ def build_payload(
tie_shared_rank: bool = DEFAULT_TIE_SHARED_RANK,
withheld_titles: bool = False,
stem_prefix: bool = DEFAULT_STEM_PREFIX,
source_quota: int | None = DEFAULT_SOURCE_QUOTA,
) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload.
@ -1648,7 +1707,13 @@ def build_payload(
)
titles_by_id = {concept.concept_id: concept.title for concept in concepts}
matched = sum(1 for _, _, lexical in ranked if lexical > 0)
delivered, withheld, reserved = cut(ranked, k=k, limit=limit, reserve_top_rank=reserve_top_rank)
delivered, withheld, reserved = cut(
ranked,
k=k,
limit=limit,
reserve_top_rank=reserve_top_rank,
source_quota=source_quota,
)
spent = sum(excerpt_weight(excerpt) for excerpt in delivered)
if matched and not delivered:
# SS 7.3: a finding requiring a decision, never something to retry
@ -1818,6 +1883,29 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
dest="stem_prefix",
help="The rule's explicit opt-out, reproducing the pre-round-10 matcher",
)
parser.add_argument(
"--source-quota",
type=int,
default=DEFAULT_SOURCE_QUOTA,
metavar="N",
help=(
"cap how many DELIVERED places one source document may take, "
"filling the freed places from the next candidate so k is still "
"delivered in full. Default 2 since 2026-09-10. Measured on a "
"3206-concept bundle whose corpus holds its own table of contents: "
"that one document took 8 of 8 places and the answer was not "
"delivered at all; at 2 it takes 2 of 8 and the answer comes in at "
"rank 4. A bundle with no alternatives is unaffected -- the "
"shortlist is topped back up to k"
),
)
parser.add_argument(
"--no-source-quota",
action="store_const",
const=None,
dest="source_quota",
help="The rule's explicit opt-out, reproducing the pre-round-11 excerpt order",
)
parser.add_argument(
"--withheld-titles",
action="store_true",
@ -1864,6 +1952,7 @@ def main(argv: list[str] | None = None) -> int:
rarity_weight=args.rarity_weight,
tie_shared_rank=args.tie_shared_rank,
stem_prefix=args.stem_prefix,
source_quota=args.source_quota,
withheld_titles=args.withheld_titles,
)
except ConsumeError as error:

5
src/llm_ingestion_okf/skill.py
View file

@ -565,10 +565,13 @@ def _denominators(total: int, *, asked: str, delivered: int) -> str:
f"For this bundle `considered` is **{total}**, every concept the index walk\n"
"reaches, never the post-ranking shortlist. A concept dropped at the ranking\n"
"stage is `withheld` **with its rule**, not invisible, and the rules are a\n"
"closed set of six: `verdict_layer_excluded` (a verdict-layer file, § 9.1),\n"
"closed set of seven: `verdict_layer_excluded` (a verdict-layer file, § 9.1),\n"
"`verified_unreadable` (a `verified` value this reader cannot decode, so no\n"
"tier can be derived), `no_lexical_match` (the concept shares no token with\n"
"the question), `over_budget_alone` (one excerpt exceeds the whole limit),\n"
"`source_quota_exceeded` (its source document already holds as many\n"
"delivered places as `--source-quota` allows, default 2 — the freed place\n"
"goes to the next candidate, so `k` is still delivered in full),\n"
"`below_k` (ranked outside the shortlist the cut considers) and\n"
"`over_budget_after_knapsack` (it ranked inside the shortlist and the pack\n"
"had no room). Naming the rule is what makes a drop visible.\n\n"

51
tests/test_default_bundle_pin.py
View file

@ -84,11 +84,19 @@ requires_default_bundle = pytest.mark.skipif(
#: candidates it was taking with it survive. Corpus-wide, 429 -> 447 candidates
#: over 32 -> 33 documents with a plan, and characters in no segment stay 0.
EXPECTED_CONCEPTS = 453
EXPECTED_HITS = 5
#: Rank per question INDEX, `None` for the row that misses on every bundle and
#: every configuration measured so far. The identity is the index; the question
#: stays in the local-only gold set.
EXPECTED_RANKS = (1, 1, 1, 1, 1, None)
EXPECTED_HITS = 6
#: Rank per question INDEX. The identity is the index; the question stays in
#: the local-only gold set.
#:
#: ROW 6 MOVED `None` -> 5 ON 2026-09-10, on these same bytes and with no
#: bundle changing: `consume.DEFAULT_SOURCE_QUOTA` became 2. It had missed on
#: every bundle and every configuration measured until then, and the test below
#: names the quota as its cause by turning it off. What the gain is NOT: hit@8
#: scores whether the gold DOCUMENT appears among the delivered excerpts, and a
#: document quota directly raises how many distinct documents a payload holds,
#: so this metric is not neutral with respect to this rule. The five rows that
#: were already rank 1 are, and they did not move.
EXPECTED_RANKS = (1, 1, 1, 1, 1, 5)
@requires_default_bundle
@ -151,7 +159,12 @@ def test_the_reading_default_is_what_holds_row_one_on_these_bytes() -> None:
assert isinstance(excerpts, list)
ranks.append(okf_consume_measure.hit_rank(excerpts, entry["gold_document"]))
assert ranks[0] == 2, "the known-negative stopped being negative"
assert tuple(ranks[1:]) == EXPECTED_RANKS[1:]
# Rows 2-5 are untouched by the tie-break. Row 6 is NOT asserted equal to
# the pin: it is the row `DEFAULT_SOURCE_QUOTA` recovered, and the two
# rules reach it independently, so pinning it here would assert a
# coincidence rather than a cause.
assert tuple(ranks[1:5]) == EXPECTED_RANKS[1:5]
assert ranks[5] is not None
@requires_default_bundle
@ -179,6 +192,32 @@ def test_the_stem_rule_holds_every_rank_on_the_shipped_bytes() -> None:
assert tuple(ranks) == EXPECTED_RANKS
@requires_default_bundle
def test_the_document_quota_is_what_reaches_row_six_on_these_bytes() -> None:
"""The known-negative for round 11's default, on the shipped bundle.
Row 6 had missed on every bundle and every configuration measured before
2026-09-10. `DEFAULT_SOURCE_QUOTA = 2` reaches it at rank 5, and this test
is what keeps `EXPECTED_RANKS` from being green for an unstated reason:
turn the quota off on these exact bytes and the miss comes back.
The five standing rank-1 rows are asserted UNMOVED in the same run, because
"the new rule cost nothing" is the half of the claim a hit count cannot
see.
"""
questions = json.loads(GOLD_SET.read_text(encoding="utf-8"))["questions"]
ranks = []
for entry in questions:
payload = okf_consume.build_payload(
DEFAULT_BUNDLE, question=entry["question"], source_quota=None
)
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
ranks.append(okf_consume_measure.hit_rank(excerpts, entry["gold_document"]))
assert ranks[5] is None, "the known-negative stopped being negative"
assert tuple(ranks[:5]) == EXPECTED_RANKS[:5]
@requires_default_bundle
def test_the_stem_rule_is_what_keeps_bilateral_out_of_four_hundred_concepts() -> None:
"""The known-negative for the stem rule, on the shipped bundle.

133
tests/test_okf_consume.py
View file

@ -22,6 +22,7 @@ house pattern rather than new inventions:
from __future__ import annotations
import hashlib
import inspect
import json
import os
import re
@ -946,7 +947,13 @@ def test_every_gold_document_in_the_local_set_is_reached_or_named_as_a_miss() ->
hits += 1
# The published bar, and the published number. A regression that drops a
# row goes red here rather than in a document nobody re-runs.
assert hits == 5, f"hit@8 moved: {hits} of {len(questions)}"
#
# 5 -> 6 ON 2026-09-10, with no bundle changing: `DEFAULT_SOURCE_QUOTA = 2`
# reaches the one row that had missed everywhere. What that gain is not:
# this metric asks whether the gold DOCUMENT was delivered, and a document
# quota raises how many distinct documents a payload holds, so it is not
# neutral with respect to the rule that moved it.
assert hits == 6, f"hit@8 moved: {hits} of {len(questions)}"
# --- Step 9: the CLI ----------------------------------------------------------
@ -1180,13 +1187,133 @@ def test_no_corpus_document_name_reaches_any_file_this_work_tracks() -> None:
assert leak.findall(path.read_text(encoding="utf-8")) == [], path
def _quota_concept(concept_id: str, *, source_file: str) -> okf_consume.Concept:
"""A minimal concept whose only interesting property is its source document."""
return okf_consume.Concept(
path=Path(concept_id),
concept_id=concept_id,
bundle_id="quota-fixture",
bundle_id_inherited=False,
sha256="0" * 64,
okf_type="Krav",
title=concept_id,
source_file=source_file,
adjudication="unknown",
adjudication_present=False,
req_number="",
sources=(),
sources_present=False,
locators={},
frontmatter={},
body="alpha beta gamma",
)
def test_a_source_quota_caps_how_many_places_one_document_takes() -> None:
"""One source document taking most of the payload is a MEASURED defect.
Measured outside this repo on a 3206-concept bundle of a published
handbook: the code's own process overview contributes 28 of 3206 concepts
(0.87 %) and 8.0 % of the source characters, and takes 22 of 43 delivered
places on one question and 24 of 45 on the known-positive. Identical at
343 and 1651 concepts, so it is the corpus's COMPOSITION -- that it holds
its own table of contents -- and not its size.
The quota cuts where the shortlist is cut, so `k` is still delivered in
full: a concept over quota is withheld by NAME and the place goes to the
next candidate.
"""
concepts = tuple(
_quota_concept(f"c{index}", source_file="o.md" if index < 6 else f"x{index}.md")
for index in range(10)
)
ranked = tuple((concept, 1.0 / (index + 1), 3) for index, concept in enumerate(concepts))
delivered, withheld, _ = okf_consume.cut(
ranked, k=4, limit=okf_consume.DEFAULT_LIMIT, source_quota=2
)
ids = [str(entry["concept_id"]) for entry in delivered]
assert [cid for cid in ids if cid in {"c0", "c1", "c2", "c3", "c4", "c5"}] == ["c0", "c1"]
assert len(delivered) == 4, "the freed places are filled, never left empty"
assert ("c2", "source_quota_exceeded") in withheld
assert len(withheld) + len(delivered) == len(concepts)
def test_the_quota_never_shortens_a_single_document_bundle_s_payload() -> None:
"""The adverse case, recorded rather than discovered by a consumer.
Every concept in a one-document bundle shares a `source_file`, so a quota
applied literally would deliver `source_quota` excerpts where `k` were
asked for. The top-up takes the best-ranked over-quota candidates back, so
such a bundle is byte-identical to the quota being off.
"""
concepts = tuple(_quota_concept(f"c{index}", source_file="only.md") for index in range(10))
ranked = tuple((concept, 1.0 / (index + 1), 3) for index, concept in enumerate(concepts))
with_quota, _, _ = okf_consume.cut(ranked, k=4, limit=okf_consume.DEFAULT_LIMIT, source_quota=2)
without, _, _ = okf_consume.cut(ranked, k=4, limit=okf_consume.DEFAULT_LIMIT)
assert len(with_quota) == 4
assert with_quota == without
def test_the_quota_has_an_explicit_opt_out_that_restores_the_old_order() -> None:
"""`None` is the opt-out, and it must reproduce the pre-round-11 cut.
The default is 2, so this is the direction that needs proving: a consumer
pinned to the previous excerpt order passes `--no-source-quota` and gets
exactly what it got before, including no entry under the new rule.
"""
concepts = tuple(
_quota_concept(f"c{index}", source_file="o.md" if index < 6 else f"x{index}.md")
for index in range(10)
)
ranked = tuple((concept, 1.0 / (index + 1), 3) for index, concept in enumerate(concepts))
delivered, withheld, _ = okf_consume.cut(
ranked, k=4, limit=okf_consume.DEFAULT_LIMIT, source_quota=None
)
assert [str(entry["concept_id"]) for entry in delivered] == ["c0", "c1", "c2", "c3"]
assert all(rule != "source_quota_exceeded" for _, rule in withheld)
def test_the_pre_pass_signature_defaults_are_the_consume_command_defaults() -> None:
"""One flag, one default -- the O6 defect class, on the reading side.
`build_payload` is called as a Python function by the pin tests, the skill
generator and any consumer that imports this package; `okf consume` is
called by everyone else. When a flag's argparse default and its signature
default disagree, those two populations get different payloads from the
same version, and every measurement report is pinned to whichever one the
measuring script happened to use. `okf project` shipped that defect for two
rounds before O6 measured it.
"""
parsed = okf_consume.parse_args(["bundle", "--question", "q"])
signature = inspect.signature(okf_consume.build_payload)
disagreeing = {
name: (parameter.default, getattr(parsed, name))
for name, parameter in signature.parameters.items()
if hasattr(parsed, name)
and parameter.default is not inspect.Parameter.empty
and getattr(parsed, name) != parameter.default
}
assert disagreeing == {}
def test_the_quota_default_is_two_and_the_default_is_a_measurement() -> None:
"""The published default, held by a test that goes red if it drifts.
Swept over {2, 3, 4, off} on three bundles: 2 and 3 take hit@8 from 5 of 6
to 6 of 6 on both K2 bundles with all five standing rank-1 rows unmoved, 4
and off leave it at 5 of 6. 2 rather than 3 on rank -- the recovered rows
come in at 5 and 4 rather than 7 and 5.
"""
assert okf_consume.DEFAULT_SOURCE_QUOTA == 2
def test_the_readme_consume_section_states_the_rule_count_the_code_emits() -> None:
# A published number must have a test that goes red when it goes false.
readme = (PROJECT_ROOT / "README.md").read_text(encoding="utf-8")
assert readme.count("## Consume\n") == 1
assert readme.count("## Consume in Claude Code\n") == 1
assert len(okf_consume.WITHHOLDING_RULES) == 6
assert "closed set of six" in readme
assert len(okf_consume.WITHHOLDING_RULES) == 7
assert "closed set of seven" in readme
assert "tools/okf_consume.py" in readme