open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): one source document took 8 of 8 delivered places, so cap it

Browse source 
Measured outside this repository on a 3206-concept bundle of a published
handbook: the code's own process overview contributes 28 of 3206 concepts
(0.87 %) and 117 488 of 1 469 225 source characters (8.0 %), and took 8 of 8
delivered places on one question and 7 of 8 on the known-positive -- which was
not delivered at all. Identical at 343 and 1651 concepts, so the cause is the
corpus's COMPOSITION, that it holds its own table of contents, and NOT its size.
Splitting the corpus would move the defect, not remove it: any corpus with a
contents list, a project overview or a summary document has the same property.

`--source-quota N` caps how many DELIVERED places one source document may take.
It cuts where the shortlist is cut -- before the pack, never inside the DP,
which maximises a sum over a set it is handed -- so the freed place goes to the
next candidate and `k` is still delivered in full.

DEFAULT 2 SINCE TODAY, and it is the third change here that alters a payload
with NO bundle changing (after `--tie-shared-rank` and `--stem-prefix`).
Opt-out `--no-source-quota` reproduces the previous excerpt order.

Swept over {2, 3, 4, off} on three bundles, with the fasit prefixes validated
against the bundle FIRST (that control caught a defect in the measuring query
itself -- it read the last id segment where the document is the first):

- K2, both bundles: at 2 and 3, hit@8 goes 5 of 6 to 6 of 6 with all five
  standing rank-1 rows unmoved. The recovered row had missed on every bundle and
  every configuration measured until now. At 4 and off it is 5 of 6.
- The handbook bundle: hit@8 2 of 6 -> 4 of 6, the known-positive from not
  delivered to rank 4, and the dominant document's share of delivered places
  8 of 8 -> 2 of 8 (7 of 8 -> 2 of 8 on the known-positive).
- 2 rather than 3 on rank alone: the recovered rows come in at 5 and 4 rather
  than 7 and 5.

WHAT THE GAIN IS NOT. hit@8 asks whether the gold DOCUMENT appears among the
delivered excerpts, and a document quota directly raises how many distinct
documents a payload holds, so that metric is not neutral with respect to this
rule. The five rows that were already rank 1 are neutral, and they did not move.

THE ADVERSE CASE IS NAMED, not left to a consumer. A bundle built from ONE
document carries the same `source_file` on every concept, so a quota applied
literally would deliver 2 excerpts where `k` were asked for -- a rule against
dominance turned into a rule against small bundles. The shortlist is topped back
up from the best-ranked over-quota candidates, which makes such a bundle
byte-identical to the quota being off, and a test holds it.

`--rarity-weight` was measured against the same defect and does NOT repair it:
it leaves the dominant document at 8 of 8 places on the question it floods,
delivers neither that answer nor the known-positive, and holds 5 of 6 on both K2
bundles. Combined with the quota it is worse than the quota alone (the
known-positive falls back out). It stays off.

The vocabulary stays CLOSED and the new code is published in all three places a
consumer can read it: `WITHHOLDING_RULES` (six -> seven),
`docs/consumption-contract.md` 5.3, and the generated SKILL.md -- verified by
reading the generated file, not the code that writes it. `source_quota_exceeded`
is a DIVERSITY drop and not a relevance one, so folding it into
`no_lexical_match` would tell a consumer the question reached nothing in a
concept the question in fact reached. `okf check --skill --payload` stays
conformant, 0 findings over 15 rules.

Editing the contract moved the 7.4 known-positive, which is the coupling
working as intended: 12 563 -> 13 238 encoded, 12 227 -> 12 893 raw, delta
336 -> 345, updated in the constant, the instantiated skill and the shipped
example payload.

Also adds the O6 guard on the reading side: `build_payload`'s signature defaults
are asserted equal to `okf consume`'s argparse defaults for every same-named
parameter. `okf project` shipped that exact disagreement for two rounds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 00:29:02 +02:00
commit 1e9f38b125
9 changed files with 380 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

97
src/llm_ingestion_okf/consume.py
View file

@ -593,14 +593,14 @@ KNOWN_POSITIVE_CASE = "docs/consumption-contract.md, encoded as a JSON string"
#: `measure()`'s own answer for that file. Vacuous ALONE -- which is why the
#: delta below exists.
KNOWN_POSITIVE_EXPECTED = 12_563
KNOWN_POSITIVE_EXPECTED = 13_238
#: The second, independent route. `wc -c` reports 12 227 raw bytes for the same
#: The second, independent route. `wc -c` reports 12 893 raw bytes for the same
#: file; the difference is this file's JSON quoting and escaping overhead. A
#: reader can derive it without running `measure()` at all, and it moves the
#: moment `measure()` changes what it counts -- which is what stops
#: `expected == measured` from proving nothing.
KNOWN_POSITIVE_ENCODING_DELTA = 336
KNOWN_POSITIVE_ENCODING_DELTA = 345
#: The two places that file can be, resolved in this order.
#:
@ -1335,8 +1335,31 @@ WITHHOLDING_RULES = (
"over_budget_alone",
"below_k",
"over_budget_after_knapsack",
"source_quota_exceeded",
)
#: `--source-quota N` caps how many DELIVERED places one source document may
#: take, topping the shortlist back up to `k` when the bundle has no
#: alternatives to offer.
#:
#: **2 SINCE 2026-09-10, and it is the third change here that moves a payload
#: with NO bundle changing** (after `--tie-shared-rank` and `--stem-prefix`);
#: a consumer pinned to the previous excerpt order needs `--no-source-quota`.
#: The defect it repairs was measured on a 3206-concept bundle of a published
#: handbook: the code's own process overview contributes 28 of 3206 concepts
#: (0.87 %) and 8.0 % of the source characters, and took 8 of 8 delivered
#: places on one question and 7 of 8 on the known-positive -- identical at 343
#: and 1651 concepts, so the cause is the corpus's COMPOSITION (it holds its
#: own table of contents) and not its size.
#:
#: SWEPT over {2, 3, 4, off} on three bundles. At 2 and 3 hit@8 goes 5 of 6 to
#: **6 of 6 on BOTH K2 bundles** with all five standing rank-1 rows unmoved; at
#: 4 and off it is 5 of 6. On the handbook bundle hit@8 goes 2 of 6 to 4 of 6
#: and the dominant document's share of delivered places 8 of 8 to 2 of 8. 2
#: rather than 3 on rank: the recovered rows come in at 5 and 4 rather than 7
#: and 5.
DEFAULT_SOURCE_QUOTA: int | None = 2
#: The knapsack's weight granularity, in bytes. Bucketing keeps the DP table
#: small; bucketing UP the item and DOWN the capacity keeps the error one-sided,
#: so the pack may under-deliver by a bucket and can never over-spend.
@ -1437,6 +1460,7 @@ def cut(
k: int,
limit: int,
reserve_top_rank: bool = False,
source_quota: int | None = DEFAULT_SOURCE_QUOTA,
) -> tuple[tuple[dict[str, object], ...], tuple[tuple[str, str], ...], tuple[str, int] | None]:
"""The ranked concepts split into delivered excerpts, named drops, and the
reservation that was made, if any.
@ -1486,6 +1510,40 @@ def cut(
withheld.append((concept.concept_id, "over_budget_alone"))
continue
candidates.append((concept, score, excerpt, weight))
if source_quota is not None:
# THE QUOTA CUTS WHERE THE SHORTLIST IS CUT, never inside the pack. The
# DP maximises a sum over a set it is handed; a quota expressed there
# would be a constraint on the sum, which is a different problem and a
# slower one. Here it is a filter on the ranked candidate list, so `k`
# is still delivered in full and the freed place goes to the next
# candidate rather than being lost.
seen: dict[str, int] = {}
keep = [True] * len(candidates)
over: list[int] = []
for index, entry in enumerate(candidates):
document = entry[0].source_file
taken = seen.get(document, 0)
if taken >= source_quota:
keep[index] = False
over.append(index)
continue
seen[document] = taken + 1
# THE QUOTA NEVER SHORTENS THE PAYLOAD, and that is not a nicety. A
# bundle built from ONE document carries the same `source_file` on
# every concept, so a quota applied without this would deliver
# `source_quota` excerpts instead of `k` -- a rule against dominance
# turned into a rule against small bundles. `over` is in fused-rank
# order, so the top-up takes the BEST-ranked over-quota candidates
# back, and a bundle with no alternatives to offer is byte-identical
# to the quota being off.
for index in over[: max(k - sum(keep), 0)]:
keep[index] = True
withheld.extend(
(candidates[index][0].concept_id, "source_quota_exceeded")
for index in range(len(candidates))
if not keep[index]
)
candidates = [entry for index, entry in enumerate(candidates) if keep[index]]
for concept, _, _, _ in candidates[k:]:
withheld.append((concept.concept_id, "below_k"))
shortlist = candidates[:k]
@ -1571,6 +1629,7 @@ def build_payload(
tie_shared_rank: bool = DEFAULT_TIE_SHARED_RANK,
withheld_titles: bool = False,
stem_prefix: bool = DEFAULT_STEM_PREFIX,
source_quota: int | None = DEFAULT_SOURCE_QUOTA,
) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload.
@ -1648,7 +1707,13 @@ def build_payload(
)
titles_by_id = {concept.concept_id: concept.title for concept in concepts}
matched = sum(1 for _, _, lexical in ranked if lexical > 0)
delivered, withheld, reserved = cut(ranked, k=k, limit=limit, reserve_top_rank=reserve_top_rank)
delivered, withheld, reserved = cut(
ranked,
k=k,
limit=limit,
reserve_top_rank=reserve_top_rank,
source_quota=source_quota,
)
spent = sum(excerpt_weight(excerpt) for excerpt in delivered)
if matched and not delivered:
# SS 7.3: a finding requiring a decision, never something to retry
@ -1818,6 +1883,29 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
dest="stem_prefix",
help="The rule's explicit opt-out, reproducing the pre-round-10 matcher",
)
parser.add_argument(
"--source-quota",
type=int,
default=DEFAULT_SOURCE_QUOTA,
metavar="N",
help=(
"cap how many DELIVERED places one source document may take, "
"filling the freed places from the next candidate so k is still "
"delivered in full. Default 2 since 2026-09-10. Measured on a "
"3206-concept bundle whose corpus holds its own table of contents: "
"that one document took 8 of 8 places and the answer was not "
"delivered at all; at 2 it takes 2 of 8 and the answer comes in at "
"rank 4. A bundle with no alternatives is unaffected -- the "
"shortlist is topped back up to k"
),
)
parser.add_argument(
"--no-source-quota",
action="store_const",
const=None,
dest="source_quota",
help="The rule's explicit opt-out, reproducing the pre-round-11 excerpt order",
)
parser.add_argument(
"--withheld-titles",
action="store_true",
@ -1864,6 +1952,7 @@ def main(argv: list[str] | None = None) -> int:
rarity_weight=args.rarity_weight,
tie_shared_rank=args.tie_shared_rank,
stem_prefix=args.stem_prefix,
source_quota=args.source_quota,
withheld_titles=args.withheld_titles,
)
except ConsumeError as error:

5
src/llm_ingestion_okf/skill.py
View file

@ -565,10 +565,13 @@ def _denominators(total: int, *, asked: str, delivered: int) -> str:
f"For this bundle `considered` is **{total}**, every concept the index walk\n"
"reaches, never the post-ranking shortlist. A concept dropped at the ranking\n"
"stage is `withheld` **with its rule**, not invisible, and the rules are a\n"
"closed set of six: `verdict_layer_excluded` (a verdict-layer file, § 9.1),\n"
"closed set of seven: `verdict_layer_excluded` (a verdict-layer file, § 9.1),\n"
"`verified_unreadable` (a `verified` value this reader cannot decode, so no\n"
"tier can be derived), `no_lexical_match` (the concept shares no token with\n"
"the question), `over_budget_alone` (one excerpt exceeds the whole limit),\n"
"`source_quota_exceeded` (its source document already holds as many\n"
"delivered places as `--source-quota` allows, default 2 — the freed place\n"
"goes to the next candidate, so `k` is still delivered in full),\n"
"`below_k` (ranked outside the shortlist the cut considers) and\n"
"`over_budget_after_knapsack` (it ranked inside the shortlist and the pack\n"
"had no room). Naming the rule is what makes a drop visible.\n\n"