open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): one source document took 8 of 8 delivered places, so cap it

Browse source 
Measured outside this repository on a 3206-concept bundle of a published
handbook: the code's own process overview contributes 28 of 3206 concepts
(0.87 %) and 117 488 of 1 469 225 source characters (8.0 %), and took 8 of 8
delivered places on one question and 7 of 8 on the known-positive -- which was
not delivered at all. Identical at 343 and 1651 concepts, so the cause is the
corpus's COMPOSITION, that it holds its own table of contents, and NOT its size.
Splitting the corpus would move the defect, not remove it: any corpus with a
contents list, a project overview or a summary document has the same property.

`--source-quota N` caps how many DELIVERED places one source document may take.
It cuts where the shortlist is cut -- before the pack, never inside the DP,
which maximises a sum over a set it is handed -- so the freed place goes to the
next candidate and `k` is still delivered in full.

DEFAULT 2 SINCE TODAY, and it is the third change here that alters a payload
with NO bundle changing (after `--tie-shared-rank` and `--stem-prefix`).
Opt-out `--no-source-quota` reproduces the previous excerpt order.

Swept over {2, 3, 4, off} on three bundles, with the fasit prefixes validated
against the bundle FIRST (that control caught a defect in the measuring query
itself -- it read the last id segment where the document is the first):

- K2, both bundles: at 2 and 3, hit@8 goes 5 of 6 to 6 of 6 with all five
  standing rank-1 rows unmoved. The recovered row had missed on every bundle and
  every configuration measured until now. At 4 and off it is 5 of 6.
- The handbook bundle: hit@8 2 of 6 -> 4 of 6, the known-positive from not
  delivered to rank 4, and the dominant document's share of delivered places
  8 of 8 -> 2 of 8 (7 of 8 -> 2 of 8 on the known-positive).
- 2 rather than 3 on rank alone: the recovered rows come in at 5 and 4 rather
  than 7 and 5.

WHAT THE GAIN IS NOT. hit@8 asks whether the gold DOCUMENT appears among the
delivered excerpts, and a document quota directly raises how many distinct
documents a payload holds, so that metric is not neutral with respect to this
rule. The five rows that were already rank 1 are neutral, and they did not move.

THE ADVERSE CASE IS NAMED, not left to a consumer. A bundle built from ONE
document carries the same `source_file` on every concept, so a quota applied
literally would deliver 2 excerpts where `k` were asked for -- a rule against
dominance turned into a rule against small bundles. The shortlist is topped back
up from the best-ranked over-quota candidates, which makes such a bundle
byte-identical to the quota being off, and a test holds it.

`--rarity-weight` was measured against the same defect and does NOT repair it:
it leaves the dominant document at 8 of 8 places on the question it floods,
delivers neither that answer nor the known-positive, and holds 5 of 6 on both K2
bundles. Combined with the quota it is worse than the quota alone (the
known-positive falls back out). It stays off.

The vocabulary stays CLOSED and the new code is published in all three places a
consumer can read it: `WITHHOLDING_RULES` (six -> seven),
`docs/consumption-contract.md` 5.3, and the generated SKILL.md -- verified by
reading the generated file, not the code that writes it. `source_quota_exceeded`
is a DIVERSITY drop and not a relevance one, so folding it into
`no_lexical_match` would tell a consumer the question reached nothing in a
concept the question in fact reached. `okf check --skill --payload` stays
conformant, 0 findings over 15 rules.

Editing the contract moved the 7.4 known-positive, which is the coupling
working as intended: 12 563 -> 13 238 encoded, 12 227 -> 12 893 raw, delta
336 -> 345, updated in the constant, the instantiated skill and the shipped
example payload.

Also adds the O6 guard on the reading side: `build_payload`'s signature defaults
are asserted equal to `okf consume`'s argparse defaults for every same-named
parameter. `okf project` shipped that exact disagreement for two rounds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 00:29:02 +02:00
commit 1e9f38b125
9 changed files with 380 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

133
tests/test_okf_consume.py
View file

@ -22,6 +22,7 @@ house pattern rather than new inventions:
from __future__ import annotations
import hashlib
import inspect
import json
import os
import re
@ -946,7 +947,13 @@ def test_every_gold_document_in_the_local_set_is_reached_or_named_as_a_miss() ->
hits += 1
# The published bar, and the published number. A regression that drops a
# row goes red here rather than in a document nobody re-runs.
assert hits == 5, f"hit@8 moved: {hits} of {len(questions)}"
#
# 5 -> 6 ON 2026-09-10, with no bundle changing: `DEFAULT_SOURCE_QUOTA = 2`
# reaches the one row that had missed everywhere. What that gain is not:
# this metric asks whether the gold DOCUMENT was delivered, and a document
# quota raises how many distinct documents a payload holds, so it is not
# neutral with respect to the rule that moved it.
assert hits == 6, f"hit@8 moved: {hits} of {len(questions)}"
# --- Step 9: the CLI ----------------------------------------------------------
@ -1180,13 +1187,133 @@ def test_no_corpus_document_name_reaches_any_file_this_work_tracks() -> None:
assert leak.findall(path.read_text(encoding="utf-8")) == [], path
def _quota_concept(concept_id: str, *, source_file: str) -> okf_consume.Concept:
"""A minimal concept whose only interesting property is its source document."""
return okf_consume.Concept(
path=Path(concept_id),
concept_id=concept_id,
bundle_id="quota-fixture",
bundle_id_inherited=False,
sha256="0" * 64,
okf_type="Krav",
title=concept_id,
source_file=source_file,
adjudication="unknown",
adjudication_present=False,
req_number="",
sources=(),
sources_present=False,
locators={},
frontmatter={},
body="alpha beta gamma",
)
def test_a_source_quota_caps_how_many_places_one_document_takes() -> None:
"""One source document taking most of the payload is a MEASURED defect.
Measured outside this repo on a 3206-concept bundle of a published
handbook: the code's own process overview contributes 28 of 3206 concepts
(0.87 %) and 8.0 % of the source characters, and takes 22 of 43 delivered
places on one question and 24 of 45 on the known-positive. Identical at
343 and 1651 concepts, so it is the corpus's COMPOSITION -- that it holds
its own table of contents -- and not its size.
The quota cuts where the shortlist is cut, so `k` is still delivered in
full: a concept over quota is withheld by NAME and the place goes to the
next candidate.
"""
concepts = tuple(
_quota_concept(f"c{index}", source_file="o.md" if index < 6 else f"x{index}.md")
for index in range(10)
)
ranked = tuple((concept, 1.0 / (index + 1), 3) for index, concept in enumerate(concepts))
delivered, withheld, _ = okf_consume.cut(
ranked, k=4, limit=okf_consume.DEFAULT_LIMIT, source_quota=2
)
ids = [str(entry["concept_id"]) for entry in delivered]
assert [cid for cid in ids if cid in {"c0", "c1", "c2", "c3", "c4", "c5"}] == ["c0", "c1"]
assert len(delivered) == 4, "the freed places are filled, never left empty"
assert ("c2", "source_quota_exceeded") in withheld
assert len(withheld) + len(delivered) == len(concepts)
def test_the_quota_never_shortens_a_single_document_bundle_s_payload() -> None:
"""The adverse case, recorded rather than discovered by a consumer.
Every concept in a one-document bundle shares a `source_file`, so a quota
applied literally would deliver `source_quota` excerpts where `k` were
asked for. The top-up takes the best-ranked over-quota candidates back, so
such a bundle is byte-identical to the quota being off.
"""
concepts = tuple(_quota_concept(f"c{index}", source_file="only.md") for index in range(10))
ranked = tuple((concept, 1.0 / (index + 1), 3) for index, concept in enumerate(concepts))
with_quota, _, _ = okf_consume.cut(ranked, k=4, limit=okf_consume.DEFAULT_LIMIT, source_quota=2)
without, _, _ = okf_consume.cut(ranked, k=4, limit=okf_consume.DEFAULT_LIMIT)
assert len(with_quota) == 4
assert with_quota == without
def test_the_quota_has_an_explicit_opt_out_that_restores_the_old_order() -> None:
"""`None` is the opt-out, and it must reproduce the pre-round-11 cut.
The default is 2, so this is the direction that needs proving: a consumer
pinned to the previous excerpt order passes `--no-source-quota` and gets
exactly what it got before, including no entry under the new rule.
"""
concepts = tuple(
_quota_concept(f"c{index}", source_file="o.md" if index < 6 else f"x{index}.md")
for index in range(10)
)
ranked = tuple((concept, 1.0 / (index + 1), 3) for index, concept in enumerate(concepts))
delivered, withheld, _ = okf_consume.cut(
ranked, k=4, limit=okf_consume.DEFAULT_LIMIT, source_quota=None
)
assert [str(entry["concept_id"]) for entry in delivered] == ["c0", "c1", "c2", "c3"]
assert all(rule != "source_quota_exceeded" for _, rule in withheld)
def test_the_pre_pass_signature_defaults_are_the_consume_command_defaults() -> None:
"""One flag, one default -- the O6 defect class, on the reading side.
`build_payload` is called as a Python function by the pin tests, the skill
generator and any consumer that imports this package; `okf consume` is
called by everyone else. When a flag's argparse default and its signature
default disagree, those two populations get different payloads from the
same version, and every measurement report is pinned to whichever one the
measuring script happened to use. `okf project` shipped that defect for two
rounds before O6 measured it.
"""
parsed = okf_consume.parse_args(["bundle", "--question", "q"])
signature = inspect.signature(okf_consume.build_payload)
disagreeing = {
name: (parameter.default, getattr(parsed, name))
for name, parameter in signature.parameters.items()
if hasattr(parsed, name)
and parameter.default is not inspect.Parameter.empty
and getattr(parsed, name) != parameter.default
}
assert disagreeing == {}
def test_the_quota_default_is_two_and_the_default_is_a_measurement() -> None:
"""The published default, held by a test that goes red if it drifts.
Swept over {2, 3, 4, off} on three bundles: 2 and 3 take hit@8 from 5 of 6
to 6 of 6 on both K2 bundles with all five standing rank-1 rows unmoved, 4
and off leave it at 5 of 6. 2 rather than 3 on rank -- the recovered rows
come in at 5 and 4 rather than 7 and 5.
"""
assert okf_consume.DEFAULT_SOURCE_QUOTA == 2
def test_the_readme_consume_section_states_the_rule_count_the_code_emits() -> None:
# A published number must have a test that goes red when it goes false.
readme = (PROJECT_ROOT / "README.md").read_text(encoding="utf-8")
assert readme.count("## Consume\n") == 1
assert readme.count("## Consume in Claude Code\n") == 1
assert len(okf_consume.WITHHOLDING_RULES) == 6
assert "closed set of six" in readme
assert len(okf_consume.WITHHOLDING_RULES) == 7
assert "closed set of seven" in readme
assert "tools/okf_consume.py" in readme