open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): measure the below-k blind spot, add one flag-gated vocabulary bridge

Browse source 
The consumer report (portfolio-optimiser, S7 SS 2) found that a mandate-shaped
cost question withheld the corpus's one priced table under `below_k`. Measured
here, on a bundle proven byte-identical to a fresh HEAD rebuild:

- The mechanism is a VOCABULARY gap, not a `k` defect: two of three ranking
  signals are exactly 0.0 and the concept is candidate 249 of 269.
- The k-sweep buys nothing: k in {8,12,16,24,32,64,128} all withhold it, at
  +9.5 % tokens. It also found a regression -- for the question that WORKS,
  k >= 16 EVICTS the gold concept, because one 67 838 B excerpt is 56.5 % of
  the budget and the knapsack maximises a sum.
- Two proposed rules were falsified BEFORE any code: number/table density ranks
  the priced table 178/165/46 of 269 (the form is unfilled, so it is
  number-poor), and per-document spread puts its document 30th of 35.

Built instead, behind `--cost-vocabulary` (default OFF, DEFAULT byte-identical):
one declared cost/price/quantity vocabulary family that bridges a question and a
document naming money with different words. It moves the concept from candidate
rank 249 to 10 -- and does NOT close the blind spot: the budget still refuses
it, which is now a separately measured second lock.

Seven RED tests first; six mutations of the rule, six red (two survived the
first version of the tests and the tests were strengthened). Control: a question
with no cost term produces a byte-identical payload with the flag on, at every
k, on the real corpus. Known-positive: 164 987 B / 40 425 o200k tokens, equal to
the published pair.

Report: docs/2026-09-08-blindsone-below-k-k2.md
Suite 1268 green, mypy --strict clean over 28 files, both goldens unchanged.

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 04:55:45 +02:00
commit 4c699fdbb1
5 changed files with 616 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

111
tools/okf_consume.py
View file

@ -547,18 +547,77 @@ def tokens_match(left: str, right: str) -> bool:
return shared >= MIN_SHARED_PREFIX
def _overlap(question_tokens: Sequence[str], candidate: str) -> int:
#: One declared vocabulary family, spelled once: within it, any term answers to
#: any other. OFF by default and reachable only through `--cost-vocabulary`.
#:
#: WHAT IT IS FOR, and the failure it addresses. Measured 2026-09-08 over a
#: 629-concept corpus: a mandate-shaped question about cost ranked that
#: corpus's one priced table 249th of 269 lexical candidates, because the
#: question said `kostnadsbesparelser` and the document said `pris` -- two
#: words with no shared prefix. Its title score and its document score were
#: both 0. No value of `k` closes a gap in the VOCABULARY.
#:
#: WHAT IT IS NOT. Each member must be at least `MIN_SHARED_PREFIX` characters
#: or it can never match anything (`sum` is 3 and does not match `Summen`; it
#: was dropped for that reason, not by taste). The bridge is symmetric and
#: needs a family term on BOTH sides, so it can widen a cost question towards a
#: cost document and never towards an arbitrary one. Norwegian, and stated as
#: such: a corpus in another language gets nothing from it.
#:
#: HONESTY, measured rather than asserted: on that corpus the whole effect
#: rests on `kost` and `pris`. Removing either returns the priced table to rank
#: 249; removing any other member moves it not at all, and three members match
#: nothing in that corpus. They are kept because dropping a term for being
#: absent from ONE corpus fits the list to that corpus.
COST_VOCABULARY = (
"beløp",
"budsjett",
"enhet",
"honorar",
"kost",
"kroner",
"mengde",
"pris",
"utgift",
"vederlag",
)
def in_cost_vocabulary(token: str) -> bool:
"""Whether one token belongs to the declared family, by the same prefix rule."""
return any(tokens_match(token, member) for member in COST_VOCABULARY)
def question_uses_cost_vocabulary(question: str) -> bool:
"""Whether the QUESTION opens the bridge. The gate is the question, never the flag.
A question naming no term in the family gets byte-identical bytes with the
flag set, which is what keeps the flag a widening of one question class
rather than a second ranker.
"""
return any(in_cost_vocabulary(token) for token in normalise(question))
def _overlap(
question_tokens: Sequence[str], candidate: str, *, cost_vocabulary: bool = False
) -> int:
"""How many of the question's tokens the candidate text answers to."""
candidate_tokens = normalise(candidate)
bridged = cost_vocabulary and any(in_cost_vocabulary(token) for token in candidate_tokens)
return sum(
1
for token in question_tokens
if any(tokens_match(token, other) for other in candidate_tokens)
or (bridged and in_cost_vocabulary(token))
)
def document_scores(
bundle_root: Path, question: str, *, profile: BundleProfile = DEFAULT_PROFILE
bundle_root: Path,
question: str,
*,
profile: BundleProfile = DEFAULT_PROFILE,
cost_vocabulary: bool = False,
) -> dict[str, float]:
"""One score per top-level document, from the indexes and the paths alone.
@ -588,6 +647,7 @@ def document_scores(
in this command (SS 9.2).
"""
question_tokens = normalise(question)
bridge = cost_vocabulary and question_uses_cost_vocabulary(question)
indexes, concepts = _walk_index_tree(bundle_root, profile=profile)
totals: dict[str, float] = {}
units: dict[str, int] = {}
@ -597,7 +657,10 @@ def document_scores(
units[document] = units.get(document, 0) + 1
for concept_id in concepts:
record(concept_id.split("/", 1)[0], _overlap(question_tokens, concept_id.replace("/", " ")))
record(
concept_id.split("/", 1)[0],
_overlap(question_tokens, concept_id.replace("/", " "), cost_vocabulary=bridge),
)
for relative in indexes:
document = relative.split("/", 1)[0]
if document == profile.index.name:
@ -606,7 +669,7 @@ def document_scores(
entry = profile.index.parse_entry(line)
if entry is None:
continue
record(document, _overlap(question_tokens, entry.label))
record(document, _overlap(question_tokens, entry.label, cost_vocabulary=bridge))
return {document: totals[document] / units[document] for document in totals}
@ -628,6 +691,8 @@ def concept_scores(
concepts: Sequence[Concept],
question: str,
document_score: Mapping[str, float],
*,
cost_vocabulary: bool = False,
) -> list[tuple[Concept, float, int]]:
"""Every concept, ordered best first, fused from three signals by RRF.
@ -650,15 +715,22 @@ def concept_scores(
must not deliver.
"""
question_tokens = normalise(question)
bridge = cost_vocabulary and question_uses_cost_vocabulary(question)
signals: list[dict[str, float]] = [
{
concept.concept_id: float(
_overlap(question_tokens, f"{concept.title} {concept.concept_id.replace('/', ' ')}")
_overlap(
question_tokens,
f"{concept.title} {concept.concept_id.replace('/', ' ')}",
cost_vocabulary=bridge,
)
)
for concept in concepts
},
{
concept.concept_id: float(_overlap(question_tokens, concept.body))
concept.concept_id: float(
_overlap(question_tokens, concept.body, cost_vocabulary=bridge)
)
for concept in concepts
},
{
@ -858,12 +930,13 @@ def build_payload(
k: int = DEFAULT_K,
limit: int = DEFAULT_LIMIT,
profile: BundleProfile = DEFAULT_PROFILE,
cost_vocabulary: bool = False,
) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload.
Pure with respect to the clock and the network: the same
`(bundle_root, question, k, limit)` at the same bytes returns the same
object, every time.
`(bundle_root, question, k, limit, cost_vocabulary)` at the same bytes
returns the same object, every time.
"""
case, expected, measured = known_positive()
if expected != measured:
@ -902,7 +975,10 @@ def build_payload(
for concept_id in concept_ids
]
ranked = concept_scores(
concepts, question, document_scores(bundle_root, question, profile=profile)
concepts,
question,
document_scores(bundle_root, question, profile=profile, cost_vocabulary=cost_vocabulary),
cost_vocabulary=cost_vocabulary,
)
matched = sum(1 for _, _, lexical in ranked if lexical > 0)
delivered, withheld = cut(ranked, k=k, limit=limit)
@ -987,6 +1063,15 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
default=DEFAULT_LIMIT,
help=f"budget in {BUDGET_UNIT} (default {DEFAULT_LIMIT})",
)
parser.add_argument(
"--cost-vocabulary",
action="store_true",
help=(
"let the declared cost/price/quantity vocabulary bridge a question "
"and a document that share no word. OFF by default; a question "
"naming no term in that vocabulary is unaffected either way"
),
)
parser.add_argument("--out", type=Path, default=None, help="write here instead of stdout")
parser.add_argument(
"--ref",
@ -1013,7 +1098,13 @@ def main(argv: list[str] | None = None) -> int:
print(f"okf_consume: FAILED - {args.bundle} is not a directory", file=sys.stderr)
return 2
try:
payload = build_payload(args.bundle, question=args.question, k=args.k, limit=args.limit)
payload = build_payload(
args.bundle,
question=args.question,
k=args.k,
limit=args.limit,
cost_vocabulary=args.cost_vocabulary,
)
except ConsumeError as error:
print(f"okf_consume: FAILED - {error}", file=sys.stderr)
return 1