open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(consume): the payload says what of the question it reached, and row 4 reads it

Browse source 
`coverage` carries three lists: the terms the pre-pass read the question as,
the terms no concept in the bundle answers, and the terms no delivered
excerpt answers. Without it a reader holding eight excerpts cannot tell a
bundle that ANSWERED its question from one that merely ranked something --
the two payloads have the same shape.

FACTS, AND NO VERDICT, which is a measurement and not caution. Two readings
were built and both falsified over 81 questions (16 synthetic, 65 across the
three real sets, 2026-09-20): the share of question terms a delivered
excerpt answers separates the synthetic controls at 0.33 against 0.50 and
REVERSES on real data (covered questions down to 0.27, one genuinely
uncovered question at 0.71); the share of a bundle tying the best lexical
match is ~0.00 for every real question either way. Question style dominates
the first, corpus size the second.

The one bar this repository declares is the gate's: `UNANSWERED_BAR = 2/3`
over `unanswered_in_bundle`, swept and collapsing at both ends -- at 0.50
eleven real covered questions are marked, at 0.70 the row falls to 5 of 6,
at 2/3 the row is 6 of 6 and 0 of 65 real questions are marked. The margin
is thin (0.6087 against 0.6667) and is published that way, together with
what it does not catch: r761-sk2's own known-negative sits at 0.2857.

Row 4: 3 of 6 RED -> 6 of 6 GREEN, with the 10 answered synthetic questions
held unmarked as the known-negative. The contract's SS 8 gains point 7, the
consumption skill is told to read the block, and the SS 7.4 known-positive
moves with the document (14 721/375 -> 16 389/417). Suite 2292 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-20 08:44:06 +02:00
commit 05cb19087a
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
8 changed files with 216 additions and 42 deletions
Download patch file Download diff file Expand all files Collapse all files

100
src/llm_ingestion_okf/consume.py
View file

@ -661,14 +661,14 @@ KNOWN_POSITIVE_CASE = "docs/consumption-contract.md, encoded as a JSON string"
#: `measure()`'s own answer for that file. Vacuous ALONE -- which is why the
#: delta below exists.
KNOWN_POSITIVE_EXPECTED = 14_721
KNOWN_POSITIVE_EXPECTED = 16_389
#: The second, independent route. `wc -c` reports 14 346 raw bytes for the same
#: The second, independent route. `wc -c` reports 15 972 raw bytes for the same
#: file; the difference is this file's JSON quoting and escaping overhead. A
#: reader can derive it without running `measure()` at all, and it moves the
#: moment `measure()` changes what it counts -- which is what stops
#: `expected == measured` from proving nothing.
KNOWN_POSITIVE_ENCODING_DELTA = 375
KNOWN_POSITIVE_ENCODING_DELTA = 417
#: The two places that file can be, resolved in this order.
#:
@ -2091,6 +2091,58 @@ def root_bundle_id_of(bundle_root: Path, *, profile: BundleProfile = DEFAULT_PRO
return declared
def excerpt_text(excerpt: Mapping[str, object]) -> str:
"""The strings a delivered excerpt actually shows its reader.
Title, id and body, which is the same union the two lexical signals score
-- a coverage report counting a term the consumer cannot see would be a
report about the ranker rather than about the payload.
"""
return " ".join(str(excerpt.get(key, "")) for key in ("title", "concept_id", "text"))
def unanswered_terms(
terms: Sequence[str],
tokenised: Sequence[Sequence[str]],
*,
stems: frozenset[str] | None,
) -> list[str]:
"""The question terms none of `tokenised` answers, in the question's order.
FACTS, NEVER A VERDICT, and that is a measurement rather than caution. Two
readings were built and both were falsified against 81 questions (16
synthetic, 65 over the three real sets, 2026-09-20):
- the SHARE of question terms a delivered excerpt answers separates the
synthetic controls at 0.33 against 0.50 and then reverses on real data,
where covered questions run down to 0.27 and one real uncovered question
sits at 0.71. Question STYLE dominates it: a question carrying its own
instructions has a long tail of words no corpus answers.
- the share of the bundle that ties the best lexical match separates the
synthetic corpus and is ~0.00 for every real question, covered or not,
because a 4 369-concept bundle has no ties to speak of.
So this module states what it measured and stops there. A bar over these
lists is the READER's -- a model weighing "the bundle answers none of
`koster`, `doegn`, `gjester`" is doing something no constant here can do
across corpora, and `tools/okf_retrieval_gate.py`'s row 4 carries the one
bar this repository does declare, with its own measurement beside it.
The matcher is the ranker's own (`tokens_match`, same `stems`). The cost
vocabulary's bridge is NOT applied: it is off by default and it is a rule
about documents naming money in other words, which would make a term read
as answered by a text that never carries it.
"""
remaining = set(terms)
for tokens in tokenised:
if not remaining:
break
for term in tuple(remaining):
if any(tokens_match(term, other, stems=stems) for other in tokens):
remaining.discard(term)
return [term for term in terms if term in remaining]
def build_payload(
bundle_root: Path,
*,
@ -2165,28 +2217,17 @@ def build_payload(
for concept_id in concept_ids
]
)
# The text the two lexical signals read, tokenised ONCE: the stem
# vocabulary, the rarity `df` and the coverage report below all count over
# the same strings, so none of them can weigh a token by how rare it is
# somewhere it is not read.
texts = searchable_text(concepts, link_in_signal=link_in_signal)
tokenised = [normalise(text) for text in texts]
# The bundle's OWN vocabulary, and the reason the rule is a set rather than
# a threshold: `pris` is a word here and `bila` is not, which is what
# separates a Norwegian compound from four coincidental characters. One
# pass, over the same text the ranking reads.
stems = (
frozenset(
token
for text in searchable_text(concepts, link_in_signal=link_in_signal)
for token in normalise(text)
)
if stem_prefix
else None
)
weights = (
rarity_weights(
normalise(question),
searchable_text(concepts, link_in_signal=link_in_signal),
stems=stems,
)
if rarity_weight
else None
)
# separates a Norwegian compound from four coincidental characters.
stems = frozenset(token for tokens in tokenised for token in tokens) if stem_prefix else None
weights = rarity_weights(normalise(question), texts, stems=stems) if rarity_weight else None
ranked = concept_scores(
concepts,
question,
@ -2238,6 +2279,7 @@ def build_payload(
# one-sided, and kept because SS 7.3 is a MUST about the emitted
# payload rather than about the algorithm that produced it.
raise ConsumeError(f"spent ({spent}) exceeds limit ({limit})", code="budget_exceeded")
question_terms = list(dict.fromkeys(normalise(question)))
return {
"contract": CONTRACT_REVISION,
"bundle": {
@ -2275,6 +2317,18 @@ def build_payload(
"delivered": len(delivered),
},
"question": question,
# SS 8: what of the QUESTION this payload reaches. Facts, and no
# verdict -- see `unanswered_terms` for the two readings that were
# measured and felled.
"coverage": {
"question_terms": question_terms,
"unanswered_in_bundle": unanswered_terms(question_terms, tokenised, stems=stems),
"unanswered_in_payload": unanswered_terms(
question_terms,
[normalise(excerpt_text(excerpt)) for excerpt in delivered],
stems=stems,
),
},
"excerpts": list(delivered),
# Emitted only under the flag, and then only where the concept carries
# a title, so a bundle whose concepts have none produces the same bytes