feat(consume): the payload says what of the question it reached, and row 4 reads it
`coverage` carries three lists: the terms the pre-pass read the question as, the terms no concept in the bundle answers, and the terms no delivered excerpt answers. Without it a reader holding eight excerpts cannot tell a bundle that ANSWERED its question from one that merely ranked something -- the two payloads have the same shape. FACTS, AND NO VERDICT, which is a measurement and not caution. Two readings were built and both falsified over 81 questions (16 synthetic, 65 across the three real sets, 2026-09-20): the share of question terms a delivered excerpt answers separates the synthetic controls at 0.33 against 0.50 and REVERSES on real data (covered questions down to 0.27, one genuinely uncovered question at 0.71); the share of a bundle tying the best lexical match is ~0.00 for every real question either way. Question style dominates the first, corpus size the second. The one bar this repository declares is the gate's: `UNANSWERED_BAR = 2/3` over `unanswered_in_bundle`, swept and collapsing at both ends -- at 0.50 eleven real covered questions are marked, at 0.70 the row falls to 5 of 6, at 2/3 the row is 6 of 6 and 0 of 65 real questions are marked. The margin is thin (0.6087 against 0.6667) and is published that way, together with what it does not catch: r761-sk2's own known-negative sits at 0.2857. Row 4: 3 of 6 RED -> 6 of 6 GREEN, with the 10 answered synthetic questions held unmarked as the known-negative. The contract's SS 8 gains point 7, the consumption skill is told to read the block, and the SS 7.4 known-positive moves with the document (14 721/375 -> 16 389/417). Suite 2292 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
90394c383d
commit
05cb19087a
8 changed files with 216 additions and 42 deletions
|
|
@ -1157,21 +1157,68 @@ def row_three(cases: Sequence[Case]) -> Row:
|
|||
)
|
||||
|
||||
|
||||
#: Row 4's bar, and the ONE threshold this gate applies to a payload.
|
||||
#:
|
||||
#: SWEPT over 81 questions on 2026-09-20 -- the 16 of the synthetic sets and
|
||||
#: the 65 of the three real sets -- against `coverage.unanswered_in_bundle` as
|
||||
#: a share of the question's own terms:
|
||||
#:
|
||||
#: - at **0.50** row 4 is 6 of 6 and ELEVEN real questions whose fasit is in
|
||||
#: their bundle come back marked;
|
||||
#: - at **2/3** row 4 is 6 of 6 and **0 of 65** real questions are marked; the
|
||||
#: highest a real covered question reaches is 0.6087 (two of the vegnormal
|
||||
#: set's, 14 of 23 terms), and the lowest an uncovered control reaches is
|
||||
#: 0.6667 (N3, 4 of 6);
|
||||
#: - at **0.70** N3 falls under the bar and row 4 is 5 of 6.
|
||||
#:
|
||||
#: It collapses at both ends, which is what makes 2/3 a reading of the
|
||||
#: distribution rather than a number picked to pass. The margin is THIN --
|
||||
#: 0.6087 against 0.6667 -- and it is published that way.
|
||||
#:
|
||||
#: WHAT IT DOES NOT CATCH, measured on the same data: `r761-sk2`'s own
|
||||
#: known-negative, a question naming a section number the document does not
|
||||
#: have, sits at 0.2857 -- every other word of it is in a 2 756-concept road
|
||||
#: standard. One term absent among seven is not a share, and no bar over this
|
||||
#: list reaches it. Row 4's denominator is the synthetic controls, so this
|
||||
#: costs the row nothing and is stated rather than implied.
|
||||
UNANSWERED_BAR = 2 / 3
|
||||
|
||||
|
||||
def marked(payload: Mapping[str, object]) -> bool:
|
||||
"""Can a consumer READ, from the payload alone, that the bundle does not
|
||||
cover the question?
|
||||
|
||||
Today exactly one reading does that: nothing was delivered, and the
|
||||
withheld rules say why. Eight excerpts with no score, no confidence and no
|
||||
statement is not a marked answer -- it is the same payload an answered
|
||||
question gets. When the payload grows a field that says so, this function
|
||||
is where it is read, and the row moves with it.
|
||||
Two readings, and both are facts the payload carries:
|
||||
|
||||
- nothing was delivered, and the withheld rules say why; or
|
||||
- the bundle answers none of at least `UNANSWERED_BAR` of the question's
|
||||
own terms (`coverage.unanswered_in_bundle`, added to the payload
|
||||
2026-09-20 for exactly this reading).
|
||||
|
||||
Eight excerpts with no score, no confidence and no statement is not a
|
||||
marked answer -- it is the same payload an answered question gets, which
|
||||
is what took this row to 3 of 6 on N3, N4 and N5.
|
||||
|
||||
THE BAR IS THIS GATE'S AND NOT THE LIBRARY'S. `consume.unanswered_terms`
|
||||
records the two readings that were built and falsified against real
|
||||
corpora; a library asserting "this bundle does not cover your question"
|
||||
would be asserting across corpora what was measured on one. A gate is
|
||||
where a declared bar belongs, and this one carries its sweep above.
|
||||
"""
|
||||
counts = payload.get("denominators")
|
||||
assert isinstance(counts, dict)
|
||||
delivered = counts["delivered"]
|
||||
assert isinstance(delivered, int)
|
||||
return delivered == 0
|
||||
if delivered == 0:
|
||||
return True
|
||||
coverage = payload.get("coverage")
|
||||
assert isinstance(coverage, dict), "the payload carries no coverage block to read"
|
||||
terms = coverage["question_terms"]
|
||||
unanswered = coverage["unanswered_in_bundle"]
|
||||
assert isinstance(terms, list) and isinstance(unanswered, list)
|
||||
if not terms:
|
||||
return True
|
||||
return len(unanswered) / len(terms) >= UNANSWERED_BAR
|
||||
|
||||
|
||||
def row_four(cases: Sequence[Case]) -> Row:
|
||||
|
|
@ -1195,7 +1242,8 @@ def row_four(cases: Sequence[Case]) -> Row:
|
|||
"covered by the bundle and MARKED anyway"
|
||||
if control.covered
|
||||
else f"not covered and NOT marked -- {counts['delivered']} "
|
||||
"excerpts, no score, no statement"
|
||||
"excerpts, and the bundle answers enough of the question's "
|
||||
"terms that nothing in the payload says so"
|
||||
)
|
||||
)
|
||||
return _row(
|
||||
|
|
@ -1203,7 +1251,8 @@ def row_four(cases: Sequence[Case]) -> Row:
|
|||
"an uncovered question comes back marked, a covered one does not",
|
||||
correct,
|
||||
total,
|
||||
"marked = a reading the consumer can act on (today: nothing delivered)",
|
||||
"marked = a reading the consumer can act on: nothing delivered, or the "
|
||||
f"bundle answers none of >= {UNANSWERED_BAR:.0%} of the question's own terms",
|
||||
details,
|
||||
)
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue