test(consume): the retrieval gate, written red over nine rows
One command, one exit code, and no capability: nothing here changes the ranking, the fusion, the tokenisation or the cut. For a frozen question set against a bundle it asks how many of N measurement units the payload carries the fasit for, gives every miss exactly ONE class, and asks the thing the payload does not do at all -- say so when it does not know. Today, on the synthetic corpus this repository generates and the five sets it ships pinned by sha256: rows 1 (9 of 9), 2 (7 of 7) and 6 (9 of 9) green; 3 (2 of 5), 4 (3 of 6), 5 (0 of 1), 7 (11 of 13), 8 (not run) and 9 (0 of 6) red. GATE RED, exit 1. - Row 3 is the finding to read first. In a bundle built from ONE source document, every concept past the first two carries that document's `source_file`, so a concept the RANK had already lost is withheld as `source_quota_exceeded`. The truth is decided by the SAME cut run without the quota -- PM measured the same lie on 13 of 25 real misses 2026-09-17. - Row 4 is the second. No payload key says "this bundle does not answer that", so the only honest reading today is `delivered == 0`; three of six controls come out right and the three that do not get eight excerpts and no statement. A control the bundle DOES answer must come back unmarked, or the marking says nothing, so both directions sit in the same row. - A miss has exactly one class -- (a) not in the bundle, (b) below k, (c) cut by quota or budget, (d) delivered without the citation, (e) other -- and each is FORCED by its own fixture. An unplaced miss takes the row to 0 of N. - The judge opens the bundle. Every concept is read off disk before a payload is built, a delivered excerpt whose text is not the concept file's bytes is not a hit, and the two title forms come from `quality` rather than being written a second time. - FELLED means a row got WORSE, never merely that something changed: removing the quota makes row 3 green, and counting that as a kill would credit this gate with a check it does not have. The two survivors are reported with what they measured -- the document prior dead and RRF_K flattened each moved 0 ranks and 0 deliveries on these fixtures -- and with a mechanism for each. - Rows 8 and 9 are never green by leaving something out. The three real sets are read from path + sha256 and never committed: a gold set names a consumer's documents and this repository is public. K2 is 0 of 6 because the gold set exists nowhere while its bundles do. Denominators re-measured against the sources 2026-09-19, and all four hold: wiki 20 questions / 29 fasit entries (sha 972d0f57...), R761 8 (S1-S6 + KP + KN), vegnormal 32 questions / 43 must_cite, K2 6. PM's 45 of 70 is itself a mixed sum -- wiki at question level, vegnormal at citation level -- so this gate carries it labelled as recorded and never sums the two granularities. 46 tests, each failing row driven both ways through inputs rather than by editing the row. No network (the socket guard is fired before its silence counts), no private corpus, no clock. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
ae441abec4
commit
2c8296b807
9 changed files with 2748 additions and 0 deletions
87
tests/fixtures/retrieval/set-classes.json
vendored
Normal file
87
tests/fixtures/retrieval/set-classes.json
vendored
Normal file
|
|
@ -0,0 +1,87 @@
|
|||
{
|
||||
"set_id": "synthetic-classes",
|
||||
"bundle": "positive",
|
||||
"note": "One question per class, each forcing its own. A class that a fixture does not force is a class nobody has shown this gate can tell apart. B3 is the same class in a ONE-SOURCE bundle, where the payload prints a different reason for it -- row 3's fixture.",
|
||||
"questions": [
|
||||
{
|
||||
"id": "A1",
|
||||
"question": "Hvem har ansvaret for broeyting av parkeringsplassen?",
|
||||
"bundle": "positive",
|
||||
"expect_class": "a",
|
||||
"fasit": [
|
||||
{
|
||||
"by": "concept",
|
||||
"value": "haandbok/broeyteansvar",
|
||||
"quote": "broeyteansvaret ligger hos vaktmesteren"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "B2",
|
||||
"question": "Hvor mange medlemmer maa stemme for en endring av vedtektene?",
|
||||
"bundle": "miss",
|
||||
"expect_class": "b",
|
||||
"fasit": [
|
||||
{
|
||||
"by": "concept",
|
||||
"value": "vedtekter/flertallskrav",
|
||||
"quote": "To tredjedeler"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "C1",
|
||||
"question": "Hvor foeres dugnadstimene?",
|
||||
"bundle": "budget",
|
||||
"expect_class": "c",
|
||||
"limit": 8000,
|
||||
"fasit": [
|
||||
{
|
||||
"by": "concept",
|
||||
"value": "tabell/dugnadstabell",
|
||||
"quote": "rad for rad"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "D1",
|
||||
"question": "Hvor lenge arkiveres avviksskjemaet?",
|
||||
"bundle": "positive",
|
||||
"expect_class": "d",
|
||||
"fasit": [
|
||||
{
|
||||
"by": "concept",
|
||||
"value": "skjema/avviksskjema",
|
||||
"quote": "arkiveres i fem aar"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "E1",
|
||||
"question": "Hvilken safran passer til fiskesuppe?",
|
||||
"bundle": "positive",
|
||||
"expect_class": "e",
|
||||
"fasit": [
|
||||
{
|
||||
"by": "concept",
|
||||
"value": "haandbok/vinterberedskap",
|
||||
"quote": "innen 1. november"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "B3",
|
||||
"question": "Hva sier beretningen om snoedybden og dugnaden i klubben?",
|
||||
"bundle": "single-source",
|
||||
"expect_class": "b",
|
||||
"fasit": [
|
||||
{
|
||||
"by": "concept",
|
||||
"value": "aarsberetning/loypekjoring",
|
||||
"quote": "snoedybden passerer tretti centimeter"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"controls": []
|
||||
}
|
||||
14
tests/fixtures/retrieval/set-controls.json
vendored
Normal file
14
tests/fixtures/retrieval/set-controls.json
vendored
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
{
|
||||
"set_id": "synthetic-controls",
|
||||
"bundle": "positive",
|
||||
"note": "Row 4's denominator: the five negative control classes measured 2026-09-17, plus one control the bundle DOES answer. A marking that also fires on a covered question says nothing, so both directions are in the same row.",
|
||||
"questions": [],
|
||||
"controls": [
|
||||
{"id": "N1", "kind": "an unrelated topic", "question": "Hvilken safran passer til fiskesuppe?"},
|
||||
{"id": "N2", "kind": "a foreign language", "question": "How do I rotate the database credentials?"},
|
||||
{"id": "N3", "kind": "a plausible term the bundle does not cover", "question": "Hvilken kontroll gjelder for broeytebilen om vinteren?"},
|
||||
{"id": "N4", "kind": "a lookup on an identifier that does not exist", "question": "Hva sier punkt 99.9 i haandboka?"},
|
||||
{"id": "N5", "kind": "the set's own known-negative", "question": "Hva koster et doegn paa hytta for gjester?"},
|
||||
{"id": "P0", "kind": "covered by the bundle", "question": "Naar kontrolleres vinterberedskapen?", "covered": true}
|
||||
]
|
||||
}
|
||||
14
tests/fixtures/retrieval/set-miss.json
vendored
Normal file
14
tests/fixtures/retrieval/set-miss.json
vendored
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
{
|
||||
"set_id": "synthetic-miss",
|
||||
"bundle": "miss",
|
||||
"note": "The other direction of row 1, and the fixture class (b) is forced on: the fasit is in the bundle and carries the citation, the question reaches it with one common token, and fifteen decoys answer four. Nothing about the fasit is missing -- the ranking does not reach it.",
|
||||
"questions": [
|
||||
{
|
||||
"id": "B1",
|
||||
"question": "Hvor mange medlemmer maa stemme for en endring av vedtektene?",
|
||||
"expect_class": "b",
|
||||
"fasit": [{"by": "concept", "value": "vedtekter/flertallskrav", "quote": "To tredjedeler"}]
|
||||
}
|
||||
],
|
||||
"controls": []
|
||||
}
|
||||
38
tests/fixtures/retrieval/set-positive.json
vendored
Normal file
38
tests/fixtures/retrieval/set-positive.json
vendored
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
{
|
||||
"set_id": "synthetic-positive",
|
||||
"bundle": "positive",
|
||||
"note": "The instrument's known-positive: six questions whose fasit the ranker must reach at the shipped defaults. Invented subject matter, named after nothing. A question here declares no expect_class, so row 1 counts it.",
|
||||
"questions": [
|
||||
{
|
||||
"id": "P1",
|
||||
"question": "Naar kontrolleres vinterberedskapen?",
|
||||
"fasit": [{"by": "concept", "value": "haandbok/vinterberedskap", "quote": "innen 1. november"}]
|
||||
},
|
||||
{
|
||||
"id": "P2",
|
||||
"question": "Hvordan kvitteres noekkelen ut?",
|
||||
"fasit": [{"by": "concept", "value": "haandbok/noekkelkvittering", "quote": "signatur i noekkelboka"}]
|
||||
},
|
||||
{
|
||||
"id": "P3",
|
||||
"question": "Hvor ofte byttes batteriet i roekvarsleren?",
|
||||
"fasit": [{"by": "concept", "value": "haandbok/brannvarsling", "quote": "en gang i aaret"}]
|
||||
},
|
||||
{
|
||||
"id": "P4",
|
||||
"question": "Hva vedtok aarsmoetet om kontingenten?",
|
||||
"fasit": [{"by": "concept", "value": "referat/aarsmote-kontingent", "quote": "480 kroner"}]
|
||||
},
|
||||
{
|
||||
"id": "P5",
|
||||
"question": "Hvor lenge velges revisoren?",
|
||||
"fasit": [{"by": "concept", "value": "referat/aarsmote-valg", "quote": "to aar av gangen"}]
|
||||
},
|
||||
{
|
||||
"id": "P6",
|
||||
"question": "Hva skal foelge soeknaden om vinteropphold?",
|
||||
"fasit": [{"by": "concept", "value": "skjema/utstyrsliste", "quote": "Utstyrslisten skal foelge soeknaden"}]
|
||||
}
|
||||
],
|
||||
"controls": []
|
||||
}
|
||||
26
tests/fixtures/retrieval/set-signals.json
vendored
Normal file
26
tests/fixtures/retrieval/set-signals.json
vendored
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
{
|
||||
"set_id": "synthetic-signals",
|
||||
"bundle": "lookup",
|
||||
"note": "Three questions the shipped ranker must answer through three different mechanisms: the identifier lookup, the covered title, and the source quota. Each bundle answers the question's other words several times over, so a mechanism that stops firing takes its own fasit out of k and row 1 sees it.",
|
||||
"questions": [
|
||||
{
|
||||
"id": "L1",
|
||||
"question": "Hva staar i punkt 4.2 om kontrollen av hytta?",
|
||||
"bundle": "lookup",
|
||||
"fasit": [{"by": "concept", "value": "rutiner/vakthold-4-2", "quote": "fredag til soendag"}]
|
||||
},
|
||||
{
|
||||
"id": "L2",
|
||||
"question": "Hva sier noekkelrutine om kontrollen av hytta?",
|
||||
"bundle": "lookup",
|
||||
"fasit": [{"by": "concept", "value": "rutiner/noekkelrutine", "quote": "skifte av laas"}]
|
||||
},
|
||||
{
|
||||
"id": "Q1",
|
||||
"question": "Hva sier oversikten om dugnaden og kontrollen av broennen?",
|
||||
"bundle": "quota",
|
||||
"fasit": [{"by": "concept", "value": "svar/broennproeve", "quote": "proevetas i juni"}]
|
||||
}
|
||||
],
|
||||
"controls": []
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue