Compare commits

..

No commits in common. "main" and "v1.1.0" have entirely different histories.

8 changed files with 103 additions and 1049 deletions

View file

@ -1318,37 +1318,15 @@ and fixtures, never code.
what a bundle looks like, this asks whether the payload for a question what a bundle looks like, this asks whether the payload for a question
carries the fasit. **Written RED, no capability**: eight rows, exit 0 only carries the fasit. **Written RED, no capability**: eight rows, exit 0 only
when all are green, 1 otherwise, 2 on wrong input, `--json`. Rows 1-4, 6, 7 when all are green, 1 otherwise, 2 on wrong input, `--json`. Rows 1-4, 6, 7
run against a synthetic corpus this repo GENERATES and eight sets it ships run against a synthetic corpus this repo GENERATES and six sets it ships
pinned by sha256 -- no network, no private corpus, no clock; a set whose pinned by sha256 -- no network, no private corpus, no clock; a set whose
bytes moved is exit 2, never a quiet continuation, because **a gold set is bytes moved is exit 2, never a quiet continuation, because **a gold set is
an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and
this repo is public). **THE CORPUS IS PINNED THE SAME WAY** since this repo is public). **THE CORPUS IS PINNED THE SAME WAY** since
2026-09-19 (`SPECS_SHA256`). Today (measured on the default run 2026-09-19 (`SPECS_SHA256`). Today (measured on the default run
2026-09-22): **1 (14/14) - 2 (7/7) - 3 (5/5) - 4 (6/6) - 5 (0/1) - 2026-09-21, v1.1 K): **1 (10/10) - 2 (7/7) - 3 (5/5) - 4 (6/6) - 5 (0/1) -
6 (14/14) - 7 (15/15) - 8 (0 of 1 set, NOT RUN without `--real`)**, 6 (10/10) - 7 (11/14) - 8 (0 of 1 set, NOT RUN without `--real`)**,
`GATE RED: rows 5, 8`. **Row 7 fells 14 of 14 since 2026-09-22**: the three `GATE RED: rows 5, 7, 8`.
that survived v1.1 (M06 passage body, M07 title/path weight, M10
`bm25.RRF_K`) each got a fixture that makes its mechanism decide a delivery
(`PASSAGE`, `PATH`, `FUSION`, set `set-mechanisms.json`; the fusion one keeps
its lead through K = 180 and loses it at 190), with `src/` untouched.
**THE JUDGE READS A PASSAGE AS A PASSAGE (2026-09-22).** It predated v1.1
C1 and required a delivered text to be the whole concept's bytes, so every
excerpt `as_passage` cut was class (e) even when its span carried the
citation -- the dominant miss class when row 8 was run on the real set.
`passage_span` now accepts a passage only when the delivered text IS its
exact reconstruction: the span is `body[start:end]` byte for byte, `of` is
the body's length, a `[...]` stands exactly where text is left out, and the
one other line allowed is a heading line of the body above the span (or a
prefix of one); WHICH heading is the product's choice and is not re-derived.
The citation is read in the SPAN alone, never in the heading, the markers or
across the seam. Two traps are pinned by tests: the offsets count in the
DELIVERED body (read in the concept file they land a frontmatter's length
off), and "the span occurs somewhere in the text" accepts an invented
sentence beside it. **M15** (a passage carrying one sentence the file does
not, M14's shape for passages) is row 7's fifteenth mutant, felled through a
new fixture that delivers a passage (`DELIVERY`, set `set-passage.json`),
because no synthetic concept was long enough to be cut. `src/` untouched;
`MUTANT_BAR` unchanged.
**THE K2 TEST TRACK IS RETIRED (operator 2026-09-21, v1.1 K)**: material **THE K2 TEST TRACK IS RETIRED (operator 2026-09-21, v1.1 K)**: material
tied to the operator's employer is not re-measured and not frozen, and tied to the operator's employer is not re-measured and not frozen, and
public tests and gates run on invented material. Row 9 (K2) and the `r761` public tests and gates run on invented material. Row 9 (K2) and the `r761`
@ -1357,8 +1335,8 @@ and fixtures, never code.
the R761/N101/N200 deliveries; `okf_accounting_gate.py` has no default real the R761/N101/N200 deliveries; `okf_accounting_gate.py` has no default real
corpus. The synthetic corpus was re-measured for BM25 (`MISS`, `LOOKUP`, corpus. The synthetic corpus was re-measured for BM25 (`MISS`, `LOOKUP`,
`QUOTA`), so rows 2 and 3 are green through their own fixtures again, and `QUOTA`), so rows 2 and 3 are green through their own fixtures again, and
row 7's mutants patch `bm25` where the default runs there; the three that row 7's mutants patch `bm25` where the default runs there; three survive
survived with 0 ranks moved are felled since 2026-09-22 (above). What (passage body, field title weight, `bm25.RRF_K`) with 0 ranks moved. What
the text below says about rows 8/9, K2 and the three sets is history. the text below says about rows 8/9, K2 and the three sets is history.
**FOUR OF EIGHT CHEATING ATTACKS WENT THROUGH IT AND ARE CLOSED **FOUR OF EIGHT CHEATING ATTACKS WENT THROUGH IT AND ARE CLOSED
(2026-09-19).** PM's checkpoint on `2c8296b` found rows 3, 5, 7 and 8 could (2026-09-19).** PM's checkpoint on `2c8296b` found rows 3, 5, 7 and 8 could
@ -1385,13 +1363,7 @@ and fixtures, never code.
itself; three now read GIT (committed and unmodified, its commit is not itself; three now read GIT (committed and unmodified, its commit is not
itself a ranking change, a ranking change landed AFTER it), the last being itself a ranking change, a ranking change landed AFTER it), the last being
the one that cannot be self-attested. What git cannot prove -- that nobody the one that cannot be self-attested. What git cannot prove -- that nobody
read the number first -- is stated, not implied. Since 2026-09-23 row 5 read the number first -- is stated, not implied. **Row 7's bar** was 90 % of
reads the hold-out set in the schema the SET declares
(`HOLD_OUT_SCHEMAS`: `fase-sporsmaal/1` goes through row 8's own adapter
and `hit_rule`, no `schema` is the gate's synthetic form, any other schema
is a NO naming it) -- it read every set with the synthetic reader, so a set
in a consumer's schema was a NO nothing could turn. It prints the share and
never which hold-out question missed, and a test holds that. **Row 7's bar** was 90 % of
a list in the file a capability session edits, so seven duplicate `k = 1` a list in the file a capability session edits, so seven duplicate `k = 1`
mutants read `18 of 20 GREEN`; `MUTANT_ROSTER` and `MUTANT_COUNT` are pinned mutants read `18 of 20 GREEN`; `MUTANT_ROSTER` and `MUTANT_COUNT` are pinned
apart from `MUTANTS`, duplicates are refused, and the bar is the roster's. apart from `MUTANTS`, duplicates are refused, and the bar is the roster's.

View file

@ -84,13 +84,13 @@ here was measured; none of it is a plan.
`--no-assets` takes the image path out entirely. `--no-assets` takes the image path out entirely.
- **Three of this repository's own gates are RED, and each red row is a stated - **Three of this repository's own gates are RED, and each red row is a stated
finding rather than a bug to be surprised by.** The retrieval gate is red on finding rather than a bug to be surprised by.** The retrieval gate is red on
rows 5 and 8, the MCP gate on row 2, and the content accounting's judge rows 5, 7 and 8, the MCP gate on row 2, and the content accounting's judge
on rows 2, 3 and 6. For a user that means: no ranking change has yet been on rows 2, 3 and 6. For a user that means: no ranking change has yet been
registered against a held-out set (row 5), the real-corpus row is NOT RUN registered against a held-out set (row 5), the real-corpus row is NOT RUN
without its private set and so is red rather than green by omission without its private sets and so is red rather than green by omission
(row 8) -- and since 2026-09-22 its judge reads an excerpt v1.1 cuts to its (row 8), three mechanical mutants of the BM25 ranking survive with 0 ranks
answering passage as that passage, accepted only when it is the exact moved — the passage body, the title field weight and the fusion constant —
reconstruction from the bundle's bytes. MCP anchors and concept ids are different so the gate cannot yet tell those parts from their absence (row 7), MCP anchors and concept ids are different
vocabularies so `okf_fetch` cannot be addressed with a set's anchor (row 2), vocabularies so `okf_fetch` cannot be addressed with a set's anchor (row 2),
and the accounting still reports real losses on the reference corpus (rows 2, and the accounting still reports real losses on the reference corpus (rows 2,
3, 6). The rows and their numbers are under [Judge the 3, 6). The rows and their numbers are under [Judge the
@ -1196,18 +1196,18 @@ set names a consumer's documents and this repository is public. **The corpus
is pinned the same way** (`SPECS_SHA256`): every row counts against those is pinned the same way** (`SPECS_SHA256`): every row counts against those
documents, so moving them without moving the pin is exit 2. documents, so moving them without moving the pin is exit 2.
**It is RED today, on rows 5 and 8**, and each of those is a **It is RED today, on rows 5, 7 and 8**, and each of those is a
finding rather than a defect in the gate: finding rather than a defect in the gate:
| row | what it asks | today | | row | what it asks | today |
|---|---|---| |---|---|---|
| 1 | hit@payload, one fasit entry = one unit | 14 of 14 | | 1 | hit@payload, one fasit entry = one unit | 10 of 10 |
| 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 | | 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 |
| 3 | the `rule` the payload prints for a withheld fasit is the true one | 5 of 5 | | 3 | the `rule` the payload prints for a withheld fasit is the true one | 5 of 5 |
| 4 | an uncovered question comes back marked, a covered one does not | 6 of 6 | | 4 | an uncovered question comes back marked, a covered one does not | 6 of 6 |
| 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 | | 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 |
| 6 | every delivery confirmed against the bundle's own bytes | 14 of 14 | | 6 | every delivery confirmed against the bundle's own bytes | 10 of 10 |
| 7 | mechanical mutants of the ranking and the cut, felled | 15 of 15 | | 7 | mechanical mutants of the ranking and the cut, felled | 11 of 14 |
| 8 | the real set, from path + sha256 | 0 of 1 set, NOT RUN without `--real` | | 8 | the real set, from path + sha256 | 0 of 1 set, NOT RUN without `--real` |
**The public rows run on invented material only** (operator decision **The public rows run on invented material only** (operator decision
@ -1230,28 +1230,13 @@ delivered excerpt answers — facts, no verdict, and the one bar is the gate's
own `UNANSWERED_BAR = 2/3` — 3 of 6 to 6 of 6. own `UNANSWERED_BAR = 2/3` — 3 of 6 to 6 of 6.
Row 7's mutants patch the code the DEFAULT ranking runs (BM25 since v1.1), Row 7's mutants patch the code the DEFAULT ranking runs (BM25 since v1.1),
and it fells all 15. Until 2026-09-22 three survived with **0 ranks and 0 and it reports three survivors with what they moved rather than with a shrug:
deliveries** moved -- the passage signal reading no body, the field signal the passage signal reading no body, the field signal weighing no title and no
weighing no title and no path, and a flattened fusion (`bm25.RRF_K`) -- because path, and a flattened fusion (`bm25.RRF_K`) each moved **0 ranks and 0
every synthetic concept was short and opened with its own title, so none of the deliveries** on these fixtures. Each has a mechanism: the field signal still
three ever decided a delivery. Three fixtures now make each one decide one reads every body, every synthetic body opens with its own title as a heading,
(`set-mechanisms.json`): a long concept answered in one window of its body and over two signals `K` only reorders a concept whose two ranks lie far apart,
against short concepts whose titles carry the question's words; a concept named which no fixture holds.
by its path alone against decoys denser in the body; and a concept first in one
signal and 21st in the other against a decoy 10th and 11th, which a flattened
fusion ranks first because only the rank sum then counts. Each fixture is
delivered at the shipped defaults and falls out of `k` with its own mechanism
switched off; `src/` did not change.
A delivery is judged against the bundle's own bytes (row 6), and since v1.1 a
concept over 4 000 characters is delivered as its answering passage: the
nearest heading above it, `[...]`, the span, `[...]`. The judge counts such a
passage only when it is EXACTLY that reconstruction -- the span byte for byte
the delivered body at the passage's own offsets, markers only where text is
left out, the heading a line of the body above the span -- and it reads the
citation in the span alone. A looser rule ("the span occurs somewhere in the
text") would accept an invented sentence beside the span; the fifteenth
mutant (M15) is exactly that, and a fixture that delivers a passage fells it.
Row 8 is never green by leaving something out: it requires every named set and Row 8 is never green by leaving something out: it requires every named set and
is NOT RUN until it has them — one set of three used to read `6 of 6 GREEN`. is NOT RUN until it has them — one set of three used to read `6 of 6 GREEN`.

View file

@ -1,13 +0,0 @@
{
"set": "~/repos/claude-code-llm-wiki/eval/okf-holdout-sporsmaal.json",
"sha256": "7c45e2dc787b72dec3db2252f6ebc2d14b28d55f7e220bf76b43bfbe28adab82",
"bundle": "~/okf/gate-bundles/claude-code-wiki-7141ab3f",
"bundle_ref": "sha256-tree:7141ab3f00a20387caa2c22e43761ba625fbe596091303ca6c2069d77d20095c",
"bundle_built_with": "llm-ingestion-okf v1.1.0 (43499c0), --gate none; the bundle row 8 measures, copied to a durable path with its ref measured identical",
"metric": "questions answered over questions asked",
"threshold": 0.8,
"threshold_written_at": "2026-09-23T06:56:47Z",
"written_by": "claude-code-llm-wiki, order 4350316446, commit 33735b4: written blind, not the session that changes the ranking",
"registered_by": "llm-ingestion-okf, order 20260923T080931Z-3809217500, a session that changes no ranking",
"readings": []
}

View file

@ -1,28 +0,0 @@
{
"set_id": "synthetic-mechanisms",
"bundle": "passage",
"note": "Three questions, one per mechanism the default BM25 ranking runs and no earlier fixture made decide a delivery: the passage signal's body windows (bundle `passage`), the field signal's title and path weight (bundle `path`), and the rank-fusion constant (bundle `fusion`). Each is delivered at the shipped defaults and falls out of k when its own mechanism is switched off, so row 1 is what fells M06, M07 and M10.",
"questions": [
{
"id": "PA1",
"question": "Hvem kontrollerer varmekablene i taket?",
"bundle": "passage",
"fasit": [{"by": "concept", "value": "aarbok/kapittel-tre", "quote": "kontrolleres av vaktmesteren"}]
},
{
"id": "PT1",
"question": "Hva gjelder for vester ved kanopadling?",
"bundle": "path",
"k": 6,
"fasit": [{"by": "concept", "value": "kanopadling/regel", "quote": "brukes paa vannet"}]
},
{
"id": "FU1",
"question": "Hva gjelder for isbading?",
"bundle": "fusion",
"k": 1,
"fasit": [{"by": "concept", "value": "badebok/kapittel", "quote": "isbading om morgenen"}]
}
],
"controls": []
}

View file

@ -1,13 +0,0 @@
{
"set_id": "synthetic-passage",
"bundle": "delivery",
"note": "One question whose answer lies in a concept over consume.PASSAGE_CHARS, so it is delivered as a passage: heading, [...], span, [...]. The span carries the citation. Rows 1 and 6 count it only when the judge accepts a passage that is its exact reconstruction from the bundle's bytes, and M15 (one sentence added to the passage) fells both.",
"questions": [
{
"id": "PD1",
"question": "Hvem gir varmepumpa i kjelleren service?",
"fasit": [{"by": "concept", "value": "husbok/drift", "quote": "service av roerleggeren"}]
}
],
"controls": []
}

View file

@ -1071,7 +1071,6 @@ ARM_ATTRIBUTION = {
def test_each_arm_flag_carries_its_attribution_inside_its_own_option_chunk( def test_each_arm_flag_carries_its_attribution_inside_its_own_option_chunk(
capsys: pytest.CaptureFixture[str], capsys: pytest.CaptureFixture[str],
monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
"""Each attribution must sit in its own option's help, not merely in the file. """Each attribution must sit in its own option's help, not merely in the file.
@ -1085,15 +1084,7 @@ def test_each_arm_flag_carries_its_attribution_inside_its_own_option_chunk(
written down. A hard-coded `== 1` is correct only until a second arm is written down. A hard-coded `== 1` is correct only until a second arm is
attributed the same way, and it then goes red on an axis that has nothing attributed the same way, and it then goes red on an axis that has nothing
to do with whether the new arm is right. to do with whether the new arm is right.
Colour is switched off for the call. Python 3.14's argparse colours
`--help` when `FORCE_COLOR` is set -- a background session inherits
`FORCE_COLOR=3` -- and an escape code before ` --outline-run` leaves no
chunk starting with the flag. `PYTHON_COLORS=0` is the variable
`_colorize.can_colorize` reads FIRST, so it outranks `FORCE_COLOR` and
`NO_COLOR` alike; the help's words are the same either way.
""" """
monkeypatch.setenv("PYTHON_COLORS", "0")
with pytest.raises(SystemExit) as exit_info: with pytest.raises(SystemExit) as exit_info:
okf_propose_segments.main(["--help"]) okf_propose_segments.main(["--help"])
assert exit_info.value.code == 0 assert exit_info.value.code == 0

View file

@ -394,8 +394,8 @@ def test_row_five_is_red_while_no_hold_out_is_registered(tmp_path: Path) -> None
assert (row.k, row.m, row.status) == (0, 1, gate.RED) assert (row.k, row.m, row.status) == (0, 1, gate.RED)
def test_row_five_is_green_for_a_registration_that_carries_all_twelve( def test_row_five_is_green_for_a_registration_that_carries_all_eleven(
tmp_path: Path, home: Path tmp_path: Path,
) -> None: ) -> None:
registration = _registration( registration = _registration(
tmp_path, tmp_path,
@ -404,11 +404,11 @@ def test_row_five_is_green_for_a_registration_that_carries_all_twelve(
threshold=0.8, threshold=0.8,
) )
row = gate.row_five(registration, provenance=lambda _: _carried_by_git()) row = gate.row_five(registration, provenance=lambda _: _carried_by_git())
assert (row.k, row.m, row.status) == (12, 12, gate.GREEN) assert (row.k, row.m, row.status) == (11, 11, gate.GREEN)
def test_row_five_falls_on_a_number_read_before_its_threshold_was_written( def test_row_five_falls_on_a_number_read_before_its_threshold_was_written(
tmp_path: Path, home: Path tmp_path: Path,
) -> None: ) -> None:
registration = _registration( registration = _registration(
tmp_path, tmp_path,
@ -420,12 +420,12 @@ def test_row_five_falls_on_a_number_read_before_its_threshold_was_written(
spec["readings"] = [{"at": "2026-09-18T09:00:00Z", "value": "0.62"}] spec["readings"] = [{"at": "2026-09-18T09:00:00Z", "value": "0.62"}]
registration.write_text(json.dumps(spec), encoding="utf-8") registration.write_text(json.dumps(spec), encoding="utf-8")
row = gate.row_five(registration, provenance=lambda _: _carried_by_git()) row = gate.row_five(registration, provenance=lambda _: _carried_by_git())
assert (row.k, row.m, row.status) == (11, 12, gate.RED) assert (row.k, row.m, row.status) == (10, 11, gate.RED)
assert any("no reading predates the threshold: NO" in detail for detail in row.details) assert any("no reading predates the threshold: NO" in detail for detail in row.details)
def test_row_five_falls_when_the_registration_rides_in_on_the_ranking_change( def test_row_five_falls_when_the_registration_rides_in_on_the_ranking_change(
tmp_path: Path, home: Path tmp_path: Path,
) -> None: ) -> None:
"""The three git checks, each driven red on its own: a file nobody """The three git checks, each driven red on its own: a file nobody
committed, a threshold committed together with the ranking change, and a committed, a threshold committed together with the ranking change, and a
@ -524,37 +524,6 @@ def test_a_mutant_is_felled_by_the_row_that_got_worse_and_never_by_one_that_did_
assert after[3] >= before[3] assert after[3] >= before[3]
@pytest.mark.parametrize(
"prefix",
[
# The three that survived v1.1: each is a mechanism the default ranking
# RUNS, so a gate that cannot see it removed cannot see it break.
"M06 ", # the passage signal reads no body
"M07 ", # the field signal weighs no title and no path
"M10 ", # the fusion is flattened
# A passage carrying one sentence the file does not: M14's shape for
# the delivery form v1.1 C1 added.
"M15 ",
],
)
def test_row_seven_fells_every_mutant_of_a_mechanism_the_ranking_runs(
tmp_path: Path, prefix: str
) -> None:
"""A survivor of row 7 is a mechanism the synthetic corpus never makes
decide a delivery. Felled means a row got WORSE on the pinned sets and the
pinned corpus -- never a row that merely moved."""
(mutant,) = [m for m in gate.MUTANTS if m.label.startswith(prefix)]
cases, _ = gate.synthetic_cases(tmp_path / "bundles", FIXTURES)
before = {row.number: row.k - row.m for row in gate.deterministic_rows(cases)}
with mutant.patch():
rows = gate.deterministic_rows(
[gate.measure_case(case.question_set, case.bundles) for case in cases]
)
after = {row.number: row.k - row.m for row in rows}
worse = [number for number in sorted(before) if after[number] < before[number]]
assert worse, f"{mutant.label} survives: no row got worse ({before} -> {after})"
# --- row 8 -------------------------------------------------------------------- # --- row 8 --------------------------------------------------------------------
@ -635,18 +604,14 @@ def test_the_gate_is_red_today_and_says_which_rows(tmp_path: Path) -> None:
by_number = {row.number: row for row in rows} by_number = {row.number: row for row in rows}
assert sorted(by_number) == [1, 2, 3, 4, 5, 6, 7, 8] assert sorted(by_number) == [1, 2, 3, 4, 5, 6, 7, 8]
# Rows 2 and 3 are green again since the synthetic corpus was re-measured # Rows 2 and 3 are green again since the synthetic corpus was re-measured
# for BM25 (2026-09-21). Row 7 fells all 14 since 2026-09-22, when the three # for BM25 (2026-09-21). Row 7 is red on three survivors, each printed with
# v1.1 survivors each got a fixture that makes their mechanism decide a # what it moved.
# delivery (`set-mechanisms.json`, three more row-1 units), and all 15 assert [row.number for row in rows if row.fails] == [5, 7, 8]
# since the judge reads a passage as its exact reconstruction and one assert (by_number[1].k, by_number[1].m) == (10, 10)
# fixture delivers a passage (`set-passage.json`, one more unit), so M15
# has something to fell.
assert [row.number for row in rows if row.fails] == [5, 8]
assert (by_number[1].k, by_number[1].m) == (14, 14)
assert (by_number[2].k, by_number[2].m) == (7, 7) assert (by_number[2].k, by_number[2].m) == (7, 7)
assert (by_number[3].k, by_number[3].m) == (5, 5) assert (by_number[3].k, by_number[3].m) == (5, 5)
assert (by_number[6].k, by_number[6].m) == (14, 14) assert (by_number[6].k, by_number[6].m) == (10, 10)
assert (by_number[7].k, by_number[7].m) == (15, 15) assert (by_number[7].k, by_number[7].m) == (11, 14)
def test_the_same_tree_measures_the_same_twice(tmp_path: Path) -> None: def test_the_same_tree_measures_the_same_twice(tmp_path: Path) -> None:
@ -662,7 +627,7 @@ def test_the_command_exits_one_and_prints_every_row(
printed = capsys.readouterr().out printed = capsys.readouterr().out
for number in range(1, 9): for number in range(1, 9):
assert f"\n{number} " in f"\n{printed}" assert f"\n{number} " in f"\n{printed}"
assert "GATE RED: rows 5, 8" in printed assert "GATE RED: rows 5, 7, 8" in printed
def test_the_json_form_carries_the_same_rows(capsys: pytest.CaptureFixture[str]) -> None: def test_the_json_form_carries_the_same_rows(capsys: pytest.CaptureFixture[str]) -> None:
@ -964,7 +929,7 @@ def test_a_forced_fixture_that_stops_missing_counts_against_row_two(
assert any("Z9" in detail and "premise" in detail for detail in row.details) assert any("Z9" in detail and "premise" in detail for detail in row.details)
def test_j1_a_registration_this_session_wrote_is_not_a_hold_out(tmp_path: Path, home: Path) -> None: def test_j1_a_registration_this_session_wrote_is_not_a_hold_out(tmp_path: Path) -> None:
"""PM's J1: two files written by the session under test came back """PM's J1: two files written by the session under test came back
`7 of 7 GREEN`. Every check was an assertion the registration made about `7 of 7 GREEN`. Every check was an assertion the registration made about
itself -- `written_by` is `bool()` of a string the file sets, and the itself -- `written_by` is `bool()` of a string the file sets, and the
@ -1034,140 +999,6 @@ def test_the_mutant_roster_is_pinned_apart_from_the_list_it_names() -> None:
assert len(set(gate.MUTANT_ROSTER)) == len(gate.MUTANT_ROSTER) assert len(set(gate.MUTANT_ROSTER)) == len(gate.MUTANT_ROSTER)
# --- a passage is judged as a passage ------------------------------------------
#
# Since v1.1 C1 a concept over `consume.PASSAGE_CHARS` is delivered as its
# answering passage: the nearest heading above the span, `[...]`, the span,
# `[...]`. The row-6 identity (delivered text == the whole body) is older than
# that and read every such delivery as class e. The judge must now accept a
# passage that IS its exact reconstruction from the bundle's bytes, and nothing
# looser: "the span occurs somewhere in the text" accepts an invented sentence
# beside it.
_HOUSE_NOTE = "Styret gjennomgaar notatet og foerer det inn i arkivet. Sekretaeren sender kopi."
_INVENTED = "En setning som ikke staar i konseptfila."
def _house_bundle(tmp_path: Path) -> Path:
"""One concept of about 6 600 characters: a heading ten lines in, the
answer thirty lines below it, and a literal `[...]` on the LAST line, far
outside any span centred on the answer."""
body = "\n".join(
[_HOUSE_NOTE] * 10
+ ["## Teknisk rom"]
+ [_HOUSE_NOTE] * 30
+ ["Varmepumpa i kjelleren faar service av roerleggeren hvert aar."]
+ [_HOUSE_NOTE] * 40
+ ["Se vedlegg [...] i arkivet."]
)
spec = gate.BundleSpec(
"retrieval-house",
(
gate.DocumentSpec(
"husbok", "husbok.md", (gate.ConceptSpec("drift", "Drift av huset", body),)
),
*(
gate.DocumentSpec(
f"skriv-{number}",
f"skriv-{number}.md",
(gate.ConceptSpec("skriv", f"Skriv {number}", "Skrivet er arkivert."),),
)
for number in range(3)
),
),
)
return gate.build_bundle(tmp_path / "house", spec)
def _house_question(*quotes: str) -> gate.Question:
return gate.Question(
id="H1",
question="Hvem gir varmepumpa i kjelleren service?",
fasit=tuple(gate.Fasit("concept", "husbok/drift", quote) for quote in quotes),
)
def _inject(where: str) -> pytest.MonkeyPatch:
"""A passage carrying one sentence the concept file does not: at the end of
the span, or between the heading's marker line and the span."""
original = consume.as_passage
def mutant(excerpt: dict[str, object], window: int) -> dict[str, object]:
out = original(excerpt, window)
if "passage" in out:
text = str(out["text"])
if where == "span":
cut = text.rindex(f"\n{consume.PASSAGE_ELISION}")
out["text"] = f"{text[:cut]} {_INVENTED}{text[cut:]}"
else:
marker = f"{consume.PASSAGE_ELISION}\n"
cut = text.index(marker) + len(marker)
out["text"] = f"{text[:cut]}{_INVENTED}\n{text[cut:]}"
return out
patch = pytest.MonkeyPatch()
patch.setattr(consume, "as_passage", mutant)
return patch
def test_a_passage_whose_span_carries_the_quote_is_a_hit(tmp_path: Path) -> None:
bundle = _house_bundle(tmp_path)
gate.bundle_index(bundle)
question = _house_question("service av roerleggeren")
payload = consume.build_payload(bundle, question=question.question, withheld_full=True)
(excerpt,) = [e for e in payload["excerpts"] if e["concept_id"] == "husbok/drift"]
passage = excerpt["passage"]
assert isinstance(passage, dict), "the premise: this concept is delivered as a passage"
start, end = passage["start"], passage["end"]
assert 0 < start and end < passage["of"], "the premise: `[...]` on both sides"
# FELLE 1, pinned: the offsets count in the DELIVERED body. Read against
# the concept file they land a frontmatter's length off.
view = gate.bundle_index(bundle).concepts["husbok/drift"]
assert view.body[start:end] in str(excerpt["text"])
assert view.whole[start:end] not in str(excerpt["text"])
(unit,) = gate.measure_units(bundle, question)
assert (unit.hit, unit.confirmed, unit.klass) == (True, True, None), unit
@pytest.mark.parametrize("where", ["span", "between heading and span"])
def test_a_passage_carrying_an_invented_sentence_is_not_a_hit(tmp_path: Path, where: str) -> None:
"""FELLE 2, pinned: the quote is still in the span and the span is still in
the text, so "the span occurs somewhere" would count this. The delivered
text must BE the reconstruction."""
bundle = _house_bundle(tmp_path)
gate.bundle_index(bundle) # the judge reads the bundle BEFORE the payload
question = _house_question("service av roerleggeren")
patch = _inject(where)
try:
(unit,) = gate.measure_units(bundle, question)
finally:
patch.undo()
assert unit.hit is False
assert unit.confirmed is False
assert unit.klass == "e"
assert unit.detail == "the delivered passage is not its reconstruction from the bundle's bytes"
def test_a_quote_the_passage_carries_only_outside_its_span_is_not_a_hit(tmp_path: Path) -> None:
"""The heading and the markers are delivered and are not the span: a quote
found only there was delivered without the citation (class d). The first
unit is the known-positive of the same run."""
bundle = _house_bundle(tmp_path)
units = gate.measure_units(
bundle, _house_question("service av roerleggeren", "Teknisk rom", "[...]")
)
assert [(unit.hit, unit.klass) for unit in units] == [
(True, None),
(False, "d"),
(False, "d"),
], units
def test_row_seven_names_passage_cheating() -> None:
assert gate.MUTANT_COUNT == 15
assert gate.MUTANT_ROSTER[-1] == "M15 the passage carries a sentence the concept file does not"
def test_the_synthetic_corpus_is_pinned_like_the_sets(tmp_path: Path) -> None: def test_the_synthetic_corpus_is_pinned_like_the_sets(tmp_path: Path) -> None:
assert gate.specs_digest() == gate.SPECS_SHA256 assert gate.specs_digest() == gate.SPECS_SHA256
tuned = dict(gate.SPECS) tuned = dict(gate.SPECS)
@ -1294,18 +1125,6 @@ def test_a_wiki_set_of_one_question_is_refused_on_the_command_line(
# --- step 0: the threshold is a number, and it is compared with something ---- # --- step 0: the threshold is a number, and it is compared with something ----
@pytest.fixture
def home(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
"""Row 5 reads a registration's paths as `~/...`, expanded against the
home directory; every test writing one makes its `tmp_path` that home."""
monkeypatch.setenv("HOME", str(tmp_path))
return tmp_path
def _under_home(path: Path, home: Path) -> str:
return "~/" + path.relative_to(home).as_posix()
def _registration( def _registration(
tmp_path: Path, tmp_path: Path,
*, *,
@ -1313,26 +1132,14 @@ def _registration(
bundle: Path, bundle: Path,
threshold: object, threshold: object,
name: str = "registration.json", name: str = "registration.json",
bundle_ref: str | None = None,
) -> Path: ) -> Path:
"""A registration in the form the committed one must have: its paths
under the home directory, written `~/...`, and the bundle's tree ref
pinned. The home is `tmp_path` (the `home` fixture); a set living outside
it is copied in byte for byte, so the sha256 pinned is the set's own."""
assert Path.home() == tmp_path, "a row-5 test must request the `home` fixture"
if not set_path.is_relative_to(tmp_path):
copied = tmp_path / "sets" / set_path.name
copied.parent.mkdir(exist_ok=True)
copied.write_bytes(set_path.read_bytes())
set_path = copied
registration = tmp_path / name registration = tmp_path / name
registration.write_text( registration.write_text(
json.dumps( json.dumps(
{ {
"set": _under_home(set_path, tmp_path), "set": str(set_path),
"sha256": gate.sha256_of(set_path), "sha256": gate.sha256_of(set_path),
"bundle": _under_home(bundle, tmp_path), "bundle": str(bundle),
"bundle_ref": consume.bundle_ref(bundle) if bundle_ref is None else bundle_ref,
"metric": "questions answered over questions asked", "metric": "questions answered over questions asked",
"threshold": threshold, "threshold": threshold,
"threshold_written_at": "2026-09-19T10:00:00Z", "threshold_written_at": "2026-09-19T10:00:00Z",
@ -1345,7 +1152,7 @@ def _registration(
return registration return registration
def test_a_threshold_that_is_not_a_number_is_refused(tmp_path: Path, home: Path) -> None: def test_a_threshold_that_is_not_a_number_is_refused(tmp_path: Path) -> None:
"""`bool(threshold)` was the whole check, so `report-only; any number is """`bool(threshold)` was the whole check, so `report-only; any number is
acceptable for v1` read as `a threshold is written: yes`. A threshold that acceptable for v1` read as `a threshold is written: yes`. A threshold that
cannot be compared with a number cannot fell anything.""" cannot be compared with a number cannot fell anything."""
@ -1361,7 +1168,7 @@ def test_a_threshold_that_is_not_a_number_is_refused(tmp_path: Path, home: Path)
assert row.fails assert row.fails
def test_the_threshold_is_compared_with_the_measured_hold_out(tmp_path: Path, home: Path) -> None: def test_the_threshold_is_compared_with_the_measured_hold_out(tmp_path: Path) -> None:
"""Both directions, from the same code path: a set the bundle answers """Both directions, from the same code path: a set the bundle answers
clears a threshold under it, and a set it does not answer falls under one clears a threshold under it, and a set it does not answer falls under one
over it. The measured share is counted here as well, off the set's own over it. The measured share is counted here as well, off the set's own
@ -1378,7 +1185,7 @@ def test_the_threshold_is_compared_with_the_measured_hold_out(tmp_path: Path, ho
provenance=lambda _: _carried_by_git(), provenance=lambda _: _carried_by_git(),
) )
assert any("clears the threshold: yes" in detail for detail in clears.details) assert any("clears the threshold: yes" in detail for detail in clears.details)
assert (clears.k, clears.m, clears.status) == (12, 12, gate.GREEN) assert (clears.k, clears.m, clears.status) == (11, 11, gate.GREEN)
# The independent count: set-miss carries one question, forced to miss. # The independent count: set-miss carries one question, forced to miss.
missing = json.loads((FIXTURES / "set-miss.json").read_text(encoding="utf-8")) missing = json.loads((FIXTURES / "set-miss.json").read_text(encoding="utf-8"))
@ -1394,7 +1201,7 @@ def test_the_threshold_is_compared_with_the_measured_hold_out(tmp_path: Path, ho
provenance=lambda _: _carried_by_git(), provenance=lambda _: _carried_by_git(),
) )
assert any("clears the threshold: NO" in detail for detail in falls.details) assert any("clears the threshold: NO" in detail for detail in falls.details)
assert (falls.k, falls.m, falls.status) == (11, 12, gate.RED) assert (falls.k, falls.m, falls.status) == (10, 11, gate.RED)
# --- step 1: row 4, the marking a consumer can act on ------------------------- # --- step 1: row 4, the marking a consumer can act on -------------------------
@ -1482,7 +1289,7 @@ def test_a_set_of_the_right_size_and_the_wrong_bytes_is_still_refused(
assert "sha256" in str(refusal.value) assert "sha256" in str(refusal.value)
def test_a_threshold_outside_nought_to_one_is_not_a_share(tmp_path: Path, home: Path) -> None: def test_a_threshold_outside_nought_to_one_is_not_a_share(tmp_path: Path) -> None:
"""`80` is either 80 % written wrongly or a bar no run can clear, and """`80` is either 80 % written wrongly or a bar no run can clear, and
guessing which is not this row's job.""" guessing which is not this row's job."""
bundles = _bundles(tmp_path) bundles = _bundles(tmp_path)
@ -1512,7 +1319,7 @@ def test_a_threshold_outside_nought_to_one_is_not_a_share(tmp_path: Path, home:
assert any("the threshold is a number: yes" in detail for detail in inside.details) assert any("the threshold is a number: yes" in detail for detail in inside.details)
def test_a_hold_out_set_with_no_question_clears_no_threshold(tmp_path: Path, home: Path) -> None: def test_a_hold_out_set_with_no_question_clears_no_threshold(tmp_path: Path) -> None:
"""A share over a denominator of nought is not a number, and an empty set """A share over a denominator of nought is not a number, and an empty set
was the shape every row-5 test used before 2026-09-20 -- so `>=` over it was the shape every row-5 test used before 2026-09-20 -- so `>=` over it
would have made the comparison vacuous the moment it was added.""" would have made the comparison vacuous the moment it was added."""
@ -1534,265 +1341,3 @@ def test_a_hold_out_set_with_no_question_clears_no_threshold(tmp_path: Path, hom
"clears the threshold: NO (the hold-out set carries no question" in detail "clears the threshold: NO (the hold-out set carries no question" in detail
for detail in row.details for detail in row.details
), row.details ), row.details
# --- row 5 reads a hold-out set in the schema it declares --------------------
#
# The registered hold-out set is written in a consumer's own question-set
# schema, the one row 8 already reads, and row 5 used to read every set with
# the gate's synthetic reader: `not a question set this gate can read:
# 'set_id'`, a NO no set in that schema could ever turn. Every file below is
# written in that shape with INVENTED content over the synthetic bundle; the
# real hold-out set is never read by a test.
#: Distinctive on purpose: a row that printed a question's id or text would
#: carry one of these, and nothing else in the gate's output can.
_HOLD_OUT_MISS_ID = "HOLDOUT-MISS-7Q"
_HOLD_OUT_MISS_TEXT = "Hvilken farge har den fraflyttede fyrlykta paa Utvaer?"
_HOLD_OUT_HIT_IDS = ("HOLDOUT-H1", "HOLDOUT-H2", "HOLDOUT-H3", "HOLDOUT-H4")
def _fase_set(path: Path, *, schema: str = "fase-sporsmaal/1") -> Path:
"""Four questions the positive bundle answers and one it cannot: a
share of 4 of 5, counted here off the file and not off the row."""
hits = [
("Naar kontrolleres vinterberedskapen?", "haandbok", "innen 1. november"),
("Hvordan kvitteres noekkelen ut?", "haandbok", "signatur i noekkelboka"),
("Hvor ofte byttes batteriet i roekvarsleren?", "haandbok", "en gang i aaret"),
("Hva vedtok aarsmoetet om kontingenten?", "referat", "480 kroner"),
]
questions = [
{"id": qid, "question": text, "fasit": [{"doc": doc, "quote": quote}]}
for qid, (text, doc, quote) in zip(_HOLD_OUT_HIT_IDS, hits)
]
questions.append(
{
"id": _HOLD_OUT_MISS_ID,
"question": _HOLD_OUT_MISS_TEXT,
"fasit": [{"doc": "haandbok", "quote": "fyrlykta er malt i gult og fiolett"}],
}
)
path.write_text(
json.dumps(
{
"schema": schema,
"hit_rule": "an excerpt whose source_file is <doc>.md AND carries the quote",
"questions": questions,
},
ensure_ascii=False,
),
encoding="utf-8",
)
return path
def test_row_five_measures_a_hold_out_set_in_the_wiki_schema(tmp_path: Path, home: Path) -> None:
"""Both directions through the same set and the same bundle: 4 of 5
clears 0.8 and falls under 0.9. The share is counted off the file first,
so the row is not the only thing that knows it."""
held_out = _fase_set(tmp_path / "held-out.json")
written = json.loads(held_out.read_text(encoding="utf-8"))
assert len(written["questions"]) == 5
bundle = _bundles(tmp_path)["positive"]
clears = gate.row_five(
_registration(tmp_path, set_path=held_out, bundle=bundle, threshold=0.8, name="c.json"),
provenance=lambda _: _carried_by_git(),
)
assert any(
"the measured hold-out clears the threshold: yes (4 of 5 = 0.8000 against 0.8000)" in detail
for detail in clears.details
), clears.details
assert (clears.k, clears.m, clears.status) == (12, 12, gate.GREEN)
falls = gate.row_five(
_registration(tmp_path, set_path=held_out, bundle=bundle, threshold=0.9, name="f.json"),
provenance=lambda _: _carried_by_git(),
)
assert any(
"the measured hold-out clears the threshold: NO (4 of 5 = 0.8000 against 0.9000)" in detail
for detail in falls.details
), falls.details
assert (falls.k, falls.m, falls.status) == (11, 12, gate.RED)
def test_a_hold_out_set_in_an_unknown_schema_is_a_named_no(tmp_path: Path, home: Path) -> None:
"""Fail closed: a set read in a shape it was not written in measures
nothing, so an unknown schema is a NO that NAMES the schema -- never a
guess at a reader and never an exception."""
held_out = _fase_set(tmp_path / "held-out.json", schema="noe-annet/9")
row = gate.row_five(
_registration(
tmp_path, set_path=held_out, bundle=_bundles(tmp_path)["positive"], threshold=0.8
),
provenance=lambda _: _carried_by_git(),
)
verdict = [d for d in row.details if "the measured hold-out clears the threshold" in d]
assert len(verdict) == 1, row.details
assert ": NO (" in verdict[0]
assert "schema `noe-annet/9`" in verdict[0], verdict[0]
assert "fase-sporsmaal/1" in verdict[0], verdict[0]
assert row.fails
def test_row_five_prints_the_share_and_never_which_hold_out_question_missed(
tmp_path: Path, home: Path
) -> None:
"""A hold-out set whose misses are printed is a tuning set by the next
session. The known-positive in the same run: the row DID measure, one
question missed, and the share is printed -- so the absence of the ids
and the text below is the row's silence and not a row that never ran."""
held_out = _fase_set(tmp_path / "held-out.json")
for threshold in (0.8, 0.9):
row = gate.row_five(
_registration(
tmp_path,
set_path=held_out,
bundle=_bundles(tmp_path)["positive"],
threshold=threshold,
name=f"r{threshold}.json",
),
provenance=lambda _: _carried_by_git(),
)
text = gate.render([row]) + json.dumps(row.to_json(), ensure_ascii=False)
assert "4 of 5 = 0.8000" in text, text
assert _HOLD_OUT_MISS_ID not in text
assert _HOLD_OUT_MISS_TEXT not in text
assert "fyrlykta" not in text
for hit in _HOLD_OUT_HIT_IDS:
assert hit not in text
# --- row 5 names no machine, and measures the bundle it registered ------------
#
# The registration is committed to a PUBLIC repository, so the paths it names
# are written relative to the home directory and expanded when read: an
# absolute path names one machine and is a NO, never a path the row follows.
# And the bundle it names is only the registered bundle while its tree still
# measures the registered `bundle_ref`: a bundle path is a place, and a place
# can be filled with other bytes.
def _row_five_line(row: gate.Row, name: str) -> str:
lines = [detail for detail in row.details if detail.strip().startswith(f"{name}:")]
assert len(lines) == 1, (name, row.details)
return lines[0]
def test_an_absolute_path_in_the_registration_is_a_named_no(tmp_path: Path, home: Path) -> None:
"""Both paths, each on its own: the set and the bundle written as the
absolute path of the very file the `~/` form names. The row does not
follow either, and says why."""
held_out = _fase_set(tmp_path / "held-out.json")
bundle = _bundles(tmp_path)["positive"]
registration = _registration(tmp_path, set_path=held_out, bundle=bundle, threshold=0.8)
written = json.loads(registration.read_text(encoding="utf-8"))
for key, absolute, line in (
("set", str(held_out), "a set is named"),
("bundle", str(bundle), "the bundle is the registered tree"),
):
spec = dict(written, **{key: absolute})
registration.write_text(json.dumps(spec), encoding="utf-8")
row = gate.row_five(registration, provenance=lambda _: _carried_by_git())
verdict = _row_five_line(row, line)
assert ": NO (" in verdict, verdict
assert "absolute" in verdict, verdict
assert "~/" in verdict, verdict
assert row.status == gate.RED, key
# A path the row does not follow is a set it does not measure.
assert "4 of 5" not in gate.render([row]), key
def test_a_home_relative_path_is_expanded_and_read(tmp_path: Path, home: Path) -> None:
"""The known-positive for the rule above: the registration's own text
carries `~/` and no path of this machine, and the row expands it, reads
the set and measures the bundle -- 4 of 5, counted off the file."""
held_out = _fase_set(tmp_path / "held-out.json")
registration = _registration(
tmp_path, set_path=held_out, bundle=_bundles(tmp_path)["positive"], threshold=0.8
)
text = registration.read_text(encoding="utf-8")
spec = json.loads(text)
assert spec["set"].startswith("~/") and spec["bundle"].startswith("~/"), spec
assert str(tmp_path) not in text
row = gate.row_five(registration, provenance=lambda _: _carried_by_git())
assert ": yes (" in _row_five_line(row, "the pinned bytes are the bytes on disk")
assert ": yes (" in _row_five_line(row, "the bundle is the registered tree")
assert "(4 of 5 = 0.8000 against 0.8000)" in _row_five_line(
row, "the measured hold-out clears the threshold"
)
assert (row.k, row.m, row.status) == (12, 12, gate.GREEN)
def test_a_bundle_whose_tree_is_not_the_registered_ref_is_a_no(tmp_path: Path, home: Path) -> None:
"""Two ways the path stays and the bytes go: a registration pinning
another bundle's ref, and the registered bundle's concepts edited after
the ref was written. Neither is measured -- a share read off other bytes
is not the registered hold-out."""
held_out = _fase_set(tmp_path / "held-out.json")
bundles = _bundles(tmp_path)
foreign = consume.bundle_ref(bundles["miss"])
assert foreign != consume.bundle_ref(bundles["positive"]), "the control must differ"
pinned_elsewhere = gate.row_five(
_registration(
tmp_path,
set_path=held_out,
bundle=bundles["positive"],
threshold=0.8,
bundle_ref=foreign,
name="foreign.json",
),
provenance=lambda _: _carried_by_git(),
)
verdict = _row_five_line(pinned_elsewhere, "the bundle is the registered tree")
assert ": NO (" in verdict and foreign in verdict, verdict
assert pinned_elsewhere.status == gate.RED
assert "4 of 5" not in gate.render([pinned_elsewhere])
registration = _registration(
tmp_path, set_path=held_out, bundle=bundles["positive"], threshold=0.8, name="edited.json"
)
concepts = [p for p in bundles["positive"].rglob("*.md") if p.name != "index.md"]
assert concepts, "the control needs a concept to edit"
for concept in concepts:
concept.write_bytes(concept.read_bytes() + b"\nOther bytes at the same path.\n")
edited = gate.row_five(registration, provenance=lambda _: _carried_by_git())
assert ": NO (" in _row_five_line(edited, "the bundle is the registered tree")
assert edited.status == gate.RED
def test_a_registered_set_or_bundle_absent_on_this_machine_is_a_named_no(
tmp_path: Path, home: Path
) -> None:
"""A registration read on a machine that does not carry what it names:
a NO with the registered `~/` path, never an exception, never green."""
held_out = _fase_set(tmp_path / "held-out.json")
bundle = _bundles(tmp_path)["positive"]
written = json.loads(
_registration(tmp_path, set_path=held_out, bundle=bundle, threshold=0.8).read_text(
encoding="utf-8"
)
)
registration = tmp_path / "absent.json"
for key, value, line in (
("set", "~/nowhere/held-out.json", "the pinned bytes are the bytes on disk"),
("bundle", "~/nowhere/bundle", "the bundle is the registered tree"),
):
registration.write_text(json.dumps(dict(written, **{key: value})), encoding="utf-8")
row = gate.row_five(registration, provenance=lambda _: _carried_by_git())
verdict = _row_five_line(row, line)
assert ": NO (" in verdict and "absent" in verdict and value in verdict, verdict
assert row.status == gate.RED, key
assert ": NO (" in _row_five_line(row, "the measured hold-out clears the threshold")
def test_the_committed_registration_names_no_machine_path() -> None:
"""The file this repository publishes: its paths are `~/...`, and no
value in it names a directory of the machine it was written on."""
text = gate.HOLDOUT_REGISTRATION.read_text(encoding="utf-8")
spec = json.loads(text)
assert spec["set"].startswith("~/"), spec["set"]
assert spec["bundle"].startswith("~/"), spec["bundle"]
assert spec["bundle_ref"].startswith("sha256-tree:"), spec["bundle_ref"]
for marker in ("/Users/", "/home/", "/private/", "/var/folders/"):
assert marker not in text, marker

View file

@ -75,8 +75,8 @@ from llm_ingestion_okf.quality import ( # noqa: E402
FIXTURES = REPO / "tests" / "fixtures" / "retrieval" FIXTURES = REPO / "tests" / "fixtures" / "retrieval"
#: The hold-out registration. The row reads a path so a test can drive both #: The hold-out registration. Absent today, which is what makes row 5 red; the
#: directions without editing code. #: row reads a path so a test can drive both directions without editing code.
HOLDOUT_REGISTRATION = FIXTURES / "holdout-registration.json" HOLDOUT_REGISTRATION = FIXTURES / "holdout-registration.json"
GREEN = "GREEN" GREEN = "GREEN"
@ -434,190 +434,6 @@ BUDGET = BundleSpec(
), ),
) )
# --- three fixtures, one per mechanism the default ranking runs ---------------
#
# Until 2026-09-22 row 7 left three mutants standing (M06, M07, M10): every
# synthetic concept was short and opened with its own title, so the passage
# signal, the title/path weight and the fusion constant never DECIDED a
# delivery here, and switching any of them off moved 0 ranks. Each bundle below
# is built so that exactly one of them does. None of them is a tuning of the
# ranker: the corpus is pinned (`SPECS_SHA256`) and `src/` is untouched.
_NOTE = "Styret gjennomgaar notatet og foerer det inn i arkivet. Sekretaeren sender kopi. "
#: THE PASSAGE SIGNAL. One long concept answers in ONE window of its body; ten
#: short concepts carry the question's words in their TITLES and none in their
#: bodies. The field signal reads ten title-weighted matches in short documents
#: above one long body, so only the passage signal -- the best WINDOW, not the
#: whole concept -- puts the gold first. Remove the body windows (M06) and the
#: passage signal reads titles alone, where the gold has none of the words.
PASSAGE = BundleSpec(
"retrieval-passage",
(
DocumentSpec(
"aarbok",
"aarbok.md",
(
ConceptSpec(
"kapittel-tre",
"Kapittel tre i aarboka",
_NOTE * 15
+ "Varmekablene i taket kontrolleres av vaktmesteren hver oktober. "
+ _NOTE * 15,
),
),
),
*(
DocumentSpec(
f"rundskriv-{number:02d}",
f"rundskriv-{number:02d}.md",
(
ConceptSpec(
f"skriv-{number:02d}",
f"Varmekablene i taket, skriv {number:02d}",
"Skrivet er sendt ut og arkivert.",
),
),
)
for number in range(1, 11)
),
),
)
#: THE TITLE/PATH WEIGHT. The gold is named by its PATH alone -- the question's
#: `kanopadling` is its directory and its source file and occurs in no body --
#: while ten decoys carry the question's other word more densely than the gold
#: does, one fewer time each. The path weight puts the gold first in the field
#: signal, which lifts it into k (rank 4 at `k` = 6). Weigh no title and no path
#: (M07) and both signals read bodies only, where the decoys outrank it and it
#: falls out of k.
PATH = BundleSpec(
"retrieval-path",
(
DocumentSpec(
"kanopadling",
"kanopadling.md",
(ConceptSpec("regel", "Regel om vester", "Vester brukes paa vannet."),),
),
*(
DocumentSpec(
f"turnotat-{number:02d}",
f"turnotat-{number:02d}.md",
(
ConceptSpec(
f"notat-{number:02d}",
f"Turnotat {number:02d}",
" ".join(["Vester henger i boden."] * (11 - number))
+ " Notatet er arkivert.",
),
),
)
for number in range(1, 11)
),
),
)
_FUSION_NOTE = "Styret gjennomgaar saken og sekretaeren arkiverer den. "
#: THE FUSION CONSTANT. Reciprocal-rank fusion trades a concept with one very
#: good rank against one with two middling ranks, and `RRF_K` sets the price.
#: The gold is FIRST in the passage signal and 21st in the field signal (a long
#: body, below ten short `isbading-*` path matches, nine longer ones and the
#: decoy); the one decoy is 10th and 11th, a smaller rank SUM (19 against 20).
#: A flattened fusion (M10, `RRF_K` = 10 000, where only the sum counts) puts
#: the decoy first and `k` = 1 no longer reaches the gold. At the shipped
#: `RRF_K` = 60 the gold leads, and measured over K in steps of 10 it keeps the
#: lead through K = 180 and loses it at 190. The eight long log pages hold the
#: decoy's passage rank at 10th and rank below the gold in both signals.
FUSION = BundleSpec(
"retrieval-fusion",
(
DocumentSpec(
"badebok",
"badebok.md",
(
ConceptSpec(
"kapittel",
"Kapittel i badeboka",
_FUSION_NOTE
* 12
+ "Isbading ved brygga, isbading om morgenen, isbading med vakt, "
"isbading hver dag. " + _FUSION_NOTE * 12,
),
),
),
*(
DocumentSpec(
f"logg-{number:02d}",
f"logg-{number:02d}.md",
(
ConceptSpec(
"side",
f"Loggside {number:02d}",
_FUSION_NOTE * 16
+ "Isbading ved brygga, isbading om kvelden, isbading med vakt. "
+ _FUSION_NOTE * 16,
),
),
)
for number in range(8)
),
DocumentSpec(
"isbading-lapp-00",
"isbading-lapp-00.md",
(ConceptSpec("lapp", "Lapp 00", _FUSION_NOTE * 5 + "Isbading er tillatt."),),
),
*(
DocumentSpec(
f"isbading-{number:02d}",
f"isbading-{number:02d}.md",
(ConceptSpec("oppslag", f"Oppslag {number:02d}", "Oppslaget er arkivert."),),
)
for number in range(10)
),
*(
DocumentSpec(
f"isbading-arkiv-{number:02d}",
f"isbading-arkiv-{number:02d}.md",
(ConceptSpec("mappe", f"Mappe {number:02d}", _FUSION_NOTE * 10),),
)
for number in range(9)
),
),
)
_ARCHIVE_LINE = "Styret gjennomgaar notatet og foerer det inn i arkivet. Sekretaeren sender kopi."
#: THE PASSAGE DELIVERY. Every concept above is shorter than
#: `consume.PASSAGE_CHARS`, so no synthetic payload carried a passage and a
#: mutant of one (M15) could not be felled. One concept of about 6 600
#: characters: a heading ten lines in, the answer thirty lines below it, so the
#: delivery is heading + `[...]` + span + `[...]` and the span carries the
#: citation. Rows 1 and 6 read it as a hit only because the judge reads a
#: passage as its exact reconstruction; M15 adds one sentence and fells both.
DELIVERY = BundleSpec(
"retrieval-delivery",
(
DocumentSpec(
"husbok",
"husbok.md",
(
ConceptSpec(
"drift",
"Drift av huset",
"\n".join(
[_ARCHIVE_LINE] * 10
+ ["## Teknisk rom"]
+ [_ARCHIVE_LINE] * 30
+ ["Varmepumpa i kjelleren faar service av roerleggeren hvert aar."]
+ [_ARCHIVE_LINE] * 40
),
),
),
),
),
)
SPECS: Mapping[str, BundleSpec] = { SPECS: Mapping[str, BundleSpec] = {
"positive": POSITIVE, "positive": POSITIVE,
"miss": MISS, "miss": MISS,
@ -625,10 +441,6 @@ SPECS: Mapping[str, BundleSpec] = {
"budget": BUDGET, "budget": BUDGET,
"lookup": LOOKUP, "lookup": LOOKUP,
"quota": QUOTA, "quota": QUOTA,
"passage": PASSAGE,
"path": PATH,
"fusion": FUSION,
"delivery": DELIVERY,
} }
@ -671,7 +483,7 @@ def specs_digest(specs: Mapping[str, BundleSpec] = SPECS) -> str:
#: The synthetic corpus, pinned the way the sets are. #: The synthetic corpus, pinned the way the sets are.
SPECS_SHA256 = "089a6a9770c03c59fff2da22430e5f9aeb2b2374c49aed419bae99409a2a5cb9" SPECS_SHA256 = "23c447add9a58ee2e454ef5d5b92d5c3c97d441cd883d203aaf2f275dd2f68f9"
def synthetic_bundles(root: Path, specs: Mapping[str, BundleSpec] = SPECS) -> dict[str, Path]: def synthetic_bundles(root: Path, specs: Mapping[str, BundleSpec] = SPECS) -> dict[str, Path]:
@ -836,8 +648,6 @@ SYNTHETIC_SETS: dict[str, str] = {
"set-signals.json": "17d83f305a10af8dd2a45b72943c1285704d23ed074ecb549c05e5fac3e137e9", "set-signals.json": "17d83f305a10af8dd2a45b72943c1285704d23ed074ecb549c05e5fac3e137e9",
"set-quota.json": "61fda652719d7403ddf9701d50e914c46572a4fda77dc0336838695e18c498cf", "set-quota.json": "61fda652719d7403ddf9701d50e914c46572a4fda77dc0336838695e18c498cf",
"set-controls.json": "c2894656326e5a69ec7063fdc280e763910d20b4cbc2c0124f639283864ce506", "set-controls.json": "c2894656326e5a69ec7063fdc280e763910d20b4cbc2c0124f639283864ce506",
"set-mechanisms.json": "435ce620c30c94b0d151335883583520490482be2ade6797524ae928a9f1eab7",
"set-passage.json": "93b6de8ef7012c84516fe9aee560924d4b579669fb046e6eadc70dbf444eb627",
} }
@ -1001,79 +811,6 @@ def _carries(text: str, quote: str) -> bool:
return not quote or _flat(quote) in _flat(text) return not quote or _flat(quote) in _flat(text)
#: What class e says when a passage is not its own reconstruction.
PASSAGE_NOT_RECONSTRUCTED = (
"the delivered passage is not its reconstruction from the bundle's bytes"
)
def passage_span(text: str, passage: object, body: str) -> str | None:
"""The span a passage delivery carries, or None when `text` is not EXACTLY
what the bundle's bytes rebuild.
Since v1.1 C1 a long concept is delivered as `[heading]`, `[...]`, the
span, `[...]` (`consume.as_passage`), with `passage = {start, end, of}`.
Two traps PM measured 2026-09-22, both held here:
- **The offsets count in the DELIVERED body** (`ConceptView.body`), never
in the concept file: read in the file they land a frontmatter's length
off, and 0 of 12 real spans matched.
- **"The span occurs somewhere in the text" is not a check.** It accepts
an invented sentence beside the span. Every character of `text` must be
accounted for: the span is `body[start:end]` byte for byte, a marker
stands exactly where the span leaves text out and nowhere else, and the
one other line allowed is a heading line of the body ABOVE the span (or a
prefix of one -- `as_passage` cuts a long heading). WHICH heading is the
product's choice and is not re-derived here; that it is the bundle's
bytes is the guarantee.
The citation is then read in the span alone -- not in the heading, not in
the markers, and not across the seam between them, which is no sequence
the concept file holds.
"""
if not isinstance(passage, Mapping):
return None
start, end, of = passage.get("start"), passage.get("end"), passage.get("of")
if not all(
isinstance(value, int) and not isinstance(value, bool) for value in (start, end, of)
):
return None
assert isinstance(start, int) and isinstance(end, int)
if of != len(body) or not 0 <= start < end <= len(body):
return None
span = body[start:end]
tail = f"\n{consume.PASSAGE_ELISION}" if end < len(body) else ""
if not text.endswith(span + tail):
return None
head = text[: len(text) - len(span + tail)]
if start == 0:
return span if head == "" else None
marker = f"{consume.PASSAGE_ELISION}\n"
if head == marker:
return span
if not head.endswith(f"\n{marker}"):
return None
heading = head[: -len(f"\n{marker}")]
above = body[:start].split("\n")
if heading.startswith("#") and any(
line.startswith("#") and line.startswith(heading) for line in above
):
return span
return None
def _judged(excerpt: Mapping[str, object], body: str) -> tuple[str, bool, str]:
"""(the text a citation is looked for in, whether it is the bundle's
bytes, the detail when it is not) for one delivered excerpt."""
text = str(excerpt.get("text", ""))
if "passage" in excerpt:
span = passage_span(text, excerpt["passage"], body)
if span is None:
return text, False, PASSAGE_NOT_RECONSTRUCTED
return span, True, ""
return text, _flat(text) == _flat(body), "the delivered text is not the bundle's bytes"
def measure_units(bundle: Path, question: Question) -> list[Unit]: def measure_units(bundle: Path, question: Question) -> list[Unit]:
"""One question, measured at the shipped defaults, plus the quota-off run """One question, measured at the shipped defaults, plus the quota-off run
that says what the truth of a withheld concept is. that says what the truth of a withheld concept is.
@ -1120,21 +857,18 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
for concept_id in holding for concept_id in holding
if _carries(index.concepts[concept_id].body, fasit.quote) if _carries(index.concepts[concept_id].body, fasit.quote)
) )
# A whole-body delivery is judged whole; a passage is judged as its hit_ids = [
# exact reconstruction, and its citation is read in its span alone. concept_id
judged = {
concept_id: _judged(delivered[concept_id], index.concepts[concept_id].body)
for concept_id in holding for concept_id in holding
if concept_id in delivered if concept_id in delivered
} and _carries(str(delivered[concept_id].get("text", "")), fasit.quote)
hit_ids = [
concept_id for concept_id, (text, _, _) in judged.items() if _carries(text, fasit.quote)
] ]
# The payload SAYS it delivered this; the bundle says what it is. # The payload SAYS it delivered this; the bundle says what it is.
confirmed: bool | None = None confirmed: bool | None = None
not_bytes = ""
for concept_id in hit_ids: for concept_id in hit_ids:
_, confirmed, not_bytes = judged[concept_id] confirmed = _flat(str(delivered[concept_id].get("text", ""))) == _flat(
index.concepts[concept_id].body
)
if confirmed: if confirmed:
break break
# The concept this entry is really about: the one the bundle holds the # The concept this entry is really about: the one the bundle holds the
@ -1164,7 +898,7 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
else "no concept in the bundle answers to this name" else "no concept in the bundle answers to this name"
) )
elif hit_ids and not confirmed: elif hit_ids and not confirmed:
klass, detail = "e", not_bytes klass, detail = "e", "the delivered text is not the bundle's bytes"
elif target in delivered: elif target in delivered:
klass = "d" klass = "d"
detail = "delivered without the citation" + ( detail = "delivered without the citation" + (
@ -1642,55 +1376,9 @@ def _as_share(value: object) -> float | None:
return number return number
#: A registration is committed to a PUBLIC repository and read on more than
#: one machine, so every path it names is written under the home directory
#: and expanded when read. An absolute path names ONE machine -- it is refused
#: with its reason and never followed, so a machine path cannot be registered
#: again without row 5 saying so.
HOME_PREFIX = "~/"
def _home_path(value: object) -> tuple[Path | None, str]:
"""A registered path expanded against the home directory, or None and
the reason it is refused. The text returned beside a path is the
registration's own `~/...`, so a row names what was registered and never
the machine it was read on."""
text = value.strip() if isinstance(value, str) else ""
if not text:
return None, "not named"
if not text.startswith(HOME_PREFIX):
kind = "an absolute path" if Path(text).is_absolute() else "not under the home directory"
return None, f"`{text}` is {kind}; a registration names `{HOME_PREFIX}...` and no machine"
return Path(text).expanduser(), text
def _registered_bundle(spec: Mapping[str, object]) -> tuple[Path | None, str]:
"""The bundle the registration names, IF its tree still measures the
`bundle_ref` the registration pins; otherwise None and why.
A bundle path is a place, and a place can be filled with other bytes: the
ref is what makes the path the registered bundle."""
path, text = _home_path(spec.get("bundle"))
if path is None:
return None, text
registered = spec.get("bundle_ref")
if not isinstance(registered, str) or not registered:
return None, "the registration pins no bundle_ref"
if not path.is_dir():
return None, f"the bundle is absent at {text}"
try:
measured = consume.bundle_ref(path)
except Exception as error: # a bundle that cannot be measured is a NO
return None, f"no ref could be measured at {text}: {type(error).__name__}: {error}"
if measured != registered:
return None, f"{text} measures {measured}, the registration pins {registered}"
return path, f"{text} measures the pinned {registered[:24]}"
def _hold_out_verdict( def _hold_out_verdict(
set_path: Path | None, spec: Mapping[str, object],
set_text: str, set_path: Path,
bundle: Path | None,
pinned: str, pinned: str,
threshold: float | None, threshold: float | None,
) -> tuple[bool, str]: ) -> tuple[bool, str]:
@ -1699,20 +1387,19 @@ def _hold_out_verdict(
Every refusal is a NO with its reason and never an exception: a Every refusal is a NO with its reason and never an exception: a
registration naming an absent set is a finding about the registration. registration naming an absent set is a finding about the registration.
A bundle that is not the registered tree is never measured -- a share
read off other bytes is not the registered hold-out's.
""" """
if threshold is None: if threshold is None:
return False, "no numeric threshold to compare with" return False, "no numeric threshold to compare with"
if bundle is None: bundle = str(spec.get("bundle", ""))
return False, "not run: the bundle is not the registered tree" if not bundle:
if set_path is None: return False, "the registration names no bundle to measure against"
return False, f"not run: the set is {set_text}"
if not set_path.is_file(): if not set_path.is_file():
return False, f"the set is absent at {set_text}" return False, f"the set is absent at {_display(set_path)}"
try: try:
question_set = read_hold_out_set(set_path, pinned) question_set = load_set(set_path, pinned)
bundles = {question_set.bundle: bundle} bundles = _bundle_map(bundle)
if list(bundles) == [""] and question_set.bundle:
bundles = {question_set.bundle: bundles[""]}
units = [ units = [
unit unit
for question in question_set.questions for question in question_set.questions
@ -1757,15 +1444,6 @@ def row_five(
registration first and the ranking change second, and it is red today registration first and the ranking change second, and it is red today
because neither has happened. because neither has happened.
AND IT NAMES NO MACHINE, AND THE BUNDLE IT NAMES IS THE ONE IT PINNED.
Until 2026-09-23 the committed registration carried two absolute paths of
the machine it was written on, in a public repository, and `bundle_ref`
was a field no line read -- other bytes at the registered bundle path
would have been measured as the registered bundle. Its paths are now
`~/...`, expanded when read, and an absolute one is a NO; the bundle's
tree is measured with `consume.bundle_ref` and a ref other than the pinned
one is a NO, with the hold-out NOT run against it.
WHAT GIT CANNOT PROVE, stated rather than implied: that nobody read the WHAT GIT CANNOT PROVE, stated rather than implied: that nobody read the
number before writing the threshold. A number can be read from an number before writing the threshold. A number can be read from an
uncommitted working tree, and no history shows that. What history does uncommitted working tree, and no history shows that. What history does
@ -1788,9 +1466,6 @@ def row_five(
" and a bundle named, so the set can be RUN and its share put " " and a bundle named, so the set can be RUN and its share put "
"beside the threshold. A number never compared with a measurement " "beside the threshold. A number never compared with a measurement "
"is a note", "is a note",
" both paths written `~/...` -- the file is public and an "
"absolute path names one machine -- and the bundle's tree ref "
"pinned as `bundle_ref`, so other bytes at that path are not measured",
" and COMMITTED before the ranking moves: git must show the " " and COMMITTED before the ranking moves: git must show the "
f"registration in a commit of its own, with a later commit to " f"registration in a commit of its own, with a later commit to "
f"{RANKING_PATH}. That is the half a session cannot write about itself", f"{RANKING_PATH}. That is the half a session cannot write about itself",
@ -1803,10 +1478,10 @@ def row_five(
checks: list[tuple[str, bool, str]] = [] checks: list[tuple[str, bool, str]] = []
threshold = str(spec.get("threshold", "")) threshold = str(spec.get("threshold", ""))
written_at = str(spec.get("threshold_written_at", "")) written_at = str(spec.get("threshold_written_at", ""))
set_path, set_text = _home_path(spec.get("set")) set_path = Path(str(spec.get("set", "")))
pinned = str(spec.get("sha256", "")) pinned = str(spec.get("sha256", ""))
readings = spec.get("readings", []) readings = spec.get("readings", [])
checks.append(("a set is named", set_path is not None, set_text)) checks.append(("a set is named", bool(str(spec.get("set", ""))), str(set_path)))
checks.append(("a sha256 is pinned", len(pinned) == 64, pinned[:12])) checks.append(("a sha256 is pinned", len(pinned) == 64, pinned[:12]))
number = _as_share(threshold) number = _as_share(threshold)
checks.append( checks.append(
@ -1824,16 +1499,13 @@ def row_five(
str(spec.get("written_by", "")), str(spec.get("written_by", "")),
) )
) )
present = set_path is not None and set_path.is_file()
checks.append( checks.append(
( (
"the pinned bytes are the bytes on disk", "the pinned bytes are the bytes on disk",
present and set_path is not None and sha256_of(set_path) == pinned, set_path.is_file() and sha256_of(set_path) == pinned,
f"present at {set_text}" if present else f"absent at {set_text}", "present" if set_path.is_file() else "absent",
) )
) )
bundle, bundle_note = _registered_bundle(spec)
checks.append(("the bundle is the registered tree", bundle is not None, bundle_note))
early = [ early = [
reading reading
for reading in readings for reading in readings
@ -1851,7 +1523,7 @@ def row_five(
# whole check until 2026-09-20, and `report-only; any number is acceptable # whole check until 2026-09-20, and `report-only; any number is acceptable
# for v1` passed it. The hold-out is run HERE, against the bundle the # for v1` passed it. The hold-out is run HERE, against the bundle the
# registration names, and the row says what it measured. # registration names, and the row says what it measured.
cleared, note = _hold_out_verdict(set_path, set_text, bundle, pinned, number) cleared, note = _hold_out_verdict(spec, set_path, pinned, number)
checks.append(("the measured hold-out clears the threshold", cleared, note)) checks.append(("the measured hold-out clears the threshold", cleared, note))
history = provenance(registration) history = provenance(registration)
@ -1927,10 +1599,8 @@ class Mutant:
expected_row: int expected_row: int
patch: Callable[[], contextlib.AbstractContextManager[None]] patch: Callable[[], contextlib.AbstractContextManager[None]]
#: Why this one may be invisible to a gate that judges DELIVERY. Written #: Why this one may be invisible to a gate that judges DELIVERY. Written
#: for a mutant that survives, so a survivor is a statement about the #: for the two that survive, so a survivor is a statement about the
#: mechanism and not a shrug. None does since 2026-09-22: the three that #: mechanism and not a shrug.
#: carried a note (M06, M07, M10) each got a fixture that makes their
#: mechanism decide a delivery (`PASSAGE`, `PATH`, `FUSION`).
note: str = "" note: str = ""
@ -2017,25 +1687,6 @@ def _extend_delivered() -> contextlib.AbstractContextManager[None]:
return _patched(delivered_text=mutant) return _patched(delivered_text=mutant)
def _extend_passage() -> contextlib.AbstractContextManager[None]:
"""M14's shape for the delivery form v1.1 C1 added: the passage still
CARRIES the citation in its span, and one sentence at the span's end is not
the concept file's. A judge that asked only whether the span occurs
somewhere in the text would count it."""
original = consume.as_passage
def mutant(excerpt: dict[str, object], window: int) -> dict[str, object]:
out = original(excerpt, window)
if "passage" in out:
text = str(out["text"])
tail = f"\n{consume.PASSAGE_ELISION}"
cut = len(text) - len(tail) if text.endswith(tail) else len(text)
out["text"] = f"{text[:cut]} En setning som ikke staar i konseptfila.{text[cut:]}"
return out
return _patched(as_passage=mutant)
def _drop_text_key() -> contextlib.AbstractContextManager[None]: def _drop_text_key() -> contextlib.AbstractContextManager[None]:
original = consume.excerpt_for original = consume.excerpt_for
@ -2066,12 +1717,23 @@ MUTANTS: tuple[Mutant, ...] = (
Mutant("M04 the ranking is reversed", 1, _reverse_scores), Mutant("M04 the ranking is reversed", 1, _reverse_scores),
Mutant("M05 the source quota is removed", 3, lambda: _wrap_cut(source_quota=None)), Mutant("M05 the source quota is removed", 3, lambda: _wrap_cut(source_quota=None)),
Mutant( Mutant(
"M06 the passage signal reads no body", 1, lambda: _patched_bm25(windows=lambda body: []) "M06 the passage signal reads no body",
1,
lambda: _patched_bm25(windows=lambda body: []),
note=(
"the field signal still reads every body, and no synthetic concept is "
"long enough for its best window to decide a delivery"
),
), ),
Mutant( Mutant(
"M07 the field signal weighs no title and no path", "M07 the field signal weighs no title and no path",
1, 1,
lambda: _patched_bm25(TITLE_WEIGHT=0, PATH_WEIGHT=0), lambda: _patched_bm25(TITLE_WEIGHT=0, PATH_WEIGHT=0),
note=(
"every synthetic body opens with its own title as a heading "
"(`build_bundle`), so the weight only re-weights words the body "
"already carries"
),
), ),
Mutant("M08 every token matches every token", 4, _every_term_everything), Mutant("M08 every token matches every token", 4, _every_term_everything),
Mutant("M09 no stem, no prefix: equality only", 1, _equality_only), Mutant("M09 no stem, no prefix: equality only", 1, _equality_only),
@ -2079,6 +1741,10 @@ MUTANTS: tuple[Mutant, ...] = (
"M10 the fusion is flattened (bm25.RRF_K = 10 000)", "M10 the fusion is flattened (bm25.RRF_K = 10 000)",
1, 1,
lambda: _patched_bm25(RRF_K=10_000), lambda: _patched_bm25(RRF_K=10_000),
note=(
"over two signals K reorders only a concept whose two ranks lie far "
"apart against one whose ranks are close, and no fixture here holds that"
),
), ),
Mutant("M11 the cut takes the LAST k", 1, _last_k), Mutant("M11 the cut takes the LAST k", 1, _last_k),
Mutant("M12 the delivered text is truncated to 40 characters", 6, _truncate_delivered), Mutant("M12 the delivered text is truncated to 40 characters", 6, _truncate_delivered),
@ -2086,7 +1752,6 @@ MUTANTS: tuple[Mutant, ...] = (
Mutant( Mutant(
"M14 the delivered text carries a sentence the concept file does not", 6, _extend_delivered "M14 the delivered text carries a sentence the concept file does not", 6, _extend_delivered
), ),
Mutant("M15 the passage carries a sentence the concept file does not", 6, _extend_passage),
) )
@ -2113,11 +1778,10 @@ MUTANT_ROSTER: tuple[str, ...] = (
"M12 the delivered text is truncated to 40 characters", "M12 the delivered text is truncated to 40 characters",
"M13 the excerpt carries no text", "M13 the excerpt carries no text",
"M14 the delivered text carries a sentence the concept file does not", "M14 the delivered text carries a sentence the concept file does not",
"M15 the passage carries a sentence the concept file does not",
) )
#: The roster's length, written as a number so appending is not one edit. #: The roster's length, written as a number so appending is not one edit.
MUTANT_COUNT = 15 MUTANT_COUNT = 14
def _score(rows: Sequence[Row]) -> dict[int, int]: def _score(rows: Sequence[Row]) -> dict[int, int]:
@ -2274,72 +1938,23 @@ def read_real_set(name: str, path: Path, expected_sha256: str) -> QuestionSet:
) )
spec = json.loads(raw.decode("utf-8")) spec = json.loads(raw.decode("utf-8"))
if name == "wiki": if name == "wiki":
return QuestionSet("wiki-20", "wiki", path, measured, _wiki_questions(spec), ()) # The set's own `hit_rule`, verbatim: an excerpt whose `source_file` is
# the fasit's document AND whose text carries the fasit's quote.
questions = tuple(
Question(
id=str(entry["id"]),
question=str(entry["question"]),
fasit=tuple(
Fasit(by="source_file", value=f"{item['doc']}.md", quote=str(item["quote"]))
for item in entry["fasit"]
),
)
for entry in spec["questions"]
)
return QuestionSet("wiki-20", "wiki", path, measured, questions, ())
raise GateUsage(f"unknown real set `{name}`; the one real set is `wiki`") raise GateUsage(f"unknown real set `{name}`; the one real set is `wiki`")
def _wiki_questions(spec: Mapping[str, Any]) -> tuple[Question, ...]:
"""The wiki set's own `hit_rule`, verbatim: an excerpt whose `source_file`
is the fasit's document AND whose text carries the fasit's quote. One
reading of the schema, shared by row 8 and row 5."""
return tuple(
Question(
id=str(entry["id"]),
question=str(entry["question"]),
fasit=tuple(
Fasit(by="source_file", value=f"{item['doc']}.md", quote=str(item["quote"]))
for item in entry["fasit"]
),
)
for entry in spec["questions"]
)
#: The schemas a registered hold-out set may declare in its own `schema`
#: field, and the bundle key its questions are read under. The SET says how
#: it is read, so the registration carries nothing about it; a set with no
#: `schema` is in this gate's own synthetic form (`load_set`), and a schema
#: not named here is a NO with its name -- a set read in a shape it was not
#: written in measures nothing.
HOLD_OUT_SCHEMAS: Mapping[str, str] = {"fase-sporsmaal/1": "wiki"}
def read_hold_out_set(path: Path, expected_sha256: str) -> QuestionSet:
"""The registered hold-out set, read in the schema it declares.
Raises GateUsage for a set that is not the pinned bytes or declares a
schema this gate does not read; `_hold_out_verdict` turns either into a
NO with its reason.
"""
try:
raw = path.read_bytes()
except OSError as error:
raise GateUsage(f"cannot read the question set {path}: {error}") from error
measured = hashlib.sha256(raw).hexdigest()
if measured != expected_sha256:
raise GateUsage(
f"{path}: expected sha256 {expected_sha256}, measured {measured}; "
"refusing to measure a set that is not the set that was pinned"
)
try:
spec = json.loads(raw.decode("utf-8"))
except (UnicodeDecodeError, json.JSONDecodeError) as error:
raise GateUsage(f"{path}: not readable as JSON: {error}") from error
schema = spec.get("schema") if isinstance(spec, dict) else None
if schema is None:
return load_set(path, expected_sha256)
if schema not in HOLD_OUT_SCHEMAS:
raise GateUsage(
f"the set's schema `{schema}` is not one this gate reads "
f"({', '.join(HOLD_OUT_SCHEMAS)}, or no `schema` for this gate's own form)"
)
try:
questions = _wiki_questions(spec)
except (KeyError, TypeError, ValueError) as error:
raise GateUsage(f"{path}: not a `{schema}` set this gate can read: {error}") from error
return QuestionSet("hold-out", HOLD_OUT_SCHEMAS[schema], path, measured, questions, ())
#: The sets row 8 is the measurement of, by name: a run that hands over some #: The sets row 8 is the measurement of, by name: a run that hands over some
#: of them has measured some of them, and the row says so. Left to #: of them has measured some of them, and the row says so. Left to
#: `len(real)` the row came back `6 of 6 GREEN` on one set of three (PM's J2, #: `len(real)` the row came back `6 of 6 GREEN` on one set of three (PM's J2,