Compare commits
No commits in common. "main" and "v1.1.0" have entirely different histories.
8 changed files with 103 additions and 1049 deletions
42
CLAUDE.md
42
CLAUDE.md
|
|
@ -1318,37 +1318,15 @@ and fixtures, never code.
|
||||||
what a bundle looks like, this asks whether the payload for a question
|
what a bundle looks like, this asks whether the payload for a question
|
||||||
carries the fasit. **Written RED, no capability**: eight rows, exit 0 only
|
carries the fasit. **Written RED, no capability**: eight rows, exit 0 only
|
||||||
when all are green, 1 otherwise, 2 on wrong input, `--json`. Rows 1-4, 6, 7
|
when all are green, 1 otherwise, 2 on wrong input, `--json`. Rows 1-4, 6, 7
|
||||||
run against a synthetic corpus this repo GENERATES and eight sets it ships
|
run against a synthetic corpus this repo GENERATES and six sets it ships
|
||||||
pinned by sha256 -- no network, no private corpus, no clock; a set whose
|
pinned by sha256 -- no network, no private corpus, no clock; a set whose
|
||||||
bytes moved is exit 2, never a quiet continuation, because **a gold set is
|
bytes moved is exit 2, never a quiet continuation, because **a gold set is
|
||||||
an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and
|
an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and
|
||||||
this repo is public). **THE CORPUS IS PINNED THE SAME WAY** since
|
this repo is public). **THE CORPUS IS PINNED THE SAME WAY** since
|
||||||
2026-09-19 (`SPECS_SHA256`). Today (measured on the default run
|
2026-09-19 (`SPECS_SHA256`). Today (measured on the default run
|
||||||
2026-09-22): **1 (14/14) - 2 (7/7) - 3 (5/5) - 4 (6/6) - 5 (0/1) -
|
2026-09-21, v1.1 K): **1 (10/10) - 2 (7/7) - 3 (5/5) - 4 (6/6) - 5 (0/1) -
|
||||||
6 (14/14) - 7 (15/15) - 8 (0 of 1 set, NOT RUN without `--real`)**,
|
6 (10/10) - 7 (11/14) - 8 (0 of 1 set, NOT RUN without `--real`)**,
|
||||||
`GATE RED: rows 5, 8`. **Row 7 fells 14 of 14 since 2026-09-22**: the three
|
`GATE RED: rows 5, 7, 8`.
|
||||||
that survived v1.1 (M06 passage body, M07 title/path weight, M10
|
|
||||||
`bm25.RRF_K`) each got a fixture that makes its mechanism decide a delivery
|
|
||||||
(`PASSAGE`, `PATH`, `FUSION`, set `set-mechanisms.json`; the fusion one keeps
|
|
||||||
its lead through K = 180 and loses it at 190), with `src/` untouched.
|
|
||||||
**THE JUDGE READS A PASSAGE AS A PASSAGE (2026-09-22).** It predated v1.1
|
|
||||||
C1 and required a delivered text to be the whole concept's bytes, so every
|
|
||||||
excerpt `as_passage` cut was class (e) even when its span carried the
|
|
||||||
citation -- the dominant miss class when row 8 was run on the real set.
|
|
||||||
`passage_span` now accepts a passage only when the delivered text IS its
|
|
||||||
exact reconstruction: the span is `body[start:end]` byte for byte, `of` is
|
|
||||||
the body's length, a `[...]` stands exactly where text is left out, and the
|
|
||||||
one other line allowed is a heading line of the body above the span (or a
|
|
||||||
prefix of one); WHICH heading is the product's choice and is not re-derived.
|
|
||||||
The citation is read in the SPAN alone, never in the heading, the markers or
|
|
||||||
across the seam. Two traps are pinned by tests: the offsets count in the
|
|
||||||
DELIVERED body (read in the concept file they land a frontmatter's length
|
|
||||||
off), and "the span occurs somewhere in the text" accepts an invented
|
|
||||||
sentence beside it. **M15** (a passage carrying one sentence the file does
|
|
||||||
not, M14's shape for passages) is row 7's fifteenth mutant, felled through a
|
|
||||||
new fixture that delivers a passage (`DELIVERY`, set `set-passage.json`),
|
|
||||||
because no synthetic concept was long enough to be cut. `src/` untouched;
|
|
||||||
`MUTANT_BAR` unchanged.
|
|
||||||
**THE K2 TEST TRACK IS RETIRED (operator 2026-09-21, v1.1 K)**: material
|
**THE K2 TEST TRACK IS RETIRED (operator 2026-09-21, v1.1 K)**: material
|
||||||
tied to the operator's employer is not re-measured and not frozen, and
|
tied to the operator's employer is not re-measured and not frozen, and
|
||||||
public tests and gates run on invented material. Row 9 (K2) and the `r761`
|
public tests and gates run on invented material. Row 9 (K2) and the `r761`
|
||||||
|
|
@ -1357,8 +1335,8 @@ and fixtures, never code.
|
||||||
the R761/N101/N200 deliveries; `okf_accounting_gate.py` has no default real
|
the R761/N101/N200 deliveries; `okf_accounting_gate.py` has no default real
|
||||||
corpus. The synthetic corpus was re-measured for BM25 (`MISS`, `LOOKUP`,
|
corpus. The synthetic corpus was re-measured for BM25 (`MISS`, `LOOKUP`,
|
||||||
`QUOTA`), so rows 2 and 3 are green through their own fixtures again, and
|
`QUOTA`), so rows 2 and 3 are green through their own fixtures again, and
|
||||||
row 7's mutants patch `bm25` where the default runs there; the three that
|
row 7's mutants patch `bm25` where the default runs there; three survive
|
||||||
survived with 0 ranks moved are felled since 2026-09-22 (above). What
|
(passage body, field title weight, `bm25.RRF_K`) with 0 ranks moved. What
|
||||||
the text below says about rows 8/9, K2 and the three sets is history.
|
the text below says about rows 8/9, K2 and the three sets is history.
|
||||||
**FOUR OF EIGHT CHEATING ATTACKS WENT THROUGH IT AND ARE CLOSED
|
**FOUR OF EIGHT CHEATING ATTACKS WENT THROUGH IT AND ARE CLOSED
|
||||||
(2026-09-19).** PM's checkpoint on `2c8296b` found rows 3, 5, 7 and 8 could
|
(2026-09-19).** PM's checkpoint on `2c8296b` found rows 3, 5, 7 and 8 could
|
||||||
|
|
@ -1385,13 +1363,7 @@ and fixtures, never code.
|
||||||
itself; three now read GIT (committed and unmodified, its commit is not
|
itself; three now read GIT (committed and unmodified, its commit is not
|
||||||
itself a ranking change, a ranking change landed AFTER it), the last being
|
itself a ranking change, a ranking change landed AFTER it), the last being
|
||||||
the one that cannot be self-attested. What git cannot prove -- that nobody
|
the one that cannot be self-attested. What git cannot prove -- that nobody
|
||||||
read the number first -- is stated, not implied. Since 2026-09-23 row 5
|
read the number first -- is stated, not implied. **Row 7's bar** was 90 % of
|
||||||
reads the hold-out set in the schema the SET declares
|
|
||||||
(`HOLD_OUT_SCHEMAS`: `fase-sporsmaal/1` goes through row 8's own adapter
|
|
||||||
and `hit_rule`, no `schema` is the gate's synthetic form, any other schema
|
|
||||||
is a NO naming it) -- it read every set with the synthetic reader, so a set
|
|
||||||
in a consumer's schema was a NO nothing could turn. It prints the share and
|
|
||||||
never which hold-out question missed, and a test holds that. **Row 7's bar** was 90 % of
|
|
||||||
a list in the file a capability session edits, so seven duplicate `k = 1`
|
a list in the file a capability session edits, so seven duplicate `k = 1`
|
||||||
mutants read `18 of 20 GREEN`; `MUTANT_ROSTER` and `MUTANT_COUNT` are pinned
|
mutants read `18 of 20 GREEN`; `MUTANT_ROSTER` and `MUTANT_COUNT` are pinned
|
||||||
apart from `MUTANTS`, duplicates are refused, and the bar is the roster's.
|
apart from `MUTANTS`, duplicates are refused, and the bar is the roster's.
|
||||||
|
|
|
||||||
47
README.md
47
README.md
|
|
@ -84,13 +84,13 @@ here was measured; none of it is a plan.
|
||||||
`--no-assets` takes the image path out entirely.
|
`--no-assets` takes the image path out entirely.
|
||||||
- **Three of this repository's own gates are RED, and each red row is a stated
|
- **Three of this repository's own gates are RED, and each red row is a stated
|
||||||
finding rather than a bug to be surprised by.** The retrieval gate is red on
|
finding rather than a bug to be surprised by.** The retrieval gate is red on
|
||||||
rows 5 and 8, the MCP gate on row 2, and the content accounting's judge
|
rows 5, 7 and 8, the MCP gate on row 2, and the content accounting's judge
|
||||||
on rows 2, 3 and 6. For a user that means: no ranking change has yet been
|
on rows 2, 3 and 6. For a user that means: no ranking change has yet been
|
||||||
registered against a held-out set (row 5), the real-corpus row is NOT RUN
|
registered against a held-out set (row 5), the real-corpus row is NOT RUN
|
||||||
without its private set and so is red rather than green by omission
|
without its private sets and so is red rather than green by omission
|
||||||
(row 8) -- and since 2026-09-22 its judge reads an excerpt v1.1 cuts to its
|
(row 8), three mechanical mutants of the BM25 ranking survive with 0 ranks
|
||||||
answering passage as that passage, accepted only when it is the exact
|
moved — the passage body, the title field weight and the fusion constant —
|
||||||
reconstruction from the bundle's bytes. MCP anchors and concept ids are different
|
so the gate cannot yet tell those parts from their absence (row 7), MCP anchors and concept ids are different
|
||||||
vocabularies so `okf_fetch` cannot be addressed with a set's anchor (row 2),
|
vocabularies so `okf_fetch` cannot be addressed with a set's anchor (row 2),
|
||||||
and the accounting still reports real losses on the reference corpus (rows 2,
|
and the accounting still reports real losses on the reference corpus (rows 2,
|
||||||
3, 6). The rows and their numbers are under [Judge the
|
3, 6). The rows and their numbers are under [Judge the
|
||||||
|
|
@ -1196,18 +1196,18 @@ set names a consumer's documents and this repository is public. **The corpus
|
||||||
is pinned the same way** (`SPECS_SHA256`): every row counts against those
|
is pinned the same way** (`SPECS_SHA256`): every row counts against those
|
||||||
documents, so moving them without moving the pin is exit 2.
|
documents, so moving them without moving the pin is exit 2.
|
||||||
|
|
||||||
**It is RED today, on rows 5 and 8**, and each of those is a
|
**It is RED today, on rows 5, 7 and 8**, and each of those is a
|
||||||
finding rather than a defect in the gate:
|
finding rather than a defect in the gate:
|
||||||
|
|
||||||
| row | what it asks | today |
|
| row | what it asks | today |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| 1 | hit@payload, one fasit entry = one unit | 14 of 14 |
|
| 1 | hit@payload, one fasit entry = one unit | 10 of 10 |
|
||||||
| 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 |
|
| 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 |
|
||||||
| 3 | the `rule` the payload prints for a withheld fasit is the true one | 5 of 5 |
|
| 3 | the `rule` the payload prints for a withheld fasit is the true one | 5 of 5 |
|
||||||
| 4 | an uncovered question comes back marked, a covered one does not | 6 of 6 |
|
| 4 | an uncovered question comes back marked, a covered one does not | 6 of 6 |
|
||||||
| 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 |
|
| 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 |
|
||||||
| 6 | every delivery confirmed against the bundle's own bytes | 14 of 14 |
|
| 6 | every delivery confirmed against the bundle's own bytes | 10 of 10 |
|
||||||
| 7 | mechanical mutants of the ranking and the cut, felled | 15 of 15 |
|
| 7 | mechanical mutants of the ranking and the cut, felled | 11 of 14 |
|
||||||
| 8 | the real set, from path + sha256 | 0 of 1 set, NOT RUN without `--real` |
|
| 8 | the real set, from path + sha256 | 0 of 1 set, NOT RUN without `--real` |
|
||||||
|
|
||||||
**The public rows run on invented material only** (operator decision
|
**The public rows run on invented material only** (operator decision
|
||||||
|
|
@ -1230,28 +1230,13 @@ delivered excerpt answers — facts, no verdict, and the one bar is the gate's
|
||||||
own `UNANSWERED_BAR = 2/3` — 3 of 6 to 6 of 6.
|
own `UNANSWERED_BAR = 2/3` — 3 of 6 to 6 of 6.
|
||||||
|
|
||||||
Row 7's mutants patch the code the DEFAULT ranking runs (BM25 since v1.1),
|
Row 7's mutants patch the code the DEFAULT ranking runs (BM25 since v1.1),
|
||||||
and it fells all 15. Until 2026-09-22 three survived with **0 ranks and 0
|
and it reports three survivors with what they moved rather than with a shrug:
|
||||||
deliveries** moved -- the passage signal reading no body, the field signal
|
the passage signal reading no body, the field signal weighing no title and no
|
||||||
weighing no title and no path, and a flattened fusion (`bm25.RRF_K`) -- because
|
path, and a flattened fusion (`bm25.RRF_K`) each moved **0 ranks and 0
|
||||||
every synthetic concept was short and opened with its own title, so none of the
|
deliveries** on these fixtures. Each has a mechanism: the field signal still
|
||||||
three ever decided a delivery. Three fixtures now make each one decide one
|
reads every body, every synthetic body opens with its own title as a heading,
|
||||||
(`set-mechanisms.json`): a long concept answered in one window of its body
|
and over two signals `K` only reorders a concept whose two ranks lie far apart,
|
||||||
against short concepts whose titles carry the question's words; a concept named
|
which no fixture holds.
|
||||||
by its path alone against decoys denser in the body; and a concept first in one
|
|
||||||
signal and 21st in the other against a decoy 10th and 11th, which a flattened
|
|
||||||
fusion ranks first because only the rank sum then counts. Each fixture is
|
|
||||||
delivered at the shipped defaults and falls out of `k` with its own mechanism
|
|
||||||
switched off; `src/` did not change.
|
|
||||||
|
|
||||||
A delivery is judged against the bundle's own bytes (row 6), and since v1.1 a
|
|
||||||
concept over 4 000 characters is delivered as its answering passage: the
|
|
||||||
nearest heading above it, `[...]`, the span, `[...]`. The judge counts such a
|
|
||||||
passage only when it is EXACTLY that reconstruction -- the span byte for byte
|
|
||||||
the delivered body at the passage's own offsets, markers only where text is
|
|
||||||
left out, the heading a line of the body above the span -- and it reads the
|
|
||||||
citation in the span alone. A looser rule ("the span occurs somewhere in the
|
|
||||||
text") would accept an invented sentence beside the span; the fifteenth
|
|
||||||
mutant (M15) is exactly that, and a fixture that delivers a passage fells it.
|
|
||||||
|
|
||||||
Row 8 is never green by leaving something out: it requires every named set and
|
Row 8 is never green by leaving something out: it requires every named set and
|
||||||
is NOT RUN until it has them — one set of three used to read `6 of 6 GREEN`.
|
is NOT RUN until it has them — one set of three used to read `6 of 6 GREEN`.
|
||||||
|
|
|
||||||
|
|
@ -1,13 +0,0 @@
|
||||||
{
|
|
||||||
"set": "~/repos/claude-code-llm-wiki/eval/okf-holdout-sporsmaal.json",
|
|
||||||
"sha256": "7c45e2dc787b72dec3db2252f6ebc2d14b28d55f7e220bf76b43bfbe28adab82",
|
|
||||||
"bundle": "~/okf/gate-bundles/claude-code-wiki-7141ab3f",
|
|
||||||
"bundle_ref": "sha256-tree:7141ab3f00a20387caa2c22e43761ba625fbe596091303ca6c2069d77d20095c",
|
|
||||||
"bundle_built_with": "llm-ingestion-okf v1.1.0 (43499c0), --gate none; the bundle row 8 measures, copied to a durable path with its ref measured identical",
|
|
||||||
"metric": "questions answered over questions asked",
|
|
||||||
"threshold": 0.8,
|
|
||||||
"threshold_written_at": "2026-09-23T06:56:47Z",
|
|
||||||
"written_by": "claude-code-llm-wiki, order 4350316446, commit 33735b4: written blind, not the session that changes the ranking",
|
|
||||||
"registered_by": "llm-ingestion-okf, order 20260923T080931Z-3809217500, a session that changes no ranking",
|
|
||||||
"readings": []
|
|
||||||
}
|
|
||||||
28
tests/fixtures/retrieval/set-mechanisms.json
vendored
28
tests/fixtures/retrieval/set-mechanisms.json
vendored
|
|
@ -1,28 +0,0 @@
|
||||||
{
|
|
||||||
"set_id": "synthetic-mechanisms",
|
|
||||||
"bundle": "passage",
|
|
||||||
"note": "Three questions, one per mechanism the default BM25 ranking runs and no earlier fixture made decide a delivery: the passage signal's body windows (bundle `passage`), the field signal's title and path weight (bundle `path`), and the rank-fusion constant (bundle `fusion`). Each is delivered at the shipped defaults and falls out of k when its own mechanism is switched off, so row 1 is what fells M06, M07 and M10.",
|
|
||||||
"questions": [
|
|
||||||
{
|
|
||||||
"id": "PA1",
|
|
||||||
"question": "Hvem kontrollerer varmekablene i taket?",
|
|
||||||
"bundle": "passage",
|
|
||||||
"fasit": [{"by": "concept", "value": "aarbok/kapittel-tre", "quote": "kontrolleres av vaktmesteren"}]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "PT1",
|
|
||||||
"question": "Hva gjelder for vester ved kanopadling?",
|
|
||||||
"bundle": "path",
|
|
||||||
"k": 6,
|
|
||||||
"fasit": [{"by": "concept", "value": "kanopadling/regel", "quote": "brukes paa vannet"}]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "FU1",
|
|
||||||
"question": "Hva gjelder for isbading?",
|
|
||||||
"bundle": "fusion",
|
|
||||||
"k": 1,
|
|
||||||
"fasit": [{"by": "concept", "value": "badebok/kapittel", "quote": "isbading om morgenen"}]
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"controls": []
|
|
||||||
}
|
|
||||||
13
tests/fixtures/retrieval/set-passage.json
vendored
13
tests/fixtures/retrieval/set-passage.json
vendored
|
|
@ -1,13 +0,0 @@
|
||||||
{
|
|
||||||
"set_id": "synthetic-passage",
|
|
||||||
"bundle": "delivery",
|
|
||||||
"note": "One question whose answer lies in a concept over consume.PASSAGE_CHARS, so it is delivered as a passage: heading, [...], span, [...]. The span carries the citation. Rows 1 and 6 count it only when the judge accepts a passage that is its exact reconstruction from the bundle's bytes, and M15 (one sentence added to the passage) fells both.",
|
|
||||||
"questions": [
|
|
||||||
{
|
|
||||||
"id": "PD1",
|
|
||||||
"question": "Hvem gir varmepumpa i kjelleren service?",
|
|
||||||
"fasit": [{"by": "concept", "value": "husbok/drift", "quote": "service av roerleggeren"}]
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"controls": []
|
|
||||||
}
|
|
||||||
|
|
@ -1071,7 +1071,6 @@ ARM_ATTRIBUTION = {
|
||||||
|
|
||||||
def test_each_arm_flag_carries_its_attribution_inside_its_own_option_chunk(
|
def test_each_arm_flag_carries_its_attribution_inside_its_own_option_chunk(
|
||||||
capsys: pytest.CaptureFixture[str],
|
capsys: pytest.CaptureFixture[str],
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Each attribution must sit in its own option's help, not merely in the file.
|
"""Each attribution must sit in its own option's help, not merely in the file.
|
||||||
|
|
||||||
|
|
@ -1085,15 +1084,7 @@ def test_each_arm_flag_carries_its_attribution_inside_its_own_option_chunk(
|
||||||
written down. A hard-coded `== 1` is correct only until a second arm is
|
written down. A hard-coded `== 1` is correct only until a second arm is
|
||||||
attributed the same way, and it then goes red on an axis that has nothing
|
attributed the same way, and it then goes red on an axis that has nothing
|
||||||
to do with whether the new arm is right.
|
to do with whether the new arm is right.
|
||||||
|
|
||||||
Colour is switched off for the call. Python 3.14's argparse colours
|
|
||||||
`--help` when `FORCE_COLOR` is set -- a background session inherits
|
|
||||||
`FORCE_COLOR=3` -- and an escape code before ` --outline-run` leaves no
|
|
||||||
chunk starting with the flag. `PYTHON_COLORS=0` is the variable
|
|
||||||
`_colorize.can_colorize` reads FIRST, so it outranks `FORCE_COLOR` and
|
|
||||||
`NO_COLOR` alike; the help's words are the same either way.
|
|
||||||
"""
|
"""
|
||||||
monkeypatch.setenv("PYTHON_COLORS", "0")
|
|
||||||
with pytest.raises(SystemExit) as exit_info:
|
with pytest.raises(SystemExit) as exit_info:
|
||||||
okf_propose_segments.main(["--help"])
|
okf_propose_segments.main(["--help"])
|
||||||
assert exit_info.value.code == 0
|
assert exit_info.value.code == 0
|
||||||
|
|
|
||||||
|
|
@ -394,8 +394,8 @@ def test_row_five_is_red_while_no_hold_out_is_registered(tmp_path: Path) -> None
|
||||||
assert (row.k, row.m, row.status) == (0, 1, gate.RED)
|
assert (row.k, row.m, row.status) == (0, 1, gate.RED)
|
||||||
|
|
||||||
|
|
||||||
def test_row_five_is_green_for_a_registration_that_carries_all_twelve(
|
def test_row_five_is_green_for_a_registration_that_carries_all_eleven(
|
||||||
tmp_path: Path, home: Path
|
tmp_path: Path,
|
||||||
) -> None:
|
) -> None:
|
||||||
registration = _registration(
|
registration = _registration(
|
||||||
tmp_path,
|
tmp_path,
|
||||||
|
|
@ -404,11 +404,11 @@ def test_row_five_is_green_for_a_registration_that_carries_all_twelve(
|
||||||
threshold=0.8,
|
threshold=0.8,
|
||||||
)
|
)
|
||||||
row = gate.row_five(registration, provenance=lambda _: _carried_by_git())
|
row = gate.row_five(registration, provenance=lambda _: _carried_by_git())
|
||||||
assert (row.k, row.m, row.status) == (12, 12, gate.GREEN)
|
assert (row.k, row.m, row.status) == (11, 11, gate.GREEN)
|
||||||
|
|
||||||
|
|
||||||
def test_row_five_falls_on_a_number_read_before_its_threshold_was_written(
|
def test_row_five_falls_on_a_number_read_before_its_threshold_was_written(
|
||||||
tmp_path: Path, home: Path
|
tmp_path: Path,
|
||||||
) -> None:
|
) -> None:
|
||||||
registration = _registration(
|
registration = _registration(
|
||||||
tmp_path,
|
tmp_path,
|
||||||
|
|
@ -420,12 +420,12 @@ def test_row_five_falls_on_a_number_read_before_its_threshold_was_written(
|
||||||
spec["readings"] = [{"at": "2026-09-18T09:00:00Z", "value": "0.62"}]
|
spec["readings"] = [{"at": "2026-09-18T09:00:00Z", "value": "0.62"}]
|
||||||
registration.write_text(json.dumps(spec), encoding="utf-8")
|
registration.write_text(json.dumps(spec), encoding="utf-8")
|
||||||
row = gate.row_five(registration, provenance=lambda _: _carried_by_git())
|
row = gate.row_five(registration, provenance=lambda _: _carried_by_git())
|
||||||
assert (row.k, row.m, row.status) == (11, 12, gate.RED)
|
assert (row.k, row.m, row.status) == (10, 11, gate.RED)
|
||||||
assert any("no reading predates the threshold: NO" in detail for detail in row.details)
|
assert any("no reading predates the threshold: NO" in detail for detail in row.details)
|
||||||
|
|
||||||
|
|
||||||
def test_row_five_falls_when_the_registration_rides_in_on_the_ranking_change(
|
def test_row_five_falls_when_the_registration_rides_in_on_the_ranking_change(
|
||||||
tmp_path: Path, home: Path
|
tmp_path: Path,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""The three git checks, each driven red on its own: a file nobody
|
"""The three git checks, each driven red on its own: a file nobody
|
||||||
committed, a threshold committed together with the ranking change, and a
|
committed, a threshold committed together with the ranking change, and a
|
||||||
|
|
@ -524,37 +524,6 @@ def test_a_mutant_is_felled_by_the_row_that_got_worse_and_never_by_one_that_did_
|
||||||
assert after[3] >= before[3]
|
assert after[3] >= before[3]
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
"prefix",
|
|
||||||
[
|
|
||||||
# The three that survived v1.1: each is a mechanism the default ranking
|
|
||||||
# RUNS, so a gate that cannot see it removed cannot see it break.
|
|
||||||
"M06 ", # the passage signal reads no body
|
|
||||||
"M07 ", # the field signal weighs no title and no path
|
|
||||||
"M10 ", # the fusion is flattened
|
|
||||||
# A passage carrying one sentence the file does not: M14's shape for
|
|
||||||
# the delivery form v1.1 C1 added.
|
|
||||||
"M15 ",
|
|
||||||
],
|
|
||||||
)
|
|
||||||
def test_row_seven_fells_every_mutant_of_a_mechanism_the_ranking_runs(
|
|
||||||
tmp_path: Path, prefix: str
|
|
||||||
) -> None:
|
|
||||||
"""A survivor of row 7 is a mechanism the synthetic corpus never makes
|
|
||||||
decide a delivery. Felled means a row got WORSE on the pinned sets and the
|
|
||||||
pinned corpus -- never a row that merely moved."""
|
|
||||||
(mutant,) = [m for m in gate.MUTANTS if m.label.startswith(prefix)]
|
|
||||||
cases, _ = gate.synthetic_cases(tmp_path / "bundles", FIXTURES)
|
|
||||||
before = {row.number: row.k - row.m for row in gate.deterministic_rows(cases)}
|
|
||||||
with mutant.patch():
|
|
||||||
rows = gate.deterministic_rows(
|
|
||||||
[gate.measure_case(case.question_set, case.bundles) for case in cases]
|
|
||||||
)
|
|
||||||
after = {row.number: row.k - row.m for row in rows}
|
|
||||||
worse = [number for number in sorted(before) if after[number] < before[number]]
|
|
||||||
assert worse, f"{mutant.label} survives: no row got worse ({before} -> {after})"
|
|
||||||
|
|
||||||
|
|
||||||
# --- row 8 --------------------------------------------------------------------
|
# --- row 8 --------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -635,18 +604,14 @@ def test_the_gate_is_red_today_and_says_which_rows(tmp_path: Path) -> None:
|
||||||
by_number = {row.number: row for row in rows}
|
by_number = {row.number: row for row in rows}
|
||||||
assert sorted(by_number) == [1, 2, 3, 4, 5, 6, 7, 8]
|
assert sorted(by_number) == [1, 2, 3, 4, 5, 6, 7, 8]
|
||||||
# Rows 2 and 3 are green again since the synthetic corpus was re-measured
|
# Rows 2 and 3 are green again since the synthetic corpus was re-measured
|
||||||
# for BM25 (2026-09-21). Row 7 fells all 14 since 2026-09-22, when the three
|
# for BM25 (2026-09-21). Row 7 is red on three survivors, each printed with
|
||||||
# v1.1 survivors each got a fixture that makes their mechanism decide a
|
# what it moved.
|
||||||
# delivery (`set-mechanisms.json`, three more row-1 units), and all 15
|
assert [row.number for row in rows if row.fails] == [5, 7, 8]
|
||||||
# since the judge reads a passage as its exact reconstruction and one
|
assert (by_number[1].k, by_number[1].m) == (10, 10)
|
||||||
# fixture delivers a passage (`set-passage.json`, one more unit), so M15
|
|
||||||
# has something to fell.
|
|
||||||
assert [row.number for row in rows if row.fails] == [5, 8]
|
|
||||||
assert (by_number[1].k, by_number[1].m) == (14, 14)
|
|
||||||
assert (by_number[2].k, by_number[2].m) == (7, 7)
|
assert (by_number[2].k, by_number[2].m) == (7, 7)
|
||||||
assert (by_number[3].k, by_number[3].m) == (5, 5)
|
assert (by_number[3].k, by_number[3].m) == (5, 5)
|
||||||
assert (by_number[6].k, by_number[6].m) == (14, 14)
|
assert (by_number[6].k, by_number[6].m) == (10, 10)
|
||||||
assert (by_number[7].k, by_number[7].m) == (15, 15)
|
assert (by_number[7].k, by_number[7].m) == (11, 14)
|
||||||
|
|
||||||
|
|
||||||
def test_the_same_tree_measures_the_same_twice(tmp_path: Path) -> None:
|
def test_the_same_tree_measures_the_same_twice(tmp_path: Path) -> None:
|
||||||
|
|
@ -662,7 +627,7 @@ def test_the_command_exits_one_and_prints_every_row(
|
||||||
printed = capsys.readouterr().out
|
printed = capsys.readouterr().out
|
||||||
for number in range(1, 9):
|
for number in range(1, 9):
|
||||||
assert f"\n{number} " in f"\n{printed}"
|
assert f"\n{number} " in f"\n{printed}"
|
||||||
assert "GATE RED: rows 5, 8" in printed
|
assert "GATE RED: rows 5, 7, 8" in printed
|
||||||
|
|
||||||
|
|
||||||
def test_the_json_form_carries_the_same_rows(capsys: pytest.CaptureFixture[str]) -> None:
|
def test_the_json_form_carries_the_same_rows(capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
|
|
@ -964,7 +929,7 @@ def test_a_forced_fixture_that_stops_missing_counts_against_row_two(
|
||||||
assert any("Z9" in detail and "premise" in detail for detail in row.details)
|
assert any("Z9" in detail and "premise" in detail for detail in row.details)
|
||||||
|
|
||||||
|
|
||||||
def test_j1_a_registration_this_session_wrote_is_not_a_hold_out(tmp_path: Path, home: Path) -> None:
|
def test_j1_a_registration_this_session_wrote_is_not_a_hold_out(tmp_path: Path) -> None:
|
||||||
"""PM's J1: two files written by the session under test came back
|
"""PM's J1: two files written by the session under test came back
|
||||||
`7 of 7 GREEN`. Every check was an assertion the registration made about
|
`7 of 7 GREEN`. Every check was an assertion the registration made about
|
||||||
itself -- `written_by` is `bool()` of a string the file sets, and the
|
itself -- `written_by` is `bool()` of a string the file sets, and the
|
||||||
|
|
@ -1034,140 +999,6 @@ def test_the_mutant_roster_is_pinned_apart_from_the_list_it_names() -> None:
|
||||||
assert len(set(gate.MUTANT_ROSTER)) == len(gate.MUTANT_ROSTER)
|
assert len(set(gate.MUTANT_ROSTER)) == len(gate.MUTANT_ROSTER)
|
||||||
|
|
||||||
|
|
||||||
# --- a passage is judged as a passage ------------------------------------------
|
|
||||||
#
|
|
||||||
# Since v1.1 C1 a concept over `consume.PASSAGE_CHARS` is delivered as its
|
|
||||||
# answering passage: the nearest heading above the span, `[...]`, the span,
|
|
||||||
# `[...]`. The row-6 identity (delivered text == the whole body) is older than
|
|
||||||
# that and read every such delivery as class e. The judge must now accept a
|
|
||||||
# passage that IS its exact reconstruction from the bundle's bytes, and nothing
|
|
||||||
# looser: "the span occurs somewhere in the text" accepts an invented sentence
|
|
||||||
# beside it.
|
|
||||||
|
|
||||||
_HOUSE_NOTE = "Styret gjennomgaar notatet og foerer det inn i arkivet. Sekretaeren sender kopi."
|
|
||||||
_INVENTED = "En setning som ikke staar i konseptfila."
|
|
||||||
|
|
||||||
|
|
||||||
def _house_bundle(tmp_path: Path) -> Path:
|
|
||||||
"""One concept of about 6 600 characters: a heading ten lines in, the
|
|
||||||
answer thirty lines below it, and a literal `[...]` on the LAST line, far
|
|
||||||
outside any span centred on the answer."""
|
|
||||||
body = "\n".join(
|
|
||||||
[_HOUSE_NOTE] * 10
|
|
||||||
+ ["## Teknisk rom"]
|
|
||||||
+ [_HOUSE_NOTE] * 30
|
|
||||||
+ ["Varmepumpa i kjelleren faar service av roerleggeren hvert aar."]
|
|
||||||
+ [_HOUSE_NOTE] * 40
|
|
||||||
+ ["Se vedlegg [...] i arkivet."]
|
|
||||||
)
|
|
||||||
spec = gate.BundleSpec(
|
|
||||||
"retrieval-house",
|
|
||||||
(
|
|
||||||
gate.DocumentSpec(
|
|
||||||
"husbok", "husbok.md", (gate.ConceptSpec("drift", "Drift av huset", body),)
|
|
||||||
),
|
|
||||||
*(
|
|
||||||
gate.DocumentSpec(
|
|
||||||
f"skriv-{number}",
|
|
||||||
f"skriv-{number}.md",
|
|
||||||
(gate.ConceptSpec("skriv", f"Skriv {number}", "Skrivet er arkivert."),),
|
|
||||||
)
|
|
||||||
for number in range(3)
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
return gate.build_bundle(tmp_path / "house", spec)
|
|
||||||
|
|
||||||
|
|
||||||
def _house_question(*quotes: str) -> gate.Question:
|
|
||||||
return gate.Question(
|
|
||||||
id="H1",
|
|
||||||
question="Hvem gir varmepumpa i kjelleren service?",
|
|
||||||
fasit=tuple(gate.Fasit("concept", "husbok/drift", quote) for quote in quotes),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _inject(where: str) -> pytest.MonkeyPatch:
|
|
||||||
"""A passage carrying one sentence the concept file does not: at the end of
|
|
||||||
the span, or between the heading's marker line and the span."""
|
|
||||||
original = consume.as_passage
|
|
||||||
|
|
||||||
def mutant(excerpt: dict[str, object], window: int) -> dict[str, object]:
|
|
||||||
out = original(excerpt, window)
|
|
||||||
if "passage" in out:
|
|
||||||
text = str(out["text"])
|
|
||||||
if where == "span":
|
|
||||||
cut = text.rindex(f"\n{consume.PASSAGE_ELISION}")
|
|
||||||
out["text"] = f"{text[:cut]} {_INVENTED}{text[cut:]}"
|
|
||||||
else:
|
|
||||||
marker = f"{consume.PASSAGE_ELISION}\n"
|
|
||||||
cut = text.index(marker) + len(marker)
|
|
||||||
out["text"] = f"{text[:cut]}{_INVENTED}\n{text[cut:]}"
|
|
||||||
return out
|
|
||||||
|
|
||||||
patch = pytest.MonkeyPatch()
|
|
||||||
patch.setattr(consume, "as_passage", mutant)
|
|
||||||
return patch
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_passage_whose_span_carries_the_quote_is_a_hit(tmp_path: Path) -> None:
|
|
||||||
bundle = _house_bundle(tmp_path)
|
|
||||||
gate.bundle_index(bundle)
|
|
||||||
question = _house_question("service av roerleggeren")
|
|
||||||
payload = consume.build_payload(bundle, question=question.question, withheld_full=True)
|
|
||||||
(excerpt,) = [e for e in payload["excerpts"] if e["concept_id"] == "husbok/drift"]
|
|
||||||
passage = excerpt["passage"]
|
|
||||||
assert isinstance(passage, dict), "the premise: this concept is delivered as a passage"
|
|
||||||
start, end = passage["start"], passage["end"]
|
|
||||||
assert 0 < start and end < passage["of"], "the premise: `[...]` on both sides"
|
|
||||||
# FELLE 1, pinned: the offsets count in the DELIVERED body. Read against
|
|
||||||
# the concept file they land a frontmatter's length off.
|
|
||||||
view = gate.bundle_index(bundle).concepts["husbok/drift"]
|
|
||||||
assert view.body[start:end] in str(excerpt["text"])
|
|
||||||
assert view.whole[start:end] not in str(excerpt["text"])
|
|
||||||
(unit,) = gate.measure_units(bundle, question)
|
|
||||||
assert (unit.hit, unit.confirmed, unit.klass) == (True, True, None), unit
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("where", ["span", "between heading and span"])
|
|
||||||
def test_a_passage_carrying_an_invented_sentence_is_not_a_hit(tmp_path: Path, where: str) -> None:
|
|
||||||
"""FELLE 2, pinned: the quote is still in the span and the span is still in
|
|
||||||
the text, so "the span occurs somewhere" would count this. The delivered
|
|
||||||
text must BE the reconstruction."""
|
|
||||||
bundle = _house_bundle(tmp_path)
|
|
||||||
gate.bundle_index(bundle) # the judge reads the bundle BEFORE the payload
|
|
||||||
question = _house_question("service av roerleggeren")
|
|
||||||
patch = _inject(where)
|
|
||||||
try:
|
|
||||||
(unit,) = gate.measure_units(bundle, question)
|
|
||||||
finally:
|
|
||||||
patch.undo()
|
|
||||||
assert unit.hit is False
|
|
||||||
assert unit.confirmed is False
|
|
||||||
assert unit.klass == "e"
|
|
||||||
assert unit.detail == "the delivered passage is not its reconstruction from the bundle's bytes"
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_quote_the_passage_carries_only_outside_its_span_is_not_a_hit(tmp_path: Path) -> None:
|
|
||||||
"""The heading and the markers are delivered and are not the span: a quote
|
|
||||||
found only there was delivered without the citation (class d). The first
|
|
||||||
unit is the known-positive of the same run."""
|
|
||||||
bundle = _house_bundle(tmp_path)
|
|
||||||
units = gate.measure_units(
|
|
||||||
bundle, _house_question("service av roerleggeren", "Teknisk rom", "[...]")
|
|
||||||
)
|
|
||||||
assert [(unit.hit, unit.klass) for unit in units] == [
|
|
||||||
(True, None),
|
|
||||||
(False, "d"),
|
|
||||||
(False, "d"),
|
|
||||||
], units
|
|
||||||
|
|
||||||
|
|
||||||
def test_row_seven_names_passage_cheating() -> None:
|
|
||||||
assert gate.MUTANT_COUNT == 15
|
|
||||||
assert gate.MUTANT_ROSTER[-1] == "M15 the passage carries a sentence the concept file does not"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_synthetic_corpus_is_pinned_like_the_sets(tmp_path: Path) -> None:
|
def test_the_synthetic_corpus_is_pinned_like_the_sets(tmp_path: Path) -> None:
|
||||||
assert gate.specs_digest() == gate.SPECS_SHA256
|
assert gate.specs_digest() == gate.SPECS_SHA256
|
||||||
tuned = dict(gate.SPECS)
|
tuned = dict(gate.SPECS)
|
||||||
|
|
@ -1294,18 +1125,6 @@ def test_a_wiki_set_of_one_question_is_refused_on_the_command_line(
|
||||||
# --- step 0: the threshold is a number, and it is compared with something ----
|
# --- step 0: the threshold is a number, and it is compared with something ----
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def home(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
|
||||||
"""Row 5 reads a registration's paths as `~/...`, expanded against the
|
|
||||||
home directory; every test writing one makes its `tmp_path` that home."""
|
|
||||||
monkeypatch.setenv("HOME", str(tmp_path))
|
|
||||||
return tmp_path
|
|
||||||
|
|
||||||
|
|
||||||
def _under_home(path: Path, home: Path) -> str:
|
|
||||||
return "~/" + path.relative_to(home).as_posix()
|
|
||||||
|
|
||||||
|
|
||||||
def _registration(
|
def _registration(
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
*,
|
*,
|
||||||
|
|
@ -1313,26 +1132,14 @@ def _registration(
|
||||||
bundle: Path,
|
bundle: Path,
|
||||||
threshold: object,
|
threshold: object,
|
||||||
name: str = "registration.json",
|
name: str = "registration.json",
|
||||||
bundle_ref: str | None = None,
|
|
||||||
) -> Path:
|
) -> Path:
|
||||||
"""A registration in the form the committed one must have: its paths
|
|
||||||
under the home directory, written `~/...`, and the bundle's tree ref
|
|
||||||
pinned. The home is `tmp_path` (the `home` fixture); a set living outside
|
|
||||||
it is copied in byte for byte, so the sha256 pinned is the set's own."""
|
|
||||||
assert Path.home() == tmp_path, "a row-5 test must request the `home` fixture"
|
|
||||||
if not set_path.is_relative_to(tmp_path):
|
|
||||||
copied = tmp_path / "sets" / set_path.name
|
|
||||||
copied.parent.mkdir(exist_ok=True)
|
|
||||||
copied.write_bytes(set_path.read_bytes())
|
|
||||||
set_path = copied
|
|
||||||
registration = tmp_path / name
|
registration = tmp_path / name
|
||||||
registration.write_text(
|
registration.write_text(
|
||||||
json.dumps(
|
json.dumps(
|
||||||
{
|
{
|
||||||
"set": _under_home(set_path, tmp_path),
|
"set": str(set_path),
|
||||||
"sha256": gate.sha256_of(set_path),
|
"sha256": gate.sha256_of(set_path),
|
||||||
"bundle": _under_home(bundle, tmp_path),
|
"bundle": str(bundle),
|
||||||
"bundle_ref": consume.bundle_ref(bundle) if bundle_ref is None else bundle_ref,
|
|
||||||
"metric": "questions answered over questions asked",
|
"metric": "questions answered over questions asked",
|
||||||
"threshold": threshold,
|
"threshold": threshold,
|
||||||
"threshold_written_at": "2026-09-19T10:00:00Z",
|
"threshold_written_at": "2026-09-19T10:00:00Z",
|
||||||
|
|
@ -1345,7 +1152,7 @@ def _registration(
|
||||||
return registration
|
return registration
|
||||||
|
|
||||||
|
|
||||||
def test_a_threshold_that_is_not_a_number_is_refused(tmp_path: Path, home: Path) -> None:
|
def test_a_threshold_that_is_not_a_number_is_refused(tmp_path: Path) -> None:
|
||||||
"""`bool(threshold)` was the whole check, so `report-only; any number is
|
"""`bool(threshold)` was the whole check, so `report-only; any number is
|
||||||
acceptable for v1` read as `a threshold is written: yes`. A threshold that
|
acceptable for v1` read as `a threshold is written: yes`. A threshold that
|
||||||
cannot be compared with a number cannot fell anything."""
|
cannot be compared with a number cannot fell anything."""
|
||||||
|
|
@ -1361,7 +1168,7 @@ def test_a_threshold_that_is_not_a_number_is_refused(tmp_path: Path, home: Path)
|
||||||
assert row.fails
|
assert row.fails
|
||||||
|
|
||||||
|
|
||||||
def test_the_threshold_is_compared_with_the_measured_hold_out(tmp_path: Path, home: Path) -> None:
|
def test_the_threshold_is_compared_with_the_measured_hold_out(tmp_path: Path) -> None:
|
||||||
"""Both directions, from the same code path: a set the bundle answers
|
"""Both directions, from the same code path: a set the bundle answers
|
||||||
clears a threshold under it, and a set it does not answer falls under one
|
clears a threshold under it, and a set it does not answer falls under one
|
||||||
over it. The measured share is counted here as well, off the set's own
|
over it. The measured share is counted here as well, off the set's own
|
||||||
|
|
@ -1378,7 +1185,7 @@ def test_the_threshold_is_compared_with_the_measured_hold_out(tmp_path: Path, ho
|
||||||
provenance=lambda _: _carried_by_git(),
|
provenance=lambda _: _carried_by_git(),
|
||||||
)
|
)
|
||||||
assert any("clears the threshold: yes" in detail for detail in clears.details)
|
assert any("clears the threshold: yes" in detail for detail in clears.details)
|
||||||
assert (clears.k, clears.m, clears.status) == (12, 12, gate.GREEN)
|
assert (clears.k, clears.m, clears.status) == (11, 11, gate.GREEN)
|
||||||
|
|
||||||
# The independent count: set-miss carries one question, forced to miss.
|
# The independent count: set-miss carries one question, forced to miss.
|
||||||
missing = json.loads((FIXTURES / "set-miss.json").read_text(encoding="utf-8"))
|
missing = json.loads((FIXTURES / "set-miss.json").read_text(encoding="utf-8"))
|
||||||
|
|
@ -1394,7 +1201,7 @@ def test_the_threshold_is_compared_with_the_measured_hold_out(tmp_path: Path, ho
|
||||||
provenance=lambda _: _carried_by_git(),
|
provenance=lambda _: _carried_by_git(),
|
||||||
)
|
)
|
||||||
assert any("clears the threshold: NO" in detail for detail in falls.details)
|
assert any("clears the threshold: NO" in detail for detail in falls.details)
|
||||||
assert (falls.k, falls.m, falls.status) == (11, 12, gate.RED)
|
assert (falls.k, falls.m, falls.status) == (10, 11, gate.RED)
|
||||||
|
|
||||||
|
|
||||||
# --- step 1: row 4, the marking a consumer can act on -------------------------
|
# --- step 1: row 4, the marking a consumer can act on -------------------------
|
||||||
|
|
@ -1482,7 +1289,7 @@ def test_a_set_of_the_right_size_and_the_wrong_bytes_is_still_refused(
|
||||||
assert "sha256" in str(refusal.value)
|
assert "sha256" in str(refusal.value)
|
||||||
|
|
||||||
|
|
||||||
def test_a_threshold_outside_nought_to_one_is_not_a_share(tmp_path: Path, home: Path) -> None:
|
def test_a_threshold_outside_nought_to_one_is_not_a_share(tmp_path: Path) -> None:
|
||||||
"""`80` is either 80 % written wrongly or a bar no run can clear, and
|
"""`80` is either 80 % written wrongly or a bar no run can clear, and
|
||||||
guessing which is not this row's job."""
|
guessing which is not this row's job."""
|
||||||
bundles = _bundles(tmp_path)
|
bundles = _bundles(tmp_path)
|
||||||
|
|
@ -1512,7 +1319,7 @@ def test_a_threshold_outside_nought_to_one_is_not_a_share(tmp_path: Path, home:
|
||||||
assert any("the threshold is a number: yes" in detail for detail in inside.details)
|
assert any("the threshold is a number: yes" in detail for detail in inside.details)
|
||||||
|
|
||||||
|
|
||||||
def test_a_hold_out_set_with_no_question_clears_no_threshold(tmp_path: Path, home: Path) -> None:
|
def test_a_hold_out_set_with_no_question_clears_no_threshold(tmp_path: Path) -> None:
|
||||||
"""A share over a denominator of nought is not a number, and an empty set
|
"""A share over a denominator of nought is not a number, and an empty set
|
||||||
was the shape every row-5 test used before 2026-09-20 -- so `>=` over it
|
was the shape every row-5 test used before 2026-09-20 -- so `>=` over it
|
||||||
would have made the comparison vacuous the moment it was added."""
|
would have made the comparison vacuous the moment it was added."""
|
||||||
|
|
@ -1534,265 +1341,3 @@ def test_a_hold_out_set_with_no_question_clears_no_threshold(tmp_path: Path, hom
|
||||||
"clears the threshold: NO (the hold-out set carries no question" in detail
|
"clears the threshold: NO (the hold-out set carries no question" in detail
|
||||||
for detail in row.details
|
for detail in row.details
|
||||||
), row.details
|
), row.details
|
||||||
|
|
||||||
|
|
||||||
# --- row 5 reads a hold-out set in the schema it declares --------------------
|
|
||||||
#
|
|
||||||
# The registered hold-out set is written in a consumer's own question-set
|
|
||||||
# schema, the one row 8 already reads, and row 5 used to read every set with
|
|
||||||
# the gate's synthetic reader: `not a question set this gate can read:
|
|
||||||
# 'set_id'`, a NO no set in that schema could ever turn. Every file below is
|
|
||||||
# written in that shape with INVENTED content over the synthetic bundle; the
|
|
||||||
# real hold-out set is never read by a test.
|
|
||||||
|
|
||||||
#: Distinctive on purpose: a row that printed a question's id or text would
|
|
||||||
#: carry one of these, and nothing else in the gate's output can.
|
|
||||||
_HOLD_OUT_MISS_ID = "HOLDOUT-MISS-7Q"
|
|
||||||
_HOLD_OUT_MISS_TEXT = "Hvilken farge har den fraflyttede fyrlykta paa Utvaer?"
|
|
||||||
_HOLD_OUT_HIT_IDS = ("HOLDOUT-H1", "HOLDOUT-H2", "HOLDOUT-H3", "HOLDOUT-H4")
|
|
||||||
|
|
||||||
|
|
||||||
def _fase_set(path: Path, *, schema: str = "fase-sporsmaal/1") -> Path:
|
|
||||||
"""Four questions the positive bundle answers and one it cannot: a
|
|
||||||
share of 4 of 5, counted here off the file and not off the row."""
|
|
||||||
hits = [
|
|
||||||
("Naar kontrolleres vinterberedskapen?", "haandbok", "innen 1. november"),
|
|
||||||
("Hvordan kvitteres noekkelen ut?", "haandbok", "signatur i noekkelboka"),
|
|
||||||
("Hvor ofte byttes batteriet i roekvarsleren?", "haandbok", "en gang i aaret"),
|
|
||||||
("Hva vedtok aarsmoetet om kontingenten?", "referat", "480 kroner"),
|
|
||||||
]
|
|
||||||
questions = [
|
|
||||||
{"id": qid, "question": text, "fasit": [{"doc": doc, "quote": quote}]}
|
|
||||||
for qid, (text, doc, quote) in zip(_HOLD_OUT_HIT_IDS, hits)
|
|
||||||
]
|
|
||||||
questions.append(
|
|
||||||
{
|
|
||||||
"id": _HOLD_OUT_MISS_ID,
|
|
||||||
"question": _HOLD_OUT_MISS_TEXT,
|
|
||||||
"fasit": [{"doc": "haandbok", "quote": "fyrlykta er malt i gult og fiolett"}],
|
|
||||||
}
|
|
||||||
)
|
|
||||||
path.write_text(
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"schema": schema,
|
|
||||||
"hit_rule": "an excerpt whose source_file is <doc>.md AND carries the quote",
|
|
||||||
"questions": questions,
|
|
||||||
},
|
|
||||||
ensure_ascii=False,
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
return path
|
|
||||||
|
|
||||||
|
|
||||||
def test_row_five_measures_a_hold_out_set_in_the_wiki_schema(tmp_path: Path, home: Path) -> None:
|
|
||||||
"""Both directions through the same set and the same bundle: 4 of 5
|
|
||||||
clears 0.8 and falls under 0.9. The share is counted off the file first,
|
|
||||||
so the row is not the only thing that knows it."""
|
|
||||||
held_out = _fase_set(tmp_path / "held-out.json")
|
|
||||||
written = json.loads(held_out.read_text(encoding="utf-8"))
|
|
||||||
assert len(written["questions"]) == 5
|
|
||||||
bundle = _bundles(tmp_path)["positive"]
|
|
||||||
|
|
||||||
clears = gate.row_five(
|
|
||||||
_registration(tmp_path, set_path=held_out, bundle=bundle, threshold=0.8, name="c.json"),
|
|
||||||
provenance=lambda _: _carried_by_git(),
|
|
||||||
)
|
|
||||||
assert any(
|
|
||||||
"the measured hold-out clears the threshold: yes (4 of 5 = 0.8000 against 0.8000)" in detail
|
|
||||||
for detail in clears.details
|
|
||||||
), clears.details
|
|
||||||
assert (clears.k, clears.m, clears.status) == (12, 12, gate.GREEN)
|
|
||||||
|
|
||||||
falls = gate.row_five(
|
|
||||||
_registration(tmp_path, set_path=held_out, bundle=bundle, threshold=0.9, name="f.json"),
|
|
||||||
provenance=lambda _: _carried_by_git(),
|
|
||||||
)
|
|
||||||
assert any(
|
|
||||||
"the measured hold-out clears the threshold: NO (4 of 5 = 0.8000 against 0.9000)" in detail
|
|
||||||
for detail in falls.details
|
|
||||||
), falls.details
|
|
||||||
assert (falls.k, falls.m, falls.status) == (11, 12, gate.RED)
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_hold_out_set_in_an_unknown_schema_is_a_named_no(tmp_path: Path, home: Path) -> None:
|
|
||||||
"""Fail closed: a set read in a shape it was not written in measures
|
|
||||||
nothing, so an unknown schema is a NO that NAMES the schema -- never a
|
|
||||||
guess at a reader and never an exception."""
|
|
||||||
held_out = _fase_set(tmp_path / "held-out.json", schema="noe-annet/9")
|
|
||||||
row = gate.row_five(
|
|
||||||
_registration(
|
|
||||||
tmp_path, set_path=held_out, bundle=_bundles(tmp_path)["positive"], threshold=0.8
|
|
||||||
),
|
|
||||||
provenance=lambda _: _carried_by_git(),
|
|
||||||
)
|
|
||||||
verdict = [d for d in row.details if "the measured hold-out clears the threshold" in d]
|
|
||||||
assert len(verdict) == 1, row.details
|
|
||||||
assert ": NO (" in verdict[0]
|
|
||||||
assert "schema `noe-annet/9`" in verdict[0], verdict[0]
|
|
||||||
assert "fase-sporsmaal/1" in verdict[0], verdict[0]
|
|
||||||
assert row.fails
|
|
||||||
|
|
||||||
|
|
||||||
def test_row_five_prints_the_share_and_never_which_hold_out_question_missed(
|
|
||||||
tmp_path: Path, home: Path
|
|
||||||
) -> None:
|
|
||||||
"""A hold-out set whose misses are printed is a tuning set by the next
|
|
||||||
session. The known-positive in the same run: the row DID measure, one
|
|
||||||
question missed, and the share is printed -- so the absence of the ids
|
|
||||||
and the text below is the row's silence and not a row that never ran."""
|
|
||||||
held_out = _fase_set(tmp_path / "held-out.json")
|
|
||||||
for threshold in (0.8, 0.9):
|
|
||||||
row = gate.row_five(
|
|
||||||
_registration(
|
|
||||||
tmp_path,
|
|
||||||
set_path=held_out,
|
|
||||||
bundle=_bundles(tmp_path)["positive"],
|
|
||||||
threshold=threshold,
|
|
||||||
name=f"r{threshold}.json",
|
|
||||||
),
|
|
||||||
provenance=lambda _: _carried_by_git(),
|
|
||||||
)
|
|
||||||
text = gate.render([row]) + json.dumps(row.to_json(), ensure_ascii=False)
|
|
||||||
assert "4 of 5 = 0.8000" in text, text
|
|
||||||
assert _HOLD_OUT_MISS_ID not in text
|
|
||||||
assert _HOLD_OUT_MISS_TEXT not in text
|
|
||||||
assert "fyrlykta" not in text
|
|
||||||
for hit in _HOLD_OUT_HIT_IDS:
|
|
||||||
assert hit not in text
|
|
||||||
|
|
||||||
|
|
||||||
# --- row 5 names no machine, and measures the bundle it registered ------------
|
|
||||||
#
|
|
||||||
# The registration is committed to a PUBLIC repository, so the paths it names
|
|
||||||
# are written relative to the home directory and expanded when read: an
|
|
||||||
# absolute path names one machine and is a NO, never a path the row follows.
|
|
||||||
# And the bundle it names is only the registered bundle while its tree still
|
|
||||||
# measures the registered `bundle_ref`: a bundle path is a place, and a place
|
|
||||||
# can be filled with other bytes.
|
|
||||||
|
|
||||||
|
|
||||||
def _row_five_line(row: gate.Row, name: str) -> str:
|
|
||||||
lines = [detail for detail in row.details if detail.strip().startswith(f"{name}:")]
|
|
||||||
assert len(lines) == 1, (name, row.details)
|
|
||||||
return lines[0]
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_absolute_path_in_the_registration_is_a_named_no(tmp_path: Path, home: Path) -> None:
|
|
||||||
"""Both paths, each on its own: the set and the bundle written as the
|
|
||||||
absolute path of the very file the `~/` form names. The row does not
|
|
||||||
follow either, and says why."""
|
|
||||||
held_out = _fase_set(tmp_path / "held-out.json")
|
|
||||||
bundle = _bundles(tmp_path)["positive"]
|
|
||||||
registration = _registration(tmp_path, set_path=held_out, bundle=bundle, threshold=0.8)
|
|
||||||
written = json.loads(registration.read_text(encoding="utf-8"))
|
|
||||||
for key, absolute, line in (
|
|
||||||
("set", str(held_out), "a set is named"),
|
|
||||||
("bundle", str(bundle), "the bundle is the registered tree"),
|
|
||||||
):
|
|
||||||
spec = dict(written, **{key: absolute})
|
|
||||||
registration.write_text(json.dumps(spec), encoding="utf-8")
|
|
||||||
row = gate.row_five(registration, provenance=lambda _: _carried_by_git())
|
|
||||||
verdict = _row_five_line(row, line)
|
|
||||||
assert ": NO (" in verdict, verdict
|
|
||||||
assert "absolute" in verdict, verdict
|
|
||||||
assert "~/" in verdict, verdict
|
|
||||||
assert row.status == gate.RED, key
|
|
||||||
# A path the row does not follow is a set it does not measure.
|
|
||||||
assert "4 of 5" not in gate.render([row]), key
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_home_relative_path_is_expanded_and_read(tmp_path: Path, home: Path) -> None:
|
|
||||||
"""The known-positive for the rule above: the registration's own text
|
|
||||||
carries `~/` and no path of this machine, and the row expands it, reads
|
|
||||||
the set and measures the bundle -- 4 of 5, counted off the file."""
|
|
||||||
held_out = _fase_set(tmp_path / "held-out.json")
|
|
||||||
registration = _registration(
|
|
||||||
tmp_path, set_path=held_out, bundle=_bundles(tmp_path)["positive"], threshold=0.8
|
|
||||||
)
|
|
||||||
text = registration.read_text(encoding="utf-8")
|
|
||||||
spec = json.loads(text)
|
|
||||||
assert spec["set"].startswith("~/") and spec["bundle"].startswith("~/"), spec
|
|
||||||
assert str(tmp_path) not in text
|
|
||||||
row = gate.row_five(registration, provenance=lambda _: _carried_by_git())
|
|
||||||
assert ": yes (" in _row_five_line(row, "the pinned bytes are the bytes on disk")
|
|
||||||
assert ": yes (" in _row_five_line(row, "the bundle is the registered tree")
|
|
||||||
assert "(4 of 5 = 0.8000 against 0.8000)" in _row_five_line(
|
|
||||||
row, "the measured hold-out clears the threshold"
|
|
||||||
)
|
|
||||||
assert (row.k, row.m, row.status) == (12, 12, gate.GREEN)
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_bundle_whose_tree_is_not_the_registered_ref_is_a_no(tmp_path: Path, home: Path) -> None:
|
|
||||||
"""Two ways the path stays and the bytes go: a registration pinning
|
|
||||||
another bundle's ref, and the registered bundle's concepts edited after
|
|
||||||
the ref was written. Neither is measured -- a share read off other bytes
|
|
||||||
is not the registered hold-out."""
|
|
||||||
held_out = _fase_set(tmp_path / "held-out.json")
|
|
||||||
bundles = _bundles(tmp_path)
|
|
||||||
foreign = consume.bundle_ref(bundles["miss"])
|
|
||||||
assert foreign != consume.bundle_ref(bundles["positive"]), "the control must differ"
|
|
||||||
pinned_elsewhere = gate.row_five(
|
|
||||||
_registration(
|
|
||||||
tmp_path,
|
|
||||||
set_path=held_out,
|
|
||||||
bundle=bundles["positive"],
|
|
||||||
threshold=0.8,
|
|
||||||
bundle_ref=foreign,
|
|
||||||
name="foreign.json",
|
|
||||||
),
|
|
||||||
provenance=lambda _: _carried_by_git(),
|
|
||||||
)
|
|
||||||
verdict = _row_five_line(pinned_elsewhere, "the bundle is the registered tree")
|
|
||||||
assert ": NO (" in verdict and foreign in verdict, verdict
|
|
||||||
assert pinned_elsewhere.status == gate.RED
|
|
||||||
assert "4 of 5" not in gate.render([pinned_elsewhere])
|
|
||||||
|
|
||||||
registration = _registration(
|
|
||||||
tmp_path, set_path=held_out, bundle=bundles["positive"], threshold=0.8, name="edited.json"
|
|
||||||
)
|
|
||||||
concepts = [p for p in bundles["positive"].rglob("*.md") if p.name != "index.md"]
|
|
||||||
assert concepts, "the control needs a concept to edit"
|
|
||||||
for concept in concepts:
|
|
||||||
concept.write_bytes(concept.read_bytes() + b"\nOther bytes at the same path.\n")
|
|
||||||
edited = gate.row_five(registration, provenance=lambda _: _carried_by_git())
|
|
||||||
assert ": NO (" in _row_five_line(edited, "the bundle is the registered tree")
|
|
||||||
assert edited.status == gate.RED
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_registered_set_or_bundle_absent_on_this_machine_is_a_named_no(
|
|
||||||
tmp_path: Path, home: Path
|
|
||||||
) -> None:
|
|
||||||
"""A registration read on a machine that does not carry what it names:
|
|
||||||
a NO with the registered `~/` path, never an exception, never green."""
|
|
||||||
held_out = _fase_set(tmp_path / "held-out.json")
|
|
||||||
bundle = _bundles(tmp_path)["positive"]
|
|
||||||
written = json.loads(
|
|
||||||
_registration(tmp_path, set_path=held_out, bundle=bundle, threshold=0.8).read_text(
|
|
||||||
encoding="utf-8"
|
|
||||||
)
|
|
||||||
)
|
|
||||||
registration = tmp_path / "absent.json"
|
|
||||||
for key, value, line in (
|
|
||||||
("set", "~/nowhere/held-out.json", "the pinned bytes are the bytes on disk"),
|
|
||||||
("bundle", "~/nowhere/bundle", "the bundle is the registered tree"),
|
|
||||||
):
|
|
||||||
registration.write_text(json.dumps(dict(written, **{key: value})), encoding="utf-8")
|
|
||||||
row = gate.row_five(registration, provenance=lambda _: _carried_by_git())
|
|
||||||
verdict = _row_five_line(row, line)
|
|
||||||
assert ": NO (" in verdict and "absent" in verdict and value in verdict, verdict
|
|
||||||
assert row.status == gate.RED, key
|
|
||||||
assert ": NO (" in _row_five_line(row, "the measured hold-out clears the threshold")
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_committed_registration_names_no_machine_path() -> None:
|
|
||||||
"""The file this repository publishes: its paths are `~/...`, and no
|
|
||||||
value in it names a directory of the machine it was written on."""
|
|
||||||
text = gate.HOLDOUT_REGISTRATION.read_text(encoding="utf-8")
|
|
||||||
spec = json.loads(text)
|
|
||||||
assert spec["set"].startswith("~/"), spec["set"]
|
|
||||||
assert spec["bundle"].startswith("~/"), spec["bundle"]
|
|
||||||
assert spec["bundle_ref"].startswith("sha256-tree:"), spec["bundle_ref"]
|
|
||||||
for marker in ("/Users/", "/home/", "/private/", "/var/folders/"):
|
|
||||||
assert marker not in text, marker
|
|
||||||
|
|
|
||||||
|
|
@ -75,8 +75,8 @@ from llm_ingestion_okf.quality import ( # noqa: E402
|
||||||
|
|
||||||
FIXTURES = REPO / "tests" / "fixtures" / "retrieval"
|
FIXTURES = REPO / "tests" / "fixtures" / "retrieval"
|
||||||
|
|
||||||
#: The hold-out registration. The row reads a path so a test can drive both
|
#: The hold-out registration. Absent today, which is what makes row 5 red; the
|
||||||
#: directions without editing code.
|
#: row reads a path so a test can drive both directions without editing code.
|
||||||
HOLDOUT_REGISTRATION = FIXTURES / "holdout-registration.json"
|
HOLDOUT_REGISTRATION = FIXTURES / "holdout-registration.json"
|
||||||
|
|
||||||
GREEN = "GREEN"
|
GREEN = "GREEN"
|
||||||
|
|
@ -434,190 +434,6 @@ BUDGET = BundleSpec(
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
# --- three fixtures, one per mechanism the default ranking runs ---------------
|
|
||||||
#
|
|
||||||
# Until 2026-09-22 row 7 left three mutants standing (M06, M07, M10): every
|
|
||||||
# synthetic concept was short and opened with its own title, so the passage
|
|
||||||
# signal, the title/path weight and the fusion constant never DECIDED a
|
|
||||||
# delivery here, and switching any of them off moved 0 ranks. Each bundle below
|
|
||||||
# is built so that exactly one of them does. None of them is a tuning of the
|
|
||||||
# ranker: the corpus is pinned (`SPECS_SHA256`) and `src/` is untouched.
|
|
||||||
|
|
||||||
_NOTE = "Styret gjennomgaar notatet og foerer det inn i arkivet. Sekretaeren sender kopi. "
|
|
||||||
|
|
||||||
#: THE PASSAGE SIGNAL. One long concept answers in ONE window of its body; ten
|
|
||||||
#: short concepts carry the question's words in their TITLES and none in their
|
|
||||||
#: bodies. The field signal reads ten title-weighted matches in short documents
|
|
||||||
#: above one long body, so only the passage signal -- the best WINDOW, not the
|
|
||||||
#: whole concept -- puts the gold first. Remove the body windows (M06) and the
|
|
||||||
#: passage signal reads titles alone, where the gold has none of the words.
|
|
||||||
PASSAGE = BundleSpec(
|
|
||||||
"retrieval-passage",
|
|
||||||
(
|
|
||||||
DocumentSpec(
|
|
||||||
"aarbok",
|
|
||||||
"aarbok.md",
|
|
||||||
(
|
|
||||||
ConceptSpec(
|
|
||||||
"kapittel-tre",
|
|
||||||
"Kapittel tre i aarboka",
|
|
||||||
_NOTE * 15
|
|
||||||
+ "Varmekablene i taket kontrolleres av vaktmesteren hver oktober. "
|
|
||||||
+ _NOTE * 15,
|
|
||||||
),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
*(
|
|
||||||
DocumentSpec(
|
|
||||||
f"rundskriv-{number:02d}",
|
|
||||||
f"rundskriv-{number:02d}.md",
|
|
||||||
(
|
|
||||||
ConceptSpec(
|
|
||||||
f"skriv-{number:02d}",
|
|
||||||
f"Varmekablene i taket, skriv {number:02d}",
|
|
||||||
"Skrivet er sendt ut og arkivert.",
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
for number in range(1, 11)
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
#: THE TITLE/PATH WEIGHT. The gold is named by its PATH alone -- the question's
|
|
||||||
#: `kanopadling` is its directory and its source file and occurs in no body --
|
|
||||||
#: while ten decoys carry the question's other word more densely than the gold
|
|
||||||
#: does, one fewer time each. The path weight puts the gold first in the field
|
|
||||||
#: signal, which lifts it into k (rank 4 at `k` = 6). Weigh no title and no path
|
|
||||||
#: (M07) and both signals read bodies only, where the decoys outrank it and it
|
|
||||||
#: falls out of k.
|
|
||||||
PATH = BundleSpec(
|
|
||||||
"retrieval-path",
|
|
||||||
(
|
|
||||||
DocumentSpec(
|
|
||||||
"kanopadling",
|
|
||||||
"kanopadling.md",
|
|
||||||
(ConceptSpec("regel", "Regel om vester", "Vester brukes paa vannet."),),
|
|
||||||
),
|
|
||||||
*(
|
|
||||||
DocumentSpec(
|
|
||||||
f"turnotat-{number:02d}",
|
|
||||||
f"turnotat-{number:02d}.md",
|
|
||||||
(
|
|
||||||
ConceptSpec(
|
|
||||||
f"notat-{number:02d}",
|
|
||||||
f"Turnotat {number:02d}",
|
|
||||||
" ".join(["Vester henger i boden."] * (11 - number))
|
|
||||||
+ " Notatet er arkivert.",
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
for number in range(1, 11)
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
_FUSION_NOTE = "Styret gjennomgaar saken og sekretaeren arkiverer den. "
|
|
||||||
|
|
||||||
#: THE FUSION CONSTANT. Reciprocal-rank fusion trades a concept with one very
|
|
||||||
#: good rank against one with two middling ranks, and `RRF_K` sets the price.
|
|
||||||
#: The gold is FIRST in the passage signal and 21st in the field signal (a long
|
|
||||||
#: body, below ten short `isbading-*` path matches, nine longer ones and the
|
|
||||||
#: decoy); the one decoy is 10th and 11th, a smaller rank SUM (19 against 20).
|
|
||||||
#: A flattened fusion (M10, `RRF_K` = 10 000, where only the sum counts) puts
|
|
||||||
#: the decoy first and `k` = 1 no longer reaches the gold. At the shipped
|
|
||||||
#: `RRF_K` = 60 the gold leads, and measured over K in steps of 10 it keeps the
|
|
||||||
#: lead through K = 180 and loses it at 190. The eight long log pages hold the
|
|
||||||
#: decoy's passage rank at 10th and rank below the gold in both signals.
|
|
||||||
FUSION = BundleSpec(
|
|
||||||
"retrieval-fusion",
|
|
||||||
(
|
|
||||||
DocumentSpec(
|
|
||||||
"badebok",
|
|
||||||
"badebok.md",
|
|
||||||
(
|
|
||||||
ConceptSpec(
|
|
||||||
"kapittel",
|
|
||||||
"Kapittel i badeboka",
|
|
||||||
_FUSION_NOTE
|
|
||||||
* 12
|
|
||||||
+ "Isbading ved brygga, isbading om morgenen, isbading med vakt, "
|
|
||||||
"isbading hver dag. " + _FUSION_NOTE * 12,
|
|
||||||
),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
*(
|
|
||||||
DocumentSpec(
|
|
||||||
f"logg-{number:02d}",
|
|
||||||
f"logg-{number:02d}.md",
|
|
||||||
(
|
|
||||||
ConceptSpec(
|
|
||||||
"side",
|
|
||||||
f"Loggside {number:02d}",
|
|
||||||
_FUSION_NOTE * 16
|
|
||||||
+ "Isbading ved brygga, isbading om kvelden, isbading med vakt. "
|
|
||||||
+ _FUSION_NOTE * 16,
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
for number in range(8)
|
|
||||||
),
|
|
||||||
DocumentSpec(
|
|
||||||
"isbading-lapp-00",
|
|
||||||
"isbading-lapp-00.md",
|
|
||||||
(ConceptSpec("lapp", "Lapp 00", _FUSION_NOTE * 5 + "Isbading er tillatt."),),
|
|
||||||
),
|
|
||||||
*(
|
|
||||||
DocumentSpec(
|
|
||||||
f"isbading-{number:02d}",
|
|
||||||
f"isbading-{number:02d}.md",
|
|
||||||
(ConceptSpec("oppslag", f"Oppslag {number:02d}", "Oppslaget er arkivert."),),
|
|
||||||
)
|
|
||||||
for number in range(10)
|
|
||||||
),
|
|
||||||
*(
|
|
||||||
DocumentSpec(
|
|
||||||
f"isbading-arkiv-{number:02d}",
|
|
||||||
f"isbading-arkiv-{number:02d}.md",
|
|
||||||
(ConceptSpec("mappe", f"Mappe {number:02d}", _FUSION_NOTE * 10),),
|
|
||||||
)
|
|
||||||
for number in range(9)
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
_ARCHIVE_LINE = "Styret gjennomgaar notatet og foerer det inn i arkivet. Sekretaeren sender kopi."
|
|
||||||
|
|
||||||
#: THE PASSAGE DELIVERY. Every concept above is shorter than
|
|
||||||
#: `consume.PASSAGE_CHARS`, so no synthetic payload carried a passage and a
|
|
||||||
#: mutant of one (M15) could not be felled. One concept of about 6 600
|
|
||||||
#: characters: a heading ten lines in, the answer thirty lines below it, so the
|
|
||||||
#: delivery is heading + `[...]` + span + `[...]` and the span carries the
|
|
||||||
#: citation. Rows 1 and 6 read it as a hit only because the judge reads a
|
|
||||||
#: passage as its exact reconstruction; M15 adds one sentence and fells both.
|
|
||||||
DELIVERY = BundleSpec(
|
|
||||||
"retrieval-delivery",
|
|
||||||
(
|
|
||||||
DocumentSpec(
|
|
||||||
"husbok",
|
|
||||||
"husbok.md",
|
|
||||||
(
|
|
||||||
ConceptSpec(
|
|
||||||
"drift",
|
|
||||||
"Drift av huset",
|
|
||||||
"\n".join(
|
|
||||||
[_ARCHIVE_LINE] * 10
|
|
||||||
+ ["## Teknisk rom"]
|
|
||||||
+ [_ARCHIVE_LINE] * 30
|
|
||||||
+ ["Varmepumpa i kjelleren faar service av roerleggeren hvert aar."]
|
|
||||||
+ [_ARCHIVE_LINE] * 40
|
|
||||||
),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
SPECS: Mapping[str, BundleSpec] = {
|
SPECS: Mapping[str, BundleSpec] = {
|
||||||
"positive": POSITIVE,
|
"positive": POSITIVE,
|
||||||
"miss": MISS,
|
"miss": MISS,
|
||||||
|
|
@ -625,10 +441,6 @@ SPECS: Mapping[str, BundleSpec] = {
|
||||||
"budget": BUDGET,
|
"budget": BUDGET,
|
||||||
"lookup": LOOKUP,
|
"lookup": LOOKUP,
|
||||||
"quota": QUOTA,
|
"quota": QUOTA,
|
||||||
"passage": PASSAGE,
|
|
||||||
"path": PATH,
|
|
||||||
"fusion": FUSION,
|
|
||||||
"delivery": DELIVERY,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -671,7 +483,7 @@ def specs_digest(specs: Mapping[str, BundleSpec] = SPECS) -> str:
|
||||||
|
|
||||||
|
|
||||||
#: The synthetic corpus, pinned the way the sets are.
|
#: The synthetic corpus, pinned the way the sets are.
|
||||||
SPECS_SHA256 = "089a6a9770c03c59fff2da22430e5f9aeb2b2374c49aed419bae99409a2a5cb9"
|
SPECS_SHA256 = "23c447add9a58ee2e454ef5d5b92d5c3c97d441cd883d203aaf2f275dd2f68f9"
|
||||||
|
|
||||||
|
|
||||||
def synthetic_bundles(root: Path, specs: Mapping[str, BundleSpec] = SPECS) -> dict[str, Path]:
|
def synthetic_bundles(root: Path, specs: Mapping[str, BundleSpec] = SPECS) -> dict[str, Path]:
|
||||||
|
|
@ -836,8 +648,6 @@ SYNTHETIC_SETS: dict[str, str] = {
|
||||||
"set-signals.json": "17d83f305a10af8dd2a45b72943c1285704d23ed074ecb549c05e5fac3e137e9",
|
"set-signals.json": "17d83f305a10af8dd2a45b72943c1285704d23ed074ecb549c05e5fac3e137e9",
|
||||||
"set-quota.json": "61fda652719d7403ddf9701d50e914c46572a4fda77dc0336838695e18c498cf",
|
"set-quota.json": "61fda652719d7403ddf9701d50e914c46572a4fda77dc0336838695e18c498cf",
|
||||||
"set-controls.json": "c2894656326e5a69ec7063fdc280e763910d20b4cbc2c0124f639283864ce506",
|
"set-controls.json": "c2894656326e5a69ec7063fdc280e763910d20b4cbc2c0124f639283864ce506",
|
||||||
"set-mechanisms.json": "435ce620c30c94b0d151335883583520490482be2ade6797524ae928a9f1eab7",
|
|
||||||
"set-passage.json": "93b6de8ef7012c84516fe9aee560924d4b579669fb046e6eadc70dbf444eb627",
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -1001,79 +811,6 @@ def _carries(text: str, quote: str) -> bool:
|
||||||
return not quote or _flat(quote) in _flat(text)
|
return not quote or _flat(quote) in _flat(text)
|
||||||
|
|
||||||
|
|
||||||
#: What class e says when a passage is not its own reconstruction.
|
|
||||||
PASSAGE_NOT_RECONSTRUCTED = (
|
|
||||||
"the delivered passage is not its reconstruction from the bundle's bytes"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def passage_span(text: str, passage: object, body: str) -> str | None:
|
|
||||||
"""The span a passage delivery carries, or None when `text` is not EXACTLY
|
|
||||||
what the bundle's bytes rebuild.
|
|
||||||
|
|
||||||
Since v1.1 C1 a long concept is delivered as `[heading]`, `[...]`, the
|
|
||||||
span, `[...]` (`consume.as_passage`), with `passage = {start, end, of}`.
|
|
||||||
Two traps PM measured 2026-09-22, both held here:
|
|
||||||
|
|
||||||
- **The offsets count in the DELIVERED body** (`ConceptView.body`), never
|
|
||||||
in the concept file: read in the file they land a frontmatter's length
|
|
||||||
off, and 0 of 12 real spans matched.
|
|
||||||
- **"The span occurs somewhere in the text" is not a check.** It accepts
|
|
||||||
an invented sentence beside the span. Every character of `text` must be
|
|
||||||
accounted for: the span is `body[start:end]` byte for byte, a marker
|
|
||||||
stands exactly where the span leaves text out and nowhere else, and the
|
|
||||||
one other line allowed is a heading line of the body ABOVE the span (or a
|
|
||||||
prefix of one -- `as_passage` cuts a long heading). WHICH heading is the
|
|
||||||
product's choice and is not re-derived here; that it is the bundle's
|
|
||||||
bytes is the guarantee.
|
|
||||||
|
|
||||||
The citation is then read in the span alone -- not in the heading, not in
|
|
||||||
the markers, and not across the seam between them, which is no sequence
|
|
||||||
the concept file holds.
|
|
||||||
"""
|
|
||||||
if not isinstance(passage, Mapping):
|
|
||||||
return None
|
|
||||||
start, end, of = passage.get("start"), passage.get("end"), passage.get("of")
|
|
||||||
if not all(
|
|
||||||
isinstance(value, int) and not isinstance(value, bool) for value in (start, end, of)
|
|
||||||
):
|
|
||||||
return None
|
|
||||||
assert isinstance(start, int) and isinstance(end, int)
|
|
||||||
if of != len(body) or not 0 <= start < end <= len(body):
|
|
||||||
return None
|
|
||||||
span = body[start:end]
|
|
||||||
tail = f"\n{consume.PASSAGE_ELISION}" if end < len(body) else ""
|
|
||||||
if not text.endswith(span + tail):
|
|
||||||
return None
|
|
||||||
head = text[: len(text) - len(span + tail)]
|
|
||||||
if start == 0:
|
|
||||||
return span if head == "" else None
|
|
||||||
marker = f"{consume.PASSAGE_ELISION}\n"
|
|
||||||
if head == marker:
|
|
||||||
return span
|
|
||||||
if not head.endswith(f"\n{marker}"):
|
|
||||||
return None
|
|
||||||
heading = head[: -len(f"\n{marker}")]
|
|
||||||
above = body[:start].split("\n")
|
|
||||||
if heading.startswith("#") and any(
|
|
||||||
line.startswith("#") and line.startswith(heading) for line in above
|
|
||||||
):
|
|
||||||
return span
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _judged(excerpt: Mapping[str, object], body: str) -> tuple[str, bool, str]:
|
|
||||||
"""(the text a citation is looked for in, whether it is the bundle's
|
|
||||||
bytes, the detail when it is not) for one delivered excerpt."""
|
|
||||||
text = str(excerpt.get("text", ""))
|
|
||||||
if "passage" in excerpt:
|
|
||||||
span = passage_span(text, excerpt["passage"], body)
|
|
||||||
if span is None:
|
|
||||||
return text, False, PASSAGE_NOT_RECONSTRUCTED
|
|
||||||
return span, True, ""
|
|
||||||
return text, _flat(text) == _flat(body), "the delivered text is not the bundle's bytes"
|
|
||||||
|
|
||||||
|
|
||||||
def measure_units(bundle: Path, question: Question) -> list[Unit]:
|
def measure_units(bundle: Path, question: Question) -> list[Unit]:
|
||||||
"""One question, measured at the shipped defaults, plus the quota-off run
|
"""One question, measured at the shipped defaults, plus the quota-off run
|
||||||
that says what the truth of a withheld concept is.
|
that says what the truth of a withheld concept is.
|
||||||
|
|
@ -1120,21 +857,18 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
|
||||||
for concept_id in holding
|
for concept_id in holding
|
||||||
if _carries(index.concepts[concept_id].body, fasit.quote)
|
if _carries(index.concepts[concept_id].body, fasit.quote)
|
||||||
)
|
)
|
||||||
# A whole-body delivery is judged whole; a passage is judged as its
|
hit_ids = [
|
||||||
# exact reconstruction, and its citation is read in its span alone.
|
concept_id
|
||||||
judged = {
|
|
||||||
concept_id: _judged(delivered[concept_id], index.concepts[concept_id].body)
|
|
||||||
for concept_id in holding
|
for concept_id in holding
|
||||||
if concept_id in delivered
|
if concept_id in delivered
|
||||||
}
|
and _carries(str(delivered[concept_id].get("text", "")), fasit.quote)
|
||||||
hit_ids = [
|
|
||||||
concept_id for concept_id, (text, _, _) in judged.items() if _carries(text, fasit.quote)
|
|
||||||
]
|
]
|
||||||
# The payload SAYS it delivered this; the bundle says what it is.
|
# The payload SAYS it delivered this; the bundle says what it is.
|
||||||
confirmed: bool | None = None
|
confirmed: bool | None = None
|
||||||
not_bytes = ""
|
|
||||||
for concept_id in hit_ids:
|
for concept_id in hit_ids:
|
||||||
_, confirmed, not_bytes = judged[concept_id]
|
confirmed = _flat(str(delivered[concept_id].get("text", ""))) == _flat(
|
||||||
|
index.concepts[concept_id].body
|
||||||
|
)
|
||||||
if confirmed:
|
if confirmed:
|
||||||
break
|
break
|
||||||
# The concept this entry is really about: the one the bundle holds the
|
# The concept this entry is really about: the one the bundle holds the
|
||||||
|
|
@ -1164,7 +898,7 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
|
||||||
else "no concept in the bundle answers to this name"
|
else "no concept in the bundle answers to this name"
|
||||||
)
|
)
|
||||||
elif hit_ids and not confirmed:
|
elif hit_ids and not confirmed:
|
||||||
klass, detail = "e", not_bytes
|
klass, detail = "e", "the delivered text is not the bundle's bytes"
|
||||||
elif target in delivered:
|
elif target in delivered:
|
||||||
klass = "d"
|
klass = "d"
|
||||||
detail = "delivered without the citation" + (
|
detail = "delivered without the citation" + (
|
||||||
|
|
@ -1642,55 +1376,9 @@ def _as_share(value: object) -> float | None:
|
||||||
return number
|
return number
|
||||||
|
|
||||||
|
|
||||||
#: A registration is committed to a PUBLIC repository and read on more than
|
|
||||||
#: one machine, so every path it names is written under the home directory
|
|
||||||
#: and expanded when read. An absolute path names ONE machine -- it is refused
|
|
||||||
#: with its reason and never followed, so a machine path cannot be registered
|
|
||||||
#: again without row 5 saying so.
|
|
||||||
HOME_PREFIX = "~/"
|
|
||||||
|
|
||||||
|
|
||||||
def _home_path(value: object) -> tuple[Path | None, str]:
|
|
||||||
"""A registered path expanded against the home directory, or None and
|
|
||||||
the reason it is refused. The text returned beside a path is the
|
|
||||||
registration's own `~/...`, so a row names what was registered and never
|
|
||||||
the machine it was read on."""
|
|
||||||
text = value.strip() if isinstance(value, str) else ""
|
|
||||||
if not text:
|
|
||||||
return None, "not named"
|
|
||||||
if not text.startswith(HOME_PREFIX):
|
|
||||||
kind = "an absolute path" if Path(text).is_absolute() else "not under the home directory"
|
|
||||||
return None, f"`{text}` is {kind}; a registration names `{HOME_PREFIX}...` and no machine"
|
|
||||||
return Path(text).expanduser(), text
|
|
||||||
|
|
||||||
|
|
||||||
def _registered_bundle(spec: Mapping[str, object]) -> tuple[Path | None, str]:
|
|
||||||
"""The bundle the registration names, IF its tree still measures the
|
|
||||||
`bundle_ref` the registration pins; otherwise None and why.
|
|
||||||
|
|
||||||
A bundle path is a place, and a place can be filled with other bytes: the
|
|
||||||
ref is what makes the path the registered bundle."""
|
|
||||||
path, text = _home_path(spec.get("bundle"))
|
|
||||||
if path is None:
|
|
||||||
return None, text
|
|
||||||
registered = spec.get("bundle_ref")
|
|
||||||
if not isinstance(registered, str) or not registered:
|
|
||||||
return None, "the registration pins no bundle_ref"
|
|
||||||
if not path.is_dir():
|
|
||||||
return None, f"the bundle is absent at {text}"
|
|
||||||
try:
|
|
||||||
measured = consume.bundle_ref(path)
|
|
||||||
except Exception as error: # a bundle that cannot be measured is a NO
|
|
||||||
return None, f"no ref could be measured at {text}: {type(error).__name__}: {error}"
|
|
||||||
if measured != registered:
|
|
||||||
return None, f"{text} measures {measured}, the registration pins {registered}"
|
|
||||||
return path, f"{text} measures the pinned {registered[:24]}"
|
|
||||||
|
|
||||||
|
|
||||||
def _hold_out_verdict(
|
def _hold_out_verdict(
|
||||||
set_path: Path | None,
|
spec: Mapping[str, object],
|
||||||
set_text: str,
|
set_path: Path,
|
||||||
bundle: Path | None,
|
|
||||||
pinned: str,
|
pinned: str,
|
||||||
threshold: float | None,
|
threshold: float | None,
|
||||||
) -> tuple[bool, str]:
|
) -> tuple[bool, str]:
|
||||||
|
|
@ -1699,20 +1387,19 @@ def _hold_out_verdict(
|
||||||
|
|
||||||
Every refusal is a NO with its reason and never an exception: a
|
Every refusal is a NO with its reason and never an exception: a
|
||||||
registration naming an absent set is a finding about the registration.
|
registration naming an absent set is a finding about the registration.
|
||||||
A bundle that is not the registered tree is never measured -- a share
|
|
||||||
read off other bytes is not the registered hold-out's.
|
|
||||||
"""
|
"""
|
||||||
if threshold is None:
|
if threshold is None:
|
||||||
return False, "no numeric threshold to compare with"
|
return False, "no numeric threshold to compare with"
|
||||||
if bundle is None:
|
bundle = str(spec.get("bundle", ""))
|
||||||
return False, "not run: the bundle is not the registered tree"
|
if not bundle:
|
||||||
if set_path is None:
|
return False, "the registration names no bundle to measure against"
|
||||||
return False, f"not run: the set is {set_text}"
|
|
||||||
if not set_path.is_file():
|
if not set_path.is_file():
|
||||||
return False, f"the set is absent at {set_text}"
|
return False, f"the set is absent at {_display(set_path)}"
|
||||||
try:
|
try:
|
||||||
question_set = read_hold_out_set(set_path, pinned)
|
question_set = load_set(set_path, pinned)
|
||||||
bundles = {question_set.bundle: bundle}
|
bundles = _bundle_map(bundle)
|
||||||
|
if list(bundles) == [""] and question_set.bundle:
|
||||||
|
bundles = {question_set.bundle: bundles[""]}
|
||||||
units = [
|
units = [
|
||||||
unit
|
unit
|
||||||
for question in question_set.questions
|
for question in question_set.questions
|
||||||
|
|
@ -1757,15 +1444,6 @@ def row_five(
|
||||||
registration first and the ranking change second, and it is red today
|
registration first and the ranking change second, and it is red today
|
||||||
because neither has happened.
|
because neither has happened.
|
||||||
|
|
||||||
AND IT NAMES NO MACHINE, AND THE BUNDLE IT NAMES IS THE ONE IT PINNED.
|
|
||||||
Until 2026-09-23 the committed registration carried two absolute paths of
|
|
||||||
the machine it was written on, in a public repository, and `bundle_ref`
|
|
||||||
was a field no line read -- other bytes at the registered bundle path
|
|
||||||
would have been measured as the registered bundle. Its paths are now
|
|
||||||
`~/...`, expanded when read, and an absolute one is a NO; the bundle's
|
|
||||||
tree is measured with `consume.bundle_ref` and a ref other than the pinned
|
|
||||||
one is a NO, with the hold-out NOT run against it.
|
|
||||||
|
|
||||||
WHAT GIT CANNOT PROVE, stated rather than implied: that nobody read the
|
WHAT GIT CANNOT PROVE, stated rather than implied: that nobody read the
|
||||||
number before writing the threshold. A number can be read from an
|
number before writing the threshold. A number can be read from an
|
||||||
uncommitted working tree, and no history shows that. What history does
|
uncommitted working tree, and no history shows that. What history does
|
||||||
|
|
@ -1788,9 +1466,6 @@ def row_five(
|
||||||
" and a bundle named, so the set can be RUN and its share put "
|
" and a bundle named, so the set can be RUN and its share put "
|
||||||
"beside the threshold. A number never compared with a measurement "
|
"beside the threshold. A number never compared with a measurement "
|
||||||
"is a note",
|
"is a note",
|
||||||
" both paths written `~/...` -- the file is public and an "
|
|
||||||
"absolute path names one machine -- and the bundle's tree ref "
|
|
||||||
"pinned as `bundle_ref`, so other bytes at that path are not measured",
|
|
||||||
" and COMMITTED before the ranking moves: git must show the "
|
" and COMMITTED before the ranking moves: git must show the "
|
||||||
f"registration in a commit of its own, with a later commit to "
|
f"registration in a commit of its own, with a later commit to "
|
||||||
f"{RANKING_PATH}. That is the half a session cannot write about itself",
|
f"{RANKING_PATH}. That is the half a session cannot write about itself",
|
||||||
|
|
@ -1803,10 +1478,10 @@ def row_five(
|
||||||
checks: list[tuple[str, bool, str]] = []
|
checks: list[tuple[str, bool, str]] = []
|
||||||
threshold = str(spec.get("threshold", ""))
|
threshold = str(spec.get("threshold", ""))
|
||||||
written_at = str(spec.get("threshold_written_at", ""))
|
written_at = str(spec.get("threshold_written_at", ""))
|
||||||
set_path, set_text = _home_path(spec.get("set"))
|
set_path = Path(str(spec.get("set", "")))
|
||||||
pinned = str(spec.get("sha256", ""))
|
pinned = str(spec.get("sha256", ""))
|
||||||
readings = spec.get("readings", [])
|
readings = spec.get("readings", [])
|
||||||
checks.append(("a set is named", set_path is not None, set_text))
|
checks.append(("a set is named", bool(str(spec.get("set", ""))), str(set_path)))
|
||||||
checks.append(("a sha256 is pinned", len(pinned) == 64, pinned[:12]))
|
checks.append(("a sha256 is pinned", len(pinned) == 64, pinned[:12]))
|
||||||
number = _as_share(threshold)
|
number = _as_share(threshold)
|
||||||
checks.append(
|
checks.append(
|
||||||
|
|
@ -1824,16 +1499,13 @@ def row_five(
|
||||||
str(spec.get("written_by", "")),
|
str(spec.get("written_by", "")),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
present = set_path is not None and set_path.is_file()
|
|
||||||
checks.append(
|
checks.append(
|
||||||
(
|
(
|
||||||
"the pinned bytes are the bytes on disk",
|
"the pinned bytes are the bytes on disk",
|
||||||
present and set_path is not None and sha256_of(set_path) == pinned,
|
set_path.is_file() and sha256_of(set_path) == pinned,
|
||||||
f"present at {set_text}" if present else f"absent at {set_text}",
|
"present" if set_path.is_file() else "absent",
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
bundle, bundle_note = _registered_bundle(spec)
|
|
||||||
checks.append(("the bundle is the registered tree", bundle is not None, bundle_note))
|
|
||||||
early = [
|
early = [
|
||||||
reading
|
reading
|
||||||
for reading in readings
|
for reading in readings
|
||||||
|
|
@ -1851,7 +1523,7 @@ def row_five(
|
||||||
# whole check until 2026-09-20, and `report-only; any number is acceptable
|
# whole check until 2026-09-20, and `report-only; any number is acceptable
|
||||||
# for v1` passed it. The hold-out is run HERE, against the bundle the
|
# for v1` passed it. The hold-out is run HERE, against the bundle the
|
||||||
# registration names, and the row says what it measured.
|
# registration names, and the row says what it measured.
|
||||||
cleared, note = _hold_out_verdict(set_path, set_text, bundle, pinned, number)
|
cleared, note = _hold_out_verdict(spec, set_path, pinned, number)
|
||||||
checks.append(("the measured hold-out clears the threshold", cleared, note))
|
checks.append(("the measured hold-out clears the threshold", cleared, note))
|
||||||
|
|
||||||
history = provenance(registration)
|
history = provenance(registration)
|
||||||
|
|
@ -1927,10 +1599,8 @@ class Mutant:
|
||||||
expected_row: int
|
expected_row: int
|
||||||
patch: Callable[[], contextlib.AbstractContextManager[None]]
|
patch: Callable[[], contextlib.AbstractContextManager[None]]
|
||||||
#: Why this one may be invisible to a gate that judges DELIVERY. Written
|
#: Why this one may be invisible to a gate that judges DELIVERY. Written
|
||||||
#: for a mutant that survives, so a survivor is a statement about the
|
#: for the two that survive, so a survivor is a statement about the
|
||||||
#: mechanism and not a shrug. None does since 2026-09-22: the three that
|
#: mechanism and not a shrug.
|
||||||
#: carried a note (M06, M07, M10) each got a fixture that makes their
|
|
||||||
#: mechanism decide a delivery (`PASSAGE`, `PATH`, `FUSION`).
|
|
||||||
note: str = ""
|
note: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -2017,25 +1687,6 @@ def _extend_delivered() -> contextlib.AbstractContextManager[None]:
|
||||||
return _patched(delivered_text=mutant)
|
return _patched(delivered_text=mutant)
|
||||||
|
|
||||||
|
|
||||||
def _extend_passage() -> contextlib.AbstractContextManager[None]:
|
|
||||||
"""M14's shape for the delivery form v1.1 C1 added: the passage still
|
|
||||||
CARRIES the citation in its span, and one sentence at the span's end is not
|
|
||||||
the concept file's. A judge that asked only whether the span occurs
|
|
||||||
somewhere in the text would count it."""
|
|
||||||
original = consume.as_passage
|
|
||||||
|
|
||||||
def mutant(excerpt: dict[str, object], window: int) -> dict[str, object]:
|
|
||||||
out = original(excerpt, window)
|
|
||||||
if "passage" in out:
|
|
||||||
text = str(out["text"])
|
|
||||||
tail = f"\n{consume.PASSAGE_ELISION}"
|
|
||||||
cut = len(text) - len(tail) if text.endswith(tail) else len(text)
|
|
||||||
out["text"] = f"{text[:cut]} En setning som ikke staar i konseptfila.{text[cut:]}"
|
|
||||||
return out
|
|
||||||
|
|
||||||
return _patched(as_passage=mutant)
|
|
||||||
|
|
||||||
|
|
||||||
def _drop_text_key() -> contextlib.AbstractContextManager[None]:
|
def _drop_text_key() -> contextlib.AbstractContextManager[None]:
|
||||||
original = consume.excerpt_for
|
original = consume.excerpt_for
|
||||||
|
|
||||||
|
|
@ -2066,12 +1717,23 @@ MUTANTS: tuple[Mutant, ...] = (
|
||||||
Mutant("M04 the ranking is reversed", 1, _reverse_scores),
|
Mutant("M04 the ranking is reversed", 1, _reverse_scores),
|
||||||
Mutant("M05 the source quota is removed", 3, lambda: _wrap_cut(source_quota=None)),
|
Mutant("M05 the source quota is removed", 3, lambda: _wrap_cut(source_quota=None)),
|
||||||
Mutant(
|
Mutant(
|
||||||
"M06 the passage signal reads no body", 1, lambda: _patched_bm25(windows=lambda body: [])
|
"M06 the passage signal reads no body",
|
||||||
|
1,
|
||||||
|
lambda: _patched_bm25(windows=lambda body: []),
|
||||||
|
note=(
|
||||||
|
"the field signal still reads every body, and no synthetic concept is "
|
||||||
|
"long enough for its best window to decide a delivery"
|
||||||
|
),
|
||||||
),
|
),
|
||||||
Mutant(
|
Mutant(
|
||||||
"M07 the field signal weighs no title and no path",
|
"M07 the field signal weighs no title and no path",
|
||||||
1,
|
1,
|
||||||
lambda: _patched_bm25(TITLE_WEIGHT=0, PATH_WEIGHT=0),
|
lambda: _patched_bm25(TITLE_WEIGHT=0, PATH_WEIGHT=0),
|
||||||
|
note=(
|
||||||
|
"every synthetic body opens with its own title as a heading "
|
||||||
|
"(`build_bundle`), so the weight only re-weights words the body "
|
||||||
|
"already carries"
|
||||||
|
),
|
||||||
),
|
),
|
||||||
Mutant("M08 every token matches every token", 4, _every_term_everything),
|
Mutant("M08 every token matches every token", 4, _every_term_everything),
|
||||||
Mutant("M09 no stem, no prefix: equality only", 1, _equality_only),
|
Mutant("M09 no stem, no prefix: equality only", 1, _equality_only),
|
||||||
|
|
@ -2079,6 +1741,10 @@ MUTANTS: tuple[Mutant, ...] = (
|
||||||
"M10 the fusion is flattened (bm25.RRF_K = 10 000)",
|
"M10 the fusion is flattened (bm25.RRF_K = 10 000)",
|
||||||
1,
|
1,
|
||||||
lambda: _patched_bm25(RRF_K=10_000),
|
lambda: _patched_bm25(RRF_K=10_000),
|
||||||
|
note=(
|
||||||
|
"over two signals K reorders only a concept whose two ranks lie far "
|
||||||
|
"apart against one whose ranks are close, and no fixture here holds that"
|
||||||
|
),
|
||||||
),
|
),
|
||||||
Mutant("M11 the cut takes the LAST k", 1, _last_k),
|
Mutant("M11 the cut takes the LAST k", 1, _last_k),
|
||||||
Mutant("M12 the delivered text is truncated to 40 characters", 6, _truncate_delivered),
|
Mutant("M12 the delivered text is truncated to 40 characters", 6, _truncate_delivered),
|
||||||
|
|
@ -2086,7 +1752,6 @@ MUTANTS: tuple[Mutant, ...] = (
|
||||||
Mutant(
|
Mutant(
|
||||||
"M14 the delivered text carries a sentence the concept file does not", 6, _extend_delivered
|
"M14 the delivered text carries a sentence the concept file does not", 6, _extend_delivered
|
||||||
),
|
),
|
||||||
Mutant("M15 the passage carries a sentence the concept file does not", 6, _extend_passage),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -2113,11 +1778,10 @@ MUTANT_ROSTER: tuple[str, ...] = (
|
||||||
"M12 the delivered text is truncated to 40 characters",
|
"M12 the delivered text is truncated to 40 characters",
|
||||||
"M13 the excerpt carries no text",
|
"M13 the excerpt carries no text",
|
||||||
"M14 the delivered text carries a sentence the concept file does not",
|
"M14 the delivered text carries a sentence the concept file does not",
|
||||||
"M15 the passage carries a sentence the concept file does not",
|
|
||||||
)
|
)
|
||||||
|
|
||||||
#: The roster's length, written as a number so appending is not one edit.
|
#: The roster's length, written as a number so appending is not one edit.
|
||||||
MUTANT_COUNT = 15
|
MUTANT_COUNT = 14
|
||||||
|
|
||||||
|
|
||||||
def _score(rows: Sequence[Row]) -> dict[int, int]:
|
def _score(rows: Sequence[Row]) -> dict[int, int]:
|
||||||
|
|
@ -2274,72 +1938,23 @@ def read_real_set(name: str, path: Path, expected_sha256: str) -> QuestionSet:
|
||||||
)
|
)
|
||||||
spec = json.loads(raw.decode("utf-8"))
|
spec = json.loads(raw.decode("utf-8"))
|
||||||
if name == "wiki":
|
if name == "wiki":
|
||||||
return QuestionSet("wiki-20", "wiki", path, measured, _wiki_questions(spec), ())
|
# The set's own `hit_rule`, verbatim: an excerpt whose `source_file` is
|
||||||
|
# the fasit's document AND whose text carries the fasit's quote.
|
||||||
|
questions = tuple(
|
||||||
|
Question(
|
||||||
|
id=str(entry["id"]),
|
||||||
|
question=str(entry["question"]),
|
||||||
|
fasit=tuple(
|
||||||
|
Fasit(by="source_file", value=f"{item['doc']}.md", quote=str(item["quote"]))
|
||||||
|
for item in entry["fasit"]
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for entry in spec["questions"]
|
||||||
|
)
|
||||||
|
return QuestionSet("wiki-20", "wiki", path, measured, questions, ())
|
||||||
raise GateUsage(f"unknown real set `{name}`; the one real set is `wiki`")
|
raise GateUsage(f"unknown real set `{name}`; the one real set is `wiki`")
|
||||||
|
|
||||||
|
|
||||||
def _wiki_questions(spec: Mapping[str, Any]) -> tuple[Question, ...]:
|
|
||||||
"""The wiki set's own `hit_rule`, verbatim: an excerpt whose `source_file`
|
|
||||||
is the fasit's document AND whose text carries the fasit's quote. One
|
|
||||||
reading of the schema, shared by row 8 and row 5."""
|
|
||||||
return tuple(
|
|
||||||
Question(
|
|
||||||
id=str(entry["id"]),
|
|
||||||
question=str(entry["question"]),
|
|
||||||
fasit=tuple(
|
|
||||||
Fasit(by="source_file", value=f"{item['doc']}.md", quote=str(item["quote"]))
|
|
||||||
for item in entry["fasit"]
|
|
||||||
),
|
|
||||||
)
|
|
||||||
for entry in spec["questions"]
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
#: The schemas a registered hold-out set may declare in its own `schema`
|
|
||||||
#: field, and the bundle key its questions are read under. The SET says how
|
|
||||||
#: it is read, so the registration carries nothing about it; a set with no
|
|
||||||
#: `schema` is in this gate's own synthetic form (`load_set`), and a schema
|
|
||||||
#: not named here is a NO with its name -- a set read in a shape it was not
|
|
||||||
#: written in measures nothing.
|
|
||||||
HOLD_OUT_SCHEMAS: Mapping[str, str] = {"fase-sporsmaal/1": "wiki"}
|
|
||||||
|
|
||||||
|
|
||||||
def read_hold_out_set(path: Path, expected_sha256: str) -> QuestionSet:
|
|
||||||
"""The registered hold-out set, read in the schema it declares.
|
|
||||||
|
|
||||||
Raises GateUsage for a set that is not the pinned bytes or declares a
|
|
||||||
schema this gate does not read; `_hold_out_verdict` turns either into a
|
|
||||||
NO with its reason.
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
raw = path.read_bytes()
|
|
||||||
except OSError as error:
|
|
||||||
raise GateUsage(f"cannot read the question set {path}: {error}") from error
|
|
||||||
measured = hashlib.sha256(raw).hexdigest()
|
|
||||||
if measured != expected_sha256:
|
|
||||||
raise GateUsage(
|
|
||||||
f"{path}: expected sha256 {expected_sha256}, measured {measured}; "
|
|
||||||
"refusing to measure a set that is not the set that was pinned"
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
spec = json.loads(raw.decode("utf-8"))
|
|
||||||
except (UnicodeDecodeError, json.JSONDecodeError) as error:
|
|
||||||
raise GateUsage(f"{path}: not readable as JSON: {error}") from error
|
|
||||||
schema = spec.get("schema") if isinstance(spec, dict) else None
|
|
||||||
if schema is None:
|
|
||||||
return load_set(path, expected_sha256)
|
|
||||||
if schema not in HOLD_OUT_SCHEMAS:
|
|
||||||
raise GateUsage(
|
|
||||||
f"the set's schema `{schema}` is not one this gate reads "
|
|
||||||
f"({', '.join(HOLD_OUT_SCHEMAS)}, or no `schema` for this gate's own form)"
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
questions = _wiki_questions(spec)
|
|
||||||
except (KeyError, TypeError, ValueError) as error:
|
|
||||||
raise GateUsage(f"{path}: not a `{schema}` set this gate can read: {error}") from error
|
|
||||||
return QuestionSet("hold-out", HOLD_OUT_SCHEMAS[schema], path, measured, questions, ())
|
|
||||||
|
|
||||||
|
|
||||||
#: The sets row 8 is the measurement of, by name: a run that hands over some
|
#: The sets row 8 is the measurement of, by name: a run that hands over some
|
||||||
#: of them has measured some of them, and the row says so. Left to
|
#: of them has measured some of them, and the row says so. Left to
|
||||||
#: `len(real)` the row came back `6 of 6 GREEN` on one set of three (PM's J2,
|
#: `len(real)` the row came back `6 of 6 GREEN` on one set of three (PM's J2,
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue