Compare commits
No commits in common. "main" and "v1.1.0" have entirely different histories.
6 changed files with 56 additions and 577 deletions
34
CLAUDE.md
34
CLAUDE.md
|
|
@ -1318,37 +1318,15 @@ and fixtures, never code.
|
||||||
what a bundle looks like, this asks whether the payload for a question
|
what a bundle looks like, this asks whether the payload for a question
|
||||||
carries the fasit. **Written RED, no capability**: eight rows, exit 0 only
|
carries the fasit. **Written RED, no capability**: eight rows, exit 0 only
|
||||||
when all are green, 1 otherwise, 2 on wrong input, `--json`. Rows 1-4, 6, 7
|
when all are green, 1 otherwise, 2 on wrong input, `--json`. Rows 1-4, 6, 7
|
||||||
run against a synthetic corpus this repo GENERATES and eight sets it ships
|
run against a synthetic corpus this repo GENERATES and six sets it ships
|
||||||
pinned by sha256 -- no network, no private corpus, no clock; a set whose
|
pinned by sha256 -- no network, no private corpus, no clock; a set whose
|
||||||
bytes moved is exit 2, never a quiet continuation, because **a gold set is
|
bytes moved is exit 2, never a quiet continuation, because **a gold set is
|
||||||
an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and
|
an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and
|
||||||
this repo is public). **THE CORPUS IS PINNED THE SAME WAY** since
|
this repo is public). **THE CORPUS IS PINNED THE SAME WAY** since
|
||||||
2026-09-19 (`SPECS_SHA256`). Today (measured on the default run
|
2026-09-19 (`SPECS_SHA256`). Today (measured on the default run
|
||||||
2026-09-22): **1 (14/14) - 2 (7/7) - 3 (5/5) - 4 (6/6) - 5 (0/1) -
|
2026-09-21, v1.1 K): **1 (10/10) - 2 (7/7) - 3 (5/5) - 4 (6/6) - 5 (0/1) -
|
||||||
6 (14/14) - 7 (15/15) - 8 (0 of 1 set, NOT RUN without `--real`)**,
|
6 (10/10) - 7 (11/14) - 8 (0 of 1 set, NOT RUN without `--real`)**,
|
||||||
`GATE RED: rows 5, 8`. **Row 7 fells 14 of 14 since 2026-09-22**: the three
|
`GATE RED: rows 5, 7, 8`.
|
||||||
that survived v1.1 (M06 passage body, M07 title/path weight, M10
|
|
||||||
`bm25.RRF_K`) each got a fixture that makes its mechanism decide a delivery
|
|
||||||
(`PASSAGE`, `PATH`, `FUSION`, set `set-mechanisms.json`; the fusion one keeps
|
|
||||||
its lead through K = 180 and loses it at 190), with `src/` untouched.
|
|
||||||
**THE JUDGE READS A PASSAGE AS A PASSAGE (2026-09-22).** It predated v1.1
|
|
||||||
C1 and required a delivered text to be the whole concept's bytes, so every
|
|
||||||
excerpt `as_passage` cut was class (e) even when its span carried the
|
|
||||||
citation -- the dominant miss class when row 8 was run on the real set.
|
|
||||||
`passage_span` now accepts a passage only when the delivered text IS its
|
|
||||||
exact reconstruction: the span is `body[start:end]` byte for byte, `of` is
|
|
||||||
the body's length, a `[...]` stands exactly where text is left out, and the
|
|
||||||
one other line allowed is a heading line of the body above the span (or a
|
|
||||||
prefix of one); WHICH heading is the product's choice and is not re-derived.
|
|
||||||
The citation is read in the SPAN alone, never in the heading, the markers or
|
|
||||||
across the seam. Two traps are pinned by tests: the offsets count in the
|
|
||||||
DELIVERED body (read in the concept file they land a frontmatter's length
|
|
||||||
off), and "the span occurs somewhere in the text" accepts an invented
|
|
||||||
sentence beside it. **M15** (a passage carrying one sentence the file does
|
|
||||||
not, M14's shape for passages) is row 7's fifteenth mutant, felled through a
|
|
||||||
new fixture that delivers a passage (`DELIVERY`, set `set-passage.json`),
|
|
||||||
because no synthetic concept was long enough to be cut. `src/` untouched;
|
|
||||||
`MUTANT_BAR` unchanged.
|
|
||||||
**THE K2 TEST TRACK IS RETIRED (operator 2026-09-21, v1.1 K)**: material
|
**THE K2 TEST TRACK IS RETIRED (operator 2026-09-21, v1.1 K)**: material
|
||||||
tied to the operator's employer is not re-measured and not frozen, and
|
tied to the operator's employer is not re-measured and not frozen, and
|
||||||
public tests and gates run on invented material. Row 9 (K2) and the `r761`
|
public tests and gates run on invented material. Row 9 (K2) and the `r761`
|
||||||
|
|
@ -1357,8 +1335,8 @@ and fixtures, never code.
|
||||||
the R761/N101/N200 deliveries; `okf_accounting_gate.py` has no default real
|
the R761/N101/N200 deliveries; `okf_accounting_gate.py` has no default real
|
||||||
corpus. The synthetic corpus was re-measured for BM25 (`MISS`, `LOOKUP`,
|
corpus. The synthetic corpus was re-measured for BM25 (`MISS`, `LOOKUP`,
|
||||||
`QUOTA`), so rows 2 and 3 are green through their own fixtures again, and
|
`QUOTA`), so rows 2 and 3 are green through their own fixtures again, and
|
||||||
row 7's mutants patch `bm25` where the default runs there; the three that
|
row 7's mutants patch `bm25` where the default runs there; three survive
|
||||||
survived with 0 ranks moved are felled since 2026-09-22 (above). What
|
(passage body, field title weight, `bm25.RRF_K`) with 0 ranks moved. What
|
||||||
the text below says about rows 8/9, K2 and the three sets is history.
|
the text below says about rows 8/9, K2 and the three sets is history.
|
||||||
**FOUR OF EIGHT CHEATING ATTACKS WENT THROUGH IT AND ARE CLOSED
|
**FOUR OF EIGHT CHEATING ATTACKS WENT THROUGH IT AND ARE CLOSED
|
||||||
(2026-09-19).** PM's checkpoint on `2c8296b` found rows 3, 5, 7 and 8 could
|
(2026-09-19).** PM's checkpoint on `2c8296b` found rows 3, 5, 7 and 8 could
|
||||||
|
|
|
||||||
47
README.md
47
README.md
|
|
@ -84,13 +84,13 @@ here was measured; none of it is a plan.
|
||||||
`--no-assets` takes the image path out entirely.
|
`--no-assets` takes the image path out entirely.
|
||||||
- **Three of this repository's own gates are RED, and each red row is a stated
|
- **Three of this repository's own gates are RED, and each red row is a stated
|
||||||
finding rather than a bug to be surprised by.** The retrieval gate is red on
|
finding rather than a bug to be surprised by.** The retrieval gate is red on
|
||||||
rows 5 and 8, the MCP gate on row 2, and the content accounting's judge
|
rows 5, 7 and 8, the MCP gate on row 2, and the content accounting's judge
|
||||||
on rows 2, 3 and 6. For a user that means: no ranking change has yet been
|
on rows 2, 3 and 6. For a user that means: no ranking change has yet been
|
||||||
registered against a held-out set (row 5), the real-corpus row is NOT RUN
|
registered against a held-out set (row 5), the real-corpus row is NOT RUN
|
||||||
without its private set and so is red rather than green by omission
|
without its private sets and so is red rather than green by omission
|
||||||
(row 8) -- and since 2026-09-22 its judge reads an excerpt v1.1 cuts to its
|
(row 8), three mechanical mutants of the BM25 ranking survive with 0 ranks
|
||||||
answering passage as that passage, accepted only when it is the exact
|
moved — the passage body, the title field weight and the fusion constant —
|
||||||
reconstruction from the bundle's bytes. MCP anchors and concept ids are different
|
so the gate cannot yet tell those parts from their absence (row 7), MCP anchors and concept ids are different
|
||||||
vocabularies so `okf_fetch` cannot be addressed with a set's anchor (row 2),
|
vocabularies so `okf_fetch` cannot be addressed with a set's anchor (row 2),
|
||||||
and the accounting still reports real losses on the reference corpus (rows 2,
|
and the accounting still reports real losses on the reference corpus (rows 2,
|
||||||
3, 6). The rows and their numbers are under [Judge the
|
3, 6). The rows and their numbers are under [Judge the
|
||||||
|
|
@ -1196,18 +1196,18 @@ set names a consumer's documents and this repository is public. **The corpus
|
||||||
is pinned the same way** (`SPECS_SHA256`): every row counts against those
|
is pinned the same way** (`SPECS_SHA256`): every row counts against those
|
||||||
documents, so moving them without moving the pin is exit 2.
|
documents, so moving them without moving the pin is exit 2.
|
||||||
|
|
||||||
**It is RED today, on rows 5 and 8**, and each of those is a
|
**It is RED today, on rows 5, 7 and 8**, and each of those is a
|
||||||
finding rather than a defect in the gate:
|
finding rather than a defect in the gate:
|
||||||
|
|
||||||
| row | what it asks | today |
|
| row | what it asks | today |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| 1 | hit@payload, one fasit entry = one unit | 14 of 14 |
|
| 1 | hit@payload, one fasit entry = one unit | 10 of 10 |
|
||||||
| 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 |
|
| 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 |
|
||||||
| 3 | the `rule` the payload prints for a withheld fasit is the true one | 5 of 5 |
|
| 3 | the `rule` the payload prints for a withheld fasit is the true one | 5 of 5 |
|
||||||
| 4 | an uncovered question comes back marked, a covered one does not | 6 of 6 |
|
| 4 | an uncovered question comes back marked, a covered one does not | 6 of 6 |
|
||||||
| 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 |
|
| 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 |
|
||||||
| 6 | every delivery confirmed against the bundle's own bytes | 14 of 14 |
|
| 6 | every delivery confirmed against the bundle's own bytes | 10 of 10 |
|
||||||
| 7 | mechanical mutants of the ranking and the cut, felled | 15 of 15 |
|
| 7 | mechanical mutants of the ranking and the cut, felled | 11 of 14 |
|
||||||
| 8 | the real set, from path + sha256 | 0 of 1 set, NOT RUN without `--real` |
|
| 8 | the real set, from path + sha256 | 0 of 1 set, NOT RUN without `--real` |
|
||||||
|
|
||||||
**The public rows run on invented material only** (operator decision
|
**The public rows run on invented material only** (operator decision
|
||||||
|
|
@ -1230,28 +1230,13 @@ delivered excerpt answers — facts, no verdict, and the one bar is the gate's
|
||||||
own `UNANSWERED_BAR = 2/3` — 3 of 6 to 6 of 6.
|
own `UNANSWERED_BAR = 2/3` — 3 of 6 to 6 of 6.
|
||||||
|
|
||||||
Row 7's mutants patch the code the DEFAULT ranking runs (BM25 since v1.1),
|
Row 7's mutants patch the code the DEFAULT ranking runs (BM25 since v1.1),
|
||||||
and it fells all 15. Until 2026-09-22 three survived with **0 ranks and 0
|
and it reports three survivors with what they moved rather than with a shrug:
|
||||||
deliveries** moved -- the passage signal reading no body, the field signal
|
the passage signal reading no body, the field signal weighing no title and no
|
||||||
weighing no title and no path, and a flattened fusion (`bm25.RRF_K`) -- because
|
path, and a flattened fusion (`bm25.RRF_K`) each moved **0 ranks and 0
|
||||||
every synthetic concept was short and opened with its own title, so none of the
|
deliveries** on these fixtures. Each has a mechanism: the field signal still
|
||||||
three ever decided a delivery. Three fixtures now make each one decide one
|
reads every body, every synthetic body opens with its own title as a heading,
|
||||||
(`set-mechanisms.json`): a long concept answered in one window of its body
|
and over two signals `K` only reorders a concept whose two ranks lie far apart,
|
||||||
against short concepts whose titles carry the question's words; a concept named
|
which no fixture holds.
|
||||||
by its path alone against decoys denser in the body; and a concept first in one
|
|
||||||
signal and 21st in the other against a decoy 10th and 11th, which a flattened
|
|
||||||
fusion ranks first because only the rank sum then counts. Each fixture is
|
|
||||||
delivered at the shipped defaults and falls out of `k` with its own mechanism
|
|
||||||
switched off; `src/` did not change.
|
|
||||||
|
|
||||||
A delivery is judged against the bundle's own bytes (row 6), and since v1.1 a
|
|
||||||
concept over 4 000 characters is delivered as its answering passage: the
|
|
||||||
nearest heading above it, `[...]`, the span, `[...]`. The judge counts such a
|
|
||||||
passage only when it is EXACTLY that reconstruction -- the span byte for byte
|
|
||||||
the delivered body at the passage's own offsets, markers only where text is
|
|
||||||
left out, the heading a line of the body above the span -- and it reads the
|
|
||||||
citation in the span alone. A looser rule ("the span occurs somewhere in the
|
|
||||||
text") would accept an invented sentence beside the span; the fifteenth
|
|
||||||
mutant (M15) is exactly that, and a fixture that delivers a passage fells it.
|
|
||||||
|
|
||||||
Row 8 is never green by leaving something out: it requires every named set and
|
Row 8 is never green by leaving something out: it requires every named set and
|
||||||
is NOT RUN until it has them — one set of three used to read `6 of 6 GREEN`.
|
is NOT RUN until it has them — one set of three used to read `6 of 6 GREEN`.
|
||||||
|
|
|
||||||
28
tests/fixtures/retrieval/set-mechanisms.json
vendored
28
tests/fixtures/retrieval/set-mechanisms.json
vendored
|
|
@ -1,28 +0,0 @@
|
||||||
{
|
|
||||||
"set_id": "synthetic-mechanisms",
|
|
||||||
"bundle": "passage",
|
|
||||||
"note": "Three questions, one per mechanism the default BM25 ranking runs and no earlier fixture made decide a delivery: the passage signal's body windows (bundle `passage`), the field signal's title and path weight (bundle `path`), and the rank-fusion constant (bundle `fusion`). Each is delivered at the shipped defaults and falls out of k when its own mechanism is switched off, so row 1 is what fells M06, M07 and M10.",
|
|
||||||
"questions": [
|
|
||||||
{
|
|
||||||
"id": "PA1",
|
|
||||||
"question": "Hvem kontrollerer varmekablene i taket?",
|
|
||||||
"bundle": "passage",
|
|
||||||
"fasit": [{"by": "concept", "value": "aarbok/kapittel-tre", "quote": "kontrolleres av vaktmesteren"}]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "PT1",
|
|
||||||
"question": "Hva gjelder for vester ved kanopadling?",
|
|
||||||
"bundle": "path",
|
|
||||||
"k": 6,
|
|
||||||
"fasit": [{"by": "concept", "value": "kanopadling/regel", "quote": "brukes paa vannet"}]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": "FU1",
|
|
||||||
"question": "Hva gjelder for isbading?",
|
|
||||||
"bundle": "fusion",
|
|
||||||
"k": 1,
|
|
||||||
"fasit": [{"by": "concept", "value": "badebok/kapittel", "quote": "isbading om morgenen"}]
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"controls": []
|
|
||||||
}
|
|
||||||
13
tests/fixtures/retrieval/set-passage.json
vendored
13
tests/fixtures/retrieval/set-passage.json
vendored
|
|
@ -1,13 +0,0 @@
|
||||||
{
|
|
||||||
"set_id": "synthetic-passage",
|
|
||||||
"bundle": "delivery",
|
|
||||||
"note": "One question whose answer lies in a concept over consume.PASSAGE_CHARS, so it is delivered as a passage: heading, [...], span, [...]. The span carries the citation. Rows 1 and 6 count it only when the judge accepts a passage that is its exact reconstruction from the bundle's bytes, and M15 (one sentence added to the passage) fells both.",
|
|
||||||
"questions": [
|
|
||||||
{
|
|
||||||
"id": "PD1",
|
|
||||||
"question": "Hvem gir varmepumpa i kjelleren service?",
|
|
||||||
"fasit": [{"by": "concept", "value": "husbok/drift", "quote": "service av roerleggeren"}]
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"controls": []
|
|
||||||
}
|
|
||||||
|
|
@ -524,37 +524,6 @@ def test_a_mutant_is_felled_by_the_row_that_got_worse_and_never_by_one_that_did_
|
||||||
assert after[3] >= before[3]
|
assert after[3] >= before[3]
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
"prefix",
|
|
||||||
[
|
|
||||||
# The three that survived v1.1: each is a mechanism the default ranking
|
|
||||||
# RUNS, so a gate that cannot see it removed cannot see it break.
|
|
||||||
"M06 ", # the passage signal reads no body
|
|
||||||
"M07 ", # the field signal weighs no title and no path
|
|
||||||
"M10 ", # the fusion is flattened
|
|
||||||
# A passage carrying one sentence the file does not: M14's shape for
|
|
||||||
# the delivery form v1.1 C1 added.
|
|
||||||
"M15 ",
|
|
||||||
],
|
|
||||||
)
|
|
||||||
def test_row_seven_fells_every_mutant_of_a_mechanism_the_ranking_runs(
|
|
||||||
tmp_path: Path, prefix: str
|
|
||||||
) -> None:
|
|
||||||
"""A survivor of row 7 is a mechanism the synthetic corpus never makes
|
|
||||||
decide a delivery. Felled means a row got WORSE on the pinned sets and the
|
|
||||||
pinned corpus -- never a row that merely moved."""
|
|
||||||
(mutant,) = [m for m in gate.MUTANTS if m.label.startswith(prefix)]
|
|
||||||
cases, _ = gate.synthetic_cases(tmp_path / "bundles", FIXTURES)
|
|
||||||
before = {row.number: row.k - row.m for row in gate.deterministic_rows(cases)}
|
|
||||||
with mutant.patch():
|
|
||||||
rows = gate.deterministic_rows(
|
|
||||||
[gate.measure_case(case.question_set, case.bundles) for case in cases]
|
|
||||||
)
|
|
||||||
after = {row.number: row.k - row.m for row in rows}
|
|
||||||
worse = [number for number in sorted(before) if after[number] < before[number]]
|
|
||||||
assert worse, f"{mutant.label} survives: no row got worse ({before} -> {after})"
|
|
||||||
|
|
||||||
|
|
||||||
# --- row 8 --------------------------------------------------------------------
|
# --- row 8 --------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -635,18 +604,14 @@ def test_the_gate_is_red_today_and_says_which_rows(tmp_path: Path) -> None:
|
||||||
by_number = {row.number: row for row in rows}
|
by_number = {row.number: row for row in rows}
|
||||||
assert sorted(by_number) == [1, 2, 3, 4, 5, 6, 7, 8]
|
assert sorted(by_number) == [1, 2, 3, 4, 5, 6, 7, 8]
|
||||||
# Rows 2 and 3 are green again since the synthetic corpus was re-measured
|
# Rows 2 and 3 are green again since the synthetic corpus was re-measured
|
||||||
# for BM25 (2026-09-21). Row 7 fells all 14 since 2026-09-22, when the three
|
# for BM25 (2026-09-21). Row 7 is red on three survivors, each printed with
|
||||||
# v1.1 survivors each got a fixture that makes their mechanism decide a
|
# what it moved.
|
||||||
# delivery (`set-mechanisms.json`, three more row-1 units), and all 15
|
assert [row.number for row in rows if row.fails] == [5, 7, 8]
|
||||||
# since the judge reads a passage as its exact reconstruction and one
|
assert (by_number[1].k, by_number[1].m) == (10, 10)
|
||||||
# fixture delivers a passage (`set-passage.json`, one more unit), so M15
|
|
||||||
# has something to fell.
|
|
||||||
assert [row.number for row in rows if row.fails] == [5, 8]
|
|
||||||
assert (by_number[1].k, by_number[1].m) == (14, 14)
|
|
||||||
assert (by_number[2].k, by_number[2].m) == (7, 7)
|
assert (by_number[2].k, by_number[2].m) == (7, 7)
|
||||||
assert (by_number[3].k, by_number[3].m) == (5, 5)
|
assert (by_number[3].k, by_number[3].m) == (5, 5)
|
||||||
assert (by_number[6].k, by_number[6].m) == (14, 14)
|
assert (by_number[6].k, by_number[6].m) == (10, 10)
|
||||||
assert (by_number[7].k, by_number[7].m) == (15, 15)
|
assert (by_number[7].k, by_number[7].m) == (11, 14)
|
||||||
|
|
||||||
|
|
||||||
def test_the_same_tree_measures_the_same_twice(tmp_path: Path) -> None:
|
def test_the_same_tree_measures_the_same_twice(tmp_path: Path) -> None:
|
||||||
|
|
@ -662,7 +627,7 @@ def test_the_command_exits_one_and_prints_every_row(
|
||||||
printed = capsys.readouterr().out
|
printed = capsys.readouterr().out
|
||||||
for number in range(1, 9):
|
for number in range(1, 9):
|
||||||
assert f"\n{number} " in f"\n{printed}"
|
assert f"\n{number} " in f"\n{printed}"
|
||||||
assert "GATE RED: rows 5, 8" in printed
|
assert "GATE RED: rows 5, 7, 8" in printed
|
||||||
|
|
||||||
|
|
||||||
def test_the_json_form_carries_the_same_rows(capsys: pytest.CaptureFixture[str]) -> None:
|
def test_the_json_form_carries_the_same_rows(capsys: pytest.CaptureFixture[str]) -> None:
|
||||||
|
|
@ -1034,140 +999,6 @@ def test_the_mutant_roster_is_pinned_apart_from_the_list_it_names() -> None:
|
||||||
assert len(set(gate.MUTANT_ROSTER)) == len(gate.MUTANT_ROSTER)
|
assert len(set(gate.MUTANT_ROSTER)) == len(gate.MUTANT_ROSTER)
|
||||||
|
|
||||||
|
|
||||||
# --- a passage is judged as a passage ------------------------------------------
|
|
||||||
#
|
|
||||||
# Since v1.1 C1 a concept over `consume.PASSAGE_CHARS` is delivered as its
|
|
||||||
# answering passage: the nearest heading above the span, `[...]`, the span,
|
|
||||||
# `[...]`. The row-6 identity (delivered text == the whole body) is older than
|
|
||||||
# that and read every such delivery as class e. The judge must now accept a
|
|
||||||
# passage that IS its exact reconstruction from the bundle's bytes, and nothing
|
|
||||||
# looser: "the span occurs somewhere in the text" accepts an invented sentence
|
|
||||||
# beside it.
|
|
||||||
|
|
||||||
_HOUSE_NOTE = "Styret gjennomgaar notatet og foerer det inn i arkivet. Sekretaeren sender kopi."
|
|
||||||
_INVENTED = "En setning som ikke staar i konseptfila."
|
|
||||||
|
|
||||||
|
|
||||||
def _house_bundle(tmp_path: Path) -> Path:
|
|
||||||
"""One concept of about 6 600 characters: a heading ten lines in, the
|
|
||||||
answer thirty lines below it, and a literal `[...]` on the LAST line, far
|
|
||||||
outside any span centred on the answer."""
|
|
||||||
body = "\n".join(
|
|
||||||
[_HOUSE_NOTE] * 10
|
|
||||||
+ ["## Teknisk rom"]
|
|
||||||
+ [_HOUSE_NOTE] * 30
|
|
||||||
+ ["Varmepumpa i kjelleren faar service av roerleggeren hvert aar."]
|
|
||||||
+ [_HOUSE_NOTE] * 40
|
|
||||||
+ ["Se vedlegg [...] i arkivet."]
|
|
||||||
)
|
|
||||||
spec = gate.BundleSpec(
|
|
||||||
"retrieval-house",
|
|
||||||
(
|
|
||||||
gate.DocumentSpec(
|
|
||||||
"husbok", "husbok.md", (gate.ConceptSpec("drift", "Drift av huset", body),)
|
|
||||||
),
|
|
||||||
*(
|
|
||||||
gate.DocumentSpec(
|
|
||||||
f"skriv-{number}",
|
|
||||||
f"skriv-{number}.md",
|
|
||||||
(gate.ConceptSpec("skriv", f"Skriv {number}", "Skrivet er arkivert."),),
|
|
||||||
)
|
|
||||||
for number in range(3)
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
return gate.build_bundle(tmp_path / "house", spec)
|
|
||||||
|
|
||||||
|
|
||||||
def _house_question(*quotes: str) -> gate.Question:
|
|
||||||
return gate.Question(
|
|
||||||
id="H1",
|
|
||||||
question="Hvem gir varmepumpa i kjelleren service?",
|
|
||||||
fasit=tuple(gate.Fasit("concept", "husbok/drift", quote) for quote in quotes),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _inject(where: str) -> pytest.MonkeyPatch:
|
|
||||||
"""A passage carrying one sentence the concept file does not: at the end of
|
|
||||||
the span, or between the heading's marker line and the span."""
|
|
||||||
original = consume.as_passage
|
|
||||||
|
|
||||||
def mutant(excerpt: dict[str, object], window: int) -> dict[str, object]:
|
|
||||||
out = original(excerpt, window)
|
|
||||||
if "passage" in out:
|
|
||||||
text = str(out["text"])
|
|
||||||
if where == "span":
|
|
||||||
cut = text.rindex(f"\n{consume.PASSAGE_ELISION}")
|
|
||||||
out["text"] = f"{text[:cut]} {_INVENTED}{text[cut:]}"
|
|
||||||
else:
|
|
||||||
marker = f"{consume.PASSAGE_ELISION}\n"
|
|
||||||
cut = text.index(marker) + len(marker)
|
|
||||||
out["text"] = f"{text[:cut]}{_INVENTED}\n{text[cut:]}"
|
|
||||||
return out
|
|
||||||
|
|
||||||
patch = pytest.MonkeyPatch()
|
|
||||||
patch.setattr(consume, "as_passage", mutant)
|
|
||||||
return patch
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_passage_whose_span_carries_the_quote_is_a_hit(tmp_path: Path) -> None:
|
|
||||||
bundle = _house_bundle(tmp_path)
|
|
||||||
gate.bundle_index(bundle)
|
|
||||||
question = _house_question("service av roerleggeren")
|
|
||||||
payload = consume.build_payload(bundle, question=question.question, withheld_full=True)
|
|
||||||
(excerpt,) = [e for e in payload["excerpts"] if e["concept_id"] == "husbok/drift"]
|
|
||||||
passage = excerpt["passage"]
|
|
||||||
assert isinstance(passage, dict), "the premise: this concept is delivered as a passage"
|
|
||||||
start, end = passage["start"], passage["end"]
|
|
||||||
assert 0 < start and end < passage["of"], "the premise: `[...]` on both sides"
|
|
||||||
# FELLE 1, pinned: the offsets count in the DELIVERED body. Read against
|
|
||||||
# the concept file they land a frontmatter's length off.
|
|
||||||
view = gate.bundle_index(bundle).concepts["husbok/drift"]
|
|
||||||
assert view.body[start:end] in str(excerpt["text"])
|
|
||||||
assert view.whole[start:end] not in str(excerpt["text"])
|
|
||||||
(unit,) = gate.measure_units(bundle, question)
|
|
||||||
assert (unit.hit, unit.confirmed, unit.klass) == (True, True, None), unit
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("where", ["span", "between heading and span"])
|
|
||||||
def test_a_passage_carrying_an_invented_sentence_is_not_a_hit(tmp_path: Path, where: str) -> None:
|
|
||||||
"""FELLE 2, pinned: the quote is still in the span and the span is still in
|
|
||||||
the text, so "the span occurs somewhere" would count this. The delivered
|
|
||||||
text must BE the reconstruction."""
|
|
||||||
bundle = _house_bundle(tmp_path)
|
|
||||||
gate.bundle_index(bundle) # the judge reads the bundle BEFORE the payload
|
|
||||||
question = _house_question("service av roerleggeren")
|
|
||||||
patch = _inject(where)
|
|
||||||
try:
|
|
||||||
(unit,) = gate.measure_units(bundle, question)
|
|
||||||
finally:
|
|
||||||
patch.undo()
|
|
||||||
assert unit.hit is False
|
|
||||||
assert unit.confirmed is False
|
|
||||||
assert unit.klass == "e"
|
|
||||||
assert unit.detail == "the delivered passage is not its reconstruction from the bundle's bytes"
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_quote_the_passage_carries_only_outside_its_span_is_not_a_hit(tmp_path: Path) -> None:
|
|
||||||
"""The heading and the markers are delivered and are not the span: a quote
|
|
||||||
found only there was delivered without the citation (class d). The first
|
|
||||||
unit is the known-positive of the same run."""
|
|
||||||
bundle = _house_bundle(tmp_path)
|
|
||||||
units = gate.measure_units(
|
|
||||||
bundle, _house_question("service av roerleggeren", "Teknisk rom", "[...]")
|
|
||||||
)
|
|
||||||
assert [(unit.hit, unit.klass) for unit in units] == [
|
|
||||||
(True, None),
|
|
||||||
(False, "d"),
|
|
||||||
(False, "d"),
|
|
||||||
], units
|
|
||||||
|
|
||||||
|
|
||||||
def test_row_seven_names_passage_cheating() -> None:
|
|
||||||
assert gate.MUTANT_COUNT == 15
|
|
||||||
assert gate.MUTANT_ROSTER[-1] == "M15 the passage carries a sentence the concept file does not"
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_synthetic_corpus_is_pinned_like_the_sets(tmp_path: Path) -> None:
|
def test_the_synthetic_corpus_is_pinned_like_the_sets(tmp_path: Path) -> None:
|
||||||
assert gate.specs_digest() == gate.SPECS_SHA256
|
assert gate.specs_digest() == gate.SPECS_SHA256
|
||||||
tuned = dict(gate.SPECS)
|
tuned = dict(gate.SPECS)
|
||||||
|
|
|
||||||
|
|
@ -434,190 +434,6 @@ BUDGET = BundleSpec(
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
# --- three fixtures, one per mechanism the default ranking runs ---------------
|
|
||||||
#
|
|
||||||
# Until 2026-09-22 row 7 left three mutants standing (M06, M07, M10): every
|
|
||||||
# synthetic concept was short and opened with its own title, so the passage
|
|
||||||
# signal, the title/path weight and the fusion constant never DECIDED a
|
|
||||||
# delivery here, and switching any of them off moved 0 ranks. Each bundle below
|
|
||||||
# is built so that exactly one of them does. None of them is a tuning of the
|
|
||||||
# ranker: the corpus is pinned (`SPECS_SHA256`) and `src/` is untouched.
|
|
||||||
|
|
||||||
_NOTE = "Styret gjennomgaar notatet og foerer det inn i arkivet. Sekretaeren sender kopi. "
|
|
||||||
|
|
||||||
#: THE PASSAGE SIGNAL. One long concept answers in ONE window of its body; ten
|
|
||||||
#: short concepts carry the question's words in their TITLES and none in their
|
|
||||||
#: bodies. The field signal reads ten title-weighted matches in short documents
|
|
||||||
#: above one long body, so only the passage signal -- the best WINDOW, not the
|
|
||||||
#: whole concept -- puts the gold first. Remove the body windows (M06) and the
|
|
||||||
#: passage signal reads titles alone, where the gold has none of the words.
|
|
||||||
PASSAGE = BundleSpec(
|
|
||||||
"retrieval-passage",
|
|
||||||
(
|
|
||||||
DocumentSpec(
|
|
||||||
"aarbok",
|
|
||||||
"aarbok.md",
|
|
||||||
(
|
|
||||||
ConceptSpec(
|
|
||||||
"kapittel-tre",
|
|
||||||
"Kapittel tre i aarboka",
|
|
||||||
_NOTE * 15
|
|
||||||
+ "Varmekablene i taket kontrolleres av vaktmesteren hver oktober. "
|
|
||||||
+ _NOTE * 15,
|
|
||||||
),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
*(
|
|
||||||
DocumentSpec(
|
|
||||||
f"rundskriv-{number:02d}",
|
|
||||||
f"rundskriv-{number:02d}.md",
|
|
||||||
(
|
|
||||||
ConceptSpec(
|
|
||||||
f"skriv-{number:02d}",
|
|
||||||
f"Varmekablene i taket, skriv {number:02d}",
|
|
||||||
"Skrivet er sendt ut og arkivert.",
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
for number in range(1, 11)
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
#: THE TITLE/PATH WEIGHT. The gold is named by its PATH alone -- the question's
|
|
||||||
#: `kanopadling` is its directory and its source file and occurs in no body --
|
|
||||||
#: while ten decoys carry the question's other word more densely than the gold
|
|
||||||
#: does, one fewer time each. The path weight puts the gold first in the field
|
|
||||||
#: signal, which lifts it into k (rank 4 at `k` = 6). Weigh no title and no path
|
|
||||||
#: (M07) and both signals read bodies only, where the decoys outrank it and it
|
|
||||||
#: falls out of k.
|
|
||||||
PATH = BundleSpec(
|
|
||||||
"retrieval-path",
|
|
||||||
(
|
|
||||||
DocumentSpec(
|
|
||||||
"kanopadling",
|
|
||||||
"kanopadling.md",
|
|
||||||
(ConceptSpec("regel", "Regel om vester", "Vester brukes paa vannet."),),
|
|
||||||
),
|
|
||||||
*(
|
|
||||||
DocumentSpec(
|
|
||||||
f"turnotat-{number:02d}",
|
|
||||||
f"turnotat-{number:02d}.md",
|
|
||||||
(
|
|
||||||
ConceptSpec(
|
|
||||||
f"notat-{number:02d}",
|
|
||||||
f"Turnotat {number:02d}",
|
|
||||||
" ".join(["Vester henger i boden."] * (11 - number))
|
|
||||||
+ " Notatet er arkivert.",
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
for number in range(1, 11)
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
_FUSION_NOTE = "Styret gjennomgaar saken og sekretaeren arkiverer den. "
|
|
||||||
|
|
||||||
#: THE FUSION CONSTANT. Reciprocal-rank fusion trades a concept with one very
|
|
||||||
#: good rank against one with two middling ranks, and `RRF_K` sets the price.
|
|
||||||
#: The gold is FIRST in the passage signal and 21st in the field signal (a long
|
|
||||||
#: body, below ten short `isbading-*` path matches, nine longer ones and the
|
|
||||||
#: decoy); the one decoy is 10th and 11th, a smaller rank SUM (19 against 20).
|
|
||||||
#: A flattened fusion (M10, `RRF_K` = 10 000, where only the sum counts) puts
|
|
||||||
#: the decoy first and `k` = 1 no longer reaches the gold. At the shipped
|
|
||||||
#: `RRF_K` = 60 the gold leads, and measured over K in steps of 10 it keeps the
|
|
||||||
#: lead through K = 180 and loses it at 190. The eight long log pages hold the
|
|
||||||
#: decoy's passage rank at 10th and rank below the gold in both signals.
|
|
||||||
FUSION = BundleSpec(
|
|
||||||
"retrieval-fusion",
|
|
||||||
(
|
|
||||||
DocumentSpec(
|
|
||||||
"badebok",
|
|
||||||
"badebok.md",
|
|
||||||
(
|
|
||||||
ConceptSpec(
|
|
||||||
"kapittel",
|
|
||||||
"Kapittel i badeboka",
|
|
||||||
_FUSION_NOTE
|
|
||||||
* 12
|
|
||||||
+ "Isbading ved brygga, isbading om morgenen, isbading med vakt, "
|
|
||||||
"isbading hver dag. " + _FUSION_NOTE * 12,
|
|
||||||
),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
*(
|
|
||||||
DocumentSpec(
|
|
||||||
f"logg-{number:02d}",
|
|
||||||
f"logg-{number:02d}.md",
|
|
||||||
(
|
|
||||||
ConceptSpec(
|
|
||||||
"side",
|
|
||||||
f"Loggside {number:02d}",
|
|
||||||
_FUSION_NOTE * 16
|
|
||||||
+ "Isbading ved brygga, isbading om kvelden, isbading med vakt. "
|
|
||||||
+ _FUSION_NOTE * 16,
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
for number in range(8)
|
|
||||||
),
|
|
||||||
DocumentSpec(
|
|
||||||
"isbading-lapp-00",
|
|
||||||
"isbading-lapp-00.md",
|
|
||||||
(ConceptSpec("lapp", "Lapp 00", _FUSION_NOTE * 5 + "Isbading er tillatt."),),
|
|
||||||
),
|
|
||||||
*(
|
|
||||||
DocumentSpec(
|
|
||||||
f"isbading-{number:02d}",
|
|
||||||
f"isbading-{number:02d}.md",
|
|
||||||
(ConceptSpec("oppslag", f"Oppslag {number:02d}", "Oppslaget er arkivert."),),
|
|
||||||
)
|
|
||||||
for number in range(10)
|
|
||||||
),
|
|
||||||
*(
|
|
||||||
DocumentSpec(
|
|
||||||
f"isbading-arkiv-{number:02d}",
|
|
||||||
f"isbading-arkiv-{number:02d}.md",
|
|
||||||
(ConceptSpec("mappe", f"Mappe {number:02d}", _FUSION_NOTE * 10),),
|
|
||||||
)
|
|
||||||
for number in range(9)
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
_ARCHIVE_LINE = "Styret gjennomgaar notatet og foerer det inn i arkivet. Sekretaeren sender kopi."
|
|
||||||
|
|
||||||
#: THE PASSAGE DELIVERY. Every concept above is shorter than
|
|
||||||
#: `consume.PASSAGE_CHARS`, so no synthetic payload carried a passage and a
|
|
||||||
#: mutant of one (M15) could not be felled. One concept of about 6 600
|
|
||||||
#: characters: a heading ten lines in, the answer thirty lines below it, so the
|
|
||||||
#: delivery is heading + `[...]` + span + `[...]` and the span carries the
|
|
||||||
#: citation. Rows 1 and 6 read it as a hit only because the judge reads a
|
|
||||||
#: passage as its exact reconstruction; M15 adds one sentence and fells both.
|
|
||||||
DELIVERY = BundleSpec(
|
|
||||||
"retrieval-delivery",
|
|
||||||
(
|
|
||||||
DocumentSpec(
|
|
||||||
"husbok",
|
|
||||||
"husbok.md",
|
|
||||||
(
|
|
||||||
ConceptSpec(
|
|
||||||
"drift",
|
|
||||||
"Drift av huset",
|
|
||||||
"\n".join(
|
|
||||||
[_ARCHIVE_LINE] * 10
|
|
||||||
+ ["## Teknisk rom"]
|
|
||||||
+ [_ARCHIVE_LINE] * 30
|
|
||||||
+ ["Varmepumpa i kjelleren faar service av roerleggeren hvert aar."]
|
|
||||||
+ [_ARCHIVE_LINE] * 40
|
|
||||||
),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
SPECS: Mapping[str, BundleSpec] = {
|
SPECS: Mapping[str, BundleSpec] = {
|
||||||
"positive": POSITIVE,
|
"positive": POSITIVE,
|
||||||
"miss": MISS,
|
"miss": MISS,
|
||||||
|
|
@ -625,10 +441,6 @@ SPECS: Mapping[str, BundleSpec] = {
|
||||||
"budget": BUDGET,
|
"budget": BUDGET,
|
||||||
"lookup": LOOKUP,
|
"lookup": LOOKUP,
|
||||||
"quota": QUOTA,
|
"quota": QUOTA,
|
||||||
"passage": PASSAGE,
|
|
||||||
"path": PATH,
|
|
||||||
"fusion": FUSION,
|
|
||||||
"delivery": DELIVERY,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -671,7 +483,7 @@ def specs_digest(specs: Mapping[str, BundleSpec] = SPECS) -> str:
|
||||||
|
|
||||||
|
|
||||||
#: The synthetic corpus, pinned the way the sets are.
|
#: The synthetic corpus, pinned the way the sets are.
|
||||||
SPECS_SHA256 = "089a6a9770c03c59fff2da22430e5f9aeb2b2374c49aed419bae99409a2a5cb9"
|
SPECS_SHA256 = "23c447add9a58ee2e454ef5d5b92d5c3c97d441cd883d203aaf2f275dd2f68f9"
|
||||||
|
|
||||||
|
|
||||||
def synthetic_bundles(root: Path, specs: Mapping[str, BundleSpec] = SPECS) -> dict[str, Path]:
|
def synthetic_bundles(root: Path, specs: Mapping[str, BundleSpec] = SPECS) -> dict[str, Path]:
|
||||||
|
|
@ -836,8 +648,6 @@ SYNTHETIC_SETS: dict[str, str] = {
|
||||||
"set-signals.json": "17d83f305a10af8dd2a45b72943c1285704d23ed074ecb549c05e5fac3e137e9",
|
"set-signals.json": "17d83f305a10af8dd2a45b72943c1285704d23ed074ecb549c05e5fac3e137e9",
|
||||||
"set-quota.json": "61fda652719d7403ddf9701d50e914c46572a4fda77dc0336838695e18c498cf",
|
"set-quota.json": "61fda652719d7403ddf9701d50e914c46572a4fda77dc0336838695e18c498cf",
|
||||||
"set-controls.json": "c2894656326e5a69ec7063fdc280e763910d20b4cbc2c0124f639283864ce506",
|
"set-controls.json": "c2894656326e5a69ec7063fdc280e763910d20b4cbc2c0124f639283864ce506",
|
||||||
"set-mechanisms.json": "435ce620c30c94b0d151335883583520490482be2ade6797524ae928a9f1eab7",
|
|
||||||
"set-passage.json": "93b6de8ef7012c84516fe9aee560924d4b579669fb046e6eadc70dbf444eb627",
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -1001,79 +811,6 @@ def _carries(text: str, quote: str) -> bool:
|
||||||
return not quote or _flat(quote) in _flat(text)
|
return not quote or _flat(quote) in _flat(text)
|
||||||
|
|
||||||
|
|
||||||
#: What class e says when a passage is not its own reconstruction.
|
|
||||||
PASSAGE_NOT_RECONSTRUCTED = (
|
|
||||||
"the delivered passage is not its reconstruction from the bundle's bytes"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def passage_span(text: str, passage: object, body: str) -> str | None:
|
|
||||||
"""The span a passage delivery carries, or None when `text` is not EXACTLY
|
|
||||||
what the bundle's bytes rebuild.
|
|
||||||
|
|
||||||
Since v1.1 C1 a long concept is delivered as `[heading]`, `[...]`, the
|
|
||||||
span, `[...]` (`consume.as_passage`), with `passage = {start, end, of}`.
|
|
||||||
Two traps PM measured 2026-09-22, both held here:
|
|
||||||
|
|
||||||
- **The offsets count in the DELIVERED body** (`ConceptView.body`), never
|
|
||||||
in the concept file: read in the file they land a frontmatter's length
|
|
||||||
off, and 0 of 12 real spans matched.
|
|
||||||
- **"The span occurs somewhere in the text" is not a check.** It accepts
|
|
||||||
an invented sentence beside the span. Every character of `text` must be
|
|
||||||
accounted for: the span is `body[start:end]` byte for byte, a marker
|
|
||||||
stands exactly where the span leaves text out and nowhere else, and the
|
|
||||||
one other line allowed is a heading line of the body ABOVE the span (or a
|
|
||||||
prefix of one -- `as_passage` cuts a long heading). WHICH heading is the
|
|
||||||
product's choice and is not re-derived here; that it is the bundle's
|
|
||||||
bytes is the guarantee.
|
|
||||||
|
|
||||||
The citation is then read in the span alone -- not in the heading, not in
|
|
||||||
the markers, and not across the seam between them, which is no sequence
|
|
||||||
the concept file holds.
|
|
||||||
"""
|
|
||||||
if not isinstance(passage, Mapping):
|
|
||||||
return None
|
|
||||||
start, end, of = passage.get("start"), passage.get("end"), passage.get("of")
|
|
||||||
if not all(
|
|
||||||
isinstance(value, int) and not isinstance(value, bool) for value in (start, end, of)
|
|
||||||
):
|
|
||||||
return None
|
|
||||||
assert isinstance(start, int) and isinstance(end, int)
|
|
||||||
if of != len(body) or not 0 <= start < end <= len(body):
|
|
||||||
return None
|
|
||||||
span = body[start:end]
|
|
||||||
tail = f"\n{consume.PASSAGE_ELISION}" if end < len(body) else ""
|
|
||||||
if not text.endswith(span + tail):
|
|
||||||
return None
|
|
||||||
head = text[: len(text) - len(span + tail)]
|
|
||||||
if start == 0:
|
|
||||||
return span if head == "" else None
|
|
||||||
marker = f"{consume.PASSAGE_ELISION}\n"
|
|
||||||
if head == marker:
|
|
||||||
return span
|
|
||||||
if not head.endswith(f"\n{marker}"):
|
|
||||||
return None
|
|
||||||
heading = head[: -len(f"\n{marker}")]
|
|
||||||
above = body[:start].split("\n")
|
|
||||||
if heading.startswith("#") and any(
|
|
||||||
line.startswith("#") and line.startswith(heading) for line in above
|
|
||||||
):
|
|
||||||
return span
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _judged(excerpt: Mapping[str, object], body: str) -> tuple[str, bool, str]:
|
|
||||||
"""(the text a citation is looked for in, whether it is the bundle's
|
|
||||||
bytes, the detail when it is not) for one delivered excerpt."""
|
|
||||||
text = str(excerpt.get("text", ""))
|
|
||||||
if "passage" in excerpt:
|
|
||||||
span = passage_span(text, excerpt["passage"], body)
|
|
||||||
if span is None:
|
|
||||||
return text, False, PASSAGE_NOT_RECONSTRUCTED
|
|
||||||
return span, True, ""
|
|
||||||
return text, _flat(text) == _flat(body), "the delivered text is not the bundle's bytes"
|
|
||||||
|
|
||||||
|
|
||||||
def measure_units(bundle: Path, question: Question) -> list[Unit]:
|
def measure_units(bundle: Path, question: Question) -> list[Unit]:
|
||||||
"""One question, measured at the shipped defaults, plus the quota-off run
|
"""One question, measured at the shipped defaults, plus the quota-off run
|
||||||
that says what the truth of a withheld concept is.
|
that says what the truth of a withheld concept is.
|
||||||
|
|
@ -1120,21 +857,18 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
|
||||||
for concept_id in holding
|
for concept_id in holding
|
||||||
if _carries(index.concepts[concept_id].body, fasit.quote)
|
if _carries(index.concepts[concept_id].body, fasit.quote)
|
||||||
)
|
)
|
||||||
# A whole-body delivery is judged whole; a passage is judged as its
|
hit_ids = [
|
||||||
# exact reconstruction, and its citation is read in its span alone.
|
concept_id
|
||||||
judged = {
|
|
||||||
concept_id: _judged(delivered[concept_id], index.concepts[concept_id].body)
|
|
||||||
for concept_id in holding
|
for concept_id in holding
|
||||||
if concept_id in delivered
|
if concept_id in delivered
|
||||||
}
|
and _carries(str(delivered[concept_id].get("text", "")), fasit.quote)
|
||||||
hit_ids = [
|
|
||||||
concept_id for concept_id, (text, _, _) in judged.items() if _carries(text, fasit.quote)
|
|
||||||
]
|
]
|
||||||
# The payload SAYS it delivered this; the bundle says what it is.
|
# The payload SAYS it delivered this; the bundle says what it is.
|
||||||
confirmed: bool | None = None
|
confirmed: bool | None = None
|
||||||
not_bytes = ""
|
|
||||||
for concept_id in hit_ids:
|
for concept_id in hit_ids:
|
||||||
_, confirmed, not_bytes = judged[concept_id]
|
confirmed = _flat(str(delivered[concept_id].get("text", ""))) == _flat(
|
||||||
|
index.concepts[concept_id].body
|
||||||
|
)
|
||||||
if confirmed:
|
if confirmed:
|
||||||
break
|
break
|
||||||
# The concept this entry is really about: the one the bundle holds the
|
# The concept this entry is really about: the one the bundle holds the
|
||||||
|
|
@ -1164,7 +898,7 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
|
||||||
else "no concept in the bundle answers to this name"
|
else "no concept in the bundle answers to this name"
|
||||||
)
|
)
|
||||||
elif hit_ids and not confirmed:
|
elif hit_ids and not confirmed:
|
||||||
klass, detail = "e", not_bytes
|
klass, detail = "e", "the delivered text is not the bundle's bytes"
|
||||||
elif target in delivered:
|
elif target in delivered:
|
||||||
klass = "d"
|
klass = "d"
|
||||||
detail = "delivered without the citation" + (
|
detail = "delivered without the citation" + (
|
||||||
|
|
@ -1865,10 +1599,8 @@ class Mutant:
|
||||||
expected_row: int
|
expected_row: int
|
||||||
patch: Callable[[], contextlib.AbstractContextManager[None]]
|
patch: Callable[[], contextlib.AbstractContextManager[None]]
|
||||||
#: Why this one may be invisible to a gate that judges DELIVERY. Written
|
#: Why this one may be invisible to a gate that judges DELIVERY. Written
|
||||||
#: for a mutant that survives, so a survivor is a statement about the
|
#: for the two that survive, so a survivor is a statement about the
|
||||||
#: mechanism and not a shrug. None does since 2026-09-22: the three that
|
#: mechanism and not a shrug.
|
||||||
#: carried a note (M06, M07, M10) each got a fixture that makes their
|
|
||||||
#: mechanism decide a delivery (`PASSAGE`, `PATH`, `FUSION`).
|
|
||||||
note: str = ""
|
note: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -1955,25 +1687,6 @@ def _extend_delivered() -> contextlib.AbstractContextManager[None]:
|
||||||
return _patched(delivered_text=mutant)
|
return _patched(delivered_text=mutant)
|
||||||
|
|
||||||
|
|
||||||
def _extend_passage() -> contextlib.AbstractContextManager[None]:
|
|
||||||
"""M14's shape for the delivery form v1.1 C1 added: the passage still
|
|
||||||
CARRIES the citation in its span, and one sentence at the span's end is not
|
|
||||||
the concept file's. A judge that asked only whether the span occurs
|
|
||||||
somewhere in the text would count it."""
|
|
||||||
original = consume.as_passage
|
|
||||||
|
|
||||||
def mutant(excerpt: dict[str, object], window: int) -> dict[str, object]:
|
|
||||||
out = original(excerpt, window)
|
|
||||||
if "passage" in out:
|
|
||||||
text = str(out["text"])
|
|
||||||
tail = f"\n{consume.PASSAGE_ELISION}"
|
|
||||||
cut = len(text) - len(tail) if text.endswith(tail) else len(text)
|
|
||||||
out["text"] = f"{text[:cut]} En setning som ikke staar i konseptfila.{text[cut:]}"
|
|
||||||
return out
|
|
||||||
|
|
||||||
return _patched(as_passage=mutant)
|
|
||||||
|
|
||||||
|
|
||||||
def _drop_text_key() -> contextlib.AbstractContextManager[None]:
|
def _drop_text_key() -> contextlib.AbstractContextManager[None]:
|
||||||
original = consume.excerpt_for
|
original = consume.excerpt_for
|
||||||
|
|
||||||
|
|
@ -2004,12 +1717,23 @@ MUTANTS: tuple[Mutant, ...] = (
|
||||||
Mutant("M04 the ranking is reversed", 1, _reverse_scores),
|
Mutant("M04 the ranking is reversed", 1, _reverse_scores),
|
||||||
Mutant("M05 the source quota is removed", 3, lambda: _wrap_cut(source_quota=None)),
|
Mutant("M05 the source quota is removed", 3, lambda: _wrap_cut(source_quota=None)),
|
||||||
Mutant(
|
Mutant(
|
||||||
"M06 the passage signal reads no body", 1, lambda: _patched_bm25(windows=lambda body: [])
|
"M06 the passage signal reads no body",
|
||||||
|
1,
|
||||||
|
lambda: _patched_bm25(windows=lambda body: []),
|
||||||
|
note=(
|
||||||
|
"the field signal still reads every body, and no synthetic concept is "
|
||||||
|
"long enough for its best window to decide a delivery"
|
||||||
|
),
|
||||||
),
|
),
|
||||||
Mutant(
|
Mutant(
|
||||||
"M07 the field signal weighs no title and no path",
|
"M07 the field signal weighs no title and no path",
|
||||||
1,
|
1,
|
||||||
lambda: _patched_bm25(TITLE_WEIGHT=0, PATH_WEIGHT=0),
|
lambda: _patched_bm25(TITLE_WEIGHT=0, PATH_WEIGHT=0),
|
||||||
|
note=(
|
||||||
|
"every synthetic body opens with its own title as a heading "
|
||||||
|
"(`build_bundle`), so the weight only re-weights words the body "
|
||||||
|
"already carries"
|
||||||
|
),
|
||||||
),
|
),
|
||||||
Mutant("M08 every token matches every token", 4, _every_term_everything),
|
Mutant("M08 every token matches every token", 4, _every_term_everything),
|
||||||
Mutant("M09 no stem, no prefix: equality only", 1, _equality_only),
|
Mutant("M09 no stem, no prefix: equality only", 1, _equality_only),
|
||||||
|
|
@ -2017,6 +1741,10 @@ MUTANTS: tuple[Mutant, ...] = (
|
||||||
"M10 the fusion is flattened (bm25.RRF_K = 10 000)",
|
"M10 the fusion is flattened (bm25.RRF_K = 10 000)",
|
||||||
1,
|
1,
|
||||||
lambda: _patched_bm25(RRF_K=10_000),
|
lambda: _patched_bm25(RRF_K=10_000),
|
||||||
|
note=(
|
||||||
|
"over two signals K reorders only a concept whose two ranks lie far "
|
||||||
|
"apart against one whose ranks are close, and no fixture here holds that"
|
||||||
|
),
|
||||||
),
|
),
|
||||||
Mutant("M11 the cut takes the LAST k", 1, _last_k),
|
Mutant("M11 the cut takes the LAST k", 1, _last_k),
|
||||||
Mutant("M12 the delivered text is truncated to 40 characters", 6, _truncate_delivered),
|
Mutant("M12 the delivered text is truncated to 40 characters", 6, _truncate_delivered),
|
||||||
|
|
@ -2024,7 +1752,6 @@ MUTANTS: tuple[Mutant, ...] = (
|
||||||
Mutant(
|
Mutant(
|
||||||
"M14 the delivered text carries a sentence the concept file does not", 6, _extend_delivered
|
"M14 the delivered text carries a sentence the concept file does not", 6, _extend_delivered
|
||||||
),
|
),
|
||||||
Mutant("M15 the passage carries a sentence the concept file does not", 6, _extend_passage),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -2051,11 +1778,10 @@ MUTANT_ROSTER: tuple[str, ...] = (
|
||||||
"M12 the delivered text is truncated to 40 characters",
|
"M12 the delivered text is truncated to 40 characters",
|
||||||
"M13 the excerpt carries no text",
|
"M13 the excerpt carries no text",
|
||||||
"M14 the delivered text carries a sentence the concept file does not",
|
"M14 the delivered text carries a sentence the concept file does not",
|
||||||
"M15 the passage carries a sentence the concept file does not",
|
|
||||||
)
|
)
|
||||||
|
|
||||||
#: The roster's length, written as a number so appending is not one edit.
|
#: The roster's length, written as a number so appending is not one edit.
|
||||||
MUTANT_COUNT = 15
|
MUTANT_COUNT = 14
|
||||||
|
|
||||||
|
|
||||||
def _score(rows: Sequence[Row]) -> dict[int, int]:
|
def _score(rows: Sequence[Row]) -> dict[int, int]:
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue