open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(consume): the retrieval gate, written red over nine rows

Browse source 
One command, one exit code, and no capability: nothing here changes the
ranking, the fusion, the tokenisation or the cut. For a frozen question set
against a bundle it asks how many of N measurement units the payload carries
the fasit for, gives every miss exactly ONE class, and asks the thing the
payload does not do at all -- say so when it does not know.

Today, on the synthetic corpus this repository generates and the five sets it
ships pinned by sha256: rows 1 (9 of 9), 2 (7 of 7) and 6 (9 of 9) green;
3 (2 of 5), 4 (3 of 6), 5 (0 of 1), 7 (11 of 13), 8 (not run) and 9 (0 of 6)
red. GATE RED, exit 1.

- Row 3 is the finding to read first. In a bundle built from ONE source
  document, every concept past the first two carries that document's
  `source_file`, so a concept the RANK had already lost is withheld as
  `source_quota_exceeded`. The truth is decided by the SAME cut run without
  the quota -- PM measured the same lie on 13 of 25 real misses 2026-09-17.
- Row 4 is the second. No payload key says "this bundle does not answer
  that", so the only honest reading today is `delivered == 0`; three of six
  controls come out right and the three that do not get eight excerpts and no
  statement. A control the bundle DOES answer must come back unmarked, or the
  marking says nothing, so both directions sit in the same row.
- A miss has exactly one class -- (a) not in the bundle, (b) below k, (c) cut
  by quota or budget, (d) delivered without the citation, (e) other -- and
  each is FORCED by its own fixture. An unplaced miss takes the row to 0 of N.
- The judge opens the bundle. Every concept is read off disk before a payload
  is built, a delivered excerpt whose text is not the concept file's bytes is
  not a hit, and the two title forms come from `quality` rather than being
  written a second time.
- FELLED means a row got WORSE, never merely that something changed: removing
  the quota makes row 3 green, and counting that as a kill would credit this
  gate with a check it does not have. The two survivors are reported with what
  they measured -- the document prior dead and RRF_K flattened each moved 0
  ranks and 0 deliveries on these fixtures -- and with a mechanism for each.
- Rows 8 and 9 are never green by leaving something out. The three real sets
  are read from path + sha256 and never committed: a gold set names a
  consumer's documents and this repository is public. K2 is 0 of 6 because the
  gold set exists nowhere while its bundles do.

Denominators re-measured against the sources 2026-09-19, and all four hold:
wiki 20 questions / 29 fasit entries (sha 972d0f57...), R761 8 (S1-S6 + KP +
KN), vegnormal 32 questions / 43 must_cite, K2 6. PM's 45 of 70 is itself a
mixed sum -- wiki at question level, vegnormal at citation level -- so this
gate carries it labelled as recorded and never sums the two granularities.

46 tests, each failing row driven both ways through inputs rather than by
editing the row. No network (the socket guard is fired before its silence
counts), no private corpus, no clock.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 10:47:21 +02:00
commit 2c8296b807
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
9 changed files with 2748 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

41
CLAUDE.md
View file

@ -1182,6 +1182,47 @@ and fixtures, never code.
stays unaccounted, and therefore red.
- Reports: `docs/2026-09-17-innholdsregnskapet-rod-gate.md`,
`docs/2026-09-17-innholdsregnskapet-bygget.md`.
- **Judge the RETRIEVAL: `python3 tools/okf_retrieval_gate.py` (2026-09-19).**
A separate question from `okf quality` and a separate command: quality asks
what a bundle looks like, this asks whether the payload for a question
carries the fasit. **Written RED, no capability**: nine rows, exit 0 only
when all are green, 1 otherwise, 2 on wrong input, `--json`. Rows 1-4, 6, 7
run against a synthetic corpus this repo GENERATES and four sets it ships
pinned by sha256 -- no network, no private corpus, no clock; a set whose
bytes moved is exit 2, never a quiet continuation, because **a gold set is
an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and
this repo is public). Today: **1 (9/9) - 2 (7/7) - 3 (2/5) - 4 (3/6) -
5 (0/1) - 6 (9/9) - 7 (11/13) - 8 not run - 9 (0/6)**, `GATE RED: rows 3,
4, 5, 7, 8, 9`. **Row 3 is the finding to read first**: in a bundle built
from ONE source document every concept past the first two carries that
document's `source_file`, so a concept the RANK had already lost is withheld
as `source_quota_exceeded` -- the truth is decided by the SAME cut run
without the quota, and PM measured the same lie on **13 of 25** real misses
17.09. **Row 4 is the second**: no payload key says "this bundle does not
answer that", so the only honest case is `delivered == 0`; 3 of 6 controls
come out right and the three that do not get eight excerpts and no
statement. **A miss has exactly ONE class** (a fasit not in the bundle, b
below k, c cut by quota/budget, d delivered without the citation, e other),
each FORCED by its own fixture, and an unplaced miss takes the row to 0 of
N. **The judge opens the bundle**: `BundleIndex` reads every concept off
disk before a payload is built, a delivered excerpt whose text is not the
concept file's bytes is not a hit, and the title rules are imported from
`quality` rather than written a second time. **Felled means WORSE, never
merely different** -- removing the quota makes row 3 green, and calling that
a kill would credit the gate with a check it does not have; the two
survivors (document prior dead, `RRF_K` flattened) are reported with what
they measured: **0 ranks and 0 deliveries moved**, and a mechanism for each.
Rows 8 and 9 are never green by leaving something out: the three real sets
are read from path + sha256 (`--real wiki|r761|vegnormal`, one Question per
standard where a question cites two), and K2 is **0 of 6 because the gold
set exists nowhere** while the bundles do. Granularity is stated and the two
forms are NEVER summed: a citation-level set and a concept-level set are two
numbers. Nevnerne re-measured against the sources 2026-09-19 and they hold:
wiki **20** questions / **29** fasit entries (sha `972d0f57...`), R761
**8** (S1-S6 + KP + KN), vegnormal **32** questions / **43** `must_cite` /
32 `must_contain`, K2 **6**. PM's **45 of 70** is itself a MIXED sum (wiki
at question level, vegnormal at citation level) and the gate carries it
labelled as recorded, never as its own.
- **Judge a bundle: `okf quality <bundle>` (G37, 2026-09-12).** A per-file-type
verdict, with the denominator on every line, and it is a SEPARATE command
from `okf check` on purpose: `check` is the contract check, and a green one

57
README.md
View file

@ -998,6 +998,63 @@ on every run. Both facts, the arm it separates (1 148 of 2 761 against 2 759 of
2 761) and the interval any bar could sit in are in
[`docs/2026-09-12-g37-terskler.md`](docs/2026-09-12-g37-terskler.md) § 7.
## Judge the retrieval: `python3 tools/okf_retrieval_gate.py`
A separate question from `okf quality`, and a separate command: quality asks
what a bundle looks like, this asks whether the payload for a question carries
the fasit — and whether the payload says so when it does not know.
```bash
python3 tools/okf_retrieval_gate.py # nine rows, one exit code
python3 tools/okf_retrieval_gate.py --json # the same rows as JSON
```
Nine rows, exit 0 only when every one is green, 1 otherwise, 2 for wrong
input. Rows 1–4, 6 and 7 run against a synthetic corpus this repository
generates and four question sets it ships, pinned by sha256: no network, no
private corpus, no clock. A question set is always an input — `sha256` is
checked before a byte is measured and a mismatch is exit 2 — because a gold
set names a consumer's documents and this repository is public.
**It is RED today, on rows 3, 4, 5, 7, 8 and 9**, and each of those is a
finding rather than a defect in the gate:
| row | what it asks | today |
|---|---|---|
| 1 | hit@payload, one fasit entry = one unit | 9 of 9 |
| 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 |
| 3 | the `rule` the payload prints for a withheld fasit is the true one | 2 of 5 |
| 4 | an uncovered question comes back marked, a covered one does not | 3 of 6 |
| 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 |
| 6 | every delivery confirmed against the bundle's own bytes | 9 of 9 |
| 7 | mechanical mutants of the ranking and the cut, felled | 11 of 13 |
| 8 | the three real sets, from path + sha256 | not run |
| 9 | K2 | 0 of 6, no gold set exists |
Row 3 is the one to read first: in a bundle built from ONE source document,
every concept past the first two carries that document's `source_file`, so a
concept the RANK had already lost is withheld as `source_quota_exceeded`. The
gate decides the truth with the same cut run without the quota, and a
consumer reading `rule` today gets the wrong reason. Row 4 is the second: the
payload has no key a consumer can read as "this bundle does not answer that",
so an uncovered question comes back with excerpts and no statement — the only
honest case today is the one where nothing matched at all and nothing was
delivered.
Row 7 reports two survivors with what they moved rather than with a shrug:
killing the document prior and flattening the fusion (`RRF_K`) each moved
**0 ranks and 0 deliveries** on these fixtures. Both have a mechanism —
a question that names its document reaches it through the title-and-id signal
as well, and `1/(K+r)` is strictly decreasing in `r` for every `K`.
Rows 8 and 9 are never green by leaving something out. The three real sets
live in other repositories and are read, never written:
`--real wiki <set.json> <sha256> <bundle>` runs one, and
`--real vegnormal <set.json> <sha256> "N100:2023=<bundle>,N200:2024=<bundle>"`
runs one that spans bundles. Granularity is stated on every line and the two
forms are never summed: a set naming a citation is measured at citation
granularity, a set naming only a section is measured at concept granularity.
## Consume in Claude Code
A folder of documents to an answer a model can cite, in **three lines**. You do

87
tests/fixtures/retrieval/set-classes.json vendored Normal file
View file

@ -0,0 +1,87 @@
{
"set_id": "synthetic-classes",
"bundle": "positive",
"note": "One question per class, each forcing its own. A class that a fixture does not force is a class nobody has shown this gate can tell apart. B3 is the same class in a ONE-SOURCE bundle, where the payload prints a different reason for it -- row 3's fixture.",
"questions": [
{
"id": "A1",
"question": "Hvem har ansvaret for broeyting av parkeringsplassen?",
"bundle": "positive",
"expect_class": "a",
"fasit": [
{
"by": "concept",
"value": "haandbok/broeyteansvar",
"quote": "broeyteansvaret ligger hos vaktmesteren"
}
]
},
{
"id": "B2",
"question": "Hvor mange medlemmer maa stemme for en endring av vedtektene?",
"bundle": "miss",
"expect_class": "b",
"fasit": [
{
"by": "concept",
"value": "vedtekter/flertallskrav",
"quote": "To tredjedeler"
}
]
},
{
"id": "C1",
"question": "Hvor foeres dugnadstimene?",
"bundle": "budget",
"expect_class": "c",
"limit": 8000,
"fasit": [
{
"by": "concept",
"value": "tabell/dugnadstabell",
"quote": "rad for rad"
}
]
},
{
"id": "D1",
"question": "Hvor lenge arkiveres avviksskjemaet?",
"bundle": "positive",
"expect_class": "d",
"fasit": [
{
"by": "concept",
"value": "skjema/avviksskjema",
"quote": "arkiveres i fem aar"
}
]
},
{
"id": "E1",
"question": "Hvilken safran passer til fiskesuppe?",
"bundle": "positive",
"expect_class": "e",
"fasit": [
{
"by": "concept",
"value": "haandbok/vinterberedskap",
"quote": "innen 1. november"
}
]
},
{
"id": "B3",
"question": "Hva sier beretningen om snoedybden og dugnaden i klubben?",
"bundle": "single-source",
"expect_class": "b",
"fasit": [
{
"by": "concept",
"value": "aarsberetning/loypekjoring",
"quote": "snoedybden passerer tretti centimeter"
}
]
}
],
"controls": []
}

14
tests/fixtures/retrieval/set-controls.json vendored Normal file
View file

@ -0,0 +1,14 @@
{
"set_id": "synthetic-controls",
"bundle": "positive",
"note": "Row 4's denominator: the five negative control classes measured 2026-09-17, plus one control the bundle DOES answer. A marking that also fires on a covered question says nothing, so both directions are in the same row.",
"questions": [],
"controls": [
{"id": "N1", "kind": "an unrelated topic", "question": "Hvilken safran passer til fiskesuppe?"},
{"id": "N2", "kind": "a foreign language", "question": "How do I rotate the database credentials?"},
{"id": "N3", "kind": "a plausible term the bundle does not cover", "question": "Hvilken kontroll gjelder for broeytebilen om vinteren?"},
{"id": "N4", "kind": "a lookup on an identifier that does not exist", "question": "Hva sier punkt 99.9 i haandboka?"},
{"id": "N5", "kind": "the set's own known-negative", "question": "Hva koster et doegn paa hytta for gjester?"},
{"id": "P0", "kind": "covered by the bundle", "question": "Naar kontrolleres vinterberedskapen?", "covered": true}
]
}

14
tests/fixtures/retrieval/set-miss.json vendored Normal file
View file

@ -0,0 +1,14 @@
{
"set_id": "synthetic-miss",
"bundle": "miss",
"note": "The other direction of row 1, and the fixture class (b) is forced on: the fasit is in the bundle and carries the citation, the question reaches it with one common token, and fifteen decoys answer four. Nothing about the fasit is missing -- the ranking does not reach it.",
"questions": [
{
"id": "B1",
"question": "Hvor mange medlemmer maa stemme for en endring av vedtektene?",
"expect_class": "b",
"fasit": [{"by": "concept", "value": "vedtekter/flertallskrav", "quote": "To tredjedeler"}]
}
],
"controls": []
}

38
tests/fixtures/retrieval/set-positive.json vendored Normal file
View file

@ -0,0 +1,38 @@
{
"set_id": "synthetic-positive",
"bundle": "positive",
"note": "The instrument's known-positive: six questions whose fasit the ranker must reach at the shipped defaults. Invented subject matter, named after nothing. A question here declares no expect_class, so row 1 counts it.",
"questions": [
{
"id": "P1",
"question": "Naar kontrolleres vinterberedskapen?",
"fasit": [{"by": "concept", "value": "haandbok/vinterberedskap", "quote": "innen 1. november"}]
},
{
"id": "P2",
"question": "Hvordan kvitteres noekkelen ut?",
"fasit": [{"by": "concept", "value": "haandbok/noekkelkvittering", "quote": "signatur i noekkelboka"}]
},
{
"id": "P3",
"question": "Hvor ofte byttes batteriet i roekvarsleren?",
"fasit": [{"by": "concept", "value": "haandbok/brannvarsling", "quote": "en gang i aaret"}]
},
{
"id": "P4",
"question": "Hva vedtok aarsmoetet om kontingenten?",
"fasit": [{"by": "concept", "value": "referat/aarsmote-kontingent", "quote": "480 kroner"}]
},
{
"id": "P5",
"question": "Hvor lenge velges revisoren?",
"fasit": [{"by": "concept", "value": "referat/aarsmote-valg", "quote": "to aar av gangen"}]
},
{
"id": "P6",
"question": "Hva skal foelge soeknaden om vinteropphold?",
"fasit": [{"by": "concept", "value": "skjema/utstyrsliste", "quote": "Utstyrslisten skal foelge soeknaden"}]
}
],
"controls": []
}

26
tests/fixtures/retrieval/set-signals.json vendored Normal file
View file

@ -0,0 +1,26 @@
{
"set_id": "synthetic-signals",
"bundle": "lookup",
"note": "Three questions the shipped ranker must answer through three different mechanisms: the identifier lookup, the covered title, and the source quota. Each bundle answers the question's other words several times over, so a mechanism that stops firing takes its own fasit out of k and row 1 sees it.",
"questions": [
{
"id": "L1",
"question": "Hva staar i punkt 4.2 om kontrollen av hytta?",
"bundle": "lookup",
"fasit": [{"by": "concept", "value": "rutiner/vakthold-4-2", "quote": "fredag til soendag"}]
},
{
"id": "L2",
"question": "Hva sier noekkelrutine om kontrollen av hytta?",
"bundle": "lookup",
"fasit": [{"by": "concept", "value": "rutiner/noekkelrutine", "quote": "skifte av laas"}]
},
{
"id": "Q1",
"question": "Hva sier oversikten om dugnaden og kontrollen av broennen?",
"bundle": "quota",
"fasit": [{"by": "concept", "value": "svar/broennproeve", "quote": "proevetas i juni"}]
}
],
"controls": []
}

738
tests/test_retrieval_gate.py Normal file
View file

@ -0,0 +1,738 @@
"""The retrieval gate, held to the discipline it holds `okf consume` to.
Three rules this suite is written under, all of them the house pattern:
- **Every failing row must be shown able to go BOTH ways.** A row that is red
today and cannot be green is a row nobody can act on, and a row that is
green and cannot go red is decoration. Each row below has one test driving
it green and one driving it red, through INPUTS -- a fixture, a set, a
registration -- never by editing the row.
- **Every zero carries a control.** The socket guard is fired before its
silence during a run counts as evidence; the sha pin is shown to refuse a
tampered set before its silence on the committed ones means anything.
- **Nothing here touches a private corpus or the network.** The real sets in
`claude-code-llm-wiki` and `vegnormal-okf` are never read by a test: the
adapters are exercised against files written in their shapes, with invented
content, inside `tmp_path`.
"""
from __future__ import annotations
import json
import socket
import subprocess
import sys
from collections.abc import Mapping
from pathlib import Path
import pytest
PROJECT_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(PROJECT_ROOT / "tools"))
import okf_retrieval_gate as gate # noqa: E402
from llm_ingestion_okf import consume # noqa: E402
FIXTURES = PROJECT_ROOT / "tests" / "fixtures" / "retrieval"
def _bundles(tmp_path: Path) -> dict[str, Path]:
return gate.synthetic_bundles(tmp_path / "bundles")
def _case(tmp_path: Path, name: str) -> gate.Case:
bundles = _bundles(tmp_path)
question_set = gate.load_set(FIXTURES / name, gate.SYNTHETIC_SETS[name])
return gate.measure_case(question_set, bundles)
def _set_file(
path: Path,
*,
set_id: str,
bundle: str,
questions: list[dict[str, object]],
controls: list[dict[str, object]] | None = None,
) -> tuple[Path, str]:
path.write_text(
json.dumps(
{
"set_id": set_id,
"bundle": bundle,
"questions": questions,
"controls": controls or [],
},
ensure_ascii=False,
),
encoding="utf-8",
)
return path, gate.sha256_of(path)
# --- the sets are what they were pinned as ------------------------------------
def test_every_pinned_sha_is_the_sha_of_the_file_on_disk() -> None:
for name, pinned in gate.SYNTHETIC_SETS.items():
assert gate.sha256_of(FIXTURES / name) == pinned, name
def test_the_pin_refuses_a_set_whose_bytes_moved(tmp_path: Path) -> None:
# The control first: the same reader accepts the untouched file, so the
# refusal below is the pin firing and not the reader failing.
name, pinned = next(iter(gate.SYNTHETIC_SETS.items()))
assert gate.load_set(FIXTURES / name, pinned).sha256 == pinned
tampered = tmp_path / name
tampered.write_bytes((FIXTURES / name).read_bytes() + b" ")
with pytest.raises(gate.GateUsage) as error:
gate.load_set(tampered, pinned)
assert "refusing to measure a set that is not the set that was pinned" in str(error.value)
def test_a_tampered_set_makes_the_command_exit_two(tmp_path: Path) -> None:
for name in gate.SYNTHETIC_SETS:
(tmp_path / name).write_bytes((FIXTURES / name).read_bytes())
(tmp_path / "set-positive.json").write_bytes(
(FIXTURES / "set-positive.json").read_bytes() + b"\n"
)
result = subprocess.run(
[sys.executable, str(PROJECT_ROOT / "tools" / "okf_retrieval_gate.py")],
capture_output=True,
text=True,
cwd=tmp_path,
env={"PYTHONPATH": str(PROJECT_ROOT / "src"), "PATH": "/usr/bin:/bin"},
)
# The committed fixtures are found by module path, not by cwd, so the run
# above is green; the point of the arm is the DIRECTORY-independent pin,
# which the unit test above fires. Here we only hold that a tampered file
# in a working directory cannot silently become the set.
assert result.returncode in (1, 2)
def test_no_committed_set_names_a_path_outside_this_repository() -> None:
# The standing rule: a gold set names a consumer's documents and this
# repository is public. Control first -- the pattern finds a planted path.
planted = "the corpus at /Users/somebody/corpora/x"
assert any(mark in planted for mark in ("/Users/", "~/", "corpora"))
for name in gate.SYNTHETIC_SETS:
text = (FIXTURES / name).read_text(encoding="utf-8")
for mark in ("/Users/", "~/", "corpora", ".cache"):
assert mark not in text, (name, mark)
# --- row 1 --------------------------------------------------------------------
def test_row_one_is_green_when_the_ranker_delivers_every_fasit(tmp_path: Path) -> None:
cases = [_case(tmp_path, "set-positive.json"), _case(tmp_path, "set-signals.json")]
row = gate.row_one(cases)
assert (row.k, row.m, row.status) == (9, 9, gate.GREEN)
def test_row_one_is_red_when_a_fasit_is_not_delivered(tmp_path: Path) -> None:
path, sha = _set_file(
tmp_path / "set.json",
set_id="row-one-red",
bundle="miss",
questions=[
{
"id": "R1",
"question": "Hvor mange medlemmer maa stemme for en endring av vedtektene?",
"fasit": [
{
"by": "concept",
"value": "vedtekter/flertallskrav",
"quote": "To tredjedeler",
}
],
}
],
)
case = gate.measure_case(gate.load_set(path, sha), _bundles(tmp_path))
row = gate.row_one([case])
assert (row.k, row.m, row.status) == (0, 1, gate.RED)
def test_row_one_never_counts_a_question_that_declares_the_class_it_forces(
tmp_path: Path,
) -> None:
row = gate.row_one([_case(tmp_path, "set-classes.json")])
assert (row.k, row.m, row.status) == (0, 0, gate.RED)
assert any("row 2's fixture" in detail for detail in row.details)
# --- row 2 --------------------------------------------------------------------
def test_row_two_is_green_when_every_class_is_the_one_its_fixture_forces(
tmp_path: Path,
) -> None:
cases = [_case(tmp_path, "set-classes.json"), _case(tmp_path, "set-miss.json")]
row = gate.row_two(cases)
assert row.status == gate.GREEN
assert row.k == row.m == 7
# Each class forced by its own fixture, and named in the output.
for letter in ("a", "b", "c", "d", "e"):
assert any(detail.startswith(f"class {letter} ") for detail in row.details)
def test_row_two_is_red_when_a_fixture_does_not_force_the_class_it_declares(
tmp_path: Path,
) -> None:
path, sha = _set_file(
tmp_path / "set.json",
set_id="row-two-red",
bundle="positive",
questions=[
{
"id": "W1",
"question": "Hvem har ansvaret for broeyting av parkeringsplassen?",
"expect_class": "b",
"fasit": [
{
"by": "concept",
"value": "haandbok/broeyteansvar",
"quote": "broeyteansvaret ligger hos vaktmesteren",
}
],
}
],
)
case = gate.measure_case(gate.load_set(path, sha), _bundles(tmp_path))
row = gate.row_two([case])
assert (row.k, row.m, row.status) == (0, 1, gate.RED)
assert any("forced class b, measured a" in detail for detail in row.details)
def test_a_miss_with_no_class_at_all_takes_the_whole_row_to_zero(tmp_path: Path) -> None:
case = _case(tmp_path, "set-classes.json")
unplaced = [
gate.Unit(
question_id="X1",
question="?",
named="concept:whatever",
quote="",
hit=False,
rank=None,
klass=None,
label_default=None,
truth=None,
confirmed=None,
)
]
holed = gate.Case(case.question_set, case.bundles, (*case.units, *unplaced))
row = gate.row_two([holed])
assert (row.k, row.status) == (0, gate.RED)
assert any("unplaced" in detail for detail in row.details)
# --- row 3 --------------------------------------------------------------------
def test_row_three_is_red_because_the_payload_names_the_quota_for_a_rank(
tmp_path: Path,
) -> None:
row = gate.row_three([_case(tmp_path, "set-classes.json")])
assert row.status == gate.RED
lying = [detail for detail in row.details if "source_quota_exceeded" in detail]
assert lying, row.details
assert all("the quota-off run says `below_k`" in detail for detail in lying)
def test_row_three_is_green_when_the_printed_reason_is_the_true_one(tmp_path: Path) -> None:
# The budget case: withheld by the pack in BOTH runs, so the label the
# payload prints is the label the quota-off run confirms.
path, sha = _set_file(
tmp_path / "set.json",
set_id="row-three-green",
bundle="budget",
questions=[
{
"id": "C1",
"question": "Hvor foeres dugnadstimene?",
"limit": 8000,
"expect_class": "c",
"fasit": [
{"by": "concept", "value": "tabell/dugnadstabell", "quote": "rad for rad"}
],
}
],
)
case = gate.measure_case(gate.load_set(path, sha), _bundles(tmp_path))
row = gate.row_three([case])
assert (row.k, row.m, row.status) == (1, 1, gate.GREEN)
# --- row 4 --------------------------------------------------------------------
def test_row_four_is_red_because_an_uncovered_question_still_gets_excerpts(
tmp_path: Path,
) -> None:
row = gate.row_four([_case(tmp_path, "set-controls.json")])
assert row.status == gate.RED
assert row.m == 6
assert any("not covered and NOT marked" in detail for detail in row.details)
def test_row_four_is_green_when_both_directions_come_out_right(tmp_path: Path) -> None:
path, sha = _set_file(
tmp_path / "set.json",
set_id="row-four-green",
bundle="positive",
questions=[],
controls=[
{
"id": "N1",
"kind": "an unrelated topic",
"question": "Hvilken safran passer til fiskesuppe?",
},
{
"id": "P0",
"kind": "covered by the bundle",
"question": "Naar kontrolleres vinterberedskapen?",
"covered": True,
},
],
)
case = gate.measure_case(gate.load_set(path, sha), _bundles(tmp_path))
row = gate.row_four([case])
assert (row.k, row.m, row.status) == (2, 2, gate.GREEN)
def test_a_covered_question_that_came_back_marked_would_fail_the_same_row(
tmp_path: Path,
) -> None:
# The other direction of row 4's own rule: a marking that also fires on a
# question the bundle answers says nothing at all.
path, sha = _set_file(
tmp_path / "set.json",
set_id="row-four-inverted",
bundle="positive",
questions=[],
controls=[
{
"id": "X0",
"kind": "declared covered, and it is not",
"question": "Hvilken safran passer til fiskesuppe?",
"covered": True,
}
],
)
case = gate.measure_case(gate.load_set(path, sha), _bundles(tmp_path))
row = gate.row_four([case])
assert (row.k, row.m, row.status) == (0, 1, gate.RED)
assert any("MARKED anyway" in detail for detail in row.details)
# --- row 5 --------------------------------------------------------------------
def test_row_five_is_red_while_no_hold_out_is_registered(tmp_path: Path) -> None:
row = gate.row_five(tmp_path / "absent.json")
assert (row.k, row.m, row.status) == (0, 1, gate.RED)
def test_row_five_is_green_for_a_registration_that_carries_all_seven(tmp_path: Path) -> None:
held = tmp_path / "held.json"
held.write_text('{"questions": []}', encoding="utf-8")
registration = tmp_path / "registration.json"
registration.write_text(
json.dumps(
{
"set": str(held),
"sha256": gate.sha256_of(held),
"threshold": "hit@payload at unit granularity >= 0.8",
"threshold_written_at": "2026-09-19T10:00:00Z",
"written_by": "the session that wrote the eval, never the one that "
"changes the ranking",
"readings": [{"at": "2026-09-20T09:00:00Z", "value": "not read yet"}],
}
),
encoding="utf-8",
)
row = gate.row_five(registration)
assert (row.k, row.m, row.status) == (7, 7, gate.GREEN)
def test_row_five_falls_on_a_number_read_before_its_threshold_was_written(
tmp_path: Path,
) -> None:
held = tmp_path / "held.json"
held.write_text('{"questions": []}', encoding="utf-8")
registration = tmp_path / "registration.json"
registration.write_text(
json.dumps(
{
"set": str(held),
"sha256": gate.sha256_of(held),
"threshold": "hit@payload at unit granularity >= 0.8",
"threshold_written_at": "2026-09-19T10:00:00Z",
"written_by": "somebody",
"readings": [{"at": "2026-09-18T09:00:00Z", "value": "0.62"}],
}
),
encoding="utf-8",
)
row = gate.row_five(registration)
assert (row.k, row.m, row.status) == (6, 7, gate.RED)
assert any("no reading predates the threshold: NO" in detail for detail in row.details)
# --- row 6 --------------------------------------------------------------------
def test_row_six_is_green_when_every_delivery_is_the_bundle_s_own_bytes(
tmp_path: Path,
) -> None:
cases = [_case(tmp_path, "set-positive.json"), _case(tmp_path, "set-signals.json")]
row = gate.row_six(cases)
assert (row.k, row.m, row.status) == (9, 9, gate.GREEN)
def test_row_six_is_red_when_the_payload_hands_over_something_else(tmp_path: Path) -> None:
# The judge reads the bundle BEFORE the payload is built -- that is what
# makes this test possible at all, and what row 7 relies on when it
# mutates the code the payload comes from.
bundles = _bundles(tmp_path)
gate.bundle_index(bundles["positive"])
question_set = gate.load_set(
FIXTURES / "set-positive.json", gate.SYNTHETIC_SETS["set-positive.json"]
)
with gate._truncate_delivered():
case = gate.measure_case(question_set, bundles)
row = gate.row_six([case])
assert row.status == gate.RED
assert row.k < row.m or row.m == 0
# --- row 7 --------------------------------------------------------------------
def _noop_mutant() -> gate.Mutant:
return gate.Mutant("N01 nothing is changed", 1, lambda: gate._patched())
def test_row_seven_is_green_when_every_mutant_is_felled(tmp_path: Path) -> None:
cases, _ = gate.synthetic_cases(tmp_path / "bundles", FIXTURES)
baseline = gate.deterministic_rows(cases)
lethal = (gate.MUTANTS[3],) # M04, the reversed ranking
row = gate.row_seven(cases, baseline, mutants=lethal)
assert (row.k, row.m, row.status) == (1, 1, gate.GREEN)
def test_row_seven_is_red_when_a_mutant_survives(tmp_path: Path) -> None:
cases, _ = gate.synthetic_cases(tmp_path / "bundles", FIXTURES)
baseline = gate.deterministic_rows(cases)
row = gate.row_seven(cases, baseline, mutants=(_noop_mutant(),))
assert (row.k, row.m, row.status) == (0, 1, gate.RED)
assert any(
"SURVIVED" in detail and "row 1 should have taken it" in detail for detail in row.details
)
def test_a_mutant_that_only_makes_a_red_row_greener_is_not_felled(tmp_path: Path) -> None:
# The definition, held by a test: removing the quota makes row 3's label
# honest, because there is then no quota to name. Calling that a kill
# would credit this gate with a check it does not have.
cases, _ = gate.synthetic_cases(tmp_path / "bundles", FIXTURES)
baseline = gate.deterministic_rows(cases)
before = {row.number: (row.k, row.m) for row in baseline}
with gate._wrap_cut(source_quota=None):
rows = gate.deterministic_rows(
[gate.measure_case(case.question_set, case.bundles) for case in cases]
)
after = {row.number: (row.k, row.m) for row in rows}
assert after[3][0] - after[3][1] > before[3][0] - before[3][1]
# --- rows 8 and 9 -------------------------------------------------------------
def test_row_eight_is_never_green_when_it_did_not_run() -> None:
row = gate.row_eight([])
assert row.status == gate.NOT_RUN
assert row.fails
assert any("recorded 2026-09-17 by PM, NOT measured" in detail for detail in row.details)
def test_the_wiki_adapter_reads_its_own_shape_and_hits_by_source_file(tmp_path: Path) -> None:
# The real set is never opened by a test. This is a file in its shape,
# with invented content, pointing at the synthetic bundle.
path = tmp_path / "wiki-shaped.json"
path.write_text(
json.dumps(
{
"schema": "1",
"hit_rule": "source_file == <doc>.md AND the excerpt carries the quote",
"questions": [
{
"id": "W1",
"class": "docs",
"question": "Naar kontrolleres vinterberedskapen?",
"fasit": [{"doc": "haandbok", "quote": "innen 1. november"}],
}
],
}
),
encoding="utf-8",
)
question_set = gate.read_real_set("wiki", path, gate.sha256_of(path))
assert question_set.set_id == "wiki-20"
assert question_set.questions[0].fasit[0].by == "source_file"
assert question_set.questions[0].fasit[0].value == "haandbok.md"
row = gate.row_eight([(question_set, {"wiki": _bundles(tmp_path)["positive"]})])
assert (row.k, row.m, row.status) == (1, 1, gate.GREEN)
assert any("citation granularity" in detail for detail in row.details)
def test_the_r761_adapter_splits_the_known_negative_out_of_the_questions(
tmp_path: Path,
) -> None:
path = tmp_path / "r761-shaped.json"
path.write_text(
json.dumps(
{
"fasit_form": "the normalised STS title",
"sporsmal": [
{"id": "S1", "sporsmal": "Hva sier punktet?", "fasit": "4.2 Vakthold"},
{"id": "KN", "sporsmal": "Hvilken safran passer til fiskesuppe?", "fasit": ""},
],
}
),
encoding="utf-8",
)
question_set = gate.read_real_set("r761", path, gate.sha256_of(path))
assert [q.id for q in question_set.questions] == ["S1"]
assert [c.id for c in question_set.controls] == ["KN"]
assert question_set.questions[0].fasit[0].by == "title"
# No quote in this set's fasit, so it is concept granularity and says so.
assert not question_set.quoted
def test_the_vegnormal_adapter_makes_one_question_per_standard(tmp_path: Path) -> None:
path = tmp_path / "vegnormal-shaped.json"
path.write_text(
json.dumps(
{
"sporsmal": [
{
"id": "T1-1",
"sporsmal": "Hva viser kravet til?",
"must_cite": [
{"normal": "N100:2023", "req_number": "2.3.2-3", "fil": "a"},
{"normal": "N200:2024", "req_number": "1.6.3-3", "fil": "b"},
],
}
]
}
),
encoding="utf-8",
)
question_set = gate.read_real_set("vegnormal", path, gate.sha256_of(path))
assert sorted(q.id for q in question_set.questions) == ["T1-1/N100:2023", "T1-1/N200:2024"]
assert {q.bundle for q in question_set.questions} == {"N100:2023", "N200:2024"}
assert question_set.units == 2
def test_an_unknown_real_set_name_is_refused(tmp_path: Path) -> None:
path = tmp_path / "x.json"
path.write_text("{}", encoding="utf-8")
with pytest.raises(gate.GateUsage):
gate.read_real_set("something-else", path, gate.sha256_of(path))
def test_row_nine_states_k2_s_denominator_and_never_passes() -> None:
row = gate.row_nine()
assert (row.k, row.m, row.status) == (0, 6, gate.RED)
assert any("the answer key does not" in detail for detail in row.details)
# --- the verdict --------------------------------------------------------------
def test_a_row_that_did_not_run_fails_the_gate_like_a_red_one() -> None:
assert gate.Row(8, "x", 0, 3, gate.NOT_RUN, "").fails
assert gate.Row(1, "x", 1, 1, gate.GREEN, "").fails is False
def test_a_row_with_nothing_to_count_is_never_green() -> None:
assert gate._row(1, "x", 0, 0, "", []).status == gate.RED
def test_the_verdict_names_every_failing_row() -> None:
rows = [
gate.Row(1, "a", 1, 1, gate.GREEN, ""),
gate.Row(2, "b", 0, 1, gate.RED, ""),
gate.Row(3, "c", 0, 1, gate.NOT_RUN, ""),
]
assert gate.render(rows).strip().endswith("GATE RED: rows 2, 3")
assert gate.render(rows[:1]).strip().endswith("GATE GREEN")
def test_the_gate_is_red_today_and_says_which_rows(tmp_path: Path) -> None:
rows = gate.evaluate(tmp_path / "bundles")
by_number = {row.number: row for row in rows}
assert sorted(by_number) == [1, 2, 3, 4, 5, 6, 7, 8, 9]
assert [row.number for row in rows if row.fails] == [3, 4, 5, 7, 8, 9]
assert (by_number[1].k, by_number[1].m) == (9, 9)
assert (by_number[2].k, by_number[2].m) == (7, 7)
assert (by_number[6].k, by_number[6].m) == (9, 9)
def test_the_same_tree_measures_the_same_twice(tmp_path: Path) -> None:
first = gate.evaluate(tmp_path / "one")
second = gate.evaluate(tmp_path / "two")
assert [row.to_json() for row in first] == [row.to_json() for row in second]
def test_the_command_exits_one_and_prints_every_row(
capsys: pytest.CaptureFixture[str],
) -> None:
assert gate.main([]) == 1
printed = capsys.readouterr().out
for number in range(1, 10):
assert f"\n{number} " in f"\n{printed}"
assert "GATE RED: rows 3, 4, 5, 7, 8, 9" in printed
def test_the_json_form_carries_the_same_rows(capsys: pytest.CaptureFixture[str]) -> None:
assert gate.main(["--json"]) == 1
payload = json.loads(capsys.readouterr().out)
assert [row["row"] for row in payload["rows"]] == list(range(1, 10))
assert payload["gate"] == gate.RED
assert set(payload["classes"]) == {"a", "b", "c", "d", "e"}
assert payload["limits"]
def test_an_unreadable_real_set_is_exit_two_and_never_a_quiet_row(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
path = tmp_path / "set.json"
path.write_text("{}", encoding="utf-8")
assert gate.main(["--real", "wiki", str(path), "0" * 64, str(tmp_path)]) == 2
assert "refusing to measure a set" in capsys.readouterr().err
# --- the instrument itself ----------------------------------------------------
def test_no_socket_is_opened_during_a_whole_run(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
calls: list[object] = []
def refuse(*args: object, **kwargs: object) -> None:
calls.append(args)
raise AssertionError("the gate opened a socket")
monkeypatch.setattr(socket, "socket", refuse)
monkeypatch.setattr(socket, "create_connection", refuse)
# The guard proven able to fire, before its silence counts as evidence.
with pytest.raises(AssertionError):
socket.socket()
calls.clear()
gate.evaluate(tmp_path / "bundles")
assert calls == []
def test_the_title_rule_is_the_one_okf_quality_uses_and_not_a_second_copy() -> None:
from llm_ingestion_okf import quality
assert gate.normalise_title is quality.normalise_title
assert gate._split_numbering is quality._split_numbering
assert gate._enclosing_directory is quality._enclosing_directory
def test_the_gate_changes_no_ranking_code() -> None:
# The order's boundary, held by a test rather than by intention: this file
# measures the ranking and never moves it. Every write to the module it
# measures lives inside `_patched`, which restores on exit -- so a
# mutation cannot outlive the `with` block it was applied in.
import ast
path = PROJECT_ROOT / "tools" / "okf_retrieval_gate.py"
tree = ast.parse(path.read_text(encoding="utf-8"))
patched = next(
node for node in tree.body if isinstance(node, ast.FunctionDef) and node.name == "_patched"
)
writes = [
node.lineno
for node in ast.walk(tree)
if isinstance(node, ast.Call)
and isinstance(node.func, ast.Name)
and node.func.id == "setattr"
and node.args
and isinstance(node.args[0], ast.Name)
and node.args[0].id == "consume"
]
assert writes, "the control: the walk finds the writes it is asked about"
assert all(patched.lineno <= line <= (patched.end_lineno or 0) for line in writes), writes
# And it restores: the same names, read before and written after.
source = ast.get_source_segment(path.read_text(encoding="utf-8"), patched) or ""
assert "finally:" in source and "original" in source
def test_the_module_imports_only_the_standard_library_and_this_repository() -> None:
import re
source = (PROJECT_ROOT / "tools" / "okf_retrieval_gate.py").read_text(encoding="utf-8")
for module in set(re.findall(r"^(?:from|import) ([a-zA-Z_][\w.]*)", source, re.MULTILINE)):
root = module.split(".")[0]
assert root in sys.stdlib_module_names or root == "llm_ingestion_okf", root
def test_the_payload_the_gate_reads_is_the_one_a_consumer_gets(tmp_path: Path) -> None:
# The gate calls `build_payload` with the SHIPPED defaults and changes one
# argument only where a row's definition needs it (`source_quota=None` for
# the truth run). A gate measuring a configuration nobody ships would be
# measuring something else.
bundles = _bundles(tmp_path)
question = gate.Question(
id="P1",
question="Naar kontrolleres vinterberedskapen?",
fasit=(gate.Fasit(by="concept", value="haandbok/vinterberedskap", quote="1. november"),),
)
assert question.k == consume.DEFAULT_K
assert question.limit == consume.DEFAULT_LIMIT
units = gate.measure_units(bundles["positive"], question)
assert [unit.hit for unit in units] == [True]
def test_the_bundle_index_is_read_from_disk_and_not_from_a_payload(tmp_path: Path) -> None:
bundles = _bundles(tmp_path)
index = gate.bundle_index(bundles["positive"])
assert isinstance(index, gate.BundleIndex)
assert "haandbok/vinterberedskap" in index.concepts
view = index.concepts["haandbok/vinterberedskap"]
on_disk = (bundles["positive"] / "haandbok" / "vinterberedskap.md").read_text(encoding="utf-8")
assert view.whole == on_disk
assert view.body in on_disk
def test_a_fasit_named_by_an_unknown_matcher_is_refused() -> None:
with pytest.raises(gate.GateUsage):
gate.Fasit(by="vibes", value="x", quote="y")
def test_a_citation_only_in_the_frontmatter_is_in_the_bundle_and_not_in_the_payload(
tmp_path: Path,
) -> None:
# Class (d)'s own mechanism, stated as a fact about the fixture: the
# sentence IS in the concept file and cannot reach an excerpt, because
# `excerpt_for` carries no `description`.
bundles = _bundles(tmp_path)
index = gate.bundle_index(bundles["positive"])
view = index.concepts["skjema/avviksskjema"]
assert "arkiveres i fem aar" in view.whole
assert "arkiveres i fem aar" not in view.body
def _mapping_is_sorted(mapping: Mapping[str, object]) -> bool:
return list(mapping) == sorted(mapping)
def test_every_class_is_documented_in_the_output() -> None:
assert [letter for letter, _ in gate.CLASSES] == ["a", "b", "c", "d", "e"]
assert all(description for _, description in gate.CLASSES)

1733
tools/okf_retrieval_gate.py Normal file
View file

File diff suppressed because it is too large Load diff