open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(quality): okf quality --fasit, boundary recall against a declared structure

Browse source 
The bundle-only gate returned UNMEASURED and exit 3 on the very arm it was
built for: no metric computable from a bundle alone reaches boundary recall.
`boundary_share` -- declared boundaries that became a concept, over declared
boundaries -- is the one metric measured that orders the arms correctly, and it
needs the publisher's own structure, so it arrives as an input.

Measurement first, threshold after, which is what the order asked for.

P1, the normalisation, derived rather than guessed: stripping all whitespace
and lowercasing reproduces the fasit's own `norm` from its own `title` on
2 761 of 2 761 rows (alphanumerics-only scores 58). P1's own bar is 99 % on the
known-good arm and the literal reading of it reaches 22 of 2 761 -- not because
the normalisation is wrong but because okf's default route moves the numbering
token a publisher glues into a heading over into the concept id. The pair form
(concept's own directory, residual title) reaches 2 737, either reaches 2 759
(99.9 %). Both forms ship and neither is a fallback: `r761-2025-d1` is the
control in the opposite direction at 2 727 literal, 0 paired.

P2, the single corpus, is in the OUTPUT and not only in the document: the bar
is declared `corpora = 1`, every boundary row prints `N = 1 corpus`, and the
line states that `--fasit` is the caller's ASSERTION that this bundle is a
build of the document the fasit describes -- the posture `okf consume --ref`
has. Measured: the K2 reference and `n100-2023` score 0 of 2 761 and read FAIL,
which is the assertion being wrong rather than the bundle being bad.

One bar, at the pinned artifact's own value, 2 759/2 761. It is tight and the
cost is published rather than tuned away: 2 of 4 R761 builds fall under it
(2 752 and 2 727), while any bar between 41.6 % and 98.8 % separates the
known-bad arm from every R761 build measured. The known-bad arm
(`860019-mdb-100`) is 1 148 of 2 761 -- FAIL and exit 1, where the bundle-only
gate gave exit 3.

A fasit is validated at the door: not a list, a row missing `title` or `norm`,
or anything that is not JSON exits 2 with the reason, never a quiet UNMEASURED.
A fasit under five rows is UNMEASURED -- the document floor in the fasit's own
unit.

Without `--fasit` the command is byte-for-byte what it was, held by a test.
`okf check` is untouched; no version bump and no tag. 17 tests red on
assertions before the implementation, the two new doc pins each driven red and
back. Suite 1 869 passed / 1 skipped / 1 870 collected (base 5e5d01c: 1 851).

docs/2026-09-12-g37-terskler.md SS 7 carries the premises re-measured, the
seven bundles, the interval any bar could sit in, and the honesty limits --
including the correction of SS 2's own grep claim, which went false in the
commit that wrote it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-13 07:27:33 +02:00
commit b6da09cc97
7 changed files with 868 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

27
tests/test_docs_promises.py
View file

@ -253,3 +253,30 @@ def test_the_readme_quality_section_does_not_promise_a_quality_claim() -> None:
text = README.read_text(encoding="utf-8").lower()
assert "okf quality" in text
assert "regression bar against a pinned artifact" in text
# The `boundary_share` bar. Published in three places -- the code, the README
# and the threshold document -- and a bar published without a test goes false
# the way the format list did.
_BOUNDARY_LINE = re.compile(r"^<!-- quality-boundary-threshold: (.+) -->$", re.MULTILINE)
def test_the_readme_names_the_boundary_bar_the_gate_applies() -> None:
from llm_ingestion_okf.quality import BOUNDARY_THRESHOLD
match = _BOUNDARY_LINE.search(README.read_text(encoding="utf-8"))
assert match is not None, (
"README.md carries no `<!-- quality-boundary-threshold: ... -->` marker; "
"without it the published bar can drift from the one --fasit applies"
)
assert match.group(1).strip() == BOUNDARY_THRESHOLD.as_share()
def test_the_threshold_document_carries_the_boundary_bar_and_its_single_corpus() -> None:
"""N = 1 is half of what this bar is; a copy without it publishes the other half."""
from llm_ingestion_okf.quality import BOUNDARY_THRESHOLD
text = THRESHOLD_DOCUMENT.read_text(encoding="utf-8")
assert f"`{BOUNDARY_THRESHOLD.metric}` | **2 759/2 761**" in text
assert "1 corpus" in text
assert BOUNDARY_THRESHOLD.corpora == 1

270
tests/test_quality.py
View file

@ -16,8 +16,11 @@ read off, live in `docs/2026-09-12-g37-terskler.md`.
from __future__ import annotations
import json
from pathlib import Path
import pytest
from llm_ingestion_okf import quality
_FRONTMATTER = """---
@ -40,6 +43,7 @@ def _bundle(root: Path, concepts: list[tuple[str, str, str]]) -> Path:
root.mkdir(parents=True, exist_ok=True)
lines = ["---", "okf_version: 0.2", "bundle_id: quality-fixture", "---", ""]
for stem, source_file, body in concepts:
(root / f"{stem}.md").parent.mkdir(parents=True, exist_ok=True)
(root / f"{stem}.md").write_text(
_FRONTMATTER.format(title=stem, source_file=source_file, body=body),
encoding="utf-8",
@ -224,3 +228,269 @@ def test_the_no_source_file_row_reports_no_document_count(tmp_path: Path) -> Non
assert "one-concept documents" not in rendered
assert "empty 0/2" in rendered
assert "no source_file on 2 of 2 concepts" in rendered
# --- G37b: `--fasit` and `boundary_share` -------------------------------------
#
# The metric that separates the two arms this gate could not tell apart. It is
# the ONLY one measured that orders them correctly and it needs the publisher's
# own declared structure, so it arrives as an input rather than as a constant:
# `okf quality <bundle> --fasit <json>`.
#
# Two things are pinned here that were measured before any of it was written
# (`docs/2026-09-12-g37-terskler.md` SS 7):
#
# 1. The fasit's `norm` key is reproduced from its own `title` by stripping all
# whitespace and lowercasing -- 2 761 of 2 761 rows, so the normalisation is
# not a guess.
# 2. That normalisation ALONE reaches 22 of 2 761 on the known-good arm, not
# 99 %, because okf's default route moves the numbering token STS glues into
# `<title>` over into the concept id. The second match form -- the
# (directory, residual title) pair -- is what takes it to 2 759 of 2 761.
_GOOD_ARM = Path.home() / "repos/vegnormal-okf/build/ferdig/r761-2025-generisk"
_BAD_ARM = Path.home() / "repos/vegnormal-okf/build/860019-mdb-100"
_OTHER_PRODUCT = Path.home() / "corpora/okf-telling-20260829/K2-bundle-default-20260912"
_FASIT = Path.home() / "repos/vegnormal-okf/build/sk2-fasit-2761.json"
def _fasit(path: Path, titles: list[str]) -> Path:
"""A fasit file in the shipped shape: a list of rows carrying title and norm."""
path.write_text(
json.dumps(
[{"title": title, "norm": quality.normalise_title(title)} for title in titles],
ensure_ascii=False,
),
encoding="utf-8",
newline="",
)
return path
def _declaring(titles: list[str]) -> list[tuple[str, str, str]]:
"""One concept per declared title, matching it literally."""
return [(f"c{index}", "doc.xml", "Body text.") for index, _ in enumerate(titles)]
def test_the_normalisation_reproduces_the_fasit_key() -> None:
"""`norm` is whitespace-stripped, lowercased `title` -- not a guess.
Measured over the shipped fasit before anything was written: 2 761 of 2 761
rows reproduce, and the real file pins it below where it exists.
"""
assert quality.normalise_title(" 2.1 Hoved Prosesser\n") == "2.1hovedprosesser"
assert quality.normalise_title("1Bruksområder for Prosesskoden") == (
"1bruksområderforprosesskoden"
)
def test_a_bundle_recovering_every_declared_boundary_passes(tmp_path: Path) -> None:
"""The known-positive, in miniature: every declared title became a concept."""
titles = [f"{n} Kapittel {n}" for n in range(1, 7)]
root = tmp_path / "recovered"
root.mkdir()
bundle = _bundle(root, [(quality.normalise_title(t)[:12], "doc.xml", "Body.") for t in titles])
# The concept titles ARE the declared titles: rewrite the frontmatter.
for index, title in enumerate(titles):
stem = quality.normalise_title(title)[:12]
(bundle / f"{stem}.md").write_text(
_FRONTMATTER.format(title=title, source_file="doc.xml", body="Body."),
encoding="utf-8",
newline="",
)
report = quality.measure_bundle(
bundle, fasit=quality.load_fasit(_fasit(tmp_path / "f.json", titles))
)
assert report.boundaries is not None
assert (report.boundaries.recovered, report.boundaries.declared) == (6, 6)
assert report.boundaries.verdict == "PASS"
assert report.exit_code == 0
def test_a_bundle_recovering_few_declared_boundaries_fails(tmp_path: Path) -> None:
"""The known-negative for the same rule, and the arm it was built for.
`860019-mdb-100` recovers 1 148 of 2 761. Here five of six is already worse
than the bar of 2 759/2 761 -- the bar is that tight, which is a property of
a regression bar set at its reference and is said out loud in the document.
"""
titles = [f"{n} Kapittel {n}" for n in range(1, 7)]
root = tmp_path / "lost"
root.mkdir()
bundle = _bundle(root, [(f"c{i}", "doc.md", "Body.") for i in range(5)])
for index, title in enumerate(titles[:5]):
(bundle / f"c{index}.md").write_text(
_FRONTMATTER.format(title=title, source_file="doc.md", body="Body."),
encoding="utf-8",
newline="",
)
report = quality.measure_bundle(
bundle, fasit=quality.load_fasit(_fasit(tmp_path / "f.json", titles))
)
assert report.boundaries is not None
assert (report.boundaries.recovered, report.boundaries.declared) == (5, 6)
assert report.boundaries.verdict == "FAIL"
assert report.exit_code == 1
def test_the_numbering_token_is_matched_through_the_concept_directory(tmp_path: Path) -> None:
"""The second match form, and the reason P1's literal reading is not enough.
STS glues the numbering into the title (`11.1Fastmerker`); okf's default
route moves it into the concept id (`11-1/...`) and keeps the residual as
the title. Measured on the known-good arm: the literal form alone reaches
22 of 2 761, the pair form 2 737, either 2 759.
"""
titles = ["11.1Fastmerker"]
bundle = _bundle(tmp_path / "paired", [("11-1/p1", "doc.pdf", "Body.")])
(bundle / "11-1/p1.md").write_text(
_FRONTMATTER.format(title="Fastmerker", source_file="doc.pdf", body="Body."),
encoding="utf-8",
newline="",
)
fasit = quality.load_fasit(_fasit(tmp_path / "f.json", titles))
report = quality.measure_bundle(bundle, fasit=fasit)
assert report.boundaries is not None
assert report.boundaries.literal == 0
assert report.boundaries.paired == 1
assert report.boundaries.recovered == 1
def test_without_a_fasit_the_gate_is_byte_for_byte_what_it_was(tmp_path: Path) -> None:
"""The regression the order asks for: no `--fasit`, no change of any kind."""
root = _bundle(tmp_path / "unchanged", [("a", "note.md", "Body."), ("b", "note.md", "More.")])
report = quality.measure_bundle(root)
assert report.boundaries is None
assert report.exit_code == 3
assert "boundary" not in report.render().split("## What this verdict is not")[0].lower()
def test_a_fasit_that_is_not_a_list_exits_two_rather_than_unmeasured(tmp_path: Path) -> None:
"""An unreadable fasit is a run that did not happen, never a quiet verdict."""
bad = tmp_path / "object.json"
bad.write_text('{"title": "x", "norm": "x"}', encoding="utf-8")
root = _bundle(tmp_path / "b", [("a", "note.md", "Body.")])
assert quality.main([str(root), "--fasit", str(bad)]) == 2
def test_a_fasit_row_missing_a_key_exits_two(tmp_path: Path) -> None:
bad = tmp_path / "rows.json"
bad.write_text('[{"title": "x", "norm": "x"}, {"title": "y"}]', encoding="utf-8")
root = _bundle(tmp_path / "b", [("a", "note.md", "Body.")])
assert quality.main([str(root), "--fasit", str(bad)]) == 2
def test_a_fasit_that_is_not_json_exits_two(tmp_path: Path) -> None:
bad = tmp_path / "broken.json"
bad.write_text("not json at all", encoding="utf-8")
root = _bundle(tmp_path / "b", [("a", "note.md", "Body.")])
assert quality.main([str(root), "--fasit", str(bad)]) == 2
def test_a_missing_fasit_file_exits_two(tmp_path: Path) -> None:
root = _bundle(tmp_path / "b", [("a", "note.md", "Body.")])
assert quality.main([str(root), "--fasit", str(tmp_path / "nowhere.json")]) == 2
def test_a_valid_fasit_reaches_the_gate_through_the_cli(tmp_path: Path, capsys) -> None: # type: ignore[no-untyped-def]
titles = [f"{n} Kapittel {n}" for n in range(1, 7)]
bundle = _bundle(tmp_path / "cli", [(f"c{i}", "doc.md", "Body.") for i in range(1)])
(bundle / "c0.md").write_text(
_FRONTMATTER.format(title=titles[0], source_file="doc.md", body="Body."),
encoding="utf-8",
newline="",
)
from llm_ingestion_okf import cli
assert (
cli.main(["quality", str(bundle), "--fasit", str(_fasit(tmp_path / "f.json", titles))]) == 1
)
out = capsys.readouterr().out
assert "boundary_share" in out
assert "1/6" in out
def test_a_fasit_below_the_floor_is_unmeasured_and_never_pass(tmp_path: Path) -> None:
"""A share over four declared boundaries is not a rate either."""
titles = ["1 A", "2 B", "3 C", "4 D"]
bundle = _bundle(tmp_path / "tiny", [(f"c{i}", "doc.md", "Body.") for i in range(4)])
for index, title in enumerate(titles):
(bundle / f"c{index}.md").write_text(
_FRONTMATTER.format(title=title, source_file="doc.md", body="Body."),
encoding="utf-8",
newline="",
)
report = quality.measure_bundle(
bundle, fasit=quality.load_fasit(_fasit(tmp_path / "f.json", titles))
)
assert report.boundaries is not None
assert report.boundaries.verdict == "UNMEASURED"
assert report.boundaries.recovered == 4
def test_the_boundary_row_says_its_threshold_rests_on_one_corpus(tmp_path: Path) -> None:
"""P2, in the printout rather than only in the document."""
titles = [f"{n} Kapittel {n}" for n in range(1, 7)]
bundle = _bundle(tmp_path / "caveat", [("c0", "doc.md", "Body.")])
(bundle / "c0.md").write_text(
_FRONTMATTER.format(title=titles[0], source_file="doc.md", body="Body."),
encoding="utf-8",
newline="",
)
rendered = quality.measure_bundle(
bundle, fasit=quality.load_fasit(_fasit(tmp_path / "f.json", titles))
).render()
assert "one product" in rendered
assert "N = 1" in rendered
def test_the_boundary_threshold_names_its_corpus_and_denominator() -> None:
bar = quality.BOUNDARY_THRESHOLD
assert bar.limit_declared >= quality.MIN_DECLARED_FOR_A_THRESHOLD
assert bar.corpora == 1
assert bar.source
@pytest.mark.skipif(not (_GOOD_ARM.is_dir() and _FASIT.is_file()), reason="local gold set absent")
def test_the_known_good_arm_reproduces_its_published_recall() -> None:
"""2 759 of 2 761, and the decomposition P1 asked for: 22 literal, 2 737 paired."""
report = quality.measure_bundle(_GOOD_ARM, fasit=quality.load_fasit(_FASIT))
assert report.boundaries is not None
assert (report.boundaries.recovered, report.boundaries.declared) == (2759, 2761)
assert (report.boundaries.literal, report.boundaries.paired) == (22, 2737)
assert report.boundaries.verdict == "PASS"
@pytest.mark.skipif(not (_BAD_ARM.is_dir() and _FASIT.is_file()), reason="local gold set absent")
def test_the_known_bad_arm_reproduces_its_published_recall() -> None:
"""1 148 of 2 761 -- the 41.6 % the whole gate could not previously see."""
report = quality.measure_bundle(_BAD_ARM, fasit=quality.load_fasit(_FASIT))
assert report.boundaries is not None
assert (report.boundaries.recovered, report.boundaries.declared) == (1148, 2761)
assert report.boundaries.verdict == "FAIL"
assert report.exit_code == 1
@pytest.mark.skipif(
not (_OTHER_PRODUCT.is_dir() and _FASIT.is_file()), reason="local gold set absent"
)
def test_a_bundle_of_another_product_scores_zero_and_that_is_the_assertion(tmp_path: Path) -> None:
"""The adverse case, measured rather than argued.
The fasit describes ONE product. The pinned 43-document K2 bundle is a
different corpus entirely and scores 0 of 2 761 -- a FAIL that says the
caller's assertion was wrong, not that the bundle is bad. `--fasit` is an
assertion by the caller, the way `okf consume --ref` is.
"""
report = quality.measure_bundle(_OTHER_PRODUCT, fasit=quality.load_fasit(_FASIT))
assert report.boundaries is not None
assert report.boundaries.recovered == 0
assert report.boundaries.verdict == "FAIL"
@pytest.mark.skipif(not _FASIT.is_file(), reason="local gold set absent")
def test_the_shipped_fasit_reproduces_its_own_key_under_this_normalisation() -> None:
"""2 761 of 2 761 rows: `norm` is exactly whitespace-stripped, lowercased title."""
rows = quality.load_fasit(_FASIT)
assert len(rows) == 2761
assert sum(1 for row in rows if quality.normalise_title(row.title) == row.norm) == 2761