open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(quality): okf quality --fasit, boundary recall against a declared structure

Browse source 
The bundle-only gate returned UNMEASURED and exit 3 on the very arm it was
built for: no metric computable from a bundle alone reaches boundary recall.
`boundary_share` -- declared boundaries that became a concept, over declared
boundaries -- is the one metric measured that orders the arms correctly, and it
needs the publisher's own structure, so it arrives as an input.

Measurement first, threshold after, which is what the order asked for.

P1, the normalisation, derived rather than guessed: stripping all whitespace
and lowercasing reproduces the fasit's own `norm` from its own `title` on
2 761 of 2 761 rows (alphanumerics-only scores 58). P1's own bar is 99 % on the
known-good arm and the literal reading of it reaches 22 of 2 761 -- not because
the normalisation is wrong but because okf's default route moves the numbering
token a publisher glues into a heading over into the concept id. The pair form
(concept's own directory, residual title) reaches 2 737, either reaches 2 759
(99.9 %). Both forms ship and neither is a fallback: `r761-2025-d1` is the
control in the opposite direction at 2 727 literal, 0 paired.

P2, the single corpus, is in the OUTPUT and not only in the document: the bar
is declared `corpora = 1`, every boundary row prints `N = 1 corpus`, and the
line states that `--fasit` is the caller's ASSERTION that this bundle is a
build of the document the fasit describes -- the posture `okf consume --ref`
has. Measured: the K2 reference and `n100-2023` score 0 of 2 761 and read FAIL,
which is the assertion being wrong rather than the bundle being bad.

One bar, at the pinned artifact's own value, 2 759/2 761. It is tight and the
cost is published rather than tuned away: 2 of 4 R761 builds fall under it
(2 752 and 2 727), while any bar between 41.6 % and 98.8 % separates the
known-bad arm from every R761 build measured. The known-bad arm
(`860019-mdb-100`) is 1 148 of 2 761 -- FAIL and exit 1, where the bundle-only
gate gave exit 3.

A fasit is validated at the door: not a list, a row missing `title` or `norm`,
or anything that is not JSON exits 2 with the reason, never a quiet UNMEASURED.
A fasit under five rows is UNMEASURED -- the document floor in the fasit's own
unit.

Without `--fasit` the command is byte-for-byte what it was, held by a test.
`okf check` is untouched; no version bump and no tag. 17 tests red on
assertions before the implementation, the two new doc pins each driven red and
back. Suite 1 869 passed / 1 skipped / 1 870 collected (base 5e5d01c: 1 851).

docs/2026-09-12-g37-terskler.md SS 7 carries the premises re-measured, the
seven bundles, the interval any bar could sit in, and the honesty limits --
including the correction of SS 2's own grep claim, which went false in the
commit that wrote it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-13 07:27:33 +02:00
commit b6da09cc97
7 changed files with 868 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

40
CHANGELOG.md
View file

@ -5,6 +5,46 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Added
- **`okf quality <bundle> --fasit <json>` -- boundary recall against the
structure the source itself declares (G37b).** The bundle-only gate returned
`UNMEASURED` and exit **3** on the very arm it was built for, because no
bundle-only metric reaches it; `boundary_share` -- declared boundaries that
became a concept, over declared boundaries -- is the one metric measured that
orders the arms correctly, and it needs the publisher's own structure, so it
arrives as an input rather than as a constant. The fasit is a JSON list whose
rows carry `title` and `norm`, validated at the door: a file that is not a
list, a row missing either key, or anything that is not JSON exits **2** with
the reason, never a quiet `UNMEASURED`.
**The normalisation was measured before the metric was built**: stripping all
whitespace and lowercasing reproduces the fasit's own `norm` from its own
`title` on **2 761 of 2 761** rows (keeping only alphanumerics scores 58).
**A boundary is recovered in either of two forms**, and both are load-bearing:
a concept whose normalised title equals `norm`, or the pair of the concept's
own directory and its residual title -- because the numbering token a
publisher glues into a heading lands in the concept id on one route and in the
title on another. Measured on the known-good arm, the literal form alone
reaches **22 of 2 761** where the two together reach **2 759**; on another
build of the same product the split is the exact opposite (2 727 literal, 0
paired). One bar, at the value measured on the pinned artifact: **2 759/2 761**,
`corpora = 1`. It separates the known-bad arm at **1 148 of 2 761 (41.6 %)**,
which is now a `FAIL` and exit 1 instead of exit 3. **`--fasit` is an
assertion**, like `okf consume --ref`: a bundle of another product scores 0 of
2 761 (measured on two of them) and reads `FAIL` -- the assertion being wrong,
not the bundle. The bar rests on **one product**, and the run says so on every
boundary row. `docs/2026-09-12-g37-terskler.md` SS 7 carries the premises
re-measured, the seven bundles, the interval any bar could sit in, and the two
R761 builds this one fails.
### Unchanged
- **Without `--fasit` the command is exactly what it was**, held by a test: no
boundary row, and `860019-mdb-100` still exits 3. No version bump, no tag,
`okf check` untouched.
## [0.9.0] — 2026-09-13
### Added

38
CLAUDE.md
View file

@ -788,6 +788,44 @@ and fixtures, never code.
premises re-measured: `docs/2026-09-12-g37-terskler.md`. README publishes the
bars behind `<!-- quality-thresholds: ... -->`, pinned to the code AND the
document by `tests/test_docs_promises.py`.
**`--fasit <json>` REACHES that defect (G37b, 2026-09-13)** and is the only
input this gate takes: one whole-bundle row, `boundary_share` = declared
boundaries that became a concept over declared boundaries. It is whole-bundle
and never per file type, because a fasit names ONE document's sections and a
bundle can spread them over 828 source files -- which the known-bad arm does.
**The normalisation was derived before the metric was built**, not guessed:
strip all whitespace, lowercase, reproduces the fasit's own `norm` from its
own `title` on **2 761 of 2 761** rows (alphanumerics-only scores 58 -- it
eats the `.` in `2.1Hovedprosesser`). **A boundary is recovered in EITHER of
two forms and neither is a fallback**: the concept's normalised title equal to
`norm`, or the pair `(concept's own directory, residual title)` -- the literal
form wants the declared title WITH its numbering token, the pair form WITHOUT,
and no bundle can offer both, because okf's default route moves that token
into the concept id. Measured on the known-good arm: literal **22 of 2 761**,
paired **2 737**, either **2 759 (99.9 %)**; on `r761-2025-d1` the split is
exactly inverted (2 727 literal, 0 paired), so a gate scoring one form alone
reports a 99.9 % arm as 0.8 % and calls it a segmentation defect. The two
forms are `vegnormal-okf`'s M8 correction, which they took verbatim from THIS
repository's round-14 report -- the instrument reproduces both so the two
repos cannot silently measure different things. One bar, at the pinned
artifact's own value: **2 759/2 761 with `corpora = 1`**, and P2 is in the
OUTPUT and not only in the document (`N = 1 corpus` on every boundary row).
The known-bad arm is **1 148 of 2 761 (41.6 %)**, now `FAIL` + exit 1 where
the bundle-only gate gave exit 3. **`--fasit` is an ASSERTION** (the posture
`okf consume --ref` has) that this bundle is a build of the document the fasit
describes: the K2 reference and `n100-2023` both score **0 of 2 761** and read
`FAIL`, which is the assertion being wrong and not the bundle -- a gate
telling those apart would need a bar read off the bundles it judges. The bar
is TIGHT and the cost is published: 2 of 4 R761 builds fall under it (2 752
and 2 727 of 2 761), while **any bar between 41.6 % and 98.8 % separates the
known-bad arm from every R761 build measured** -- the shipped one is the only
point in that interval read off a pinned artifact. An unreadable fasit exits
**2** with its reason, never a quiet `UNMEASURED`, and a fasit under five rows
is `UNMEASURED` (`MIN_DECLARED_FOR_A_THRESHOLD`, the document floor in the
fasit's unit). **Without `--fasit` the command is byte-for-byte what it was**,
held by a test. README publishes this bar behind
`<!-- quality-boundary-threshold: ... -->`; SS 7 of the threshold document
carries the seven bundles and the honesty limits.
- Consume a bundle: `okf consume <bundle> --question "<q>"
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
`docs/consumption-contract.md` § 1 defines, and the only reading direction

41
README.md
View file

@ -685,11 +685,42 @@ five documents on both sides — the bundle's and its own — because a `1/1` is
not a rate.
**What a `PASS` is not.** It is a regression bar against a pinned artifact, not
a claim that the cut found the document's own structure. Boundary recall and
hit@k need a fasit and are outside a bundle-only gate; the measurement that
says so, the corpora behind each number, and three candidate metrics that were
measured and not shipped are in
[`docs/2026-09-12-g37-terskler.md`](docs/2026-09-12-g37-terskler.md).
a claim that the cut found the document's own structure. hit@k needs a question
set and is not asked here; the measurement that says so, the corpora behind
each number, and three candidate metrics that were measured and not shipped are
in [`docs/2026-09-12-g37-terskler.md`](docs/2026-09-12-g37-terskler.md).
### Boundary recall: `--fasit`
Give the command the boundaries the source itself declares and it adds one
whole-bundle row, `boundary_share` — the share of them that became a concept:
```sh
okf quality .okf/my-bundle --fasit declared-sections.json
```
The fasit is a JSON list whose rows carry `title` and the key they are matched
on, `norm` (all whitespace stripped, lowercased). An unreadable one exits **2**,
never `UNMEASURED`. A boundary counts as recovered in either of two forms — a
concept whose normalised title equals `norm`, or the pair of the concept's own
directory and its residual title — because the numbering token a publisher glues
into a heading lands in the concept *id* on one route and in the *title* on
another: measured on one 2 761-section standard, the first form alone reaches
**22 of 2 761** where the two together reach **2 759**.
<!-- quality-boundary-threshold: 2759/2761 -->
| metric | threshold | N |
|---|---|---|
| `boundary_share` | 2 759/2 761 | 2 761 declared boundaries, **1 corpus** |
**`--fasit` is an assertion**, the way `okf consume --ref` is: it says this
bundle is a build of the document the fasit describes. A bundle of another
product scores near zero and reads `FAIL` — that is the assertion being wrong,
not the bundle. The bar itself rests on **one product**, which the output says
on every run. Both facts, the arm it separates (1 148 of 2 761 against 2 759 of
2 761) and the interval any bar could sit in are in
[`docs/2026-09-12-g37-terskler.md`](docs/2026-09-12-g37-terskler.md) § 7.
## Consume in Claude Code

155
docs/2026-09-12-g37-terskler.md
View file

@ -66,7 +66,7 @@ three of the five premises moved.
| 13 extensions in the registry | `_CORE_EXTRACTORS` 7 + `_OPTIONAL_EXTRACTORS` 6 = **13** | holds |
| four evidence corpora under `~/repos/vegnormal-okf/build/ferdig/` | present: `n100-2023`, `n200-2024`, `n500-2024`, `r761-2025` — plus `r761-2025-generisk` and `r761-2025-v080`, two more R761 builds the order does not name | **holds, and is incomplete** |
| "N100 skal gi PASS på de målte typene" | `grep -rl '^source_file:' n100-2023 --include='*.md' | wc -l` → **0 of 446 concepts**. Same for `n200-2024` (0 of 1 133), `n500-2024` (0 of 270) and `r761-2025` (0 of 2 756) | **cannot hold**: three of the four name no file type at all, so no type of theirs can be PASS or FAIL |
| "41,6 % grenser" | not in this repository (`grep -rn "41[.,]6" docs/ src/` → 0). It is `vegnormal-okf`'s: `docs/2026-09-10-r2-r761-produkt.md:452`, 1 148 of 2 761 declared STS sections, against the XML arm's 2 759 of 2 761 | **holds, and the owner is another repo** |
| "41,6 % grenser" | not in this repository when the order arrived — measured on the base commit, `git grep -c "41[.,]6" d310dbb -- docs src` → **0 files**. It is `vegnormal-okf`'s: `docs/2026-09-10-r2-r761-produkt.md:452`, 1 148 of 2 761 declared STS sections, against the XML arm's 2 759 of 2 761 | **holds, and the owner is another repo** |
| "0 av 6 hit@k" on the HTML arm | two published numbers for the same 828-document bundle: **0 of 6** (vegnormal, their six fasit questions, 10.09) and **2 of 6 → 4 of 6 with `--source-quota 2`** (this repo, `docs/2026-09-10-k3-runde11-html-og-dominans.md:200,217`, questions S1–S6) | **holds for one instrument, not for the other** |
The last row is the reason the gate reports a denominator on every line: two
@ -160,7 +160,158 @@ needs the publisher's own declared structure. R761 has one (2 761 STS `<sec>`
titles, `~/repos/vegnormal-okf/build/sk2-fasit-2761.json`), and 4 of the 9
bundles above have no fasit at all. A `--fasit` input, a threshold on
`boundary_share`, and the operator's bar are one further round, named in STATE
and not begun here.
and not begun here. **That round is § 7 below** (G37b, order
`20260912T223334Z-132479088`, 2026-09-13).
## § 7 — G37b: `--fasit` and `boundary_share`
Order `20260912T223334Z-132479088-from-.claude`, on base `5e5d01c` (v0.9.0,
pushed, **untagged**). Measured 2026-09-13. Python 3.14.0, no model call, no
network. The order's own instruction was measurement first: two premises
measured before any threshold is set.
### § 7.0 — the correction this section owes § 2
The row above published `grep -rn "41[.,]6" docs/ src/` → 0 as a present-tense
fact. It stopped being true in the same commit, because the document making the
claim writes the number twice. Re-measured today the working tree returns **2
hits, both in this file**. The measurement it was meant to be is the one on the
base commit, and it is now written that way. This is the repository's own trap
(`memory/falsifiser-egne-negative-paastander.md`) firing on the file that
records traps.
### § 7.1 — P1: the normalisation, measured on a known-positive first
The fasit is `~/repos/vegnormal-okf/build/sk2-fasit-2761.json` — 665 134 bytes,
a JSON list of exactly **2 761** objects, each carrying `title` and `norm`
(`sec_type`, `depth`, `label`, `outline_page` and `id` as well). Every `norm` is
unique, and so is every `title`.
**The normalisation is derived, not guessed.** Strip all whitespace, then
lowercase, reproduces the fasit's own `norm` from its own `title` on **2 761 of
2 761** rows. Two candidates were run first and fell on the same file:
lower-casing and keeping only alphanumerics scores **58 of 2 761** (it eats the
`.` in `2.1Hovedprosesser` and the `-` in `3Sted- og elementkode`). Unicode NFC
makes no difference here — 2 761 of 2 761 titles and norms are already NFC — and
the rule is the one `vegnormal-okf`'s own measuring script applies, so a number
produced here and a number produced there are the same number.
**P1's own bar is 99 % on the known-good arm, and the literal reading of it
fails — for a reason that is measured and is not the normalisation.** On
`r761-2025-generisk` (the declared-structure arm, 2 761 concepts):
| match form | recovered of 2 761 |
|---|---|
| literal: normalised concept title equals the fasit's `norm` | **22** (0.8 %) |
| pair: `(concept's own directory, normalised residual title)` | **2 737** (99.1 %) |
| either | **2 759** (99.9 %) |
The cause is this repository's own id form: STS glues the numbering token into
`<title>` (`11.1Fastmerker`), and okf's default route moves that token into the
concept id (`11-1/…`) while keeping the residual as the title. A gate scoring
the literal form alone would report a 99.9 % arm as 0.8 % and call it a
segmentation defect. Both forms therefore ship, neither as a fallback for the
other: the literal form wants the title *with* its token, the pair form *without*,
and no single bundle can offer both. The `r761-2025-d1` build is the control in
the opposite direction — literal **2 727**, paired **0**.
That decomposition is not this round's invention either. It is the M8 correction
`vegnormal-okf` took verbatim from this repository's own round-14 report; the
instrument here reproduces their two forms so the two repositories cannot
silently measure different things.
**Known-positive delivered: 2 759 of 2 761 (99.9 %)**, the published number to
the unit. **Known-negative delivered: `860019-mdb-100` at 1 148 of 2 761
(41.6 %)**, also to the unit — the arm the bundle-only gate returned
`UNMEASURED`/exit 3 for.
### § 7.2 — P2: the threshold rests on one product, and says so
The fasit describes **one** document: R761 Prosesskoden:2025. Every number
below is that product's. The bar is declared with `corpora = 1` in the code
(`quality.BOUNDARY_THRESHOLD`), the row prints `N = 1 corpus` on every run, and
the printed line also states what `--fasit` is: **an assertion by the caller**
that this bundle is a build of the document the fasit describes — the same
posture `okf consume --ref` has.
That is not a formality. Measured: the pinned 43-document K2 reference bundle
scores **0 of 2 761** and `n100-2023` scores **0 of 2 761**. Both read `FAIL`,
and in both cases the right reading is *the assertion was wrong*, not *the
bundle is bad*. A gate that could tell those two apart would need a bar on
"does this fasit describe this bundle", and that bar would have to be read off
the bundles it judges.
### § 7.3 — every bundle measured, with the fasit
`okf quality <bundle> --fasit ~/repos/vegnormal-okf/build/sk2-fasit-2761.json`
| bundle | concepts | literal | paired | recovered of 2 761 | verdict | rc |
|---|---|---|---|---|---|---|
| `r761-2025-generisk` (reference) | 2 761 | 22 | 2 737 | **2 759** (99.9 %) | PASS | 0 |
| `r761-2025-v080` | 2 761 | 22 | 2 737 | **2 759** (99.9 %) | PASS | 0 |
| `r761-2025` | 2 756 | 20 | 2 732 | **2 752** (99.7 %) | FAIL | 1 |
| `r761-2025-d1` (vegnormal's own pipeline) | 2 728 | 2 727 | 0 | **2 727** (98.8 %) | FAIL | 1 |
| `860019-mdb-100` (known-bad arm) | 3 206 | 1 130 | 28 | **1 148** (41.6 %) | FAIL | 1 |
| `n100-2023` (another product) | 446 | 0 | 0 | **0** | FAIL | 1 |
| `K2-bundle-default-20260912` (another product) | 453 | 0 | 0 | **0** | FAIL | 1 |
### § 7.4 — the bar, and why it sits where it sits
| metric | threshold | N | source |
|---|---|---|---|
| `boundary_share` | **2 759/2 761** | 2 761 declared boundaries, **1 corpus** | `r761-2025-generisk` against `sk2-fasit-2761.json` |
Set at the value measured on the pinned artifact rather than at a rounder
number nearby — the same rule the two bars in § 4 follow, and the same
semantics: **a regression bar, not a quality claim**. `PASS` means *no worse
than that build of that product*.
**It is a tight bar and the cost is stated rather than hidden:** two of the four
R761 builds measured fall under it, at 2 752 and 2 727 of 2 761 — 0.25 % and
1.2 % below the reference. The class separation is nevertheless wide: **any bar
between 1 148/2 761 (41.6 %) and 2 727/2 761 (98.8 %) separates the known-bad
arm from every R761 build measured.** The shipped bar is the strictest point of
that interval because it is the only one in it that is read off a pinned
artifact instead of off the bundles being judged. Moving it inside that interval
is the operator's call, and the interval is published here so the call can be
made on numbers.
A floor applies here too, in the fasit's own unit:
`MIN_DECLARED_FOR_A_THRESHOLD` is the same 5 as the document floor, so a fasit
of four rows gives `UNMEASURED` rather than a share.
### § 7.5 — what did not change
- **Without `--fasit`, nothing moves.** `okf quality <bundle>` on
`860019-mdb-100` still prints one `.md` row, no boundary row, and exits **3**.
A test holds it, and the word "boundary" does not appear in the per-type half
of the output.
- **No version bump, no tag, no push.** v0.9.0 was already bumped by G37 and is
still untagged; the tag is the operator's.
- `okf check` is untouched. So is `vegnormal-okf`, which was read and not
written.
### § 7.6 — honesty limits of this section
- **N = 1 product.** Every number in § 7.3 is R761's. Nothing here says what
`boundary_share` does on a second declared structure, because no second one
exists in reach.
- **The bar fails builds nobody calls bad.** Named above rather than tuned
away. The alternative — a bar at 2 727 — is a number read off a bundle the
gate judges.
- **A zero is ambiguous by construction.** 0 of 2 761 means *this fasit does not
describe this bundle* on both bundles where it occurred, and the gate cannot
distinguish that from a total segmentation failure. It prints both match-form
counts so a reader can see which it is (a real build of the product scores in
the thousands on one of the two forms).
- **One file was read outside this repository beyond the fasit.** The order said
to read only the fasit from `vegnormal-okf`; deriving the pair-key form
required reading their measuring script
(`experiments/sk2_grenser.py`, read-only). The alternative was to guess the
match rule, which is what P1 exists to forbid. Nothing there was written.
- **`boundary_share` is still not a hit@k.** It asks whether a declared section
became a concept, never whether a question reaches it.
## Honesty limits

313
src/llm_ingestion_okf/quality.py
View file

@ -34,6 +34,7 @@ recovered -- needs a fasit and no bundle-only metric reaches it.
from __future__ import annotations
import argparse
import json
import re
import sys
from collections import Counter
@ -131,6 +132,142 @@ THRESHOLDS: dict[str, Threshold] = {
#: no threshold. FAIL is reachable for every row; PASS is not.
EMPTY_BODY_LIMIT = 0
#: The floor again, in the fasit's own unit. A share over four declared
#: boundaries is not a rate any more than a share over four documents is, and
#: the number is the same honesty limit rather than a second one: it exists to
#: refuse a degenerate fasit, not to rate a corpus.
MIN_DECLARED_FOR_A_THRESHOLD = MIN_DOCUMENTS_FOR_A_THRESHOLD
class FasitError(ValueError):
"""An unreadable or malformed fasit. A run that did not happen, never a verdict.
A `ValueError`, so `main`'s existing handler turns it into exit 2: the one
thing this must never become is a quiet `UNMEASURED` row, which reads as
"no threshold for this" when the truth is "the input was broken".
"""
_WHITESPACE = re.compile(r"\s+")
#: The numbering token STS glues onto the front of a `<title>` ("11.1Fastmerker").
_NUMBERING_TOKEN = re.compile(r"^\s*(\d+(?:\.\d+)*)\s*")
def normalise_title(value: str) -> str:
"""Strip ALL whitespace, then lowercase -- the key the fasit is written on.
Not a guess and not this module's invention: measured over the shipped
2 761-row fasit before any of this was written, the rule reproduces every
row's own `norm` from its own `title`, **2 761 of 2 761**. It is also the
normalisation `vegnormal-okf`'s measuring script applies, so a number
produced here and a number produced there are the same number.
"""
return _WHITESPACE.sub("", value).lower()
def _split_numbering(title: str) -> tuple[str, str]:
"""`("11.1", "Fastmerker")`, or `("", title)` when there is no token."""
match = _NUMBERING_TOKEN.match(title)
return (match.group(1), title[match.end() :].strip()) if match else ("", title.strip())
@dataclass(frozen=True)
class DeclaredBoundary:
"""One boundary the source itself declares, in the two forms it can be met in."""
title: str
norm: str
@property
def pair_key(self) -> tuple[str, str]:
"""`(numbering token as a directory segment, normalised residual title)`."""
number, rest = _split_numbering(self.title)
return (number.replace(".", "-"), normalise_title(rest))
def load_fasit(path: Path) -> tuple[DeclaredBoundary, ...]:
"""The declared boundaries, or a refusal naming what the file is instead.
Validated at the door rather than trusted: a list, every element a mapping,
every mapping carrying `title` and `norm` as strings. Anything else raises,
and `main` turns that into exit 2 with the reason on stderr.
"""
try:
raw = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise FasitError(f"{path} is not JSON: {exc}") from exc
if not isinstance(raw, list):
raise FasitError(f"{path} is a {type(raw).__name__}, not a list of declared boundaries")
rows: list[DeclaredBoundary] = []
for index, entry in enumerate(raw):
if not isinstance(entry, dict):
raise FasitError(f"{path} row {index} is a {type(entry).__name__}, not an object")
title, norm = entry.get("title"), entry.get("norm")
if not isinstance(title, str) or not isinstance(norm, str):
raise FasitError(
f"{path} row {index} carries no `title` and `norm` pair of strings; "
"every row must name the boundary and the key it is matched on"
)
rows.append(DeclaredBoundary(title=title, norm=norm))
if not rows:
raise FasitError(f"{path} declares no boundaries at all")
return tuple(rows)
@dataclass(frozen=True)
class BoundaryThreshold:
"""The one bar that needs a fasit, and the only one pinned to a single product.
Held as the measured PAIR, like every other bar here, so the comparison is
exact integer arithmetic.
"""
metric: str
limit_recovered: int
limit_declared: int
#: Products behind the measurement. **One**, and it is printed on the row
#: rather than only recorded here: the fasit is R761's own NISO-STS
#: structure, so a bar read off it is pinned to one corpus and says nothing
#: about a document nobody has a declared structure for.
corpora: int
source: str
def undercut_by(self, recovered: int, declared: int) -> bool:
"""`recovered/declared` strictly below the reference, without floats."""
return recovered * self.limit_declared < self.limit_recovered * declared
def as_share(self) -> str:
return f"{self.limit_recovered}/{self.limit_declared}"
#: Measured 2026-09-13 on `~/repos/vegnormal-okf/build/ferdig/r761-2025-generisk`,
#: the declared-structure (`.xml`) arm of R761 Prosesskoden:2025, against that
#: publisher's own 2 761 titled `<sec>` elements. Set at the value measured
#: there rather than at a rounder number nearby, exactly like the two bars
#: above -- a REGRESSION bar against a pinned artifact, and a tight one:
#: `docs/2026-09-12-g37-terskler.md` SS 7 records that an older build of the
#: same product (2 752 of 2 761) reads FAIL under it, and says so rather than
#: moving the bar to admit it.
BOUNDARY_THRESHOLD = BoundaryThreshold(
metric="boundary_share",
limit_recovered=2759,
limit_declared=2761,
corpora=1,
source=(
"r761-2025-generisk against sk2-fasit-2761.json (R761 Prosesskoden:2025, "
"2 761 declared STS sections) -- ONE product, N = 1 corpus"
),
)
#: Printed on every boundary row, because P2 of the order that asked for this
#: is a property of the number and not a footnote to it.
SINGLE_CORPUS_CAVEAT = (
"the bar rests on one product, N = 1 corpus, and --fasit is the caller's "
"ASSERTION that this bundle is a build of the document the fasit describes"
)
PASS = "PASS"
FAIL = "FAIL"
UNMEASURED = "UNMEASURED"
@ -177,6 +314,44 @@ class TypeReport:
)
@dataclass(frozen=True)
class BoundaryReport:
"""How many boundaries the source declares became a concept, and by which form.
**Whole bundle, never per file type.** The fasit names the sections of ONE
document; in a bundle those can be spread over 828 source files (they are,
on the arm this metric was built to fell), so attributing the share to a
file type would put a product's number in a type's row.
Both match forms are counted separately and printed, because the
decomposition is the finding: on the known-good arm the literal form alone
reaches 22 of 2 761 and the pair form 2 737, so a gate scoring only the
first would report a 99.9 % arm as 0.8 % and call it a segmentation defect.
"""
declared: int
recovered: int
#: Declared boundaries met by a concept whose normalised title equals the
#: fasit's `norm` -- the form the declared-structure route produces.
literal: int
#: Declared boundaries met by the `(directory, residual title)` pair -- the
#: form okf's default route produces, having moved the numbering token into
#: the concept id.
paired: int
verdict: str
threshold: BoundaryThreshold | None
reason: str
def render(self) -> str:
bar = f"limit {self.threshold.as_share()}" if self.threshold else "no threshold"
return (
f"{'boundary_share':<18} {self.verdict:<11} "
f"recovered {self.recovered}/{self.declared} ({bar}) "
f"literal {self.literal}/{self.declared} paired {self.paired}/{self.declared}"
f" -- {self.reason}"
)
@dataclass(frozen=True)
class BundleQuality:
"""One bundle's rows, its run log if it has one, and the exit code they imply."""
@ -189,6 +364,10 @@ class BundleQuality:
#: a rejected document leaves NO concept in the bundle, so without the log
#: the gate cannot know whether a type failed to extract entirely.
run_log: str | None
#: The whole-bundle boundary row, or `None` when no `--fasit` was given.
#: `None` is the untouched gate: without a fasit this command is exactly
#: what it was, and a test holds that.
boundaries: BoundaryReport | None = None
def row(self, extension: str) -> TypeReport:
for row in self.rows:
@ -206,9 +385,12 @@ class BundleQuality:
third code exists because exit 0 over a table of `UNMEASURED` rows would
be exactly the silent pass this gate was built to stop.
"""
if any(row.verdict == FAIL for row in self.rows):
verdicts = [row.verdict for row in self.rows]
if self.boundaries is not None:
verdicts.append(self.boundaries.verdict)
if FAIL in verdicts:
return 1
if any(row.verdict == PASS for row in self.rows):
if PASS in verdicts:
return 0
return 3
@ -229,6 +411,17 @@ class BundleQuality:
"",
]
lines.extend(row.render() for row in self.rows)
if self.boundaries is not None:
lines.extend(
[
"",
"## Boundary recall against the fasit",
"",
self.boundaries.render(),
"",
f"({SINGLE_CORPUS_CAVEAT})",
]
)
lines.extend(
[
"",
@ -236,9 +429,18 @@ class BundleQuality:
"",
"A regression bar against a pinned reference bundle, per file type.",
"PASS means no worse than that reference on the metrics below; it is",
"not a claim that the cut found the document's own structure. Boundary",
"recall and hit@k need a fasit and are outside a bundle-only gate --",
"docs/2026-09-12-g37-terskler.md carries the measurement that says so.",
"not a claim that the cut found the document's own structure.",
(
"Boundary recall is measured above, against ONE product's declared"
if self.boundaries is not None
else "Boundary recall needs a fasit (--fasit) and hit@k needs a"
),
(
"structure; hit@k still needs a question set and is not asked here."
if self.boundaries is not None
else "question set as well; neither is asked by a bundle-only run."
),
"docs/2026-09-12-g37-terskler.md carries the measurements that say so.",
]
)
return "\n".join(lines) + "\n"
@ -264,7 +466,10 @@ def read_run_log(bundle_root: Path) -> str | None:
def measure_bundle(
bundle_root: Path, *, profile: BundleProfile = SEGMENTED_OKF_V0_2
bundle_root: Path,
*,
profile: BundleProfile = SEGMENTED_OKF_V0_2,
fasit: tuple[DeclaredBoundary, ...] | None = None,
) -> BundleQuality:
"""Every concept the index declares, grouped by the extension it came from.
@ -278,6 +483,8 @@ def measure_bundle(
concepts_per_extension: Counter[str] = Counter()
empty_per_extension: Counter[str] = Counter()
documents: dict[str, Counter[str]] = {}
titles: set[str] = set()
pairs: set[tuple[str, str]] = set()
for concept_id in enumerate_concepts(bundle_root, profile=profile):
concept = read_concept(
bundle_root / f"{concept_id}{profile.paths.concept_suffix}",
@ -289,6 +496,10 @@ def measure_bundle(
documents.setdefault(extension, Counter())[concept.source_file] += 1
if not "".join(concept.body.split()):
empty_per_extension[extension] += 1
if fasit is not None:
normalised = normalise_title(concept.title)
titles.add(normalised)
pairs.add((_enclosing_directory(concept_id), normalised))
rows = tuple(
_verdict(
extension,
@ -303,6 +514,74 @@ def measure_bundle(
bundle_id=root_bundle_id,
rows=rows,
run_log=read_run_log(bundle_root),
boundaries=None if fasit is None else _boundary_verdict(fasit, titles=titles, pairs=pairs),
)
def _enclosing_directory(concept_id: str) -> str:
"""The concept's own immediate directory, or `""` at the bundle root.
The segment okf's default route writes the numbering token into
(`11-1/p3`), which is the half of the pair key the bundle side supplies.
"""
if "/" not in concept_id:
return ""
return concept_id.rsplit("/", 1)[0].rsplit("/", 1)[-1]
def _boundary_verdict(
fasit: tuple[DeclaredBoundary, ...],
*,
titles: set[str],
pairs: set[tuple[str, str]],
) -> BoundaryReport:
"""One declared boundary is recovered when EITHER match form meets it.
Both forms are needed and neither is a fallback for a defect in the other:
the literal form wants the declared title WITH its numbering token, the pair
form wants it WITHOUT, and no bundle can offer both. Scoring one alone
reports the other route's segmentation as near zero -- measured, 22 of 2 761
against 2 737 of 2 761 on the same arm.
"""
declared = len(fasit)
literal = sum(1 for row in fasit if row.norm in titles)
paired = sum(1 for row in fasit if row.pair_key in pairs)
recovered = sum(1 for row in fasit if row.norm in titles or row.pair_key in pairs)
bar = BOUNDARY_THRESHOLD
if declared < MIN_DECLARED_FOR_A_THRESHOLD:
return BoundaryReport(
declared=declared,
recovered=recovered,
literal=literal,
paired=paired,
verdict=UNMEASURED,
threshold=None,
reason=(
f"{declared} declared boundary/boundaries, below the floor of "
f"{MIN_DECLARED_FOR_A_THRESHOLD}: a share over that few is not a rate"
),
)
if bar.undercut_by(recovered, declared):
return BoundaryReport(
declared=declared,
recovered=recovered,
literal=literal,
paired=paired,
verdict=FAIL,
threshold=bar,
reason=(
f"{recovered} of {declared} declared boundaries became a concept, "
f"below the reference {bar.as_share()} ({bar.source})"
),
)
return BoundaryReport(
declared=declared,
recovered=recovered,
literal=literal,
paired=paired,
verdict=PASS,
threshold=bar,
reason=f"no worse than the reference {bar.as_share()} ({bar.source})",
)
@ -381,11 +660,26 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
description=(
"Judge one bundle per file type, with the denominator. Three verdicts: "
"PASS (no worse than the pinned reference), FAIL, and UNMEASURED -- "
"which is never PASS. Exit 0 judged and clean, 1 at least one FAIL, "
"2 did not run, 3 nothing could be judged."
"which is never PASS. With --fasit, one further whole-bundle row: the "
"share of the boundaries the source declares that became a concept. "
"Exit 0 judged and clean, 1 at least one FAIL, 2 did not run, "
"3 nothing could be judged."
),
)
parser.add_argument("bundle", type=Path, help="the OKF bundle to judge")
parser.add_argument(
"--fasit",
type=Path,
default=None,
help=(
"a JSON list of the boundaries the source itself declares, each row "
"carrying `title` and `norm`. Adds one whole-bundle `boundary_share` "
"row and changes nothing else. It is an ASSERTION that this bundle is "
"a build of the document the fasit describes: a bundle of another "
"product scores near zero, which is the assertion being wrong. An "
"unreadable fasit exits 2, never UNMEASURED."
),
)
return parser.parse_args(argv)
@ -395,7 +689,8 @@ def main(argv: list[str] | None = None) -> int:
print(f"{CLI_ID}: FAILED - no such bundle: {args.bundle}", file=sys.stderr)
return 2
try:
report = measure_bundle(args.bundle)
fasit = None if args.fasit is None else load_fasit(args.fasit)
report = measure_bundle(args.bundle, fasit=fasit)
except (ConsumeError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)
return 2

27
tests/test_docs_promises.py
View file

@ -253,3 +253,30 @@ def test_the_readme_quality_section_does_not_promise_a_quality_claim() -> None:
text = README.read_text(encoding="utf-8").lower()
assert "okf quality" in text
assert "regression bar against a pinned artifact" in text
# The `boundary_share` bar. Published in three places -- the code, the README
# and the threshold document -- and a bar published without a test goes false
# the way the format list did.
_BOUNDARY_LINE = re.compile(r"^<!-- quality-boundary-threshold: (.+) -->$", re.MULTILINE)
def test_the_readme_names_the_boundary_bar_the_gate_applies() -> None:
from llm_ingestion_okf.quality import BOUNDARY_THRESHOLD
match = _BOUNDARY_LINE.search(README.read_text(encoding="utf-8"))
assert match is not None, (
"README.md carries no `<!-- quality-boundary-threshold: ... -->` marker; "
"without it the published bar can drift from the one --fasit applies"
)
assert match.group(1).strip() == BOUNDARY_THRESHOLD.as_share()
def test_the_threshold_document_carries_the_boundary_bar_and_its_single_corpus() -> None:
"""N = 1 is half of what this bar is; a copy without it publishes the other half."""
from llm_ingestion_okf.quality import BOUNDARY_THRESHOLD
text = THRESHOLD_DOCUMENT.read_text(encoding="utf-8")
assert f"`{BOUNDARY_THRESHOLD.metric}` | **2 759/2 761**" in text
assert "1 corpus" in text
assert BOUNDARY_THRESHOLD.corpora == 1

270
tests/test_quality.py
View file

@ -16,8 +16,11 @@ read off, live in `docs/2026-09-12-g37-terskler.md`.
from __future__ import annotations
import json
from pathlib import Path
import pytest
from llm_ingestion_okf import quality
_FRONTMATTER = """---
@ -40,6 +43,7 @@ def _bundle(root: Path, concepts: list[tuple[str, str, str]]) -> Path:
root.mkdir(parents=True, exist_ok=True)
lines = ["---", "okf_version: 0.2", "bundle_id: quality-fixture", "---", ""]
for stem, source_file, body in concepts:
(root / f"{stem}.md").parent.mkdir(parents=True, exist_ok=True)
(root / f"{stem}.md").write_text(
_FRONTMATTER.format(title=stem, source_file=source_file, body=body),
encoding="utf-8",
@ -224,3 +228,269 @@ def test_the_no_source_file_row_reports_no_document_count(tmp_path: Path) -> Non
assert "one-concept documents" not in rendered
assert "empty 0/2" in rendered
assert "no source_file on 2 of 2 concepts" in rendered
# --- G37b: `--fasit` and `boundary_share` -------------------------------------
#
# The metric that separates the two arms this gate could not tell apart. It is
# the ONLY one measured that orders them correctly and it needs the publisher's
# own declared structure, so it arrives as an input rather than as a constant:
# `okf quality <bundle> --fasit <json>`.
#
# Two things are pinned here that were measured before any of it was written
# (`docs/2026-09-12-g37-terskler.md` SS 7):
#
# 1. The fasit's `norm` key is reproduced from its own `title` by stripping all
# whitespace and lowercasing -- 2 761 of 2 761 rows, so the normalisation is
# not a guess.
# 2. That normalisation ALONE reaches 22 of 2 761 on the known-good arm, not
# 99 %, because okf's default route moves the numbering token STS glues into
# `<title>` over into the concept id. The second match form -- the
# (directory, residual title) pair -- is what takes it to 2 759 of 2 761.
_GOOD_ARM = Path.home() / "repos/vegnormal-okf/build/ferdig/r761-2025-generisk"
_BAD_ARM = Path.home() / "repos/vegnormal-okf/build/860019-mdb-100"
_OTHER_PRODUCT = Path.home() / "corpora/okf-telling-20260829/K2-bundle-default-20260912"
_FASIT = Path.home() / "repos/vegnormal-okf/build/sk2-fasit-2761.json"
def _fasit(path: Path, titles: list[str]) -> Path:
"""A fasit file in the shipped shape: a list of rows carrying title and norm."""
path.write_text(
json.dumps(
[{"title": title, "norm": quality.normalise_title(title)} for title in titles],
ensure_ascii=False,
),
encoding="utf-8",
newline="",
)
return path
def _declaring(titles: list[str]) -> list[tuple[str, str, str]]:
"""One concept per declared title, matching it literally."""
return [(f"c{index}", "doc.xml", "Body text.") for index, _ in enumerate(titles)]
def test_the_normalisation_reproduces_the_fasit_key() -> None:
"""`norm` is whitespace-stripped, lowercased `title` -- not a guess.
Measured over the shipped fasit before anything was written: 2 761 of 2 761
rows reproduce, and the real file pins it below where it exists.
"""
assert quality.normalise_title(" 2.1 Hoved Prosesser\n") == "2.1hovedprosesser"
assert quality.normalise_title("1Bruksområder for Prosesskoden") == (
"1bruksområderforprosesskoden"
)
def test_a_bundle_recovering_every_declared_boundary_passes(tmp_path: Path) -> None:
"""The known-positive, in miniature: every declared title became a concept."""
titles = [f"{n} Kapittel {n}" for n in range(1, 7)]
root = tmp_path / "recovered"
root.mkdir()
bundle = _bundle(root, [(quality.normalise_title(t)[:12], "doc.xml", "Body.") for t in titles])
# The concept titles ARE the declared titles: rewrite the frontmatter.
for index, title in enumerate(titles):
stem = quality.normalise_title(title)[:12]
(bundle / f"{stem}.md").write_text(
_FRONTMATTER.format(title=title, source_file="doc.xml", body="Body."),
encoding="utf-8",
newline="",
)
report = quality.measure_bundle(
bundle, fasit=quality.load_fasit(_fasit(tmp_path / "f.json", titles))
)
assert report.boundaries is not None
assert (report.boundaries.recovered, report.boundaries.declared) == (6, 6)
assert report.boundaries.verdict == "PASS"
assert report.exit_code == 0
def test_a_bundle_recovering_few_declared_boundaries_fails(tmp_path: Path) -> None:
"""The known-negative for the same rule, and the arm it was built for.
`860019-mdb-100` recovers 1 148 of 2 761. Here five of six is already worse
than the bar of 2 759/2 761 -- the bar is that tight, which is a property of
a regression bar set at its reference and is said out loud in the document.
"""
titles = [f"{n} Kapittel {n}" for n in range(1, 7)]
root = tmp_path / "lost"
root.mkdir()
bundle = _bundle(root, [(f"c{i}", "doc.md", "Body.") for i in range(5)])
for index, title in enumerate(titles[:5]):
(bundle / f"c{index}.md").write_text(
_FRONTMATTER.format(title=title, source_file="doc.md", body="Body."),
encoding="utf-8",
newline="",
)
report = quality.measure_bundle(
bundle, fasit=quality.load_fasit(_fasit(tmp_path / "f.json", titles))
)
assert report.boundaries is not None
assert (report.boundaries.recovered, report.boundaries.declared) == (5, 6)
assert report.boundaries.verdict == "FAIL"
assert report.exit_code == 1
def test_the_numbering_token_is_matched_through_the_concept_directory(tmp_path: Path) -> None:
"""The second match form, and the reason P1's literal reading is not enough.
STS glues the numbering into the title (`11.1Fastmerker`); okf's default
route moves it into the concept id (`11-1/...`) and keeps the residual as
the title. Measured on the known-good arm: the literal form alone reaches
22 of 2 761, the pair form 2 737, either 2 759.
"""
titles = ["11.1Fastmerker"]
bundle = _bundle(tmp_path / "paired", [("11-1/p1", "doc.pdf", "Body.")])
(bundle / "11-1/p1.md").write_text(
_FRONTMATTER.format(title="Fastmerker", source_file="doc.pdf", body="Body."),
encoding="utf-8",
newline="",
)
fasit = quality.load_fasit(_fasit(tmp_path / "f.json", titles))
report = quality.measure_bundle(bundle, fasit=fasit)
assert report.boundaries is not None
assert report.boundaries.literal == 0
assert report.boundaries.paired == 1
assert report.boundaries.recovered == 1
def test_without_a_fasit_the_gate_is_byte_for_byte_what_it_was(tmp_path: Path) -> None:
"""The regression the order asks for: no `--fasit`, no change of any kind."""
root = _bundle(tmp_path / "unchanged", [("a", "note.md", "Body."), ("b", "note.md", "More.")])
report = quality.measure_bundle(root)
assert report.boundaries is None
assert report.exit_code == 3
assert "boundary" not in report.render().split("## What this verdict is not")[0].lower()
def test_a_fasit_that_is_not_a_list_exits_two_rather_than_unmeasured(tmp_path: Path) -> None:
"""An unreadable fasit is a run that did not happen, never a quiet verdict."""
bad = tmp_path / "object.json"
bad.write_text('{"title": "x", "norm": "x"}', encoding="utf-8")
root = _bundle(tmp_path / "b", [("a", "note.md", "Body.")])
assert quality.main([str(root), "--fasit", str(bad)]) == 2
def test_a_fasit_row_missing_a_key_exits_two(tmp_path: Path) -> None:
bad = tmp_path / "rows.json"
bad.write_text('[{"title": "x", "norm": "x"}, {"title": "y"}]', encoding="utf-8")
root = _bundle(tmp_path / "b", [("a", "note.md", "Body.")])
assert quality.main([str(root), "--fasit", str(bad)]) == 2
def test_a_fasit_that_is_not_json_exits_two(tmp_path: Path) -> None:
bad = tmp_path / "broken.json"
bad.write_text("not json at all", encoding="utf-8")
root = _bundle(tmp_path / "b", [("a", "note.md", "Body.")])
assert quality.main([str(root), "--fasit", str(bad)]) == 2
def test_a_missing_fasit_file_exits_two(tmp_path: Path) -> None:
root = _bundle(tmp_path / "b", [("a", "note.md", "Body.")])
assert quality.main([str(root), "--fasit", str(tmp_path / "nowhere.json")]) == 2
def test_a_valid_fasit_reaches_the_gate_through_the_cli(tmp_path: Path, capsys) -> None: # type: ignore[no-untyped-def]
titles = [f"{n} Kapittel {n}" for n in range(1, 7)]
bundle = _bundle(tmp_path / "cli", [(f"c{i}", "doc.md", "Body.") for i in range(1)])
(bundle / "c0.md").write_text(
_FRONTMATTER.format(title=titles[0], source_file="doc.md", body="Body."),
encoding="utf-8",
newline="",
)
from llm_ingestion_okf import cli
assert (
cli.main(["quality", str(bundle), "--fasit", str(_fasit(tmp_path / "f.json", titles))]) == 1
)
out = capsys.readouterr().out
assert "boundary_share" in out
assert "1/6" in out
def test_a_fasit_below_the_floor_is_unmeasured_and_never_pass(tmp_path: Path) -> None:
"""A share over four declared boundaries is not a rate either."""
titles = ["1 A", "2 B", "3 C", "4 D"]
bundle = _bundle(tmp_path / "tiny", [(f"c{i}", "doc.md", "Body.") for i in range(4)])
for index, title in enumerate(titles):
(bundle / f"c{index}.md").write_text(
_FRONTMATTER.format(title=title, source_file="doc.md", body="Body."),
encoding="utf-8",
newline="",
)
report = quality.measure_bundle(
bundle, fasit=quality.load_fasit(_fasit(tmp_path / "f.json", titles))
)
assert report.boundaries is not None
assert report.boundaries.verdict == "UNMEASURED"
assert report.boundaries.recovered == 4
def test_the_boundary_row_says_its_threshold_rests_on_one_corpus(tmp_path: Path) -> None:
"""P2, in the printout rather than only in the document."""
titles = [f"{n} Kapittel {n}" for n in range(1, 7)]
bundle = _bundle(tmp_path / "caveat", [("c0", "doc.md", "Body.")])
(bundle / "c0.md").write_text(
_FRONTMATTER.format(title=titles[0], source_file="doc.md", body="Body."),
encoding="utf-8",
newline="",
)
rendered = quality.measure_bundle(
bundle, fasit=quality.load_fasit(_fasit(tmp_path / "f.json", titles))
).render()
assert "one product" in rendered
assert "N = 1" in rendered
def test_the_boundary_threshold_names_its_corpus_and_denominator() -> None:
bar = quality.BOUNDARY_THRESHOLD
assert bar.limit_declared >= quality.MIN_DECLARED_FOR_A_THRESHOLD
assert bar.corpora == 1
assert bar.source
@pytest.mark.skipif(not (_GOOD_ARM.is_dir() and _FASIT.is_file()), reason="local gold set absent")
def test_the_known_good_arm_reproduces_its_published_recall() -> None:
"""2 759 of 2 761, and the decomposition P1 asked for: 22 literal, 2 737 paired."""
report = quality.measure_bundle(_GOOD_ARM, fasit=quality.load_fasit(_FASIT))
assert report.boundaries is not None
assert (report.boundaries.recovered, report.boundaries.declared) == (2759, 2761)
assert (report.boundaries.literal, report.boundaries.paired) == (22, 2737)
assert report.boundaries.verdict == "PASS"
@pytest.mark.skipif(not (_BAD_ARM.is_dir() and _FASIT.is_file()), reason="local gold set absent")
def test_the_known_bad_arm_reproduces_its_published_recall() -> None:
"""1 148 of 2 761 -- the 41.6 % the whole gate could not previously see."""
report = quality.measure_bundle(_BAD_ARM, fasit=quality.load_fasit(_FASIT))
assert report.boundaries is not None
assert (report.boundaries.recovered, report.boundaries.declared) == (1148, 2761)
assert report.boundaries.verdict == "FAIL"
assert report.exit_code == 1
@pytest.mark.skipif(
not (_OTHER_PRODUCT.is_dir() and _FASIT.is_file()), reason="local gold set absent"
)
def test_a_bundle_of_another_product_scores_zero_and_that_is_the_assertion(tmp_path: Path) -> None:
"""The adverse case, measured rather than argued.
The fasit describes ONE product. The pinned 43-document K2 bundle is a
different corpus entirely and scores 0 of 2 761 -- a FAIL that says the
caller's assertion was wrong, not that the bundle is bad. `--fasit` is an
assertion by the caller, the way `okf consume --ref` is.
"""
report = quality.measure_bundle(_OTHER_PRODUCT, fasit=quality.load_fasit(_FASIT))
assert report.boundaries is not None
assert report.boundaries.recovered == 0
assert report.boundaries.verdict == "FAIL"
@pytest.mark.skipif(not _FASIT.is_file(), reason="local gold set absent")
def test_the_shipped_fasit_reproduces_its_own_key_under_this_normalisation() -> None:
"""2 761 of 2 761 rows: `norm` is exactly whitespace-stripped, lowercased title."""
rows = quality.load_fasit(_FASIT)
assert len(rows) == 2761
assert sum(1 for row in rows if quality.normalise_title(row.title) == row.norm) == 2761