open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(quality): okf quality --fasit, boundary recall against a declared structure

Browse source 
The bundle-only gate returned UNMEASURED and exit 3 on the very arm it was
built for: no metric computable from a bundle alone reaches boundary recall.
`boundary_share` -- declared boundaries that became a concept, over declared
boundaries -- is the one metric measured that orders the arms correctly, and it
needs the publisher's own structure, so it arrives as an input.

Measurement first, threshold after, which is what the order asked for.

P1, the normalisation, derived rather than guessed: stripping all whitespace
and lowercasing reproduces the fasit's own `norm` from its own `title` on
2 761 of 2 761 rows (alphanumerics-only scores 58). P1's own bar is 99 % on the
known-good arm and the literal reading of it reaches 22 of 2 761 -- not because
the normalisation is wrong but because okf's default route moves the numbering
token a publisher glues into a heading over into the concept id. The pair form
(concept's own directory, residual title) reaches 2 737, either reaches 2 759
(99.9 %). Both forms ship and neither is a fallback: `r761-2025-d1` is the
control in the opposite direction at 2 727 literal, 0 paired.

P2, the single corpus, is in the OUTPUT and not only in the document: the bar
is declared `corpora = 1`, every boundary row prints `N = 1 corpus`, and the
line states that `--fasit` is the caller's ASSERTION that this bundle is a
build of the document the fasit describes -- the posture `okf consume --ref`
has. Measured: the K2 reference and `n100-2023` score 0 of 2 761 and read FAIL,
which is the assertion being wrong rather than the bundle being bad.

One bar, at the pinned artifact's own value, 2 759/2 761. It is tight and the
cost is published rather than tuned away: 2 of 4 R761 builds fall under it
(2 752 and 2 727), while any bar between 41.6 % and 98.8 % separates the
known-bad arm from every R761 build measured. The known-bad arm
(`860019-mdb-100`) is 1 148 of 2 761 -- FAIL and exit 1, where the bundle-only
gate gave exit 3.

A fasit is validated at the door: not a list, a row missing `title` or `norm`,
or anything that is not JSON exits 2 with the reason, never a quiet UNMEASURED.
A fasit under five rows is UNMEASURED -- the document floor in the fasit's own
unit.

Without `--fasit` the command is byte-for-byte what it was, held by a test.
`okf check` is untouched; no version bump and no tag. 17 tests red on
assertions before the implementation, the two new doc pins each driven red and
back. Suite 1 869 passed / 1 skipped / 1 870 collected (base 5e5d01c: 1 851).

docs/2026-09-12-g37-terskler.md SS 7 carries the premises re-measured, the
seven bundles, the interval any bar could sit in, and the honesty limits --
including the correction of SS 2's own grep claim, which went false in the
commit that wrote it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-13 07:27:33 +02:00
commit b6da09cc97
7 changed files with 868 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

313
src/llm_ingestion_okf/quality.py
View file

@ -34,6 +34,7 @@ recovered -- needs a fasit and no bundle-only metric reaches it.
from __future__ import annotations
import argparse
import json
import re
import sys
from collections import Counter
@ -131,6 +132,142 @@ THRESHOLDS: dict[str, Threshold] = {
#: no threshold. FAIL is reachable for every row; PASS is not.
EMPTY_BODY_LIMIT = 0
#: The floor again, in the fasit's own unit. A share over four declared
#: boundaries is not a rate any more than a share over four documents is, and
#: the number is the same honesty limit rather than a second one: it exists to
#: refuse a degenerate fasit, not to rate a corpus.
MIN_DECLARED_FOR_A_THRESHOLD = MIN_DOCUMENTS_FOR_A_THRESHOLD
class FasitError(ValueError):
"""An unreadable or malformed fasit. A run that did not happen, never a verdict.
A `ValueError`, so `main`'s existing handler turns it into exit 2: the one
thing this must never become is a quiet `UNMEASURED` row, which reads as
"no threshold for this" when the truth is "the input was broken".
"""
_WHITESPACE = re.compile(r"\s+")
#: The numbering token STS glues onto the front of a `<title>` ("11.1Fastmerker").
_NUMBERING_TOKEN = re.compile(r"^\s*(\d+(?:\.\d+)*)\s*")
def normalise_title(value: str) -> str:
"""Strip ALL whitespace, then lowercase -- the key the fasit is written on.
Not a guess and not this module's invention: measured over the shipped
2 761-row fasit before any of this was written, the rule reproduces every
row's own `norm` from its own `title`, **2 761 of 2 761**. It is also the
normalisation `vegnormal-okf`'s measuring script applies, so a number
produced here and a number produced there are the same number.
"""
return _WHITESPACE.sub("", value).lower()
def _split_numbering(title: str) -> tuple[str, str]:
"""`("11.1", "Fastmerker")`, or `("", title)` when there is no token."""
match = _NUMBERING_TOKEN.match(title)
return (match.group(1), title[match.end() :].strip()) if match else ("", title.strip())
@dataclass(frozen=True)
class DeclaredBoundary:
"""One boundary the source itself declares, in the two forms it can be met in."""
title: str
norm: str
@property
def pair_key(self) -> tuple[str, str]:
"""`(numbering token as a directory segment, normalised residual title)`."""
number, rest = _split_numbering(self.title)
return (number.replace(".", "-"), normalise_title(rest))
def load_fasit(path: Path) -> tuple[DeclaredBoundary, ...]:
"""The declared boundaries, or a refusal naming what the file is instead.
Validated at the door rather than trusted: a list, every element a mapping,
every mapping carrying `title` and `norm` as strings. Anything else raises,
and `main` turns that into exit 2 with the reason on stderr.
"""
try:
raw = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise FasitError(f"{path} is not JSON: {exc}") from exc
if not isinstance(raw, list):
raise FasitError(f"{path} is a {type(raw).__name__}, not a list of declared boundaries")
rows: list[DeclaredBoundary] = []
for index, entry in enumerate(raw):
if not isinstance(entry, dict):
raise FasitError(f"{path} row {index} is a {type(entry).__name__}, not an object")
title, norm = entry.get("title"), entry.get("norm")
if not isinstance(title, str) or not isinstance(norm, str):
raise FasitError(
f"{path} row {index} carries no `title` and `norm` pair of strings; "
"every row must name the boundary and the key it is matched on"
)
rows.append(DeclaredBoundary(title=title, norm=norm))
if not rows:
raise FasitError(f"{path} declares no boundaries at all")
return tuple(rows)
@dataclass(frozen=True)
class BoundaryThreshold:
"""The one bar that needs a fasit, and the only one pinned to a single product.
Held as the measured PAIR, like every other bar here, so the comparison is
exact integer arithmetic.
"""
metric: str
limit_recovered: int
limit_declared: int
#: Products behind the measurement. **One**, and it is printed on the row
#: rather than only recorded here: the fasit is R761's own NISO-STS
#: structure, so a bar read off it is pinned to one corpus and says nothing
#: about a document nobody has a declared structure for.
corpora: int
source: str
def undercut_by(self, recovered: int, declared: int) -> bool:
"""`recovered/declared` strictly below the reference, without floats."""
return recovered * self.limit_declared < self.limit_recovered * declared
def as_share(self) -> str:
return f"{self.limit_recovered}/{self.limit_declared}"
#: Measured 2026-09-13 on `~/repos/vegnormal-okf/build/ferdig/r761-2025-generisk`,
#: the declared-structure (`.xml`) arm of R761 Prosesskoden:2025, against that
#: publisher's own 2 761 titled `<sec>` elements. Set at the value measured
#: there rather than at a rounder number nearby, exactly like the two bars
#: above -- a REGRESSION bar against a pinned artifact, and a tight one:
#: `docs/2026-09-12-g37-terskler.md` SS 7 records that an older build of the
#: same product (2 752 of 2 761) reads FAIL under it, and says so rather than
#: moving the bar to admit it.
BOUNDARY_THRESHOLD = BoundaryThreshold(
metric="boundary_share",
limit_recovered=2759,
limit_declared=2761,
corpora=1,
source=(
"r761-2025-generisk against sk2-fasit-2761.json (R761 Prosesskoden:2025, "
"2 761 declared STS sections) -- ONE product, N = 1 corpus"
),
)
#: Printed on every boundary row, because P2 of the order that asked for this
#: is a property of the number and not a footnote to it.
SINGLE_CORPUS_CAVEAT = (
"the bar rests on one product, N = 1 corpus, and --fasit is the caller's "
"ASSERTION that this bundle is a build of the document the fasit describes"
)
PASS = "PASS"
FAIL = "FAIL"
UNMEASURED = "UNMEASURED"
@ -177,6 +314,44 @@ class TypeReport:
)
@dataclass(frozen=True)
class BoundaryReport:
"""How many boundaries the source declares became a concept, and by which form.
**Whole bundle, never per file type.** The fasit names the sections of ONE
document; in a bundle those can be spread over 828 source files (they are,
on the arm this metric was built to fell), so attributing the share to a
file type would put a product's number in a type's row.
Both match forms are counted separately and printed, because the
decomposition is the finding: on the known-good arm the literal form alone
reaches 22 of 2 761 and the pair form 2 737, so a gate scoring only the
first would report a 99.9 % arm as 0.8 % and call it a segmentation defect.
"""
declared: int
recovered: int
#: Declared boundaries met by a concept whose normalised title equals the
#: fasit's `norm` -- the form the declared-structure route produces.
literal: int
#: Declared boundaries met by the `(directory, residual title)` pair -- the
#: form okf's default route produces, having moved the numbering token into
#: the concept id.
paired: int
verdict: str
threshold: BoundaryThreshold | None
reason: str
def render(self) -> str:
bar = f"limit {self.threshold.as_share()}" if self.threshold else "no threshold"
return (
f"{'boundary_share':<18} {self.verdict:<11} "
f"recovered {self.recovered}/{self.declared} ({bar}) "
f"literal {self.literal}/{self.declared} paired {self.paired}/{self.declared}"
f" -- {self.reason}"
)
@dataclass(frozen=True)
class BundleQuality:
"""One bundle's rows, its run log if it has one, and the exit code they imply."""
@ -189,6 +364,10 @@ class BundleQuality:
#: a rejected document leaves NO concept in the bundle, so without the log
#: the gate cannot know whether a type failed to extract entirely.
run_log: str | None
#: The whole-bundle boundary row, or `None` when no `--fasit` was given.
#: `None` is the untouched gate: without a fasit this command is exactly
#: what it was, and a test holds that.
boundaries: BoundaryReport | None = None
def row(self, extension: str) -> TypeReport:
for row in self.rows:
@ -206,9 +385,12 @@ class BundleQuality:
third code exists because exit 0 over a table of `UNMEASURED` rows would
be exactly the silent pass this gate was built to stop.
"""
if any(row.verdict == FAIL for row in self.rows):
verdicts = [row.verdict for row in self.rows]
if self.boundaries is not None:
verdicts.append(self.boundaries.verdict)
if FAIL in verdicts:
return 1
if any(row.verdict == PASS for row in self.rows):
if PASS in verdicts:
return 0
return 3
@ -229,6 +411,17 @@ class BundleQuality:
"",
]
lines.extend(row.render() for row in self.rows)
if self.boundaries is not None:
lines.extend(
[
"",
"## Boundary recall against the fasit",
"",
self.boundaries.render(),
"",
f"({SINGLE_CORPUS_CAVEAT})",
]
)
lines.extend(
[
"",
@ -236,9 +429,18 @@ class BundleQuality:
"",
"A regression bar against a pinned reference bundle, per file type.",
"PASS means no worse than that reference on the metrics below; it is",
"not a claim that the cut found the document's own structure. Boundary",
"recall and hit@k need a fasit and are outside a bundle-only gate --",
"docs/2026-09-12-g37-terskler.md carries the measurement that says so.",
"not a claim that the cut found the document's own structure.",
(
"Boundary recall is measured above, against ONE product's declared"
if self.boundaries is not None
else "Boundary recall needs a fasit (--fasit) and hit@k needs a"
),
(
"structure; hit@k still needs a question set and is not asked here."
if self.boundaries is not None
else "question set as well; neither is asked by a bundle-only run."
),
"docs/2026-09-12-g37-terskler.md carries the measurements that say so.",
]
)
return "\n".join(lines) + "\n"
@ -264,7 +466,10 @@ def read_run_log(bundle_root: Path) -> str | None:
def measure_bundle(
bundle_root: Path, *, profile: BundleProfile = SEGMENTED_OKF_V0_2
bundle_root: Path,
*,
profile: BundleProfile = SEGMENTED_OKF_V0_2,
fasit: tuple[DeclaredBoundary, ...] | None = None,
) -> BundleQuality:
"""Every concept the index declares, grouped by the extension it came from.
@ -278,6 +483,8 @@ def measure_bundle(
concepts_per_extension: Counter[str] = Counter()
empty_per_extension: Counter[str] = Counter()
documents: dict[str, Counter[str]] = {}
titles: set[str] = set()
pairs: set[tuple[str, str]] = set()
for concept_id in enumerate_concepts(bundle_root, profile=profile):
concept = read_concept(
bundle_root / f"{concept_id}{profile.paths.concept_suffix}",
@ -289,6 +496,10 @@ def measure_bundle(
documents.setdefault(extension, Counter())[concept.source_file] += 1
if not "".join(concept.body.split()):
empty_per_extension[extension] += 1
if fasit is not None:
normalised = normalise_title(concept.title)
titles.add(normalised)
pairs.add((_enclosing_directory(concept_id), normalised))
rows = tuple(
_verdict(
extension,
@ -303,6 +514,74 @@ def measure_bundle(
bundle_id=root_bundle_id,
rows=rows,
run_log=read_run_log(bundle_root),
boundaries=None if fasit is None else _boundary_verdict(fasit, titles=titles, pairs=pairs),
)
def _enclosing_directory(concept_id: str) -> str:
"""The concept's own immediate directory, or `""` at the bundle root.
The segment okf's default route writes the numbering token into
(`11-1/p3`), which is the half of the pair key the bundle side supplies.
"""
if "/" not in concept_id:
return ""
return concept_id.rsplit("/", 1)[0].rsplit("/", 1)[-1]
def _boundary_verdict(
fasit: tuple[DeclaredBoundary, ...],
*,
titles: set[str],
pairs: set[tuple[str, str]],
) -> BoundaryReport:
"""One declared boundary is recovered when EITHER match form meets it.
Both forms are needed and neither is a fallback for a defect in the other:
the literal form wants the declared title WITH its numbering token, the pair
form wants it WITHOUT, and no bundle can offer both. Scoring one alone
reports the other route's segmentation as near zero -- measured, 22 of 2 761
against 2 737 of 2 761 on the same arm.
"""
declared = len(fasit)
literal = sum(1 for row in fasit if row.norm in titles)
paired = sum(1 for row in fasit if row.pair_key in pairs)
recovered = sum(1 for row in fasit if row.norm in titles or row.pair_key in pairs)
bar = BOUNDARY_THRESHOLD
if declared < MIN_DECLARED_FOR_A_THRESHOLD:
return BoundaryReport(
declared=declared,
recovered=recovered,
literal=literal,
paired=paired,
verdict=UNMEASURED,
threshold=None,
reason=(
f"{declared} declared boundary/boundaries, below the floor of "
f"{MIN_DECLARED_FOR_A_THRESHOLD}: a share over that few is not a rate"
),
)
if bar.undercut_by(recovered, declared):
return BoundaryReport(
declared=declared,
recovered=recovered,
literal=literal,
paired=paired,
verdict=FAIL,
threshold=bar,
reason=(
f"{recovered} of {declared} declared boundaries became a concept, "
f"below the reference {bar.as_share()} ({bar.source})"
),
)
return BoundaryReport(
declared=declared,
recovered=recovered,
literal=literal,
paired=paired,
verdict=PASS,
threshold=bar,
reason=f"no worse than the reference {bar.as_share()} ({bar.source})",
)
@ -381,11 +660,26 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
description=(
"Judge one bundle per file type, with the denominator. Three verdicts: "
"PASS (no worse than the pinned reference), FAIL, and UNMEASURED -- "
"which is never PASS. Exit 0 judged and clean, 1 at least one FAIL, "
"2 did not run, 3 nothing could be judged."
"which is never PASS. With --fasit, one further whole-bundle row: the "
"share of the boundaries the source declares that became a concept. "
"Exit 0 judged and clean, 1 at least one FAIL, 2 did not run, "
"3 nothing could be judged."
),
)
parser.add_argument("bundle", type=Path, help="the OKF bundle to judge")
parser.add_argument(
"--fasit",
type=Path,
default=None,
help=(
"a JSON list of the boundaries the source itself declares, each row "
"carrying `title` and `norm`. Adds one whole-bundle `boundary_share` "
"row and changes nothing else. It is an ASSERTION that this bundle is "
"a build of the document the fasit describes: a bundle of another "
"product scores near zero, which is the assertion being wrong. An "
"unreadable fasit exits 2, never UNMEASURED."
),
)
return parser.parse_args(argv)
@ -395,7 +689,8 @@ def main(argv: list[str] | None = None) -> int:
print(f"{CLI_ID}: FAILED - no such bundle: {args.bundle}", file=sys.stderr)
return 2
try:
report = measure_bundle(args.bundle)
fasit = None if args.fasit is None else load_fasit(args.fasit)
report = measure_bundle(args.bundle, fasit=fasit)
except (ConsumeError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)
return 2