Kjell Tore Guttormsen
6ea8fcd3c7
G37. `okf check` is a CONTRACT check and a green one is not a quality gate: measured 2026-09-10 by `vegnormal-okf`, three arms over one corpus all returned 0 findings and exit 0 while their hit@k ranged from 6 of 6 to 0 of 6. `okf quality <bundle>` asks the other question, per file type, with the denominator on every line. A separate command rather than `okf check --quality`, because the two answer different questions and a caller must not be able to read one as the other. `okf check` is untouched. Three verdicts and no fourth -- PASS, FAIL, UNMEASURED -- and a type with no measured threshold is never PASS. Exit 0 judged and clean, 1 at least one FAIL, 2 did not run, 3 nothing could be judged: exit 0 over a table of unmeasured rows would be the silent pass this command exists to stop. Two bars today, both `structure_null_share` (documents of a type yielding exactly one concept), read off the pinned 43-document reference bundle: .pdf 8/32, .docx 2/5. Plus one definitional bar for every type, taken from the harness's own degenerate-merge definition: 0 concepts with an empty body, measured 0 of 8 602 concepts over four bundles. A bar needs five documents on BOTH sides -- its own and the judged bundle's -- so .xlsx (2), .xml (1) and every type with no corpus class in `extract._EVIDENCE` are UNMEASURED and print their numbers without a verdict. The floor on the judged bundle was found by RUNNING the gate, not by reading it: one PDF cut into 2 182 concepts scored 0 of 1 against the 32-document reference and read as PASS. The gate walks the index tree and never a directory (SS 9.2; controlled against the listing on four bundles, 453 / 2 761 / 3 206 / 446 either way), and prints the bundle's own run log beside its counts -- a document rejected at extraction leaves no row in the bundle, so the pinned corpus's 33 PDFs show up as 32 and the two denominators must never be read as one. Three of the order's five premises moved when re-measured, and they are in the document rather than glossed: the four evidence corpora carry `source_file` on 0 of 446, 0 of 1 133, 0 of 270 and 0 of 2 756 concepts, so they name no file type and cannot PASS; "41,6 %" is `vegnormal-okf`'s number and not in this repository; and the same 828-document bundle carries two published hit@k figures from two question sets. Three candidate metrics measured and NOT shipped: duplicate titles within a document (0 of 3 206 on the known-bad arm against 349 of 2 761 on the known-good one) and short concepts (5.6 % against 14.6 %) order the two arms the wrong way round; duplicate titles across the whole bundle order all four correctly (37.8 / 16.3 / 12.6 / 5.7 %) and still ship without a bar, because any bar separating them is read off the two bundles it would judge. 19 new tests, each rule exercised in both directions; the three README pins were each driven red before being kept. Suite 1 850 passed, 1 skipped, 1 851 collected, run after `git add` -- +19 against a base of 1 832 collected, measured on the stashed tree (STATE's 1 831 is one short of that). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
11 KiB
G37 — the thresholds, per file type, before the gate
Order 20260912T202210Z-7596115025-from-.claude, part 1. Base d310dbb
(v0.8.5). Measured 2026-09-12 evening local time; the session crossed midnight,
so the commit lands 2026-09-13. Python 3.14.0, guard 1.4.0, no model call and
no network anywhere in what follows.
The order's own words for what this part must produce: "tabell filtype ×
metrikk × terskel × N × kilde", metrics restricted to what is already measured
or measurable without a model, and "en filtype uten måling får terskel
UNMEASURED, aldri et gjettet tall".
§ 0 — measured / not measured
Measured here. Every count in §§ 2–5: concept and document counts per file
type over nine bundles, one-concept-document shares, empty-body counts,
duplicate-title shares in two forms, short-concept shares, and which of the
four evidence corpora carry a source_file key at all. Each is a shell or
okf quality command printed beside it.
Not measured here. Boundary recall against a fasit (needs a fasit; § 6),
hit@k (needs a question set and a fasit; § 6), anything about .txt, .csv,
.json, .htm, .pptx, .odt, .rtf beyond the fixtures already in the
tree, and whether any threshold below is the right bar — ratification is the
operator's, and this document sets regression bars against a pinned artifact,
which is a different thing (§ 4).
§ 1 — the three metric names, defined here because nothing defines them
The order names "grenseandel", "struktur-null-andel", "tomme konsepter",
Bundle.skipped and "hit@k". Two of those five have no definition anywhere in
this repository:
Measured on the base commit rather than on the working tree, because this document and the module beside it introduce both words:
git grep -c "grenseandel\|boundary share" d310dbb -- docs src→ 0 filesgit grep -ci "struktur.null\|structure.null" d310dbb -- docs src→ 0 files- known-positive control for the query form:
git grep -c "hit@8" d310dbb -- docs→ 19 files, so it can find.
So the definitions below are mine, written here so the gate and this document mean the same thing by them:
| name | definition | what it needs |
|---|---|---|
boundary_share |
recovered boundaries ÷ boundaries the source itself declares | a fasit |
structure_null_share |
documents of a type yielding exactly ONE concept ÷ documents of that type in the bundle | the bundle alone |
empty_concepts |
concepts whose body holds zero non-whitespace characters ÷ concepts of that type | the bundle alone |
hit@k |
questions whose gold document reaches the payload at cut k ÷ questions |
a fasit and a question set |
| run log | N, merged and coded rejections, from the bundle's own log.md |
a bundle Door B wrote |
Bundle.skipped does not exist under that name: the door's counts reach a
bundle through corpus.CorpusReport, whose written form is the run log above
(N, extracted, gated, persisted, substantive, degenerate,
rejected, plus per-code counts). It is a whole-bundle figure and carries
no file type, which is the limit in § 5.
§ 2 — the order's own numbers, re-measured first
The coordination note that arrived with the order asked for exactly this, and three of the five premises moved.
| premise in the order | re-measured | verdict |
|---|---|---|
| 13 extensions in the registry | _CORE_EXTRACTORS 7 + _OPTIONAL_EXTRACTORS 6 = 13 |
holds |
four evidence corpora under ~/repos/vegnormal-okf/build/ferdig/ |
present: n100-2023, n200-2024, n500-2024, r761-2025 — plus r761-2025-generisk and r761-2025-v080, two more R761 builds the order does not name |
holds, and is incomplete |
| "N100 skal gi PASS på de målte typene" | `grep -rl '^source_file:' n100-2023 --include='*.md' | wc -l→ **0 of 446 concepts**. Same forn200-2024(0 of 1 133),n500-2024(0 of 270) andr761-2025` (0 of 2 756) |
| "41,6 % grenser" | not in this repository (grep -rn "41[.,]6" docs/ src/ → 0). It is vegnormal-okf's: docs/2026-09-10-r2-r761-produkt.md:452, 1 148 of 2 761 declared STS sections, against the XML arm's 2 759 of 2 761 |
holds, and the owner is another repo |
| "0 av 6 hit@k" on the HTML arm | two published numbers for the same 828-document bundle: 0 of 6 (vegnormal, their six fasit questions, 10.09) and 2 of 6 → 4 of 6 with --source-quota 2 (this repo, docs/2026-09-10-k3-runde11-html-og-dominans.md:200,217, questions S1–S6) |
holds for one instrument, not for the other |
The last row is the reason the gate reports a denominator on every line: two true numbers about one bundle, four apart, because the question set differs.
§ 3 — what each corpus can answer
Command: okf quality <bundle> (part 2 of this order), cross-checked against
grep -rh '^source_file:' <bundle> --include='*.md' | sed 's/.*\.//' | sort | uniq -c. The index walk and the directory listing agree on 453, 2 761, 3 206
and 446 concepts — four bundles, both methods.
| bundle | producer | concepts | file types present |
|---|---|---|---|
K2-bundle-default-20260912 |
Door B (pinned reference) | 453 | .pdf 32 docs, .docx 5, .xlsx 2 |
r761-2025-generisk |
Door B | 2 761 | .xml 1 doc |
sk2-bundle-default |
Door B | 2 182 | .pdf 1 doc |
860019-mdb-100 |
Door B | 3 206 | .md 828 docs |
n100-2023 |
vegnormal's own pipeline | 446 | none — no source_file |
n200-2024 |
vegnormal's own pipeline | 1 133 | none |
n500-2024 |
vegnormal's own pipeline | 270 | none |
r761-2025 |
vegnormal's own pipeline | 2 756 | none |
b3 |
— | — | no index.md; the gate refuses to walk a directory (SS 9.2) and exits 2 |
The four corpora the order names as the evidence base carry no file type. The per-file-type thresholds below therefore rest on the K2 reference bundle and, for context only, on the three single-type bundles.
§ 4 — the table
Threshold semantics, stated before the numbers: a threshold here is a
regression bar against a pinned artifact, set at the value measured on that
artifact and never at a rounder number nearby. PASS means no worse than the
reference on this metric. It is not a claim that the bundle is good, and it is
not the operator's ratified quality bar — the K3 method refuses to set one
inside the measurement that produces it ("setting one inside the same work that
produces the first measurement would be fitting the bar to the number").
A rate needs a denominator. MIN_DOCUMENTS_FOR_A_THRESHOLD = 5, and it binds
both the threshold's N and the judged bundle's: this repository's own honesty
limit is "a 1/1 is not a rate" (docs/2026-09-08-k3-runde2-per-filtype.md).
| file type | metric | threshold | N | source |
|---|---|---|---|---|
| every type | empty_concepts |
0 | 8 602 concepts over 4 Door B bundles, all 0 | definitional: corpus.CorpusReport calls a zero-character merge degenerate |
.pdf |
structure_null_share |
8/32 | 32 documents | K2-bundle-default-20260912 |
.docx |
structure_null_share |
2/5 | 5 documents | K2-bundle-default-20260912 |
.xlsx |
— | UNMEASURED | 2 documents | below the floor of 5 |
.xml |
— | UNMEASURED | 1 document | below the floor of 5 |
.md |
— | UNMEASURED | 828 documents, structure_null_share 362/828 measured |
the corpus is one generator's cut of one product, and it is the bundle under suspicion; a bar read off it would pass it by construction |
.html |
— | UNMEASURED | 0 bundles in this repository | the 828-document HTML measurement was made outside it |
.htm, .txt, .csv, .json, .pptx, .odt, .rtf |
— | UNMEASURED | 0 corpus documents | no corpus class in extract._EVIDENCE; .pptx/.odt/.rtf are constructed, N = 1–2 |
Candidates measured and not shipped
Three fasit-free metrics were measured over the same four Door B bundles, with
the known-bad arm (860019-mdb-100, 1 148 of 2 761 boundaries) and the
known-good arm (r761-2025-generisk, 2 759 of 2 761) named in advance:
| candidate | known-bad arm | known-good arm | pdf arm | K2 reference | verdict |
|---|---|---|---|---|---|
| duplicate titles within a document | 0/3 206 (0.0 %) | 349/2 761 (12.6 %) | 355/2 182 | 11/453 | wrong direction |
| concepts under 40 characters | 181/3 206 (5.6 %) | 402/2 761 (14.6 %) | 1/2 182 | 25/453 | wrong direction |
| duplicate titles across the whole bundle | 1 211/3 206 (37.8 %) | 349/2 761 (12.6 %) | 355/2 182 (16.3 %) | 26/453 (5.7 %) | orders correctly, no bar |
The third orders all four bundles correctly and is still not shipped: any bar that separates them sits between 16.3 % and 37.8 %, which is a number read off the two bundles it would judge. That is the operator's call and it is stated in STATE rather than taken here.
§ 5 — three limits that are properties of the instrument
- A rejected document leaves no trace in the bundle. The pinned K2 bundle
holds 32
.pdfdocuments; the corpus held 33, and one was rejectedextractor_empty_pdf. The gate's denominator is the bundle's, never the corpus's, and it prints the run log (N = 43, merged = 39, coded rejections = 4) beside its own numbers so the two are never read as one. A bundle without a log says "no run log" rather than reporting zero rejections. - The run log carries no file type.
codesis per rejection code, not per extension, so "this type failed to extract entirely" is invisible to a bundle-only gate. Naming that would be a change incorpus.py, not here. - The reference bundle passes its own bars by construction.
okf qualityonK2-bundle-default-20260912returningPASSon.pdfand.docxis a control that the gate reproduces its own reference — nothing more.
§ 6 — what would reach the defect that started this
The 41.6 % arm is UNMEASURED under this gate, and that is the honest answer
rather than a hidden one: the metric that fells it is boundary_share, which
needs the publisher's own declared structure. R761 has one (2 761 STS <sec>
titles, ~/repos/vegnormal-okf/build/sk2-fasit-2761.json), and 4 of the 9
bundles above have no fasit at all. A --fasit input, a threshold on
boundary_share, and the operator's bar are one further round, named in STATE
and not begun here.
Honesty limits
- Two thresholds, and one of them rests on five documents.
.docxat 2/5 is exactly at the floor this document sets for itself. It is a bar against a pinned artifact, not a rate anyone should quote. structure_null_sharehas never fired on a real bundle. Of the five Door B bundles measured, none is worse than its reference; the rule'sFAILside is exercised on constructed bundles intests/test_quality.pyonly.empty_conceptsis 0 on 8 602 of 8 602 concepts. The bar is definitional and has never fired on real data either. A metric that never fires is not evidence that nothing is wrong.- One rule was found by running the gate, not by reading it. A one-document
PDF bundle scored 0 of 1 against the 32-document reference and read as
PASS. The floor now binds both denominators; the version of this document written before that run would have published the wrong rule. - Nothing here measures answer quality. hit@k asks whether a gold document reached the payload; this gate does not even ask that.