feat(quality): okf quality, a per-file-type verdict with the denominator

G37. `okf check` is a CONTRACT check and a green one is not a quality gate:
measured 2026-09-10 by `vegnormal-okf`, three arms over one corpus all
returned 0 findings and exit 0 while their hit@k ranged from 6 of 6 to 0 of 6.
`okf quality <bundle>` asks the other question, per file type, with the
denominator on every line. A separate command rather than `okf check
--quality`, because the two answer different questions and a caller must not
be able to read one as the other. `okf check` is untouched.

Three verdicts and no fourth -- PASS, FAIL, UNMEASURED -- and a type with no
measured threshold is never PASS. Exit 0 judged and clean, 1 at least one
FAIL, 2 did not run, 3 nothing could be judged: exit 0 over a table of
unmeasured rows would be the silent pass this command exists to stop.

Two bars today, both `structure_null_share` (documents of a type yielding
exactly one concept), read off the pinned 43-document reference bundle: .pdf
8/32, .docx 2/5. Plus one definitional bar for every type, taken from the
harness's own degenerate-merge definition: 0 concepts with an empty body,
measured 0 of 8 602 concepts over four bundles. A bar needs five documents on
BOTH sides -- its own and the judged bundle's -- so .xlsx (2), .xml (1) and
every type with no corpus class in `extract._EVIDENCE` are UNMEASURED and
print their numbers without a verdict.

The floor on the judged bundle was found by RUNNING the gate, not by reading
it: one PDF cut into 2 182 concepts scored 0 of 1 against the 32-document
reference and read as PASS.

The gate walks the index tree and never a directory (SS 9.2; controlled
against the listing on four bundles, 453 / 2 761 / 3 206 / 446 either way),
and prints the bundle's own run log beside its counts -- a document rejected
at extraction leaves no row in the bundle, so the pinned corpus's 33 PDFs
show up as 32 and the two denominators must never be read as one.

Three of the order's five premises moved when re-measured, and they are in the
document rather than glossed: the four evidence corpora carry `source_file` on
0 of 446, 0 of 1 133, 0 of 270 and 0 of 2 756 concepts, so they name no file
type and cannot PASS; "41,6 %" is `vegnormal-okf`'s number and not in this
repository; and the same 828-document bundle carries two published hit@k
figures from two question sets.

Three candidate metrics measured and NOT shipped: duplicate titles within a
document (0 of 3 206 on the known-bad arm against 349 of 2 761 on the
known-good one) and short concepts (5.6 % against 14.6 %) order the two arms
the wrong way round; duplicate titles across the whole bundle order all four
correctly (37.8 / 16.3 / 12.6 / 5.7 %) and still ship without a bar, because
any bar separating them is read off the two bundles it would judge.

19 new tests, each rule exercised in both directions; the three README pins
were each driven red before being kept. Suite 1 850 passed, 1 skipped, 1 851
collected, run after `git add` -- +19 against a base of 1 832 collected,
measured on the stashed tree (STATE's 1 831 is one short of that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-13 00:13:00 +02:00
commit 6ea8fcd3c7
8 changed files with 970 additions and 1 deletions

View file

@ -641,6 +641,50 @@ on a 629-concept bundle against a chance baseline of **1.35 of 6**, with one
control that failed — is `docs/2026-09-07-okf-konsumskill-maaling.md`; the copy
filled by hand for that corpus cannot ship and is no longer the one here.
## Judge a bundle: `okf quality`
`okf check` is a **contract** check — it asks whether a payload carries what a
claim must rest on. It is not a quality gate, and that is measured rather than
conceded: on 2026-09-10 three arms over one corpus all returned 0 findings and
exit 0 while their hit@k ranged from 6 of 6 to 0 of 6.
`okf quality <bundle>` asks the other question, **per file type and with the
denominator on every line**:
```sh
okf quality .okf/my-bundle
```
Three verdicts and no fourth — `PASS`, `FAIL`, `UNMEASURED` — and a type with
no measured threshold is **never** `PASS`. Exit codes: **0** judged and clean,
**1** at least one `FAIL`, **2** the run did not happen, **3** nothing could be
judged (every row `UNMEASURED`), because exit 0 over a table of unmeasured rows
would be the silent pass this command exists to stop.
Two thresholds exist today, both `structure_null_share` — the share of a type's
documents that yielded exactly one concept — read off the pinned 43-document
reference bundle:
<!-- quality-thresholds: .pdf=8/32, .docx=2/5 -->
| file type | threshold | N |
|---|---|---|
| `.pdf` | 8/32 | 32 documents |
| `.docx` | 2/5 | 5 documents |
| every type | 0 concepts with an empty body | definitional |
Every other type is `UNMEASURED`, including `.xlsx` (2 documents), `.xml`
(1 document) and `.html` (no bundle measured here). A threshold needs at least
five documents on both sides — the bundle's and its own — because a `1/1` is
not a rate.
**What a `PASS` is not.** It is a regression bar against a pinned artifact, not
a claim that the cut found the document's own structure. Boundary recall and
hit@k need a fasit and are outside a bundle-only gate; the measurement that
says so, the corpora behind each number, and three candidate metrics that were
measured and not shipped are in
[`docs/2026-09-12-g37-terskler.md`](docs/2026-09-12-g37-terskler.md).
## Consume in Claude Code
A folder of documents to an answer a model can cite, in **three lines**. You do