feat(quality): okf quality, a per-file-type verdict with the denominator
G37. `okf check` is a CONTRACT check and a green one is not a quality gate: measured 2026-09-10 by `vegnormal-okf`, three arms over one corpus all returned 0 findings and exit 0 while their hit@k ranged from 6 of 6 to 0 of 6. `okf quality <bundle>` asks the other question, per file type, with the denominator on every line. A separate command rather than `okf check --quality`, because the two answer different questions and a caller must not be able to read one as the other. `okf check` is untouched. Three verdicts and no fourth -- PASS, FAIL, UNMEASURED -- and a type with no measured threshold is never PASS. Exit 0 judged and clean, 1 at least one FAIL, 2 did not run, 3 nothing could be judged: exit 0 over a table of unmeasured rows would be the silent pass this command exists to stop. Two bars today, both `structure_null_share` (documents of a type yielding exactly one concept), read off the pinned 43-document reference bundle: .pdf 8/32, .docx 2/5. Plus one definitional bar for every type, taken from the harness's own degenerate-merge definition: 0 concepts with an empty body, measured 0 of 8 602 concepts over four bundles. A bar needs five documents on BOTH sides -- its own and the judged bundle's -- so .xlsx (2), .xml (1) and every type with no corpus class in `extract._EVIDENCE` are UNMEASURED and print their numbers without a verdict. The floor on the judged bundle was found by RUNNING the gate, not by reading it: one PDF cut into 2 182 concepts scored 0 of 1 against the 32-document reference and read as PASS. The gate walks the index tree and never a directory (SS 9.2; controlled against the listing on four bundles, 453 / 2 761 / 3 206 / 446 either way), and prints the bundle's own run log beside its counts -- a document rejected at extraction leaves no row in the bundle, so the pinned corpus's 33 PDFs show up as 32 and the two denominators must never be read as one. Three of the order's five premises moved when re-measured, and they are in the document rather than glossed: the four evidence corpora carry `source_file` on 0 of 446, 0 of 1 133, 0 of 270 and 0 of 2 756 concepts, so they name no file type and cannot PASS; "41,6 %" is `vegnormal-okf`'s number and not in this repository; and the same 828-document bundle carries two published hit@k figures from two question sets. Three candidate metrics measured and NOT shipped: duplicate titles within a document (0 of 3 206 on the known-bad arm against 349 of 2 761 on the known-good one) and short concepts (5.6 % against 14.6 %) order the two arms the wrong way round; duplicate titles across the whole bundle order all four correctly (37.8 / 16.3 / 12.6 / 5.7 %) and still ship without a bar, because any bar separating them is read off the two bundles it would judge. 19 new tests, each rule exercised in both directions; the three README pins were each driven red before being kept. Suite 1 850 passed, 1 skipped, 1 851 collected, run after `git add` -- +19 against a base of 1 832 collected, measured on the stashed tree (STATE's 1 831 is one short of that). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
d310dbb34e
commit
6ea8fcd3c7
8 changed files with 970 additions and 1 deletions
44
README.md
44
README.md
|
|
@ -641,6 +641,50 @@ on a 629-concept bundle against a chance baseline of **1.35 of 6**, with one
|
|||
control that failed — is `docs/2026-09-07-okf-konsumskill-maaling.md`; the copy
|
||||
filled by hand for that corpus cannot ship and is no longer the one here.
|
||||
|
||||
## Judge a bundle: `okf quality`
|
||||
|
||||
`okf check` is a **contract** check — it asks whether a payload carries what a
|
||||
claim must rest on. It is not a quality gate, and that is measured rather than
|
||||
conceded: on 2026-09-10 three arms over one corpus all returned 0 findings and
|
||||
exit 0 while their hit@k ranged from 6 of 6 to 0 of 6.
|
||||
|
||||
`okf quality <bundle>` asks the other question, **per file type and with the
|
||||
denominator on every line**:
|
||||
|
||||
```sh
|
||||
okf quality .okf/my-bundle
|
||||
```
|
||||
|
||||
Three verdicts and no fourth — `PASS`, `FAIL`, `UNMEASURED` — and a type with
|
||||
no measured threshold is **never** `PASS`. Exit codes: **0** judged and clean,
|
||||
**1** at least one `FAIL`, **2** the run did not happen, **3** nothing could be
|
||||
judged (every row `UNMEASURED`), because exit 0 over a table of unmeasured rows
|
||||
would be the silent pass this command exists to stop.
|
||||
|
||||
Two thresholds exist today, both `structure_null_share` — the share of a type's
|
||||
documents that yielded exactly one concept — read off the pinned 43-document
|
||||
reference bundle:
|
||||
|
||||
<!-- quality-thresholds: .pdf=8/32, .docx=2/5 -->
|
||||
|
||||
| file type | threshold | N |
|
||||
|---|---|---|
|
||||
| `.pdf` | 8/32 | 32 documents |
|
||||
| `.docx` | 2/5 | 5 documents |
|
||||
| every type | 0 concepts with an empty body | definitional |
|
||||
|
||||
Every other type is `UNMEASURED`, including `.xlsx` (2 documents), `.xml`
|
||||
(1 document) and `.html` (no bundle measured here). A threshold needs at least
|
||||
five documents on both sides — the bundle's and its own — because a `1/1` is
|
||||
not a rate.
|
||||
|
||||
**What a `PASS` is not.** It is a regression bar against a pinned artifact, not
|
||||
a claim that the cut found the document's own structure. Boundary recall and
|
||||
hit@k need a fasit and are outside a bundle-only gate; the measurement that
|
||||
says so, the corpora behind each number, and three candidate metrics that were
|
||||
measured and not shipped are in
|
||||
[`docs/2026-09-12-g37-terskler.md`](docs/2026-09-12-g37-terskler.md).
|
||||
|
||||
## Consume in Claude Code
|
||||
|
||||
A folder of documents to an answer a model can cite, in **three lines**. You do
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue