open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(quality): okf quality, a per-file-type verdict with the denominator

Browse source 
G37. `okf check` is a CONTRACT check and a green one is not a quality gate:
measured 2026-09-10 by `vegnormal-okf`, three arms over one corpus all
returned 0 findings and exit 0 while their hit@k ranged from 6 of 6 to 0 of 6.
`okf quality <bundle>` asks the other question, per file type, with the
denominator on every line. A separate command rather than `okf check
--quality`, because the two answer different questions and a caller must not
be able to read one as the other. `okf check` is untouched.

Three verdicts and no fourth -- PASS, FAIL, UNMEASURED -- and a type with no
measured threshold is never PASS. Exit 0 judged and clean, 1 at least one
FAIL, 2 did not run, 3 nothing could be judged: exit 0 over a table of
unmeasured rows would be the silent pass this command exists to stop.

Two bars today, both `structure_null_share` (documents of a type yielding
exactly one concept), read off the pinned 43-document reference bundle: .pdf
8/32, .docx 2/5. Plus one definitional bar for every type, taken from the
harness's own degenerate-merge definition: 0 concepts with an empty body,
measured 0 of 8 602 concepts over four bundles. A bar needs five documents on
BOTH sides -- its own and the judged bundle's -- so .xlsx (2), .xml (1) and
every type with no corpus class in `extract._EVIDENCE` are UNMEASURED and
print their numbers without a verdict.

The floor on the judged bundle was found by RUNNING the gate, not by reading
it: one PDF cut into 2 182 concepts scored 0 of 1 against the 32-document
reference and read as PASS.

The gate walks the index tree and never a directory (SS 9.2; controlled
against the listing on four bundles, 453 / 2 761 / 3 206 / 446 either way),
and prints the bundle's own run log beside its counts -- a document rejected
at extraction leaves no row in the bundle, so the pinned corpus's 33 PDFs
show up as 32 and the two denominators must never be read as one.

Three of the order's five premises moved when re-measured, and they are in the
document rather than glossed: the four evidence corpora carry `source_file` on
0 of 446, 0 of 1 133, 0 of 270 and 0 of 2 756 concepts, so they name no file
type and cannot PASS; "41,6 %" is `vegnormal-okf`'s number and not in this
repository; and the same 828-document bundle carries two published hit@k
figures from two question sets.

Three candidate metrics measured and NOT shipped: duplicate titles within a
document (0 of 3 206 on the known-bad arm against 349 of 2 761 on the
known-good one) and short concepts (5.6 % against 14.6 %) order the two arms
the wrong way round; duplicate titles across the whole bundle order all four
correctly (37.8 / 16.3 / 12.6 / 5.7 %) and still ship without a bar, because
any bar separating them is read off the two bundles it would judge.

19 new tests, each rule exercised in both directions; the three README pins
were each driven red before being kept. Suite 1 850 passed, 1 skipped, 1 851
collected, run after `git add` -- +19 against a base of 1 832 collected,
measured on the stashed tree (STATE's 1 831 is one short of that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-13 00:13:00 +02:00
commit 6ea8fcd3c7
8 changed files with 970 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

29
CHANGELOG.md
View file

@ -7,6 +7,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **`okf quality <bundle>` -- a per-file-type verdict, with the denominator
(G37).** `okf check` is a CONTRACT check, and a green one is not a quality
gate: measured 2026-09-10 by `vegnormal-okf`, three arms over one corpus all
returned 0 findings and exit 0 while their hit@k ranged from 6 of 6 to 0 of 6.
The new command asks the other question. Three verdicts and no fourth --
`PASS`, `FAIL`, `UNMEASURED` -- and a type with no measured threshold is never
`PASS`; exit **0** judged and clean, **1** at least one `FAIL`, **2** did not
run, **3** nothing could be judged, because exit 0 over a table of unmeasured
rows would be the silent pass the command exists to stop. Two thresholds
today, both `structure_null_share` (the share of a type's documents that
yielded exactly one concept), read off the pinned 43-document reference
bundle: `.pdf` 8/32 and `.docx` 2/5, plus one definitional bar that applies to
every type (0 concepts with an empty body, measured 0 of 8 602 over four
bundles). A bar needs at least five documents on BOTH sides -- its own and the
judged bundle's -- so `.xlsx` (2), `.xml` (1) and every type with no corpus
class in `extract._EVIDENCE` are `UNMEASURED` and print their numbers without
a verdict. The gate reads the index tree, never a directory (SS 9.2), and
prints the bundle's own run log beside its counts, because a document rejected
at extraction leaves no row in the bundle at all -- the pinned corpus holds 33
PDFs and the bundle shows 32. `okf check` is untouched.
`docs/2026-09-12-g37-terskler.md` carries the table, the nine bundles behind
it, the order's own premises re-measured (three of five moved), and three
candidate metrics measured and NOT shipped -- two of them ordering a known-bad
arm and a known-good one the wrong way round. The README publishes the bars
behind a `<!-- quality-thresholds: ... -->` marker that
`tests/test_docs_promises.py` pins to the code and to the document.
### Changed
- **The README states every file type the extractor registry reads (K3-26).**

33
CLAUDE.md
View file

@ -755,6 +755,39 @@ and fixtures, never code.
move (it is not delivered on the Arm B bundle either, by a different
mechanism). hit@8 over the six published questions holds at **5 of 6 on both
K2 bundles**, so the default move cost the retrieval side nothing.
- **Judge a bundle: `okf quality <bundle>` (G37, 2026-09-12).** A per-file-type
verdict, with the denominator on every line, and it is a SEPARATE command
from `okf check` on purpose: `check` is the contract check, and a green one
is not a quality gate -- measured 10.09 by `vegnormal-okf`, three arms over
one corpus all gave 0 findings and exit 0 while hit@k ranged 6 of 6 to 0 of
6. Three verdicts and no fourth (`PASS` / `FAIL` / `UNMEASURED`), a type with
no measured threshold is **never** `PASS`, and exit **3** exists for
"nothing could be judged" so exit 0 over a table of unmeasured rows cannot
be a silent pass (0 clean, 1 a `FAIL`, 2 did not run). Two bars today, both
`structure_null_share` off the pinned 43-document reference -- `.pdf` 8/32,
`.docx` 2/5 -- plus one definitional bar for every type (0 empty bodies,
measured **0 of 8 602** concepts over four bundles). A bar needs **five
documents on BOTH sides**, its own and the judged bundle's, which was found
by RUNNING the gate: a one-PDF bundle scored 0 of 1 against the 32-document
reference and read as `PASS`. **The bars are regression bars against a
pinned artifact, never a quality claim**, and the defect that started G37 --
the HTML arm's 1 148 of 2 761 boundaries -- is `UNMEASURED` here, because no
bundle-only metric reaches it: three candidates were measured over the same
four bundles and **two order the known-bad and known-good arms the wrong way
round** (duplicate titles within a document 0/3 206 against 349/2 761; short
concepts 5.6 % against 14.6 %), while the third (duplicate titles across the
whole bundle, 37.8 / 16.3 / 12.6 / 5.7 %) orders them correctly and ships
anyway WITHOUT a bar, since any bar separating them sits between the two
bundles that define it. **The gate's denominator is the bundle's, never the
corpus's**: a rejected document leaves no row at all (the pinned corpus holds
33 PDFs, the bundle shows 32), so the run log is printed beside the counts
and a bundle without one says so. Three of the four evidence corpora
(`n100`/`n200`/`n500`) carry `source_file` on **0 of 446 / 0 of 1 133 / 0 of
270** concepts, so they name no file type and every row is `UNMEASURED` --
the order expected them to PASS. Thresholds, the nine bundles and the
premises re-measured: `docs/2026-09-12-g37-terskler.md`. README publishes the
bars behind `<!-- quality-thresholds: ... -->`, pinned to the code AND the
document by `tests/test_docs_promises.py`.
- Consume a bundle: `okf consume <bundle> --question "<q>"
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
`docs/consumption-contract.md` § 1 defines, and the only reading direction

44
README.md
View file

@ -641,6 +641,50 @@ on a 629-concept bundle against a chance baseline of **1.35 of 6**, with one
control that failed — is `docs/2026-09-07-okf-konsumskill-maaling.md`; the copy
filled by hand for that corpus cannot ship and is no longer the one here.
## Judge a bundle: `okf quality`
`okf check` is a **contract** check — it asks whether a payload carries what a
claim must rest on. It is not a quality gate, and that is measured rather than
conceded: on 2026-09-10 three arms over one corpus all returned 0 findings and
exit 0 while their hit@k ranged from 6 of 6 to 0 of 6.
`okf quality <bundle>` asks the other question, **per file type and with the
denominator on every line**:
```sh
okf quality .okf/my-bundle
```
Three verdicts and no fourth — `PASS`, `FAIL`, `UNMEASURED` — and a type with
no measured threshold is **never** `PASS`. Exit codes: **0** judged and clean,
**1** at least one `FAIL`, **2** the run did not happen, **3** nothing could be
judged (every row `UNMEASURED`), because exit 0 over a table of unmeasured rows
would be the silent pass this command exists to stop.
Two thresholds exist today, both `structure_null_share` — the share of a type's
documents that yielded exactly one concept — read off the pinned 43-document
reference bundle:
<!-- quality-thresholds: .pdf=8/32, .docx=2/5 -->
| file type | threshold | N |
|---|---|---|
| `.pdf` | 8/32 | 32 documents |
| `.docx` | 2/5 | 5 documents |
| every type | 0 concepts with an empty body | definitional |
Every other type is `UNMEASURED`, including `.xlsx` (2 documents), `.xml`
(1 document) and `.html` (no bundle measured here). A threshold needs at least
five documents on both sides — the bundle's and its own — because a `1/1` is
not a rate.
**What a `PASS` is not.** It is a regression bar against a pinned artifact, not
a claim that the cut found the document's own structure. Boundary recall and
hit@k need a fasit and are outside a bundle-only gate; the measurement that
says so, the corpora behind each number, and three candidate metrics that were
measured and not shipped are in
[`docs/2026-09-12-g37-terskler.md`](docs/2026-09-12-g37-terskler.md).
## Consume in Claude Code
A folder of documents to an answer a model can cite, in **three lines**. You do

181
docs/2026-09-12-g37-terskler.md Normal file
View file

@ -0,0 +1,181 @@
# G37 — the thresholds, per file type, before the gate
Order `20260912T202210Z-7596115025-from-.claude`, part 1. Base `d310dbb`
(v0.8.5). Measured 2026-09-12 evening local time; the session crossed midnight,
so the commit lands 2026-09-13. Python 3.14.0, guard 1.4.0, no model call and
no network anywhere in what follows.
The order's own words for what this part must produce: "tabell filtype ×
metrikk × terskel × N × kilde", metrics restricted to what is already measured
or measurable without a model, and "en filtype uten måling får terskel
`UNMEASURED`, aldri et gjettet tall".
## § 0 — measured / not measured
**Measured here.** Every count in §§ 2–5: concept and document counts per file
type over nine bundles, one-concept-document shares, empty-body counts,
duplicate-title shares in two forms, short-concept shares, and which of the
four evidence corpora carry a `source_file` key at all. Each is a shell or
`okf quality` command printed beside it.
**Not measured here.** Boundary recall against a fasit (needs a fasit; § 6),
hit@k (needs a question set and a fasit; § 6), anything about `.txt`, `.csv`,
`.json`, `.htm`, `.pptx`, `.odt`, `.rtf` beyond the fixtures already in the
tree, and whether any threshold below is the *right* bar — ratification is the
operator's, and this document sets regression bars against a pinned artifact,
which is a different thing (§ 4).
## § 1 — the three metric names, defined here because nothing defines them
The order names "grenseandel", "struktur-null-andel", "tomme konsepter",
`Bundle.skipped` and "hit@k". Two of those five have no definition anywhere in
this repository:
Measured on the base commit rather than on the working tree, because this
document and the module beside it introduce both words:
- `git grep -c "grenseandel\|boundary share" d310dbb -- docs src` → **0 files**
- `git grep -ci "struktur.null\|structure.null" d310dbb -- docs src` → **0 files**
- known-positive control for the query form:
`git grep -c "hit@8" d310dbb -- docs` → **19 files**, so it can find.
So the definitions below are **mine**, written here so the gate and this
document mean the same thing by them:
| name | definition | what it needs |
|---|---|---|
| `boundary_share` | recovered boundaries ÷ boundaries the source itself declares | a fasit |
| `structure_null_share` | documents of a type yielding exactly ONE concept ÷ documents of that type in the bundle | the bundle alone |
| `empty_concepts` | concepts whose body holds zero non-whitespace characters ÷ concepts of that type | the bundle alone |
| `hit@k` | questions whose gold document reaches the payload at cut `k` ÷ questions | a fasit and a question set |
| run log | `N`, merged and coded rejections, from the bundle's own `log.md` | a bundle Door B wrote |
`Bundle.skipped` does not exist under that name: the door's counts reach a
bundle through `corpus.CorpusReport`, whose written form is the run log above
(`N`, `extracted`, `gated`, `persisted`, `substantive`, `degenerate`,
`rejected`, plus per-code counts). It is a **whole-bundle** figure and carries
no file type, which is the limit in § 5.
## § 2 — the order's own numbers, re-measured first
The coordination note that arrived with the order asked for exactly this, and
three of the five premises moved.
| premise in the order | re-measured | verdict |
|---|---|---|
| 13 extensions in the registry | `_CORE_EXTRACTORS` 7 + `_OPTIONAL_EXTRACTORS` 6 = **13** | holds |
| four evidence corpora under `~/repos/vegnormal-okf/build/ferdig/` | present: `n100-2023`, `n200-2024`, `n500-2024`, `r761-2025` — plus `r761-2025-generisk` and `r761-2025-v080`, two more R761 builds the order does not name | **holds, and is incomplete** |
| "N100 skal gi PASS på de målte typene" | `grep -rl '^source_file:' n100-2023 --include='*.md' | wc -l` → **0 of 446 concepts**. Same for `n200-2024` (0 of 1 133), `n500-2024` (0 of 270) and `r761-2025` (0 of 2 756) | **cannot hold**: three of the four name no file type at all, so no type of theirs can be PASS or FAIL |
| "41,6 % grenser" | not in this repository (`grep -rn "41[.,]6" docs/ src/` → 0). It is `vegnormal-okf`'s: `docs/2026-09-10-r2-r761-produkt.md:452`, 1 148 of 2 761 declared STS sections, against the XML arm's 2 759 of 2 761 | **holds, and the owner is another repo** |
| "0 av 6 hit@k" on the HTML arm | two published numbers for the same 828-document bundle: **0 of 6** (vegnormal, their six fasit questions, 10.09) and **2 of 6 → 4 of 6 with `--source-quota 2`** (this repo, `docs/2026-09-10-k3-runde11-html-og-dominans.md:200,217`, questions S1–S6) | **holds for one instrument, not for the other** |
The last row is the reason the gate reports a denominator on every line: two
true numbers about one bundle, four apart, because the question set differs.
## § 3 — what each corpus can answer
Command: `okf quality <bundle>` (part 2 of this order), cross-checked against
`grep -rh '^source_file:' <bundle> --include='*.md' | sed 's/.*\.//' | sort |
uniq -c`. The index walk and the directory listing agree on 453, 2 761, 3 206
and 446 concepts — four bundles, both methods.
| bundle | producer | concepts | file types present |
|---|---|---|---|
| `K2-bundle-default-20260912` | Door B (pinned reference) | 453 | `.pdf` 32 docs, `.docx` 5, `.xlsx` 2 |
| `r761-2025-generisk` | Door B | 2 761 | `.xml` 1 doc |
| `sk2-bundle-default` | Door B | 2 182 | `.pdf` 1 doc |
| `860019-mdb-100` | Door B | 3 206 | `.md` 828 docs |
| `n100-2023` | vegnormal's own pipeline | 446 | none — no `source_file` |
| `n200-2024` | vegnormal's own pipeline | 1 133 | none |
| `n500-2024` | vegnormal's own pipeline | 270 | none |
| `r761-2025` | vegnormal's own pipeline | 2 756 | none |
| `b3` | — | — | no `index.md`; the gate refuses to walk a directory (SS 9.2) and exits 2 |
**The four corpora the order names as the evidence base carry no file type.**
The per-file-type thresholds below therefore rest on the K2 reference bundle
and, for context only, on the three single-type bundles.
## § 4 — the table
Threshold semantics, stated before the numbers: a threshold here is a
**regression bar against a pinned artifact**, set at the value measured on that
artifact and never at a rounder number nearby. `PASS` means *no worse than the
reference on this metric*. It is not a claim that the bundle is good, and it is
not the operator's ratified quality bar — the K3 method refuses to set one
inside the measurement that produces it ("setting one inside the same work that
produces the first measurement would be fitting the bar to the number").
A rate needs a denominator. `MIN_DOCUMENTS_FOR_A_THRESHOLD = 5`, and it binds
both the threshold's N and the judged bundle's: this repository's own honesty
limit is "a `1/1` is not a rate" (`docs/2026-09-08-k3-runde2-per-filtype.md`).
| file type | metric | threshold | N | source |
|---|---|---|---|---|
| every type | `empty_concepts` | **0** | 8 602 concepts over 4 Door B bundles, all 0 | definitional: `corpus.CorpusReport` calls a zero-character merge degenerate |
| `.pdf` | `structure_null_share` | **8/32** | 32 documents | `K2-bundle-default-20260912` |
| `.docx` | `structure_null_share` | **2/5** | 5 documents | `K2-bundle-default-20260912` |
| `.xlsx` | — | **UNMEASURED** | 2 documents | below the floor of 5 |
| `.xml` | — | **UNMEASURED** | 1 document | below the floor of 5 |
| `.md` | — | **UNMEASURED** | 828 documents, `structure_null_share` 362/828 measured | the corpus is one generator's cut of one product, and it is the bundle under suspicion; a bar read off it would pass it by construction |
| `.html` | — | **UNMEASURED** | 0 bundles in this repository | the 828-document HTML measurement was made outside it |
| `.htm`, `.txt`, `.csv`, `.json`, `.pptx`, `.odt`, `.rtf` | — | **UNMEASURED** | 0 corpus documents | no corpus class in `extract._EVIDENCE`; `.pptx`/`.odt`/`.rtf` are `constructed`, N = 1–2 |
### Candidates measured and not shipped
Three fasit-free metrics were measured over the same four Door B bundles, with
the known-bad arm (`860019-mdb-100`, 1 148 of 2 761 boundaries) and the
known-good arm (`r761-2025-generisk`, 2 759 of 2 761) named in advance:
| candidate | known-bad arm | known-good arm | pdf arm | K2 reference | verdict |
|---|---|---|---|---|---|
| duplicate titles within a document | 0/3 206 (0.0 %) | 349/2 761 (12.6 %) | 355/2 182 | 11/453 | **wrong direction** |
| concepts under 40 characters | 181/3 206 (5.6 %) | 402/2 761 (14.6 %) | 1/2 182 | 25/453 | **wrong direction** |
| duplicate titles across the whole bundle | 1 211/3 206 (**37.8 %**) | 349/2 761 (12.6 %) | 355/2 182 (16.3 %) | 26/453 (5.7 %) | orders correctly, **no bar** |
The third orders all four bundles correctly and is still not shipped: any bar
that separates them sits between 16.3 % and 37.8 %, which is a number read off
the two bundles it would judge. That is the operator's call and it is stated in
STATE rather than taken here.
## § 5 — three limits that are properties of the instrument
1. **A rejected document leaves no trace in the bundle.** The pinned K2 bundle
holds 32 `.pdf` documents; the corpus held 33, and one was rejected
`extractor_empty_pdf`. The gate's denominator is the bundle's, never the
corpus's, and it prints the run log (`N = 43, merged = 39, coded rejections
= 4`) beside its own numbers so the two are never read as one. A bundle
without a log says "no run log" rather than reporting zero rejections.
2. **The run log carries no file type.** `codes` is per rejection code, not per
extension, so "this type failed to extract entirely" is invisible to a
bundle-only gate. Naming that would be a change in `corpus.py`, not here.
3. **The reference bundle passes its own bars by construction.** `okf quality`
on `K2-bundle-default-20260912` returning `PASS` on `.pdf` and `.docx` is a
control that the gate reproduces its own reference — nothing more.
## § 6 — what would reach the defect that started this
The 41.6 % arm is `UNMEASURED` under this gate, and that is the honest answer
rather than a hidden one: the metric that fells it is `boundary_share`, which
needs the publisher's own declared structure. R761 has one (2 761 STS `<sec>`
titles, `~/repos/vegnormal-okf/build/sk2-fasit-2761.json`), and 4 of the 9
bundles above have no fasit at all. A `--fasit` input, a threshold on
`boundary_share`, and the operator's bar are one further round, named in STATE
and not begun here.
## Honesty limits
- **Two thresholds, and one of them rests on five documents.** `.docx` at 2/5
is exactly at the floor this document sets for itself. It is a bar against a
pinned artifact, not a rate anyone should quote.
- **`structure_null_share` has never fired on a real bundle.** Of the five Door
B bundles measured, none is worse than its reference; the rule's `FAIL` side
is exercised on constructed bundles in `tests/test_quality.py` only.
- **`empty_concepts` is 0 on 8 602 of 8 602 concepts.** The bar is definitional
and has never fired on real data either. A metric that never fires is not
evidence that nothing is wrong.
- **One rule was found by running the gate, not by reading it.** A one-document
PDF bundle scored 0 of 1 against the 32-document reference and read as
`PASS`. The floor now binds both denominators; the version of this document
written before that run would have published the wrong rule.
- **Nothing here measures answer quality.** hit@k asks whether a gold document
reached the payload; this gate does not even ask that.

5
src/llm_ingestion_okf/cli.py
View file

@ -96,7 +96,7 @@ __all__ = ["DEFAULT_STAMP", "build", "main", "measure"]
#:
#: Imported lazily inside the dispatch: `okf build` should not pay to import
#: the ranker, and `okf consume` should not pay to import the proposer.
DELEGATED = ("consume", "check", "skill", "project")
DELEGATED = ("consume", "check", "skill", "project", "quality")
def _delegate(command: str, argv: list[str]) -> int:
@ -106,6 +106,8 @@ def _delegate(command: str, argv: list[str]) -> int:
from .contract_check import main as run
elif command == "skill":
from .skill import main as run
elif command == "quality":
from .quality import main as run
else:
from .project import main as run
return run(argv)
@ -592,6 +594,7 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
("check", "check a consumption skill and one payload against the contract"),
("skill", "instantiate the consumption skill template for one bundle"),
("project", "folder in, bundle plus skill out: build and skill in one step"),
("quality", "judge one bundle per file type, with the denominator"),
):
subcommands.add_parser(delegated, help=blurb, add_help=False)
build_parser = subcommands.add_parser(

407
src/llm_ingestion_okf/quality.py Normal file
View file

@ -0,0 +1,407 @@
"""`okf quality` -- a per-file-type verdict on one bundle, with the denominator.
**This is not `okf check`, and the separation is the point.** `okf check` reads
a consumption skill and one payload against `docs/consumption-contract.md`: it
answers whether a payload carries what a claim must rest on. Measured
2026-09-10 by `vegnormal-okf` on three arms over one corpus, it returned 0
findings and exit 0 on all three while their hit@k ranged from 6 of 6 to 0 of 6
-- a green contract check says nothing about whether the cut found anything
worth reading. This module asks that second question, and it is a SEPARATE
command rather than a `--quality` flag on the first for exactly that reason: the
two answer different questions and a caller must not be able to read one as the
other.
Three verdicts and no fourth: `PASS`, `FAIL`, `UNMEASURED`. A type with no
measured threshold is never `PASS` -- an unmeasured row that reads as a passing
one is the failure this gate exists to prevent, and it is the same failure
`extract._EVIDENCE` was built to prevent one layer down.
**What this gate can and cannot see.** Every metric here is computed from the
bundle alone: no fasit, no model call, no clock, no network. That bounds it
sharply, and the bound is measured rather than assumed.
`docs/2026-09-12-g37-terskler.md` SS 4 records three candidates measured over
the same four bundles and what became of each: duplicate titles WITHIN a
document (0 of 3 206 on the known-bad arm against 349 of 2 761 on the known-good
one -- the wrong direction) and the share of very short concepts (5.6 % against
14.6 % -- also the wrong direction) are not shipped; duplicate titles across the
WHOLE bundle order the four bundles correctly (37.8 / 16.3 / 12.6 / 5.7 %) and
are still not shipped, because a bar separating them would have to be placed
between the two bundles that define it, which is fitting the bar to the number.
The defect that started this work -- 1 148 of 2 761 declared boundaries
recovered -- needs a fasit and no bundle-only metric reaches it.
"""
from __future__ import annotations
import argparse
import re
import sys
from collections import Counter
from dataclasses import dataclass
from pathlib import Path
from .consume import ConsumeError, enumerate_concepts, read_concept, root_bundle_id_of
from .corpus import LOG_NAME
from .profiles import SEGMENTED_OKF_V0_2, BundleProfile
CLI_ID = "okf quality"
#: The row a concept lands in when it declares no `source_file`. Not a file
#: type and never treated as one: measured 2026-09-12, three of the four
#: evidence corpora (`n100-2023`, `n200-2024`, `n500-2024`) carry the key on 0
#: of 446, 0 of 1 133 and 0 of 270 concepts, because their producer is not this
#: library's Door B. A per-file-type gate has nothing to say about them, and
#: says that.
NO_SOURCE_FILE = "(no source_file)"
#: A threshold needs a denominator big enough that a single document cannot be
#: the rate. FIVE, and the number is this repository's own honesty limit rather
#: than a statistical claim: `docs/2026-09-08-k3-runde2-per-filtype.md` states
#: "Per file type the denominators are 8, 3 and 1. A `1/1` is not a rate", and
#: `docs/2026-09-04-k3-arm-c.md` says of the three office types with no corpus
#: file at all: "Unmeasured, not passing." Below this floor the row is
#: `UNMEASURED` and its numbers are still printed.
#:
#: It binds BOTH denominators -- the threshold's and the bundle's. Found by
#: running the gate rather than by reading it: one PDF cut into 2 182 concepts
#: scored 0 of 1 against the 32-document reference and read as PASS.
MIN_DOCUMENTS_FOR_A_THRESHOLD = 5
@dataclass(frozen=True)
class Threshold:
"""One measured bar, carrying the measurement it was read off.
The bar is held as the measured PAIR (`limit_null` of `limit_documents`)
rather than a float, so the comparison is exact integer arithmetic and a
bundle sitting exactly at the reference cannot fall to a rounding step.
"""
metric: str
limit_null: int
limit_documents: int
#: Documents behind the measurement. Equal to `limit_documents` today and
#: kept separate because a threshold ratified over a wider corpus than the
#: one it is expressed as would need both numbers.
documents: int
source: str
def exceeded_by(self, null: int, documents: int) -> bool:
"""`null/documents` strictly worse than the reference, without floats."""
return null * self.limit_documents > self.limit_null * documents
def as_share(self) -> str:
return f"{self.limit_null}/{self.limit_documents}"
#: The bars, per extension, and there are two of them. Read off the pinned
#: reference bundle `K2-bundle-default-20260912` (the 43-document corpus
#: `~/corpora/okf-telling-20260829/K2/trinn1`, N = 43, 39 merged) on
#: 2026-09-12, and set at the value measured there rather than at a rounder
#: number nearby: this is a REGRESSION bar against a pinned artifact, not a
#: claim that a bundle at the bar is good. `docs/2026-09-12-g37-terskler.md`
#: carries the table, the corpora and what each number does not prove.
#:
#: Every other type is absent on purpose. `.xlsx` (2 documents) and `.xml`
#: (1 document) are below the floor above; `.html` has no bundle measured in
#: this repository; `.md`, `.txt`, `.csv`, `.json`, `.htm`, `.pptx`, `.odt`
#: and `.rtf` have no corpus class in `extract._EVIDENCE` at all.
THRESHOLDS: dict[str, Threshold] = {
".pdf": Threshold(
metric="structure_null_share",
limit_null=8,
limit_documents=32,
documents=32,
source="K2-bundle-default-20260912 (43-document corpus, 32 pdf documents)",
),
".docx": Threshold(
metric="structure_null_share",
limit_null=2,
limit_documents=5,
documents=5,
source="K2-bundle-default-20260912 (43-document corpus, 5 docx documents)",
),
}
#: The one bar that needs no corpus: a concept whose body holds no
#: non-whitespace character. Taken from the harness's own definition of a
#: degenerate merge (`corpus.CorpusReport.render`: "a merge is degenerate when
#: the extracted text is zero characters after stripping whitespace -- a
#: definition, not a threshold"), so it applies to every type INCLUDING one with
#: no threshold. FAIL is reachable for every row; PASS is not.
EMPTY_BODY_LIMIT = 0
PASS = "PASS"
FAIL = "FAIL"
UNMEASURED = "UNMEASURED"
_LOG_LINE = re.compile(
r"N = (\d+).*?merged = (\d+).*?coded rejections = (\d+)",
re.DOTALL,
)
@dataclass(frozen=True)
class TypeReport:
"""One file type's numbers and its verdict. Every count carries its own N."""
extension: str
documents: int
concepts: int
empty: int
structure_null: int
verdict: str
threshold: Threshold | None
reason: str
def render(self) -> str:
"""One line, and every count on it carries its own denominator.
The `NO_SOURCE_FILE` row prints neither a document count nor a
one-concept share: those concepts all share the same empty
`source_file`, so grouping by it yields `documents 1` for a bundle of
446 -- a number that looks measured and means nothing.
"""
head = f"{self.extension:<18} {self.verdict:<11} "
if self.extension == NO_SOURCE_FILE:
return (
f"{head}concepts {self.concepts:>5} empty {self.empty}/{self.concepts} "
f"-- {self.reason}"
)
share = f"{self.structure_null}/{self.documents}"
bar = f"limit {self.threshold.as_share()}" if self.threshold else "no threshold"
return (
f"{head}documents {self.documents:>5} "
f"concepts {self.concepts:>5} empty {self.empty}/{self.concepts} "
f"one-concept documents {share} ({bar}) -- {self.reason}"
)
@dataclass(frozen=True)
class BundleQuality:
"""One bundle's rows, its run log if it has one, and the exit code they imply."""
bundle_root: Path
bundle_id: str
rows: tuple[TypeReport, ...]
#: The `N`, merged and coded-rejection counts from the bundle's own section
#: 9 log, or `None` when the bundle carries no log. Never defaulted to zero:
#: a rejected document leaves NO concept in the bundle, so without the log
#: the gate cannot know whether a type failed to extract entirely.
run_log: str | None
def row(self, extension: str) -> TypeReport:
for row in self.rows:
if row.extension == extension:
return row
raise KeyError(
f"{extension} is not a row of this bundle: {[r.extension for r in self.rows]}"
)
@property
def exit_code(self) -> int:
"""0 judged and clean, 1 at least one FAIL, 3 nothing could be judged.
`2` is reserved for "did not run" and is returned by `main` alone. The
third code exists because exit 0 over a table of `UNMEASURED` rows would
be exactly the silent pass this gate was built to stop.
"""
if any(row.verdict == FAIL for row in self.rows):
return 1
if any(row.verdict == PASS for row in self.rows):
return 0
return 3
def render(self) -> str:
lines = [
f"# {CLI_ID}: {self.bundle_id}",
"",
f"bundle: {self.bundle_root}",
(
f"run log: {self.run_log}"
if self.run_log is not None
else f"run log: no run log in the bundle ({LOG_NAME} absent) -- the "
"denominators below are the bundle's own, and a document rejected "
"at extraction leaves no row here at all"
),
"",
"## Per file type",
"",
]
lines.extend(row.render() for row in self.rows)
lines.extend(
[
"",
"## What this verdict is not",
"",
"A regression bar against a pinned reference bundle, per file type.",
"PASS means no worse than that reference on the metrics below; it is",
"not a claim that the cut found the document's own structure. Boundary",
"recall and hit@k need a fasit and are outside a bundle-only gate --",
"docs/2026-09-12-g37-terskler.md carries the measurement that says so.",
]
)
return "\n".join(lines) + "\n"
def _extension_of(source_file: str) -> str:
if not source_file.strip():
return NO_SOURCE_FILE
suffix = Path(source_file).suffix.lower()
return suffix if suffix else NO_SOURCE_FILE
def read_run_log(bundle_root: Path) -> str | None:
"""The bundle's own `N`, merged and coded-rejection counts, or `None`."""
log = bundle_root / LOG_NAME
if not log.is_file():
return None
matches = _LOG_LINE.findall(log.read_text(encoding="utf-8"))
if not matches:
return None
total, merged, rejected = matches[-1]
return f"N = {total}, merged = {merged}, coded rejections = {rejected}"
def measure_bundle(
bundle_root: Path, *, profile: BundleProfile = SEGMENTED_OKF_V0_2
) -> BundleQuality:
"""Every concept the index declares, grouped by the extension it came from.
Reached through the index tree and never `rglob`: the index is the bundle's
own statement of what it contains, and `consume.enumerate_concepts` is the
one walker in this library that reads it. Controlled 2026-09-12 against the
directory listing on four bundles -- 453, 2 761, 3 206 and 446 concepts
either way.
"""
root_bundle_id = root_bundle_id_of(bundle_root, profile=profile)
concepts_per_extension: Counter[str] = Counter()
empty_per_extension: Counter[str] = Counter()
documents: dict[str, Counter[str]] = {}
for concept_id in enumerate_concepts(bundle_root, profile=profile):
concept = read_concept(
bundle_root / f"{concept_id}{profile.paths.concept_suffix}",
bundle_root=bundle_root,
root_bundle_id=root_bundle_id,
)
extension = _extension_of(concept.source_file)
concepts_per_extension[extension] += 1
documents.setdefault(extension, Counter())[concept.source_file] += 1
if not "".join(concept.body.split()):
empty_per_extension[extension] += 1
rows = tuple(
_verdict(
extension,
documents=documents[extension],
concepts=concepts_per_extension[extension],
empty=empty_per_extension[extension],
)
for extension in sorted(concepts_per_extension)
)
return BundleQuality(
bundle_root=bundle_root,
bundle_id=root_bundle_id,
rows=rows,
run_log=read_run_log(bundle_root),
)
def _verdict(extension: str, *, documents: Counter[str], concepts: int, empty: int) -> TypeReport:
document_count = len(documents)
structure_null = sum(1 for count in documents.values() if count == 1)
threshold = THRESHOLDS.get(extension)
if extension == NO_SOURCE_FILE and empty <= EMPTY_BODY_LIMIT:
return TypeReport(
extension=extension,
documents=0,
concepts=concepts,
empty=empty,
structure_null=0,
verdict=UNMEASURED,
threshold=None,
reason=(
f"no source_file on {concepts} of {concepts} concepts, so this "
"bundle names no file type at all -- the shape three of the four "
"evidence corpora arrive in, and nothing per file type can be said"
),
)
if empty > EMPTY_BODY_LIMIT:
verdict, reason = (
FAIL,
(
f"{empty} of {concepts} concepts carry no non-whitespace body; the "
"harness calls a zero-character merge degenerate by definition"
),
)
elif threshold is None:
verdict, reason = (
UNMEASURED,
(
"no measured threshold for this type; see "
"docs/2026-09-12-g37-terskler.md, and never read this row as PASS"
),
)
elif document_count < MIN_DOCUMENTS_FOR_A_THRESHOLD:
verdict, reason = (
UNMEASURED,
(
f"{document_count} document(s) of this type in the bundle, below the "
f"floor of {MIN_DOCUMENTS_FOR_A_THRESHOLD}: a share over that few "
"documents is not a rate, whatever the threshold says"
),
)
elif threshold.exceeded_by(structure_null, document_count):
verdict, reason = (
FAIL,
(
f"{structure_null} of {document_count} documents yielded one concept, "
f"worse than the reference {threshold.as_share()} ({threshold.source})"
),
)
else:
verdict, reason = (
PASS,
(f"no worse than the reference {threshold.as_share()} ({threshold.source})"),
)
return TypeReport(
extension=extension,
documents=document_count,
concepts=concepts,
empty=empty,
structure_null=structure_null,
verdict=verdict,
threshold=threshold,
reason=reason,
)
def parse_args(argv: list[str] | None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
prog=CLI_ID,
description=(
"Judge one bundle per file type, with the denominator. Three verdicts: "
"PASS (no worse than the pinned reference), FAIL, and UNMEASURED -- "
"which is never PASS. Exit 0 judged and clean, 1 at least one FAIL, "
"2 did not run, 3 nothing could be judged."
),
)
parser.add_argument("bundle", type=Path, help="the OKF bundle to judge")
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
if not args.bundle.is_dir():
print(f"{CLI_ID}: FAILED - no such bundle: {args.bundle}", file=sys.stderr)
return 2
try:
report = measure_bundle(args.bundle)
except (ConsumeError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)
return 2
print(report.render(), end="")
return report.exit_code
if __name__ == "__main__":
raise SystemExit(main())

46
tests/test_docs_promises.py
View file

@ -207,3 +207,49 @@ def test_the_readme_carries_only_one_file_type_table() -> None:
section = text.split("### Binary extraction", 1)[1].split("\n## ", 1)[0]
rows = [line for line in section.splitlines() if line.strip().startswith("|")]
assert not rows, f"a second file-type table is back under Binary extraction: {rows}"
# The `okf quality` thresholds, in the one place the README writes them. A bar
# published without a test goes false the way the format list did.
_THRESHOLD_LINE = re.compile(r"^<!-- quality-thresholds: (.+) -->$", re.MULTILINE)
THRESHOLD_DOCUMENT = PROJECT_ROOT / "docs" / "2026-09-12-g37-terskler.md"
def _declared_thresholds() -> dict[str, str]:
match = _THRESHOLD_LINE.search(README.read_text(encoding="utf-8"))
assert match is not None, (
"README.md carries no `<!-- quality-thresholds: ... -->` marker; without "
"it the published bars can drift from the ones the gate applies"
)
pairs = (token.strip().split("=") for token in match.group(1).split(","))
return {extension: share for extension, share in pairs}
def test_the_readme_names_exactly_the_thresholds_the_gate_applies() -> None:
from llm_ingestion_okf.quality import THRESHOLDS
assert _declared_thresholds() == {
extension: threshold.as_share() for extension, threshold in THRESHOLDS.items()
}
def test_the_threshold_document_carries_the_same_bars() -> None:
"""Three copies, one measurement: the code, the README and the document.
The document is where a bar's N and corpus live, so a bar that moved in the
code without moving there would publish a number nobody measured.
"""
from llm_ingestion_okf.quality import THRESHOLDS
text = THRESHOLD_DOCUMENT.read_text(encoding="utf-8")
for extension, threshold in THRESHOLDS.items():
row = f"| `{extension}` | `{threshold.metric}` | **{threshold.as_share()}** |"
assert row in text, f"{THRESHOLD_DOCUMENT.name} carries no row {row}"
def test_the_readme_quality_section_does_not_promise_a_quality_claim() -> None:
"""The one sentence that must not come back: PASS as a statement of quality."""
text = README.read_text(encoding="utf-8").lower()
assert "okf quality" in text
assert "regression bar against a pinned artifact" in text

226
tests/test_quality.py Normal file
View file

@ -0,0 +1,226 @@
"""`okf quality` -- the per-file-type verdict, and what it refuses to say.
`okf check` is a CONTRACT check: it asks whether a payload carries what a claim
must rest on. Measured 2026-09-10 by `vegnormal-okf`, it returned 0 findings and
exit 0 on three arms over one corpus whose hit@k ranged from 6 of 6 to 0 of 6.
This module is the other question -- did the cut find anything worth reading --
and its whole discipline is that it answers it PER FILE TYPE, with the
denominator printed, and never answers PASS for a type it has no measurement
for.
Each rule below is exercised in both directions on purpose: a rule that has only
ever been run against the case it fires on has not been measured, it has been
asserted. The thresholds themselves, and the four evidence corpora they were
read off, live in `docs/2026-09-12-g37-terskler.md`.
"""
from __future__ import annotations
from pathlib import Path
from llm_ingestion_okf import quality
_FRONTMATTER = """---
type: reference
title: {title}
source_file: {source_file}
ingested_at: 1970-01-01T00:00:00Z
---
{body}
"""
def _bundle(root: Path, concepts: list[tuple[str, str, str]]) -> Path:
"""A flat bundle: one root index, one file per concept.
`concepts` is `(stem, source_file, body)`. `source_file` may be the empty
string, which is the shape a bundle from another producer arrives in.
"""
root.mkdir(parents=True, exist_ok=True)
lines = ["---", "okf_version: 0.2", "bundle_id: quality-fixture", "---", ""]
for stem, source_file, body in concepts:
(root / f"{stem}.md").write_text(
_FRONTMATTER.format(title=stem, source_file=source_file, body=body),
encoding="utf-8",
newline="",
)
lines.append(f"- [{stem}]({stem}.md)")
(root / "index.md").write_text("\n".join(lines) + "\n", encoding="utf-8", newline="")
return root
def _pdf_documents(count: int, *, single: int) -> list[tuple[str, str, str]]:
"""`count` PDF documents, `single` of them yielding one concept only."""
concepts: list[tuple[str, str, str]] = []
for index in range(count):
source = f"doc-{index}.pdf"
segments = 1 if index < single else 3
for segment in range(segments):
concepts.append((f"doc-{index}-{segment}", source, "Body text enough to count."))
return concepts
def test_a_type_at_its_measured_reference_passes(tmp_path: Path) -> None:
"""The known-positive: 8 of 32 is the reference, and 8 of 32 is not worse."""
root = _bundle(tmp_path / "at-reference", _pdf_documents(32, single=8))
report = quality.measure_bundle(root)
row = report.row(".pdf")
assert row.verdict == "PASS"
assert (row.structure_null, row.documents) == (8, 32)
def test_a_type_worse_than_its_reference_fails(tmp_path: Path) -> None:
"""The known-negative for the same rule: 9 of 32 is worse than 8 of 32."""
root = _bundle(tmp_path / "over-reference", _pdf_documents(32, single=9))
report = quality.measure_bundle(root)
assert report.row(".pdf").verdict == "FAIL"
assert report.exit_code == 1
def test_an_empty_concept_fails_a_type_that_has_no_threshold(tmp_path: Path) -> None:
"""FAIL is reachable for every type; PASS is not. `.md` has no threshold."""
root = _bundle(
tmp_path / "empty-body",
[("a", "note.md", "Body text."), ("b", "note.md", " \n")],
)
report = quality.measure_bundle(root)
row = report.row(".md")
assert row.verdict == "FAIL"
assert (row.empty, row.concepts) == (1, 2)
def test_a_healthy_type_without_a_threshold_is_unmeasured_and_never_pass(
tmp_path: Path,
) -> None:
"""The other direction of the same rule: nothing wrong, and still not PASS."""
root = _bundle(
tmp_path / "no-threshold",
[("a", "note.md", "Body text."), ("b", "note.md", "More body text.")],
)
report = quality.measure_bundle(root)
assert report.row(".md").verdict == "UNMEASURED"
assert report.exit_code == 3
def test_concepts_without_a_source_file_are_unmeasured(tmp_path: Path) -> None:
"""The shape three of the four evidence corpora arrive in.
`n100-2023`, `n200-2024` and `n500-2024` carry `source_file` on 0 of 446,
0 of 1 133 and 0 of 270 concepts, so a per-file-type gate has no type to
speak about. Measured 2026-09-12; the gate says so instead of passing them.
"""
root = _bundle(tmp_path / "foreign", [("a", "", "Body text."), ("b", "", "More text.")])
report = quality.measure_bundle(root)
row = report.row(quality.NO_SOURCE_FILE)
assert row.verdict == "UNMEASURED"
assert report.exit_code == 3
def test_a_failing_type_outweighs_a_passing_one(tmp_path: Path) -> None:
concepts = _pdf_documents(32, single=9)
for index in range(5): # five documents: the floor a rate needs to be a rate
concepts.append((f"note-{index}-0", f"note-{index}.docx", "Body text."))
concepts.append((f"note-{index}-1", f"note-{index}.docx", "More body text."))
report = quality.measure_bundle(_bundle(tmp_path / "mixed", concepts))
assert report.row(".docx").verdict == "PASS"
assert report.row(".pdf").verdict == "FAIL"
assert report.exit_code == 1
def test_every_row_prints_its_denominator(tmp_path: Path) -> None:
root = _bundle(tmp_path / "printed", _pdf_documents(32, single=8))
rendered = quality.measure_bundle(root).render()
assert "8/32" in rendered
assert "0/80" in rendered # empty concepts, over the concepts of the type
assert "documents" in rendered
def test_the_bundle_run_log_is_reported_when_the_bundle_carries_one(tmp_path: Path) -> None:
"""A rejected document leaves NO row in the bundle: the log is the only trace.
On the pinned K2 bundle the log reads `N = 43, merged 39, coded rejections
4`, and the bundle itself shows 39 documents -- so the gate's own
denominator is the bundle's, never the corpus's, and it says which one it
is using.
"""
root = _bundle(tmp_path / "logged", _pdf_documents(2, single=0))
(root / "log.md").write_text(
"* **Ingested**: /x - N = 3 (the corpus directory's file count, computed at "
"run time), merged = 2 (2 substantive, 0 degenerate), coded rejections = 1.\n",
encoding="utf-8",
newline="",
)
report = quality.measure_bundle(root)
assert report.run_log == "N = 3, merged = 2, coded rejections = 1"
assert "coded rejections = 1" in report.render()
def test_a_bundle_without_a_log_says_so_rather_than_reporting_zero(tmp_path: Path) -> None:
root = _bundle(tmp_path / "unlogged", _pdf_documents(2, single=0))
report = quality.measure_bundle(root)
assert report.run_log is None
assert "no run log" in report.render()
def test_a_missing_bundle_exits_two(tmp_path: Path, capsys) -> None: # type: ignore[no-untyped-def]
assert quality.main([str(tmp_path / "nowhere")]) == 2
def test_the_cli_reaches_the_gate_through_okf_quality(tmp_path: Path, capsys) -> None: # type: ignore[no-untyped-def]
from llm_ingestion_okf import cli
root = _bundle(tmp_path / "through-cli", _pdf_documents(32, single=9))
assert cli.main(["quality", str(root)]) == 1
assert ".pdf" in capsys.readouterr().out
def test_every_threshold_names_the_corpus_and_the_denominator_it_was_read_off() -> None:
"""A threshold without an N is the thing this whole gate exists to refuse."""
assert quality.THRESHOLDS, "no thresholds at all would make every type UNMEASURED"
for extension, threshold in quality.THRESHOLDS.items():
assert extension.startswith("."), extension
assert threshold.documents >= quality.MIN_DOCUMENTS_FOR_A_THRESHOLD
assert threshold.source, f"{extension} threshold names no source"
assert threshold.limit_documents == threshold.documents
def test_a_rate_over_too_few_documents_is_not_a_rate(tmp_path: Path) -> None:
"""FOUND BY RUNNING THE GATE, not by reading it.
`~/repos/vegnormal-okf/build/sk2-bundle-default` is one PDF cut into 2 182
concepts. Against the 32-document reference its one-concept share is 0 of 1,
which the first version of this rule read as PASS -- a verdict resting on a
denominator of one, which is the exact failure `MIN_DOCUMENTS_FOR_A_THRESHOLD`
exists to refuse. The floor binds the BUNDLE's denominator too, not only the
threshold's.
"""
root = _bundle(tmp_path / "one-document", _pdf_documents(1, single=0))
assert quality.measure_bundle(root).row(".pdf").verdict == "UNMEASURED"
def test_the_floor_admits_a_bundle_that_reaches_it(tmp_path: Path) -> None:
"""The other direction: five documents is the floor, and five is enough."""
root = _bundle(tmp_path / "five-documents", _pdf_documents(5, single=1))
assert quality.measure_bundle(root).row(".pdf").verdict == "PASS"
def test_an_empty_concept_fails_even_below_the_floor(tmp_path: Path) -> None:
"""The empty-body bar is per concept, not a rate, so the floor does not gate it."""
root = _bundle(tmp_path / "one-document-empty", [("a", "only.pdf", " \n")])
assert quality.measure_bundle(root).row(".pdf").verdict == "FAIL"
def test_the_no_source_file_row_reports_no_document_count(tmp_path: Path) -> None:
"""A row that is not a file type has no documents either.
Every concept without the key shares the same empty `source_file`, so a
naive grouping reports `documents 1` and `one-concept documents 0/1` for a
bundle of 446 concepts -- three numbers that look measured and mean nothing.
"""
root = _bundle(tmp_path / "foreign-render", [("a", "", "Body."), ("b", "", "More.")])
rendered = quality.measure_bundle(root).render()
assert "documents 1" not in rendered
assert "one-concept documents" not in rendered
assert "empty 0/2" in rendered
assert "no source_file on 2 of 2 concepts" in rendered