open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

docs: measurement prose names no document by its numbers — green

Browse source 
Section totals, a page count and build sizes identified one reference
document and one generated HTML corpus to anyone who knows them. They are
written as N, M, an offset of N or a share in docstrings, comments, help
text, reports, README, CLAUDE.md and the changelog; no measured relation
changes. Sums that let N be worked back are written in words. The
`okf quality --fasit` bar is 999/1000 (99.9 %), just under the known-good
arm's measured share, and every build measured keeps its verdict.

Red with the local list widened to those numbers: 52 tracked files. Green: 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-23 14:51:17 +02:00
commit 81d7194e20
33 changed files with 412 additions and 398 deletions
Download patch file Download diff file Expand all files Collapse all files

125
CLAUDE.md
View file

@ -27,7 +27,7 @@ one boundary rule:
`" ".join("".join(parts).split())`, and `str.split()` with no argument splits
on newlines too, so extraction of ANY HTML file returned unconditionally ONE
line while every boundary grammar in `propose` is line-anchored -- measured
outside this repo, **828 of 828** real sections gave 0 plans and exit 2 at
outside this repo, **M of M** real sections gave 0 plans and exit 2 at
every sample point, and a coarser 145-document cut gave 145 of 145. Block tags
now open their own lines and `h1`-`h6` carry the ATX marker for their OWN level
(a flat `#` would hand `_ATX` three top-level boundaries where the document
@ -35,10 +35,10 @@ one boundary rule:
same the office rows reach the proposer through, so **no HTML-only heading
grammar exists**; the fix is in the extractor and **never** the converter,
because `.html` stays out of `_PANDOC_FORMATS` on CVE-2025-51591. After:
**828 of 828 plans, exit 0, 3206 concepts / 6015 md -- the markdown path's
count EXACTLY**, and the same at 414 (1651) and 83 (343). Text preservation is
**M of M plans, exit 0, the markdown path's concept and file
count EXACTLY**, and the same at half and at a tenth of the corpus. Text preservation is
an EXACT invariant and not a percentage: strip the added ATX markers and the
non-whitespace sequence is identical to the old extractor's, **828 of 828
non-whitespace sequence is identical to the old extractor's, **M of M
files**, character ratio **1.000000**. `_SKIP_TAGS` stays `{script, style}`.
Exposure elsewhere measured rather than argued: **0 of 86** K2 corpus files and
**0 of 5** smoke-folder files are HTML, and the smoke bundle is byte-identical
@ -46,16 +46,16 @@ one boundary rule:
`measured` since 2026-09-10, the row with the most measurement behind it and
no entry in the table at all -- with the limit
that travels with it -- one product, one format, one publisher, and a
generator's cut, not 828 documents anyone wrote.
generator's cut, not M documents anyone wrote.
**`.xml` became a CORE type 2026-09-11** and it is the first row whose
ceiling is structural rather than recovered. A NISO-STS zip from a
publisher's own viewer was **110 of 110 unreadable, 0 plans, exit 2** --
`.xml` was in neither registry -- and the one xml file in it IS the whole
product: a large regulatory reference document, the one round 12 met as a
701-page PDF, carrying 7 715 `<sec>`, **2 761 with a `<title>`**, 4 954 with a
`<label>` and no title, 10 `<table-wrap>`, root `<standard>`, 0 `<!DOCTYPE`.
Its `<sec>`-nesting depths over the titled sections are
28/118/500/1141/868/97/9, row for row the fasit's own. The reader is stdlib
long PDF, carrying several thousand `<sec>`, **N (36 %) with a `<title>`**, most
(64 %) with a `<label>` and no title, 10 `<table-wrap>`, root `<standard>`,
0 `<!DOCTYPE`. Its `<sec>`-nesting depths over the titled sections are,
row for row, the fasit's own. The reader is stdlib
(`xml.etree.ElementTree`) and adds NO dependency -- `defusedxml` and `lxml`
are 0 occurrences in `uv.lock` -- so it is core beside `.html` rather than
behind `[extract]`, which would make a pure-stdlib type binary-dependent.
@ -63,17 +63,17 @@ one boundary rule:
proposer through: `<sec>` with a `<title>`
becomes one ATX line carrying `<label>` + space + `<title>` at its own
nesting depth, `<sec>` with only a `<label>` becomes a body line with the
label in front (never a heading -- 4 954 of 7 715 are lettered points and one
heading each buries the document's own 2 761), and `<table-wrap>` becomes its
label in front (never a heading -- 64 % are lettered points and one
heading each buries the document's own N), and `<table-wrap>` becomes its
label plus one markdown table through `render_table`. **`<label>` carries the
number and `<title>` carries the text** -- 2 of 2 761 titles begin with a
digit -- so emitting `<title>` alone scores 0 of 2 761 with nothing in the
number and `<title>` carries the text** -- 2 of N titles begin with a
digit -- so emitting `<title>` alone scores 0 of N with nothing in the
code looking wrong. Inline is an ALLOWLIST and block is the default, the
inverse of the HTML reader, because block-versus-inline is a property of HTML
and XML has no such universal; the allowlist is load-bearing at 1 701
`<italic>` and 1 396 `<bold>` inside that document's prose. The ATX ceiling is
6 and STS nesting reaches 7, so the depth is CLIPPED and not dropped: 9 of
2 761 sit at depth 7 and `#######` matches nothing. Since K3-21 the clip is
N sit at depth 7 and `#######` matches nothing. Since K3-21 the clip is
the HEADING's alone: the `OutlineMark` beside it carries the declared depth,
so the plan reads those 9 at 7. A `<!DOCTYPE` is REFUSED
unparsed with its own code, a guarantee about the code rather than about the
@ -91,33 +91,33 @@ one boundary rule:
"0 plans because nothing could be read" changes the outcome on **0 of the 4**
reference corpora.
**THE READER REACHED ITS CEILING IN ROUND 13 AND THE BUILD DID NOT, AND
ROUND 14 CLOSED IT AT THE SHIPPED DEFAULTS.** The reader emitted 2 761 of
2 761 heading lines while the build delivered **23 concepts and 15 of 2 761
ROUND 14 CLOSED IT AT THE SHIPPED DEFAULTS.** The reader emitted N of
N heading lines while the build delivered **23 concepts and 15 of N
boundaries** -- two steps after the reader, each measured: the orphan check
took **710 of 2 761** (710 of 710 removed headings are followed immediately
took **710 of N** (710 of 710 removed headings are followed immediately
by another heading, **0 of 2 051** delivered ones are -- they are container
sections) and Arm F took **2 066** more, 2 089 -> 23. `find_candidates`
already skipped both for `outline_marks`, which is why the PDF bookmark arm
reaches 2 762; an STS `<sec><title>` is the same class of declaration and
reaches N + 1; an STS `<sec><title>` is the same class of declaration and
only arrived as `rule:heading`. The fix is ONE new rule constant reached from
ONE row: `extract.xml_outline` reports the marks the reader WROTE ITSELF --
**no bridge, no tolerance constant, no `unresolved` bucket**, the difference
from `pdf_outline` whose naive nearest-line rule was wrong on 1 840 of 2 762
from `pdf_outline` whose naive nearest-line rule was wrong on 1 840 of N + 1
-- `propose.RULE_XML_SECTION` (`rule:xml-section`) is its own name in
`RULE_NAMES` and `_ORPHAN_EXEMPT`, and `build_plan` chooses the route by the
**ROW** (`DECLARED_STRUCTURE_IDS`), never by the text: the same markdown from
a `.md` file is still a guess and still carries `rule:heading`. At shipped
defaults, no flag: **2 761 concepts**, **2 761 of 2 761** declared sections
defaults, no flag: **N concepts**, **N of N** declared sections
became a concept with the source's own directory AND title, **0** concepts
matching no declaration, `a)`-points **0 of 4 954**, table blocks **10 of
matching no declaration, `a)`-points **0 of the label-only points**, table blocks **10 of
10**, hit@1/8/50 **3/6 / 5/6 / 6/6** from 0/6 / 0/6 / 0/6 with the
known-positive at rank 1, and **2 761 shared concept ids** with the PDF arm
(100 % of this bundle, 2 761 of 2 762 of that one) against round 13's 2 022.
known-positive at rank 1, and **N shared concept ids** with the PDF arm
(100 % of this bundle, N of N + 1 of that one) against round 13's 2 022.
**NO other file type changes one byte and it is MEASURED on the bytes**: the
whole 43-document reference corpus rebuilt is `diff -r`-identical to the
pinned bundle (865 md), the five-document folder is `diff -r`-identical,
`okf project` stays byte-equal to `okf build`, and the PDF arm still proposes
2 762. Two directories of 2 738 still hold two concepts (`11`, `12`) -- the
N + 1. Two directories of N − 23 still hold two concepts (`11`, `12`) -- the
publisher reuses a section number, the same 2 the PDF arm has, and 0 is not
reachable without inventing an id; round 13's 14 such directories were false
positives of the TEXT route reading the document's own contents listing and
@ -134,7 +134,7 @@ one boundary rule:
and never cleaned up. A titled section's `description` is its own FIRST
spec point (first `<p>` of the first DIRECT-child `sec-type="spec"`, whole),
carried by the plan entry, screened by the gate, and written only where a
YAML reader reads it verbatim (`inbox._yaml_plain`): 2 026 of 2 761 titled
YAML reader reads it verbatim (`inbox._yaml_plain`): 2 026 of N titled
sections on the reference standard carry a point, **1 807** are written (2 have no `<p>`, 217
carry `: ` and PyYAML refused exactly those frontmatters), none invented.
SS 4.1 sets no length, so the one-paragraph limit is ours. **The directory
@ -197,10 +197,10 @@ one boundary rule:
read, an STS `<graphic>` was walked past, a PDF was opened for text alone,
the converter's markdown writer dropped every picture, and the only writer
into a bundle was `materialize.write_bytes(..., content: str)`. Measured on
a 701-page reference standard: the section text is carried in full while **12
a long reference standard: the section text is carried in full while **12
`Tabell N-N` and 9 `Figur N-N`** captions stand over nothing, because that
publisher ships those tables as raster pictures in BOTH the PDF and the
NISO-STS delivery -- a section's "is given in table 84-2"
NISO-STS delivery -- a section's "is given in table N-N"
points at empty space. **Five readers PLACE and one module DECIDES:**
`assets.py` owns what an image is (sniffed from the bytes, never from the
claimed extension), what it is called (`<sha256[:12]>-<the source's own
@ -802,7 +802,7 @@ and fixtures, never code.
double-quoted where a YAML reader would not read it plain, and a flow value
goes out as given but is REFUSED (exit 2) when a leaf has no flow form both
PyYAML and the guard read -- so a `sources` URL with a query string, the form
K3-19's own flagged build wrote on 2 761 of 2 761 concepts, fails the build. It adds any key and REPLACES only
K3-19's own flagged build wrote on every concept, fails the build. It adds any key and REPLACES only
`sources` and `description`, the two with a derived layer below them:
precedence flag > what the document declares > file name. Every other key
the door writes (`inbox._door_keys`, including Door A's `ingest_manifest`,
@ -861,15 +861,15 @@ and fixtures, never code.
at the boundaries its own `/Outlines` bookmark tree declares. It is NOT Arm D
-- `--outline-run`/`--outline-gate` are a TEXT heuristic over numbered lines
in the extracted text, and this opens a structure index the file already
carries. Measured on ONE 701-page reference standard whose publisher also ships a
NISO-STS structure for it, so the fasit is the publisher's own 2 761 titled
sections: the shipped default finds **1 967 of 2 761**, **0 of its 28**
carries. Measured on ONE long reference standard whose publisher also ships a
NISO-STS structure for it, so the fasit is the publisher's own N titled
sections: the shipped default finds **1 967 of N**, **0 of its 28**
chapters, and **794 of 794** misses have their heading text PRESENT in the
extracted text -- the line is read, the boundary is never opened. With the
arm: **2 759 of 2 761 (99.9 %)**, chapter level **28 of 28**, concept titles
identical to the source after normalisation **2 761 of 2 761** (the bookmark
arm: **N − 2 of N (99.9 %)**, chapter level **28 of 28**, concept titles
identical to the source after normalisation **N of N** (the bookmark
title is complete because it does not come from the page), false positives
**163 of 2 182 -> 3 of 2 762**, directories carrying two concept files
**163 of 2 182 -> 3 of N + 1**, directories carrying two concept files
**132 -> 2** with the 65 contents-copy pairs at **0**, front-matter concepts
**72 -> 2**. Consumption: fasit present in the bundle **4 of 7 -> 7 of 7**,
hit@1/8/50 **1/6 - 2/6 - 4/6 -> 3/6 - 5/6 - 6/6**; the known-positive is a
@ -877,15 +877,15 @@ and fixtures, never code.
the segmentation half of that row is closed and the ranking half is not.
**The bridge from (page, `/XYZ` top) to a line index is the whole risk and
BOTH routes are measured:** `extract_text_lines` splits lines identically to
`extract_text` on **701 of 701** pages and that check SHIPS per page, the y
route and the title route disagree on **0 of 2 762**, flat from 0 to 8 pt and
`extract_text` on **every** page and that check SHIPS per page, the y
route and the title route disagree on **0 of N + 1**, flat from 0 to 8 pt and
collapsing at 12, so the rule carries **no tolerance constant**; the naive
nearest-line rule was wrong on **1 840 of 2 762**, one line early every time.
nearest-line rule was wrong on **1 840 of N + 1**, one line early every time.
The orphan check is NOT applied to a bookmark mark -- it asks whether anything
stands under a candidate's first line, the right question for a guess and the
wrong one for a publisher's declaration; **683 of 2 762** marks are container
wrong one for a publisher's declaration; **683 of N + 1** marks are container
sections and applying it scores 2 079. An unresolvable `/Dest` is dropped and
COUNTED (the reference standard has 0 of 2 763; one of the eight reference PDFs has 2 of 2). NO
COUNTED (the reference standard has 0 of N + 2; one of the eight reference PDFs has 2 of 2). NO
new dependency: `pdfminer.six` already ships under `pdfplumber` in
`[extract]`, so `uv.lock` is untouched and `pypdf` stays out. Cost 119.22 s ->
183.31 s wall, peak RSS 3 252 -> 3 251 MiB, pages parsed 1 -> 1. **The default
@ -1214,7 +1214,7 @@ and fixtures, never code.
column, never also `extractor_unknown`. The files come from
`ExtractedDocument.files` (what the resolver actually resolved), never
from byte equality, which would credit the reference standard's 7 unpointed duplicates.
- **A 701-page reference standard WAS refused whole by guard 1.4.0 because of its 71 U+00AD soft
- **A long reference standard WAS refused whole by guard 1.4.0 because of its 71 U+00AD soft
hyphens, and since 2026-09-19 okf removes them itself.** Operator decision
2026-09-18, the chosen alternative verbatim: **one documented
normalisation door BEFORE the guard**, `extract.normalise_extracted`,
@ -1435,15 +1435,16 @@ the first retired set **8** (S1-S6 + KP + KN), the second **32** questions /
be a silent pass (0 clean, 1 a `FAIL`, 2 did not run). Two bars today, both
`structure_null_share` off the pinned 43-document reference -- `.pdf` 8/32,
`.docx` 2/5 -- plus one definitional bar for every type (0 empty bodies,
measured **0 of 8 602** concepts over four bundles). A bar needs **five
measured **0** over every concept of four bundles). A bar needs **five
documents on BOTH sides**, its own and the judged bundle's, which was found
by RUNNING the gate: a one-PDF bundle scored 0 of 1 against the 32-document
reference and read as `PASS`. **The bars are regression bars against a
pinned artifact, never a quality claim**, and the defect that started G37 --
the HTML arm's 1 148 of 2 761 boundaries -- is `UNMEASURED` here, because no
the HTML arm's 1 148 of the reference standard's N declared boundaries --
is `UNMEASURED` here, because no
bundle-only metric reaches it: three candidates were measured over the same
four bundles and **two order the known-bad and known-good arms the wrong way
round** (duplicate titles within a document 0/3 206 against 349/2 761; short
round** (duplicate titles within a document 0.0 % against 12.6 %; short
concepts 5.6 % against 14.6 %), while the third (duplicate titles across the
whole bundle, 37.8 / 16.3 / 12.6 / 5.7 %) orders them correctly and ships
anyway WITHOUT a bar, since any bar separating them sits between the two
@ -1461,34 +1462,37 @@ the first retired set **8** (S1-S6 + KP + KN), the second **32** questions /
input this gate takes: one whole-bundle row, `boundary_share` = declared
boundaries that became a concept over declared boundaries. It is whole-bundle
and never per file type, because a fasit names ONE document's sections and a
bundle can spread them over 828 source files -- which the known-bad arm does.
bundle can spread them over hundreds of source files -- which the known-bad
arm does.
**The normalisation was derived before the metric was built**, not guessed:
strip all whitespace, lowercase, reproduces the fasit's own `norm` from its
own `title` on **2 761 of 2 761** rows (alphanumerics-only scores 58 -- it
own `title` on **all N** rows (alphanumerics-only scores 58 -- it
eats the `.` in `2.1Hovedkapitler`). **A boundary is recovered in EITHER of
two forms and neither is a fallback**: the concept's normalised title equal to
`norm`, or the pair `(concept's own directory, residual title)` -- the literal
form wants the declared title WITH its numbering token, the pair form WITHOUT,
and no bundle can offer both, because okf's default route moves that token
into the concept id. Measured on the known-good arm: literal **22 of 2 761**,
paired **2 737**, either **2 759 (99.9 %)**; on an earlier build of the same document the split is
exactly inverted (2 727 literal, 0 paired), so a gate scoring one form alone
into the concept id. Measured on the known-good arm: literal **22 of N**,
paired **N − 24**, either **N − 2 (99.9 %)**; on an earlier build of the same document the split is
exactly inverted (N − 34 literal, 0 paired), so a gate scoring one form alone
reports a 99.9 % arm as 0.8 % and calls it a segmentation defect. The two
forms are a downstream consumer repository's M8 correction, which it took verbatim from THIS
repository's round-14 report -- the instrument reproduces both so the two
repos cannot silently measure different things. One bar, at the pinned
artifact's own value: **2 759/2 761 with `corpora = 1`**, and P2 is in the
repos cannot silently measure different things. One bar, **999/1000 with
`corpora = 1`**: 99.9 %, just under the known-good arm's measured share
(N − 2 of N), and every build measured reads the same verdict under it as
under that measured fraction. P2 is in the
OUTPUT and not only in the document (`N = 1 corpus` on every boundary row).
The known-bad arm is **1 148 of 2 761 (41.6 %)**, now `FAIL` + exit 1 where
The known-bad arm is **1 148 of N (41.6 %)**, now `FAIL` + exit 1 where
the bundle-only gate gave exit 3. **`--fasit` is an ASSERTION** (the posture
`okf consume --ref` has) that this bundle is a build of the document the fasit
describes: the K2 reference and a downstream bundle of another document both score **0 of 2 761** and read
describes: the K2 reference and a downstream bundle of another document both score **0 of N** and read
`FAIL`, which is the assertion being wrong and not the bundle -- a gate
telling those apart would need a bar read off the bundles it judges. The bar
is TIGHT and the cost is published: 2 of 4 builds of that document fall under it (2 752
and 2 727 of 2 761), while **any bar between 41.6 % and 98.8 % separates the
known-bad arm from every build of that document measured** -- the shipped one is the only
point in that interval read off a pinned artifact. An unreadable fasit exits
is TIGHT and the cost is published: 2 of 4 builds of that document fall under it (N − 9
and N − 34 of N, 99.7 % and 98.8 %), while **any bar between 41.6 % and 98.8 % separates the
known-bad arm from every build of that document measured** -- the shipped one sits
above that interval, just under the one share read off a pinned artifact. An unreadable fasit exits
**2** with its reason, never a quiet `UNMEASURED`, and a fasit under five rows
is `UNMEASURED` (`MIN_DECLARED_FOR_A_THRESHOLD`, the document floor in the
fasit's unit). **Without `--fasit` the command is byte-for-byte what it was**,
@ -1508,7 +1512,8 @@ the first retired set **8** (S1-S6 + KP + KN), the second **32** questions /
removed or rebuilt while the process runs is seen by the next call with no
restart, no config edit and no code change (measured, 9 of 9 discovery checks
over three bundles written while serving). The cost is paid per call and is
published: **0.75 s** for the identity of a 2 756-concept bundle, **5.6 s**
published: **0.75 s** for the identity of a one-document bundle of a few
thousand concepts, **5.6 s**
for one ask. Containment is TWO independent checks -- the bundle's own index
must name the concept AND `connectors.safe_resolve` must place it inside the
bundle -- and a mutant removing either one alone still refuses, with a
@ -1780,11 +1785,11 @@ the first retired set **8** (S1-S6 + KP + KN), the second **32** questions /
NO bundle changing. It caps how many DELIVERED places one `source_file` may
take, cutting where `shortlist = candidates[:k]` cuts, so the freed place goes
to the next candidate and `k` is still delivered in full. The defect was
measured OUTSIDE this repo on a 3206-concept bundle of a published handbook:
the code's own process overview is **28 of 3206 concepts (0.87 %)** and **8.0 %
measured OUTSIDE this repo on a bundle of a published handbook, a few thousand
concepts: the code's own process overview is **0.87 % of its concepts** and **8.0 %
of the source characters** yet took **8 of 8** delivered places on one question
and **7 of 8** on the known-positive, which was not delivered at all --
identical at 343 and 1651 concepts, so it is the corpus's COMPOSITION (it holds
identical at a tenth and at half of the corpus, so it is the corpus's COMPOSITION (it holds
its own table of contents) and not its size, and a split would move it rather
than remove it. Swept over {2, 3, 4, off} on three bundles with the fasit
prefixes validated against the bundle FIRST (that control caught a defect in