docs: measurement prose names no document by its numbers — green
Section totals, a page count and build sizes identified one reference document and one generated HTML corpus to anyone who knows them. They are written as N, M, an offset of N or a share in docstrings, comments, help text, reports, README, CLAUDE.md and the changelog; no measured relation changes. Sums that let N be worked back are written in words. The `okf quality --fasit` bar is 999/1000 (99.9 %), just under the known-good arm's measured share, and every build measured keeps its verdict. Red with the local list widened to those numbers: 52 tracked files. Green: 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
ea8590db82
commit
81d7194e20
33 changed files with 412 additions and 398 deletions
71
README.md
71
README.md
|
|
@ -156,10 +156,10 @@ that test red.
|
|||
| `.txt` | `_extract_passthrough` | core | stdlib, no corpus class | As `.md`. A document with no headings yields no segments, which is a failed build rather than a flat bundle. |
|
||||
| `.csv` | `_extract_csv` | core | stdlib, no corpus class | Parsed with the stdlib reader and rendered as one markdown table; a file with no header row is refused. |
|
||||
| `.json` | `_extract_json` | core | stdlib, no corpus class | Fenced verbatim. No structure is derived from the keys. |
|
||||
| `.html` | `_extract_html` | core | measured | Block tags open their own lines and `h1`–`h6` carry the ATX marker for their own level. It does **not** go through the converter, although the converter reads HTML: that route would add CVE-2025-51591 (SSRF via an iframe in HTML input) and buy nothing. The denominator is 828 files — one product, one format, one publisher. |
|
||||
| `.htm` | `_extract_html` | core | stdlib, no corpus class | The same reader as `.html`. The 828-file class is recorded for `.html` alone, and this row does not borrow it. |
|
||||
| `.xml` | `_extract_xml` | core | measured | A NISO-STS document (`<standard>` root, or any `<sec>`) becomes one heading per titled section at the section's own nesting depth; any other XML keeps its text in document order and gets no invented structure. A `<!DOCTYPE` is refused unparsed. The denominator is one file, one publisher, one schema — 2 761 titled sections. |
|
||||
| `.pdf` | `_extract_pdf` | `[extract]`: pdfplumber | measured | Eight corpus documents with a hand-counted fasit, plus a 701-page reference standard whose publisher also ships its structure. Prose only: drawn content has no text to recover, and every extraction warns. OCR lives here as a reading mode for a PDF page whose own text never arrived (`--ocr`, `OCR_CID_SHARE`), never as an entry for image files. |
|
||||
| `.html` | `_extract_html` | core | measured | Block tags open their own lines and `h1`–`h6` carry the ATX marker for their own level. It does **not** go through the converter, although the converter reads HTML: that route would add CVE-2025-51591 (SSRF via an iframe in HTML input) and buy nothing. The denominator is one HTML corpus of M files — one product, one format, one publisher. |
|
||||
| `.htm` | `_extract_html` | core | stdlib, no corpus class | The same reader as `.html`. The HTML corpus class is recorded for `.html` alone, and this row does not borrow it. |
|
||||
| `.xml` | `_extract_xml` | core | measured | A NISO-STS document (`<standard>` root, or any `<sec>`) becomes one heading per titled section at the section's own nesting depth; any other XML keeps its text in document order and gets no invented structure. A `<!DOCTYPE` is refused unparsed. The denominator is one file, one publisher, one schema — N titled sections. |
|
||||
| `.pdf` | `_extract_pdf` | `[extract]`: pdfplumber | measured | Eight corpus documents with a hand-counted fasit, plus a long reference standard whose publisher also ships its structure. Prose only: drawn content has no text to recover, and every extraction warns. OCR lives here as a reading mode for a PDF page whose own text never arrived (`--ocr`, `OCR_CID_SHARE`), never as an entry for image files. |
|
||||
| `.docx` | `_extract_office` | `[extract]`: pypandoc-binary | measured | Five corpus documents. `source_lines` index the extracted text and not the original's paragraphs — the two counts agree on none of the five. |
|
||||
| `.xlsx` | `_extract_office` | `[extract]`: pypandoc-binary | measured | Written as pipe tables, one source row per line; a sheet name becomes a heading, and a row is located by `source_sheet` and `source_rows`. |
|
||||
| `.pptx` | `_extract_office` | `[extract]`: pypandoc-binary | constructed | N = 2 decks. 2 of 2 slide titles recovered on a deck that declares them, 0 of 2 on a deck that does not, where the converter writes `Slide 1` / `Slide 2` because it has no title to use. |
|
||||
|
|
@ -277,7 +277,7 @@ who meets one in an older document knows what they are looking at.
|
|||
key with an empty value. `okf.parse_frontmatter` is public API, so this
|
||||
changes what an outside caller reads: it returns a flow string where it
|
||||
returned an empty one. That string is a READING projection — PyYAML reads
|
||||
it back on 0 of the 4 605 block files measured, because the `?` opening a
|
||||
it back on 0 of the block files measured, because the `?` opening a
|
||||
query string in the source URL ends the flow scalar — and the emitter still
|
||||
writes flow, so no bundle bytes move. `okf consume` also stops scoring the
|
||||
door's own `Enclosing section:` link line, which is now the default reading;
|
||||
|
|
@ -309,13 +309,13 @@ who meets one in an older document knows what they are looking at.
|
|||
- `v0.8.1` — a question that accounts for a concept's WHOLE
|
||||
title reads that concept first (`--title-covered`, on by default, opt out
|
||||
with `--no-title-covered`). A ranking fix, no new functionality: on one
|
||||
publisher's 2 761-concept bundle the answering section was delivered at
|
||||
publisher's one-document bundle of a few thousand concepts the answering section was delivered at
|
||||
rank 1 on 3 of 6 scored questions before it and 6 of 6 after, and no other
|
||||
measured bundle's payload changed one byte.
|
||||
- `v0.8.0` — `.xml` is a core file type, read as NISO-STS through the stdlib
|
||||
parser, and a section the source DECLARES takes the
|
||||
declared-structure route — one publisher's reference standard segments at 2 761 of
|
||||
2 761 of its own declared sections at the shipped defaults. No other file
|
||||
declared-structure route — one publisher's reference standard segments at all N
|
||||
of its own declared sections at the shipped defaults. No other file
|
||||
type changes one byte, measured on the bytes.
|
||||
- `v0.7.0` — `okf project` builds the bundle `okf build` builds (they were one
|
||||
flag apart before it), and the generated skill states the question,
|
||||
|
|
@ -365,8 +365,8 @@ at the bundle root, named `<sha256[:12]>-<the source's own base name>`, and the
|
|||
concept carries a two-line pointer where the picture stood:
|
||||
|
||||
```markdown
|
||||
|
||||
Image: graphics/tabell-84-2.png (120x90 px) -- Tabell 84-2 Hevetidsklasser
|
||||
|
||||
Image: graphics/tabell-7-2.png (120x90 px) -- Tabell 7-2 Hevetidsklasser
|
||||
```
|
||||
|
||||
The first line is markdown, so a reader that renders the concept sees the
|
||||
|
|
@ -374,11 +374,11 @@ picture. The second states what the first cannot — the name the SOURCE gave th
|
|||
file and the size in pixels — which are the two facts a person checking the
|
||||
bundle against the original needs.
|
||||
|
||||
**Why it exists.** Measured on a 701-page reference standard, published as a
|
||||
**Why it exists.** Measured on a long reference standard, published as a
|
||||
PDF and as a NISO-STS delivery: the process text is carried in
|
||||
full, and 12 `Tabell N-N` and 9 `Figur N-N` captions stand over nothing,
|
||||
because the publisher ships those tables as raster images in both deliveries.
|
||||
One section says a class "is given in table 84-2" and table 84-2 is a
|
||||
One section says a class "is given in table N-N" and that table is a
|
||||
JPEG. A bundle like that reads as complete and is not.
|
||||
|
||||
**What it costs, measured on the 43-document reference corpus** (`K2/trinn1`,
|
||||
|
|
@ -605,10 +605,10 @@ u and d both stay 0 and nothing else could see the loss.
|
|||
**The soft hyphen is removed before the persist gate, and counted** (operator
|
||||
decision 2026-09-18). U+00AD is in `llm-ingestion-guard`'s zero-width set, and
|
||||
`output:zero-width-present` is an any-tier carrier: a document carrying one is
|
||||
`fail_secure` at every trust level. Measured on a 701-page reference standard — 71
|
||||
`fail_secure` at every trust level. Measured on a long reference standard — 71
|
||||
U+00AD, and 0 of U+200B, U+200C, U+200D, U+FEFF and U+2060 — those 71 are
|
||||
Norwegian hyphenation points inside words (`ar[SHY]beider`, `sur[SHY]deig`),
|
||||
so a 701-page reference standard was unreadable for the whole chain over typography.
|
||||
so that reference standard was unreadable for the whole chain over typography.
|
||||
`extract.normalise_extracted` removes that one character from every extracted
|
||||
text and reports the count as `normalised_soft_hyphen`, per document and for
|
||||
the run, in the accounting JSON and in a `**Normalisation**` bullet in
|
||||
|
|
@ -642,7 +642,7 @@ the nearest ancestor that holds text — the nearest preceding plan entry at a
|
|||
smaller level, passing over an ancestor that is empty too. It copies no text
|
||||
and moves no boundary. It exists for a document that states its points once and
|
||||
lets every nested section inherit them: measured on one reference standard, **710 of
|
||||
2 761** concepts are heading-only, and the plan's level and order name the same
|
||||
its N** concepts are heading-only, and the plan's level and order name the same
|
||||
ancestor as the document's own nesting on **710 of 710** since K3-21 (708
|
||||
before: the two others sit at depth 7, and the reader clipped their level to 6
|
||||
in the plan as well as in the markdown heading, so they pointed one level too
|
||||
|
|
@ -700,11 +700,11 @@ bookmark destination is the contents listing rather than a second copy of the
|
|||
body, so a contents entry and the section it lists stop landing as two concepts
|
||||
under one id.
|
||||
|
||||
The measurement is one 701-page reference standard whose publisher also ships a
|
||||
NISO-STS structure for it, so the fasit is the publisher's own. Under the
|
||||
shipped default that document gives 1967 of 2761 boundaries, none of its 28
|
||||
The measurement is one long reference standard whose publisher also ships a
|
||||
NISO-STS structure for it, so the fasit is the publisher's own: N declared
|
||||
sections. Under the shipped default that document gives 1967 of N boundaries, none of its 28
|
||||
chapters, and 794 of 794 misses have their heading text present in the text the
|
||||
build read. With the arm it gives 2759 of 2761 and 28 of 28. The flag stays off
|
||||
build read. With the arm it gives N − 2 of N (99.9 %) and 28 of 28. The flag stays off
|
||||
because reach is the open question, not quality: **1 of the 8** reference PDFs
|
||||
in this repository's own sample carries a usable tree, and a bookmark tree is
|
||||
the publisher's *claim* about its own structure — a stale or wrongly pointing
|
||||
|
|
@ -753,7 +753,7 @@ rank was measured in the same session and did NOT ship; see
|
|||
| `--outline-gate` (**on** by default since 2026-09-09; opt out with `--no-outline-gate`) | admit `--outline-run`'s RECOVERED headings only where the document declares none of its own, plus any one recovered heading whose span covers `OUTLINE_SHARE` (0.20) of the text. Applied at admission, before spans are closed, so the text a removed mark opened is carried by the mark above it rather than lost | on the 12-document sample: `pdf` **2 of 8 → 5 of 8** alone and **7 of 8** with the rule above, `docx` unchanged at **3 of 3**. Whole corpus: it fires on **25 of 39** readable documents, changes the plan in **15 of 39**, and removes **64 of 485** proposed entries. No plan disappears (32 → 32) |
|
||||
| `--first-span-from-zero` (**on** by default since 2026-09-10; opt out with `--no-first-span-from-zero`) | start the first concept at character 0, so the text above it belongs to a segment instead of to none. Adds no boundary and removes none | Measured over the 39-document corpus, the default before this rule left **207 435 characters — 11.92 %** — in no segment at all: **163 804 above the first entry** (in **32 of the 32** documents that get a plan), 26 041 *between* entries and 17 590 after the last. This rule closes the first part entirely, 79 % of the whole, leaving **43 631 characters (2.51 %) over 8 of 32 documents** with two named mechanisms of their own. It adds no boundary and the K2 concept count is identical with and without it (**425 = 425**); on the 12-position reference it changes **not one cell**, and hit@8 on a K2 bundle built with it holds **5 of 6 at ranks 1,1,1,1,1,–** under both tie-breaks |
|
||||
| `--close-span-gaps` (**on** by default since 2026-09-11; opt out with `--no-close-span-gaps`) | close a concept's span against the next SURVIVING concept, and the last against the end of the text. Three steps remove a candidate AFTER its neighbour's span was already closed against it — the orphan check, and `fold_units` clause 1 both between entries and on the last run — and the removed mark's text then belongs to no segment. Adds no boundary and removes none; only spans' ends move | It closes the whole remainder the rule above left: **43 631 characters, 2.51 % of the corpus over 8 of the 32 documents with a plan, to 0** — both the 26 041 between entries and the 17 590 after the last. Decomposed: orphan check **18 527** over 15 of 39 documents, clause 1 **7 514** between entries, clause 1 on the last run **all 17 590** of the tail (with `unit_fold=False` the corpus tail gap is 0). The entry count is identical (**429 = 429** on the corpus, **436 = 436** concepts on K2, **52 = 52** md on a five-document folder); on the 12-position reference it changes **not one cell** (11 of 12 under `|F|`[3]=12, 10 of 12 under `|F|`[3]=11), and hit@8 holds **5 of 6 at ranks 1,1,1,1,1,–** on the new bundle, the previous default and Arm B alike |
|
||||
| `--pdf-outline` (**off**; opt out is the default, opt in with the flag) | cut a PDF at the boundaries its own `/Outlines` bookmark tree declares, instead of at the ones the text rules recover. A SEGMENTATION arm, not a reader option: the extracted text is byte for byte the same either way, and a PDF that carries no tree builds byte-identically with the flag on. The title comes from the BOOKMARK, so it is not cut short at the page's line break, and a page before the first bookmark destination is the contents listing rather than a second copy of the body | one 701-page reference standard whose publisher also ships a NISO-STS structure for it, so the fasit is the publisher's own: boundaries **1967 of 2761 → 2759 of 2761**, chapter level **0 of 28 → 28 of 28**, concept titles identical to the source title after normalisation **2761 of 2761**, false positives **163 of 2182 → 3 of 2762**, directories carrying two concept files **132 of 2050 → 2 of 2738**, contents-copy pairs **65 → 0**. Consumption on the same eight questions: fasit present in the bundle **4 of 7 → 7 of 7**, hit@1/8/50 **1/6 · 2/6 · 4/6 → 3/6 · 5/6 · 6/6**. Cost 119.22 s → 183.31 s wall, peak RSS 3252 → 3251 MiB, no new dependency and no second parse of the pages. **Off, and the reach is why:** **1 of the 8** reference PDFs carries a usable tree at all, and a bookmark tree is the publisher's CLAIM about its own structure — a stale or wrongly pointing one carries that error straight into the segmentation |
|
||||
| `--pdf-outline` (**off**; opt out is the default, opt in with the flag) | cut a PDF at the boundaries its own `/Outlines` bookmark tree declares, instead of at the ones the text rules recover. A SEGMENTATION arm, not a reader option: the extracted text is byte for byte the same either way, and a PDF that carries no tree builds byte-identically with the flag on. The title comes from the BOOKMARK, so it is not cut short at the page's line break, and a page before the first bookmark destination is the contents listing rather than a second copy of the body | one long reference standard whose publisher also ships a NISO-STS structure for it, so the fasit is the publisher's own, N declared sections: boundaries **1967 of N → N − 2 of N (99.9 %)**, chapter level **0 of 28 → 28 of 28**, concept titles identical to the source title after normalisation **N of N**, false positives **163 of 2182 → 3 of N + 1**, directories carrying two concept files **132 of 2050 → 2 of N − 23**, contents-copy pairs **65 → 0**. Consumption on the same eight questions: fasit present in the bundle **4 of 7 → 7 of 7**, hit@1/8/50 **1/6 · 2/6 · 4/6 → 3/6 · 5/6 · 6/6**. Cost 119.22 s → 183.31 s wall, peak RSS 3252 → 3251 MiB, no new dependency and no second parse of the pages. **Off, and the reach is why:** **1 of the 8** reference PDFs carries a usable tree at all, and a bookmark tree is the publisher's CLAIM about its own structure — a stale or wrongly pointing one carries that error straight into the segmentation |
|
||||
|
||||
They compose, and the order above is the order they apply in. Measured on a
|
||||
five-document tender folder (2 `pdf`, 2 `docx`, 1 `xlsx`), concepts per
|
||||
|
|
@ -906,11 +906,11 @@ pre-exclusion, never before — and the payload then declares
|
|||
with no bundle changing. It caps how many DELIVERED places one source document
|
||||
may take, cutting where the shortlist is cut so the freed place goes to the next
|
||||
candidate and `k` is still delivered in full. The defect it repairs was measured
|
||||
outside this repository on a 3206-concept bundle of a published handbook: the
|
||||
handbook's own overview section contributes **28 of 3206 concepts (0.87 %)** and
|
||||
outside this repository on a bundle of a published handbook, a few thousand concepts: the
|
||||
handbook's own overview section contributes **0.87 % of its concepts** and
|
||||
**8.0 % of the source characters**, and took **8 of 8** delivered places on one
|
||||
question and **7 of 8** on the known-positive, which was not delivered at all.
|
||||
Identical at 343 and 1651 concepts, so the cause is the corpus's COMPOSITION —
|
||||
Identical at a tenth and at half of the corpus, so the cause is the corpus's COMPOSITION —
|
||||
that it holds its own table of contents — and not its size; any corpus with a
|
||||
contents list, a project overview or a summary document has the same property.
|
||||
Swept over {2, 3, 4, off} on three bundles: at 2 and 3 hit@8 goes **5 of 6 to
|
||||
|
|
@ -994,7 +994,7 @@ much of the CANDIDATE the question accounts for, so a section titled with the
|
|||
question's subject alone scores exactly what a narrower section titled with that
|
||||
subject plus a qualifier scores — and then loses on the body count, because a
|
||||
longer title and a longer body can only reach more of the question. Measured on
|
||||
a 2 761-concept bundle of one standard, where **none of the six flags above
|
||||
a one-document bundle of a few thousand concepts, where **none of the six flags above
|
||||
moved the number at all**: hit@1/8/50 **3 of 6 · 5 of 6 · 5 of 6 → 6 of 6 · 6 of
|
||||
6 · 6 of 6** at default `k`, the same 6 of 6 at `--k 50`, the known-positive
|
||||
holding rank 1 at both and the known-negative still not a hit. The three
|
||||
|
|
@ -1003,7 +1003,7 @@ recovered rows go 4 → 1, not-delivered → 1 and 3 → 1.
|
|||
It is a PARTITION and not a fourth RRF signal, and the arithmetic is the
|
||||
reason: RRF consumes ranks alone, so with shared ranks a rule whose positive
|
||||
group has `m` members is worth `1/61 − 1/(61 + m)`, and a rule firing on ONE
|
||||
concept of 2 761 is worth 0.00026 against a body-signal gap of 0.0029 — **a
|
||||
concept of N is worth 0.00026 against a body-signal gap of 0.0029 — **a
|
||||
precise rule is worth least under this fusion, backwards from what precision is
|
||||
for**. Measured as a signal the same predicate moves hit@1 not at all; as a
|
||||
partition it reaches 6 of 6. `lookup_hits` is the same shape for the same
|
||||
|
|
@ -1159,21 +1159,24 @@ never `UNMEASURED`. A boundary counts as recovered in either of two forms — a
|
|||
concept whose normalised title equals `norm`, or the pair of the concept's own
|
||||
directory and its residual title — because the numbering token a publisher glues
|
||||
into a heading lands in the concept *id* on one route and in the *title* on
|
||||
another: measured on one 2 761-section standard, the first form alone reaches
|
||||
**22 of 2 761** where the two together reach **2 759**.
|
||||
another: measured on one standard with N declared sections, the first form alone reaches
|
||||
**22 of N** where the two together reach **N − 2**.
|
||||
|
||||
<!-- quality-boundary-threshold: 2759/2761 -->
|
||||
<!-- quality-boundary-threshold: 999/1000 -->
|
||||
|
||||
| metric | threshold | N |
|
||||
|---|---|---|
|
||||
| `boundary_share` | 2 759/2 761 | 2 761 declared boundaries, **1 corpus** |
|
||||
| `boundary_share` | **999/1000** | N declared boundaries, **1 corpus** |
|
||||
|
||||
The bar is 99.9 %, just under the known-good arm's measured share; every build
|
||||
measured reads the same under it.
|
||||
|
||||
**`--fasit` is an assertion**, the way `okf consume --ref` is: it says this
|
||||
bundle is a build of the document the fasit describes. A bundle of another
|
||||
product scores near zero and reads `FAIL` — that is the assertion being wrong,
|
||||
not the bundle. The bar itself rests on **one product**, which the output says
|
||||
on every run. Both facts, the arm it separates (1 148 of 2 761 against 2 759 of
|
||||
2 761) and the interval any bar could sit in are in
|
||||
on every run. Both facts, the arm it separates (1 148 of N against N − 2 of
|
||||
N) and the interval any bar could sit in are in
|
||||
[`docs/2026-09-12-g37-terskler.md`](docs/2026-09-12-g37-terskler.md) § 7.
|
||||
|
||||
## Judge the retrieval: `python3 tools/okf_retrieval_gate.py`
|
||||
|
|
@ -1391,7 +1394,7 @@ re-reads the directories and recomputes the bundle's content identity, so the
|
|||
identity in an answer is a fact about the bytes at the moment of the call
|
||||
rather than at startup — a server that answered from yesterday's bundle is the
|
||||
one failure you cannot see from the outside. The cost is real and is paid per
|
||||
call: on a 2 756-concept bundle the identity is a 0.75 s hash of the whole
|
||||
call: on a one-document bundle of a few thousand concepts the identity is a 0.75 s hash of the whole
|
||||
concept tree, and one `okf_ask` is 5.6 s.
|
||||
|
||||
**Refusals are loud.** A path climbing out of the bundle, a symlink leaving the
|
||||
|
|
@ -1736,8 +1739,8 @@ blur it:
|
|||
(`materialize.parse_frontmatter`, which is public API, and the two internal
|
||||
ones). A block sequence of mappings is decoded into the flow rendering those
|
||||
readers round-trip; the entries never enter the document's key namespace.
|
||||
Measured against PyYAML 6.0.3 and the pinned guard 1.4.0 on 4 609 of 4 609
|
||||
concept files carrying a block `sources`, all three readers return the same
|
||||
Measured against PyYAML 6.0.3 and the pinned guard 1.4.0 on every concept
|
||||
file of four bundles carrying a block `sources`, all three readers return the same
|
||||
entries both references do.
|
||||
- **Every other key, flow only.** `executor`, `attester` and any other block
|
||||
mapping are still skipped rather than parsed: two block mappings that both
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue