test(fixtures): replace sector-specific example material with generic, fictitious examples — green
Every fixture, test document, tool example and document now uses an invented kitchen-and-baking handbook series, written in this repository. The package's behaviour is unchanged; src/ changes are comments and help text only. - Generated fixtures are regenerated from their generators. Their structural counts are identical before and after: elements, images, rows, cells, headings, bookmarks and the witness inventory's per-document totals. The image-inbox and accounting documents are renamed kapittel-84-*. - tools/okf_accounting_gate.py: the two options that named one real corpus each are replaced by a generic, repeatable --corpus PATH with no default. Row 5 compares the PDF pair alone. Gate verdict unchanged: RED rows 2, 3, 6. - tools/okf_witness.py: the STS JSON reader for one publisher's delivery is removed, along with its three twins and five tests. The mutation harness loses W09. - docs/: 13 dated reports that documented runs on a retired reference corpus are removed, and 40 are neutralized. Dead links are removed, and no new dangling path is introduced. - The synthetic MCP-gate corpus and the residual probe words are neutral. Valgt: keep the `okf quality --fasit` bar value (the measured fraction, one corpus) and rewrite only its provenance, because the verdict stays unchanged and the number names nothing. Term check with the local list: 0 of 411 tracked files, 0 file names, 0 of 27 binary fixtures. Suite after git add: 2457 passed, 1 skipped. The base tree had 2460 passed and 2 skipped; five tests went with the JSON reader and four were added by the term check. ruff, ruff format and mypy --strict src/ are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
1e7345a401
commit
9d1f4b14ed
174 changed files with 1889 additions and 6512 deletions
46
README.md
46
README.md
|
|
@ -159,7 +159,7 @@ that test red.
|
|||
| `.html` | `_extract_html` | core | measured | Block tags open their own lines and `h1`–`h6` carry the ATX marker for their own level. It does **not** go through the converter, although the converter reads HTML: that route would add CVE-2025-51591 (SSRF via an iframe in HTML input) and buy nothing. The denominator is 828 files — one product, one format, one publisher. |
|
||||
| `.htm` | `_extract_html` | core | stdlib, no corpus class | The same reader as `.html`. The 828-file class is recorded for `.html` alone, and this row does not borrow it. |
|
||||
| `.xml` | `_extract_xml` | core | measured | A NISO-STS document (`<standard>` root, or any `<sec>`) becomes one heading per titled section at the section's own nesting depth; any other XML keeps its text in document order and gets no invented structure. A `<!DOCTYPE` is refused unparsed. The denominator is one file, one publisher, one schema — 2 761 titled sections. |
|
||||
| `.pdf` | `_extract_pdf` | `[extract]`: pdfplumber | measured | Eight corpus documents with a hand-counted fasit, plus a 701-page process code whose publisher also ships its structure. Prose only: drawn content has no text to recover, and every extraction warns. OCR lives here as a reading mode for a PDF page whose own text never arrived (`--ocr`, `OCR_CID_SHARE`), never as an entry for image files. |
|
||||
| `.pdf` | `_extract_pdf` | `[extract]`: pdfplumber | measured | Eight corpus documents with a hand-counted fasit, plus a 701-page reference standard whose publisher also ships its structure. Prose only: drawn content has no text to recover, and every extraction warns. OCR lives here as a reading mode for a PDF page whose own text never arrived (`--ocr`, `OCR_CID_SHARE`), never as an entry for image files. |
|
||||
| `.docx` | `_extract_office` | `[extract]`: pypandoc-binary | measured | Five corpus documents. `source_lines` index the extracted text and not the original's paragraphs — the two counts agree on none of the five. |
|
||||
| `.xlsx` | `_extract_office` | `[extract]`: pypandoc-binary | measured | Written as pipe tables, one source row per line; a sheet name becomes a heading, and a row is located by `source_sheet` and `source_rows`. |
|
||||
| `.pptx` | `_extract_office` | `[extract]`: pypandoc-binary | constructed | N = 2 decks. 2 of 2 slide titles recovered on a deck that declares them, 0 of 2 on a deck that does not, where the converter writes `Slide 1` / `Slide 2` because it has no title to use. |
|
||||
|
|
@ -314,7 +314,7 @@ who meets one in an older document knows what they are looking at.
|
|||
measured bundle's payload changed one byte.
|
||||
- `v0.8.0` — `.xml` is a core file type, read as NISO-STS through the stdlib
|
||||
parser, and a section the source DECLARES takes the
|
||||
declared-structure route — one publisher's process code segments at 2 761 of
|
||||
declared-structure route — one publisher's reference standard segments at 2 761 of
|
||||
2 761 of its own declared sections at the shipped defaults. No other file
|
||||
type changes one byte, measured on the bytes.
|
||||
- `v0.7.0` — `okf project` builds the bundle `okf build` builds (they were one
|
||||
|
|
@ -365,8 +365,8 @@ at the bundle root, named `<sha256[:12]>-<the source's own base name>`, and the
|
|||
concept carries a two-line pointer where the picture stood:
|
||||
|
||||
```markdown
|
||||
|
||||
Image: graphics/tabell-84-2.png (120x90 px) -- Tabell 84-2 Toleranseklasser
|
||||
|
||||
Image: graphics/tabell-84-2.png (120x90 px) -- Tabell 84-2 Hevetidsklasser
|
||||
```
|
||||
|
||||
The first line is markdown, so a reader that renders the concept sees the
|
||||
|
|
@ -374,11 +374,11 @@ picture. The second states what the first cannot — the name the SOURCE gave th
|
|||
file and the size in pixels — which are the two facts a person checking the
|
||||
bundle against the original needs.
|
||||
|
||||
**Why it exists.** Measured on R761 Prosesskoden:2025, a process code published
|
||||
as a 701-page PDF and as a NISO-STS delivery: the process text is carried in
|
||||
**Why it exists.** Measured on a 701-page reference standard, published as a
|
||||
PDF and as a NISO-STS delivery: the process text is carried in
|
||||
full, and 12 `Tabell N-N` and 9 `Figur N-N` captions stand over nothing,
|
||||
because the publisher ships those tables as raster images in both deliveries.
|
||||
Process 84 says "toleranseklasse ... er gitt i tabell 84-2" and table 84-2 is a
|
||||
One section says a class "is given in table 84-2" and table 84-2 is a
|
||||
JPEG. A bundle like that reads as complete and is not.
|
||||
|
||||
**What it costs, measured on the 43-document reference corpus** (`K2/trinn1`,
|
||||
|
|
@ -410,7 +410,7 @@ beside it passes nothing. `log.md` says so on every run that carries one.
|
|||
|
||||
**Every carried image is one a model can be SHOWN.** A bundle that holds a
|
||||
picture nothing can read is worse than one that says the picture is missing:
|
||||
the count reports that it arrived. Measured over the frozen R761 delivery's own
|
||||
the count reports that it arrived. Measured over the frozen reference standard's own
|
||||
`assets/` (denominator 50): 29 JPEG, 2 PNG and **19 RLE8 BMP** — correct files
|
||||
that no model decodes. Every asset's type is read off its bytes and tested
|
||||
against the viewable set; a BMP is converted losslessly to PNG (8-bit
|
||||
|
|
@ -427,7 +427,7 @@ counts: images are carried on the CORE path, where `.html` and `.xml` need no
|
|||
an installed library would make a bundle's identity move with that library's
|
||||
version, which is the property page rasterisation was felled over. Pillow is
|
||||
the independent decoder in the tests instead, and against it **19 of 19** of
|
||||
R761's real RLE8 assets convert with identical RGB, 2 366 365 pixels compared.
|
||||
the reference standard's real RLE8 assets convert with identical RGB, 2 366 365 pixels compared.
|
||||
|
||||
A converted asset is ONE asset: one file in `assets/`, one pointer, one row in
|
||||
the accounting. The pointer's second line — where the source's own file name
|
||||
|
|
@ -450,7 +450,7 @@ move.
|
|||
`MAX_IMAGE_PIXELS` (40 000 000 pixels) or `MAX_IMAGE_BYTES` (256 MiB) is
|
||||
refused with `asset_too_large`, counted like every other refusal. The largest
|
||||
image in the 43-document reference corpus is 4 515 x 4 128 (18.6 MP) and the
|
||||
largest of R761's 109 pictures is 2 072 x 656 (1.4 MP), so the bound is an
|
||||
largest of the reference standard's 109 pictures is 2 072 x 656 (1.4 MP), so the bound is an
|
||||
order of magnitude above anything measured.
|
||||
|
||||
It exists because a few kilobytes can declare an enormous picture: a 9.6 KB
|
||||
|
|
@ -605,10 +605,10 @@ u and d both stay 0 and nothing else could see the loss.
|
|||
**The soft hyphen is removed before the persist gate, and counted** (operator
|
||||
decision 2026-09-18). U+00AD is in `llm-ingestion-guard`'s zero-width set, and
|
||||
`output:zero-width-present` is an any-tier carrier: a document carrying one is
|
||||
`fail_secure` at every trust level. Measured on R761 Prosesskoden:2025 — 71
|
||||
`fail_secure` at every trust level. Measured on a 701-page reference standard — 71
|
||||
U+00AD, and 0 of U+200B, U+200C, U+200D, U+FEFF and U+2060 — those 71 are
|
||||
Norwegian hyphenation points inside words (`ar[SHY]beider`, `bitu[SHY]men`),
|
||||
so a 701-page process code was unreadable for the whole chain over typography.
|
||||
Norwegian hyphenation points inside words (`ar[SHY]beider`, `sur[SHY]deig`),
|
||||
so a 701-page reference standard was unreadable for the whole chain over typography.
|
||||
`extract.normalise_extracted` removes that one character from every extracted
|
||||
text and reports the count as `normalised_soft_hyphen`, per document and for
|
||||
the run, in the accounting JSON and in a `**Normalisation**` bullet in
|
||||
|
|
@ -641,7 +641,7 @@ concept whose body is its heading alone a `parent:` naming the `segment_id` of
|
|||
the nearest ancestor that holds text — the nearest preceding plan entry at a
|
||||
smaller level, passing over an ancestor that is empty too. It copies no text
|
||||
and moves no boundary. It exists for a document that states its points once and
|
||||
lets every nested section inherit them: measured on one process code, **710 of
|
||||
lets every nested section inherit them: measured on one reference standard, **710 of
|
||||
2 761** concepts are heading-only, and the plan's level and order name the same
|
||||
ancestor as the document's own nesting on **710 of 710** since K3-21 (708
|
||||
before: the two others sit at depth 7, and the reader clipped their level to 6
|
||||
|
|
@ -700,7 +700,7 @@ bookmark destination is the contents listing rather than a second copy of the
|
|||
body, so a contents entry and the section it lists stop landing as two concepts
|
||||
under one id.
|
||||
|
||||
The measurement is one 701-page process code whose publisher also ships a
|
||||
The measurement is one 701-page reference standard whose publisher also ships a
|
||||
NISO-STS structure for it, so the fasit is the publisher's own. Under the
|
||||
shipped default that document gives 1967 of 2761 boundaries, none of its 28
|
||||
chapters, and 794 of 794 misses have their heading text present in the text the
|
||||
|
|
@ -753,7 +753,7 @@ rank was measured in the same session and did NOT ship; see
|
|||
| `--outline-gate` (**on** by default since 2026-09-09; opt out with `--no-outline-gate`) | admit `--outline-run`'s RECOVERED headings only where the document declares none of its own, plus any one recovered heading whose span covers `OUTLINE_SHARE` (0.20) of the text. Applied at admission, before spans are closed, so the text a removed mark opened is carried by the mark above it rather than lost | on the 12-document sample: `pdf` **2 of 8 → 5 of 8** alone and **7 of 8** with the rule above, `docx` unchanged at **3 of 3**. Whole corpus: it fires on **25 of 39** readable documents, changes the plan in **15 of 39**, and removes **64 of 485** proposed entries. No plan disappears (32 → 32) |
|
||||
| `--first-span-from-zero` (**on** by default since 2026-09-10; opt out with `--no-first-span-from-zero`) | start the first concept at character 0, so the text above it belongs to a segment instead of to none. Adds no boundary and removes none | Measured over the 39-document corpus, the default before this rule left **207 435 characters — 11.92 %** — in no segment at all: **163 804 above the first entry** (in **32 of the 32** documents that get a plan), 26 041 *between* entries and 17 590 after the last. This rule closes the first part entirely, 79 % of the whole, leaving **43 631 characters (2.51 %) over 8 of 32 documents** with two named mechanisms of their own. It adds no boundary and the K2 concept count is identical with and without it (**425 = 425**); on the 12-position reference it changes **not one cell**, and hit@8 on a K2 bundle built with it holds **5 of 6 at ranks 1,1,1,1,1,–** under both tie-breaks |
|
||||
| `--close-span-gaps` (**on** by default since 2026-09-11; opt out with `--no-close-span-gaps`) | close a concept's span against the next SURVIVING concept, and the last against the end of the text. Three steps remove a candidate AFTER its neighbour's span was already closed against it — the orphan check, and `fold_units` clause 1 both between entries and on the last run — and the removed mark's text then belongs to no segment. Adds no boundary and removes none; only spans' ends move | It closes the whole remainder the rule above left: **43 631 characters, 2.51 % of the corpus over 8 of the 32 documents with a plan, to 0** — both the 26 041 between entries and the 17 590 after the last. Decomposed: orphan check **18 527** over 15 of 39 documents, clause 1 **7 514** between entries, clause 1 on the last run **all 17 590** of the tail (with `unit_fold=False` the corpus tail gap is 0). The entry count is identical (**429 = 429** on the corpus, **436 = 436** concepts on K2, **52 = 52** md on a five-document folder); on the 12-position reference it changes **not one cell** (11 of 12 under `|F|`[3]=12, 10 of 12 under `|F|`[3]=11), and hit@8 holds **5 of 6 at ranks 1,1,1,1,1,–** on the new bundle, the previous default and Arm B alike |
|
||||
| `--pdf-outline` (**off**; opt out is the default, opt in with the flag) | cut a PDF at the boundaries its own `/Outlines` bookmark tree declares, instead of at the ones the text rules recover. A SEGMENTATION arm, not a reader option: the extracted text is byte for byte the same either way, and a PDF that carries no tree builds byte-identically with the flag on. The title comes from the BOOKMARK, so it is not cut short at the page's line break, and a page before the first bookmark destination is the contents listing rather than a second copy of the body | one 701-page process code whose publisher also ships a NISO-STS structure for it, so the fasit is the publisher's own: boundaries **1967 of 2761 → 2759 of 2761**, chapter level **0 of 28 → 28 of 28**, concept titles identical to the source title after normalisation **2761 of 2761**, false positives **163 of 2182 → 3 of 2762**, directories carrying two concept files **132 of 2050 → 2 of 2738**, contents-copy pairs **65 → 0**. Consumption on the same eight questions: fasit present in the bundle **4 of 7 → 7 of 7**, hit@1/8/50 **1/6 · 2/6 · 4/6 → 3/6 · 5/6 · 6/6**. Cost 119.22 s → 183.31 s wall, peak RSS 3252 → 3251 MiB, no new dependency and no second parse of the pages. **Off, and the reach is why:** **1 of the 8** reference PDFs carries a usable tree at all, and a bookmark tree is the publisher's CLAIM about its own structure — a stale or wrongly pointing one carries that error straight into the segmentation |
|
||||
| `--pdf-outline` (**off**; opt out is the default, opt in with the flag) | cut a PDF at the boundaries its own `/Outlines` bookmark tree declares, instead of at the ones the text rules recover. A SEGMENTATION arm, not a reader option: the extracted text is byte for byte the same either way, and a PDF that carries no tree builds byte-identically with the flag on. The title comes from the BOOKMARK, so it is not cut short at the page's line break, and a page before the first bookmark destination is the contents listing rather than a second copy of the body | one 701-page reference standard whose publisher also ships a NISO-STS structure for it, so the fasit is the publisher's own: boundaries **1967 of 2761 → 2759 of 2761**, chapter level **0 of 28 → 28 of 28**, concept titles identical to the source title after normalisation **2761 of 2761**, false positives **163 of 2182 → 3 of 2762**, directories carrying two concept files **132 of 2050 → 2 of 2738**, contents-copy pairs **65 → 0**. Consumption on the same eight questions: fasit present in the bundle **4 of 7 → 7 of 7**, hit@1/8/50 **1/6 · 2/6 · 4/6 → 3/6 · 5/6 · 6/6**. Cost 119.22 s → 183.31 s wall, peak RSS 3252 → 3251 MiB, no new dependency and no second parse of the pages. **Off, and the reach is why:** **1 of the 8** reference PDFs carries a usable tree at all, and a bookmark tree is the publisher's CLAIM about its own structure — a stale or wrongly pointing one carries that error straight into the segmentation |
|
||||
|
||||
They compose, and the order above is the order they apply in. Measured on a
|
||||
five-document tender folder (2 `pdf`, 2 `docx`, 1 `xlsx`), concepts per
|
||||
|
|
@ -907,7 +907,7 @@ with no bundle changing. It caps how many DELIVERED places one source document
|
|||
may take, cutting where the shortlist is cut so the freed place goes to the next
|
||||
candidate and `k` is still delivered in full. The defect it repairs was measured
|
||||
outside this repository on a 3206-concept bundle of a published handbook: the
|
||||
code's own process overview contributes **28 of 3206 concepts (0.87 %)** and
|
||||
handbook's own overview section contributes **28 of 3206 concepts (0.87 %)** and
|
||||
**8.0 % of the source characters**, and took **8 of 8** delivered places on one
|
||||
question and **7 of 8** on the known-positive, which was not delivered at all.
|
||||
Identical at 343 and 1651 concepts, so the cause is the corpus's COMPOSITION —
|
||||
|
|
@ -956,7 +956,7 @@ the whole row on Arm B, because row 1's token `prisene` reaches its gold
|
|||
document through `pris|sammenstilling` on four characters. The rule that works
|
||||
asks whether the shared prefix is a WORD the bundle uses: `bilateral` 400 → 0
|
||||
and 512 → 0, `standhaftig` 219 → 56 and 235 → 33, **every hit@8 row keeping
|
||||
rank 1 on both bundles**. `undersjøisk` stops at 162 because `under` is a word
|
||||
rank 1 on both bundles**. `under…` stops at 162 because `under` is a word
|
||||
here — a genuine Norwegian morpheme, so the residual is a different answer and
|
||||
not a ceiling.
|
||||
|
||||
|
|
@ -1211,9 +1211,8 @@ finding rather than a defect in the gate:
|
|||
| 8 | the real set, from path + sha256 | 0 of 1 set, NOT RUN without `--real` |
|
||||
|
||||
**The public rows run on invented material only** (operator decision
|
||||
2026-09-21). The test track built on material tied to the operator's employer
|
||||
is retired rather than re-measured or frozen: its row 9 is gone, row 8 no
|
||||
longer reads the two sets built on it, and no test or gate here needs that
|
||||
2026-09-21). A retired test track is not re-measured or frozen: its row 9 is
|
||||
gone, row 8 no longer reads the two sets built on it, and no test or gate here needs that
|
||||
material to run.
|
||||
|
||||
Rows 3 and 4 were this gate's two findings and both are closed, which is what
|
||||
|
|
@ -1520,8 +1519,7 @@ bundle:
|
|||
direct-child `<sec sec-type="spec">`, whole — and a section with none gets
|
||||
no `description` at all; nothing is derived from the title. A point a YAML
|
||||
reader could not read verbatim (a `: ` inside it) is left out rather than
|
||||
quoted or cleaned up. Measurements:
|
||||
[`docs/2026-09-11-k3-runde19-dokumentidentitet-og-frontmatter.md`](docs/2026-09-11-k3-runde19-dokumentidentitet-og-frontmatter.md).
|
||||
quoted or cleaned up.
|
||||
The drop directory is walked **recursively**, in sorted relative-path order:
|
||||
a file at any depth is ingested and records its path relative to the inbox
|
||||
root as its `source_file`, while dot-directories and a bundle directory
|
||||
|
|
@ -1666,7 +1664,7 @@ writes them into the concept's own frontmatter, and carries them into the index
|
|||
entry — so a consumer can reason over the bundle rather than only look things
|
||||
up in it. Every inferred field is named in a `derived` list, because an
|
||||
unmarked heuristic is worse than no heuristic: the consumer cannot know when to
|
||||
doubt it. A pointer to a document not dropped yet is rendered `N200?` rather
|
||||
doubt it. A pointer to a document not dropped yet is rendered `Q200?` rather
|
||||
than omitted, since a bundle is built up over several drops and an absence that
|
||||
leaves no trace is the dangerous kind. Carrying the metadata costs index
|
||||
characters — roughly 3x to 6x the flat index, depending on how many facets the
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue