Kjell Tore Guttormsen
9d1f4b14ed
Every fixture, test document, tool example and document now uses an invented kitchen-and-baking handbook series, written in this repository. The package's behaviour is unchanged; src/ changes are comments and help text only. - Generated fixtures are regenerated from their generators. Their structural counts are identical before and after: elements, images, rows, cells, headings, bookmarks and the witness inventory's per-document totals. The image-inbox and accounting documents are renamed kapittel-84-*. - tools/okf_accounting_gate.py: the two options that named one real corpus each are replaced by a generic, repeatable --corpus PATH with no default. Row 5 compares the PDF pair alone. Gate verdict unchanged: RED rows 2, 3, 6. - tools/okf_witness.py: the STS JSON reader for one publisher's delivery is removed, along with its three twins and five tests. The mutation harness loses W09. - docs/: 13 dated reports that documented runs on a retired reference corpus are removed, and 40 are neutralized. Dead links are removed, and no new dangling path is introduced. - The synthetic MCP-gate corpus and the residual probe words are neutral. Valgt: keep the `okf quality --fasit` bar value (the measured fraction, one corpus) and rewrite only its provenance, because the verdict stays unchanged and the number names nothing. Term check with the local list: 0 of 411 tracked files, 0 file names, 0 of 27 binary fixtures. Suite after git add: 2457 passed, 1 skipped. The base tree had 2460 passed and 2 skipped; five tests went with the JSON reader and four were added by the term check. ruff, ruff format and mypy --strict src/ are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4.1 KiB
Intake boundaries: what will never enter through the inbox
Two boundaries on Door B's intake are decided, not open. Both were left unstated for long enough to be mistaken for gaps someone would eventually close. They will not be closed, so they are written here instead — an unstated consequence is the one that bites later.
1. OCR is permanently out of scope
Decision: operator, 2026-09-01. The library does no optical character
recognition, and no work assumes it will. A PDF whose pages are raster images
carries no extractable text layer, pdfplumber returns nothing for it, and the
file is refused with extractor_empty_pdf. That refusal is correct and final,
not a placeholder.
The consequence, stated plainly. A downstream consumer's PDFs are rastered 11
of 11 (measured, docs/2026-08-29-konverteringsarmen-arm-a.md). They will
never enter through the inbox. Any statement of the form "those
PDFs in ⇒ bundle out" is not a goal this library holds, and the phrasing should
not survive in planning documents.
For the record, that premise never described anything that happened: That consumer builds its bundle from seven JSON files through Door C, not through the inbox. The inbox path has never been the route those concepts took.
A consumer with rastered source documents needs a text layer produced outside this library before the inbox is involved. Where that text comes from is the consumer's decision; this library's answer is a coded refusal, not a best-effort guess.
2. Door B lowercases every path component
materialize.reduce_to_id_grammar reduces each path component to
[a-z0-9][a-z0-9-]*, and segmentation.normalize_segment_path applies the
same grammar to segment paths. A source path of krav/Q500/id-<uuid> therefore
becomes krav/q500/id-<uuid>.
This is the library's Phase 1 grammar, not the specification's. OKF v0.2
§ 2 defines a concept ID as the file path minus .md and imposes no case rule.
The grammar is stricter than the spec requires.
Decision: keep it, 2026-09-01. The grammar is shared with Door A, which
implements a spec portfolio-optimiser-commons owns, and it is pinned by
byte-exact golden fixtures. Softening it on the segmentation path alone would
put two grammars in one codebase — the same drift class that a single ordering
helper was introduced to remove.
The consequence, stated plainly. A source whose identities are case-sensitive cannot be reproduced verbatim through Door B. This is measured, not predicted: in that consumer's B3 run against one 270-concept source, 267 of 270 concept identities matched case-folded and 0 matched verbatim — the entire difference across all 267 was case alone.
A consumer that needs verbatim case-sensitive identities should use Door C, which writes external concepts verbatim, or accept the folded form.
3. Structured sources are not segmentable as they stand
Measured by that consumer 2026-09-01 (B3, commit 0f8a27f), with a
known-positive control that held at 270 of 270 through their domain path:
- The segment proposer returned 0 segments for that source. The raw file is 1 633 250 characters on a single line, and the proposer's rules are line-oriented. The same proposer returns 3 segments for a markdown document with headings, so the zero is a measurement rather than a broken query.
- A hand-authored plan reproduced 267 of 270 identities but 0 of 267 bodies
byte-identically, because a span cuts the canonical extracted text — and
for
.jsonthat text is the raw data inside a fenced block, not prose.
The span model assumes the extracted text is the concept text. For a hierarchical element tree it is not. This is a structural limit of the current extraction layer, and unlike §§ 1 and 2 above it is open work, not a closed boundary. What is decided is only the shape of the eventual fix: the extraction layer gains a renderer a profile can name, so a structured source can be rendered rather than handed to segmentation as fenced raw bytes. A renderer that reproduces a specific domain's prose needs that domain's vocabulary and is not this library's to write without a commission.