Kjell Tore Guttormsen
9d1f4b14ed
Every fixture, test document, tool example and document now uses an invented kitchen-and-baking handbook series, written in this repository. The package's behaviour is unchanged; src/ changes are comments and help text only. - Generated fixtures are regenerated from their generators. Their structural counts are identical before and after: elements, images, rows, cells, headings, bookmarks and the witness inventory's per-document totals. The image-inbox and accounting documents are renamed kapittel-84-*. - tools/okf_accounting_gate.py: the two options that named one real corpus each are replaced by a generic, repeatable --corpus PATH with no default. Row 5 compares the PDF pair alone. Gate verdict unchanged: RED rows 2, 3, 6. - tools/okf_witness.py: the STS JSON reader for one publisher's delivery is removed, along with its three twins and five tests. The mutation harness loses W09. - docs/: 13 dated reports that documented runs on a retired reference corpus are removed, and 40 are neutralized. Dead links are removed, and no new dangling path is introduced. - The synthetic MCP-gate corpus and the residual probe words are neutral. Valgt: keep the `okf quality --fasit` bar value (the measured fraction, one corpus) and rewrite only its provenance, because the verdict stays unchanged and the number names nothing. Term check with the local list: 0 of 411 tracked files, 0 file names, 0 of 27 binary fixtures. Suite after git add: 2457 passed, 1 skipped. The base tree had 2460 passed and 2 skipped; five tests went with the JSON reader and four were added by the term check. ruff, ruff format and mypy --strict src/ are clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
77 lines
4.1 KiB
Markdown
77 lines
4.1 KiB
Markdown
# Intake boundaries: what will never enter through the inbox
|
|
|
|
Two boundaries on Door B's intake are decided, not open. Both were left
|
|
unstated for long enough to be mistaken for gaps someone would eventually
|
|
close. They will not be closed, so they are written here instead — an unstated
|
|
consequence is the one that bites later.
|
|
|
|
## 1. OCR is permanently out of scope
|
|
|
|
**Decision: operator, 2026-09-01.** The library does no optical character
|
|
recognition, and no work assumes it will. A PDF whose pages are raster images
|
|
carries no extractable text layer, `pdfplumber` returns nothing for it, and the
|
|
file is refused with `extractor_empty_pdf`. That refusal is correct and final,
|
|
not a placeholder.
|
|
|
|
**The consequence, stated plainly.** A downstream consumer's PDFs are rastered 11
|
|
of 11 (measured, `docs/2026-08-29-konverteringsarmen-arm-a.md`). They will
|
|
never enter through the inbox. Any statement of the form "those
|
|
PDFs in ⇒ bundle out" is not a goal this library holds, and the phrasing should
|
|
not survive in planning documents.
|
|
|
|
For the record, that premise never described anything that happened:
|
|
That consumer builds its bundle from seven JSON files through **Door C**, not
|
|
through the inbox. The inbox path has never been the route those concepts took.
|
|
|
|
A consumer with rastered source documents needs a text layer produced outside
|
|
this library before the inbox is involved. Where that text comes from is the
|
|
consumer's decision; this library's answer is a coded refusal, not a
|
|
best-effort guess.
|
|
|
|
## 2. Door B lowercases every path component
|
|
|
|
`materialize.reduce_to_id_grammar` reduces each path component to
|
|
`[a-z0-9][a-z0-9-]*`, and `segmentation.normalize_segment_path` applies the
|
|
same grammar to segment paths. A source path of `krav/Q500/id-<uuid>` therefore
|
|
becomes `krav/q500/id-<uuid>`.
|
|
|
|
**This is the library's Phase 1 grammar, not the specification's.** OKF v0.2
|
|
§ 2 defines a concept ID as the file path minus `.md` and imposes no case rule.
|
|
The grammar is stricter than the spec requires.
|
|
|
|
**Decision: keep it, 2026-09-01.** The grammar is shared with Door A, which
|
|
implements a spec `portfolio-optimiser-commons` owns, and it is pinned by
|
|
byte-exact golden fixtures. Softening it on the segmentation path alone would
|
|
put two grammars in one codebase — the same drift class that a single ordering
|
|
helper was introduced to remove.
|
|
|
|
**The consequence, stated plainly.** A source whose identities are
|
|
case-sensitive cannot be reproduced verbatim through Door B. This is measured,
|
|
not predicted: in that consumer's B3 run against one 270-concept source, 267 of 270
|
|
concept identities matched case-folded and 0 matched verbatim — the entire
|
|
difference across all 267 was case alone.
|
|
|
|
A consumer that needs verbatim case-sensitive identities should use Door C,
|
|
which writes external concepts verbatim, or accept the folded form.
|
|
|
|
## 3. Structured sources are not segmentable as they stand
|
|
|
|
Measured by that consumer 2026-09-01 (B3, commit `0f8a27f`), with a
|
|
known-positive control that held at 270 of 270 through their domain path:
|
|
|
|
- The segment proposer returned **0 segments** for that source. The raw file is
|
|
1 633 250 characters on a single line, and the proposer's rules are
|
|
line-oriented. The same proposer returns 3 segments for a markdown document
|
|
with headings, so the zero is a measurement rather than a broken query.
|
|
- A hand-authored plan reproduced **267 of 270 identities but 0 of 267 bodies
|
|
byte-identically**, because a span cuts the canonical extracted text — and
|
|
for `.json` that text is the raw data inside a fenced block, not prose.
|
|
|
|
The span model assumes the extracted text *is* the concept text. For a
|
|
hierarchical element tree it is not. This is a structural limit of the current
|
|
extraction layer, and unlike §§ 1 and 2 above it is **open work**, not a closed
|
|
boundary. What is decided is only the shape of the eventual fix: the extraction
|
|
layer gains a renderer a profile can name, so a structured source can be
|
|
rendered rather than handed to segmentation as fenced raw bytes. A renderer
|
|
that reproduces a specific domain's prose needs that domain's vocabulary and is
|
|
not this library's to write without a commission.
|