Three STS fixtures still carried the section titles and labels of one real
reference document, and three identifiers were copies of its codes with a
letter or a word swapped. They now describe an invented kitchen counter and
cookbook series: the titles, labels and descriptions of sts-identity.xml,
sts-inherit.xml and sts-empty-label.xml, the P350/P351 document codes, the
99-0001 delivery prefix and chapter 7 of the image and accounting corpora.
Generated fixtures are regenerated and the witness inventory's per-document
totals are identical before and after; only names and text move.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every fixture, test document, tool example and document now uses an invented
kitchen-and-baking handbook series, written in this repository. The package's
behaviour is unchanged; src/ changes are comments and help text only.
- Generated fixtures are regenerated from their generators. Their structural
counts are identical before and after: elements, images, rows, cells,
headings, bookmarks and the witness inventory's per-document totals. The
image-inbox and accounting documents are renamed kapittel-84-*.
- tools/okf_accounting_gate.py: the two options that named one real corpus
each are replaced by a generic, repeatable --corpus PATH with no default.
Row 5 compares the PDF pair alone. Gate verdict unchanged: RED rows 2, 3, 6.
- tools/okf_witness.py: the STS JSON reader for one publisher's delivery is
removed, along with its three twins and five tests. The mutation harness
loses W09.
- docs/: 13 dated reports that documented runs on a retired reference corpus
are removed, and 40 are neutralized. Dead links are removed, and no new
dangling path is introduced.
- The synthetic MCP-gate corpus and the residual probe words are neutral.
Valgt: keep the `okf quality --fasit` bar value (the measured fraction, one corpus) and
rewrite only its provenance, because the verdict stays unchanged and the
number names nothing.
Term check with the local list: 0 of 411 tracked files, 0 file names, 0 of
27 binary fixtures. Suite after git add: 2457 passed, 1 skipped. The base
tree had 2460 passed and 2 skipped; five tests went with the JSON reader and
four were added by the term check. ruff, ruff format and mypy --strict src/
are clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two findings of the 18.09 PM checkpoint of `0f308c1`. Red tests landed
first in `3b587ea`; this is what turns them green.
BLOCKER -- `_check_inflated` read `filters[0]`, measured that one link and
returned, which is not a bound: a PDF decodes a stream through a LIST of
filters. Measured in paired subprocesses from two pinned trees, idle machine:
[/FlateDecode] 400 MB 408 516 B 59 232 256 -> 62 017 536 B
[/FlateDecode x2] 400 MB 1 636 B 886 554 624 -> 52 367 360 B
[/FlateDecode x3] 400 MB 1 070 B 889 393 152 -> 61 390 848 B
[/FlateDecode x2] 1,2 GB 2 927 B 2 567 204 864 -> 60 403 712 B
542 000x the file at two links, and the picture WAS refused at the end -- by
`check_payload` after `get_data()`, once the memory was spent. The single-link
row is the control and does not move. It also left the 16 corpus objects behind
an `[/ASCII85Decode /FlateDecode]` chain unmeasured, since `filters[0]` is not
`FlateDecode` there.
`_check_stream_cost` walks every link. THREE CLASSES and no fourth
(`extract.bounded_pdf_filters`, pinned by a test): `FlateDecode` MEASURED, a
link with another expanding link behind it inflated under the same bound and
handed on; `ASCII85Decode`/`ASCIIHexDecode` bounded by their own input because
they SHRINK; `DCTDecode`/`JPXDecode`/`JBIG2Decode` PASS THROUGH. Everything
else -- `LZWDecode`, `RunLengthDecode`, `CCITTFaxDecode`, `/Crypt`, anything
written later -- is refused UNREAD with a new code `asset_pdf_unbounded`,
decided before the FIRST link is decoded so a document cannot make the run pay
for the links in front of the one we cannot bound. An encrypted stream is
deciphered and then measured, where `stream.decipher is not None` used to
return unmeasured; 0 of 5 142 objects here are in an encrypted document, which
is why nothing caught it.
NOT ONE PICTURE CHANGES HANDS, AND IT IS MEASURED BY NAME. Every PDF on this
machine -- 78 documents, K2 in both trinn1 and trinn2, the shipped fixtures and
R761 -- run through `_pdf_images` page by page from both pinned trees:
images carried 9 356 -> 9 356
documents losing one 0 of 78
documents gaining one 0 of 78
asset_pdf_unsupported 322 -> 314
asset_pdf_unbounded 0 -> 8
The 8 are the 4 `CCITTFaxDecode` stencil masks (`/ImageMask true`,
`/BitsPerComponent 1`), counted twice because trinn1 and trinn2 hold the same
document. They were refused before and are refused now, one step earlier and
under a code that says why.
MAJOR -- `check_payload(len(data))` after `get_data()` is the counted refusal
four documentation surfaces point at, and deleting exactly that line passed all
2 132 tests. It is reachable through a stream pdfminer has ALREADY decoded
(`decode()` sets `rawdata` to `None`), which is now the ONLY case outside the
bound and has a test.
Eight mutations, one line each, every one DEAD, with the unmutated tree run
first as the control: first-link-only, loop dropped, inequality reversed,
encrypted skipped, backstop deleted, unknown filter passed through,
intermediate link not carried forward, whole check removed.
`tools/okf_accounting_gate.py` gains one line, the new code in
`REJECTION_CODES` -- what a rejection code requires and nothing more. Gate
unchanged: exit 1, GATE RED rows 2, 3, 6. Version stays 0.10.1, untagged.
Suite after `git add` against a clean tree: `uv run pytest -q` ->
2152 passed, 1 skipped (226 s). ruff, ruff format --check, mypy --strict clean.
Report: docs/2026-09-18-filterkjeden-og-backstoppen.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A second independent review read `230d1cb` -- the commit that closed the
`v0.10.0` review's two MAJOR findings -- and found one of them open. The
bound read `/Width` and `/Height`, which an untrusted document writes, while
`get_data()` pays for the stream beside them; `/Length` is the COMPRESSED
length and the two numbers are independent.
Re-measured here on `ed8d9d7` before anything changed, in its own
interpreter: a 408 516-byte PDF declaring 1x1 and carrying 400 MB of deflated
zeros was CARRIED, no rejection, 891 904 000 B peak RSS. After: 0 carried,
`asset_too_large`, 57 065 472 B. At 1,2 GB of zeros, 2 436 MB -> 64 569 344 B
-- the cost no longer scales with the bomb. End to end through the CLI with
the shipped defaults: 838 000 640 B and an asset written -> exit 0,
79 650 816 B, `0 carried of 1 found`, no `assets/`.
Three numbers are bounded now, not one: what a container DECLARES, what a
carried FILE measures (`read_image`, so a 49 MP PNG of 47 705 bytes is not
passed on to a consumer), and what a PDF stream DECOMPRESSES to
(`assets.inflated_size`, a chunk at a time, output discarded, before
`get_data()`). The limit is stated rather than implied: the stream
measurement runs where `FlateDecode` is the first filter and the document is
not encrypted; every other chain is a check on the decoded length AFTER the
decode, a counted refusal and not a bounded one.
A non-positive declared dimension is `asset_size_invalid`, its own code,
raised before the stream is read. `-1 x 40000000000` is a NEGATIVE pixel
count, under which every `>` bound read as satisfied, so the check returned
silently and the refusal arrived from `encode_png` as
`asset_samples_invalid`. Its own code because a publisher shipping a picture
bigger than this package carries and a dictionary written to be read wrong
are different facts about a document.
Two smaller findings in the line that says what is missing, both introduced
by the first fix: the address was written twice, once bare, and a linkifying
renderer autolinks a bare URL -- written once now, in one code span; and
`label` became a dead parameter, so the figure's caption was dropped, a
regression against 0.10.0. It is written again in the `-- <label>` form a
carried pointer uses.
Version bumped to 0.10.1 across all ten places. Nine were unbound and stale:
four README install lines naming the previous release, two prose lines, the
"current tag" entry, `uv.lock`, and a CHANGELOG whose 0.10.1 content sat
under `[Unreleased]`. Two new packaging tests bind them to `__version__`, and
the README's guard tag to `[tool.uv.sources]`.
Every test was red first. The fate of every image is identical with and
without the new bound on three K2 PDFs carrying 800 images (464/464, 311/311
with the same 12 rejections, 25/25), and the second inflate is below the
noise floor there. 0 shipped artifacts move: no bundle under `examples/`,
`skills/` or `tests/fixtures/` carries an image pointer at all, measured
against a known-positive control.
`asset_too_large` was undocumented in the error registry; both codes are
there now. `tools/okf_accounting_gate.py` gains the new code in its closed
list -- one string, no behaviour change, stated because that file belongs to
another order.
Suite 2141 passed / 1 skipped, ruff + format + mypy --strict clean.
Report: docs/2026-09-18-bildestien-holder-0-10-1.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Until now no reader in this package fetched, named, described or copied a
single image. `<img>`'s attributes were never read, a NISO-STS `<graphic>`
was walked past, a PDF was opened for its text alone, the converter's
markdown writer dropped every picture, and the only writer into a bundle
took `content: str`. The two lossiness warnings said so on every run, which
made the loss honest and did not make it smaller.
Measured on R761 Prosesskoden:2025, published as a 701-page PDF and as a
NISO-STS delivery: the process text is carried in full while 12 `Tabell N-N`
and 9 `Figur N-N` captions stand over nothing, because that publisher ships
those tables as raster pictures in both. Process 84's "toleranseklasse ...
er gitt i tabell 84-2" points at empty space.
THE GATE WAS WRITTEN FIRST AND RED. `tests/test_asset_gate.py` reads its
denominator out of the source (`page.images`, `word/media/`, `ppt/media/`,
`<img`, `<graphic`), never from a constant here. Measured at 332961a, built
from `git archive` and not from the editable tree: carried 0 of 8 local
images across 5 documents (9 declared), and no `assets/` at all. After: 8 of
8, with the ninth a remote source carried as a pointer without a file.
FIVE READERS PLACE, ONE MODULE DECIDES. `assets.py` owns what an image is
(sniffed from the bytes, never from the claimed extension), what it is
called (`<sha256[:12]>-<the source's own basename>`) and how it is pointed
at (one two-line block, one regex). `.xlsx` is deliberately not a row: a
block inside its pipe tables would break the `source_rows` locator, and 0 of
4 K2 workbooks hold media.
A PDF stream that is already a file is carried VERBATIM (29 of R761's 50
objects are DCTDecode); raw samples are encoded to PNG with stdlib zlib, so
no new dependency. Rendering the page region was the alternative and was
felled on determinism: a rasterised crop's bytes, and therefore the asset's
content-addressed name and the bundle's digest, would depend on the
installed rasteriser. What the encoder cannot express exactly is refused
with a code and counted, never approximated.
NO SIZE FLOOR, and that is a measurement: over the 4 828 image objects of
the K2 corpus the size distribution is a broad spread with no gap, unlike
OCR_CID_SHARE's bimodal one, so a threshold would be a number we chose.
ON BY DEFAULT, AND THE CONTROL IS TWO WHOLE BUILDS. The 43-document
reference corpus at 332961a versus rebuilt at HEAD with `--no-assets`:
865 files on both sides, `diff -rq` reports ONE difference, the added
`Images: NOT CARRIED` line in log.md. Every concept byte-identical.
Against the default: 453 -> 454 concepts, 865 -> 867 md, 0 -> 2 964 assets
(2 964 carried of 3 145 found, 4 622 pointers), 4.7 MB -> 115 MB, 2 414 s ->
3 088 s, peak RSS 6.26 -> 8.74 GB, 422 of 865 md files differ. The one new
concept has a measured cause: the pointers are body text, so a section
holding 146 of that document's images grew from 19.0 % to 30.6 % of the
extracted text and crossed `--outline-gate`'s 0.20 share clause.
THE IMAGE BYTES ARE NOT SCREENED. The guard is text-only, the pointer block
passes the gate as body text, the picture beside it passes nothing, and
log.md says so on every run.
Also fixed, both found by measuring rather than by reading:
- a markdown image is no longer read as a cross-reference. `structure._LINK`
never looked at the character in front of the bracket, so every pointer
would have arrived in the index as an edge to a concept that cannot exist.
- Door C carries the assets its merged concepts point at. Before this,
importing a bundle built with `--assets` merged 6 of 6 concepts and wrote
no `assets/` at all, so every pointer named a missing file.
Report: docs/2026-09-17-bilder-i-bundlen-trinn1.md
Spec proposal: docs/plan/okf-assets-section-6-4.md
Suite 1 955 passed / 1 skipped (from 1 896), ruff and mypy --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A PDF carries no notion of a heading -- a heading in a PDF is a typographic
fact -- so the text stream `pdfplumber` hands the segment proposer has already
thrown away the only evidence there was. The `docx` path never had that problem:
the converter emits ATX headings and `_ATX` cuts on them. Two readers close the
gap, and both are OFF.
`--pdf-headings font` infers a heading from the conjunction this repository
already measured (size above the document's character-weighted body median AND
a bold font name, recall 1.000 / precision 0.846) and emits it as ATX in the
SAME markdown the office path produces, so `_ATX` applies unchanged and no
PDF-only heading grammar exists.
It stays off BY MEASUREMENT, and the measurement is the point of the round:
against the operator's unit worksheet it takes `pdf` from 2 of 8 to 0 of 8,
losing two exact matches. The mechanism of the loss is stated rather than
guessed -- on those documents the outline rule already recovers the document's
own numbered chapters, so a second heading source can only add. Whole-corpus
screen: 25 of 32 `pdf` change, 0 of 5 `docx`, 0 of 2 `xlsx`. The default bundle
is byte-identical before and after this commit (`diff -r`, exit 0).
`--ocr` reads a page as an image when its own text never arrived: empty, or
`(cid:N)` placeholder codes at or above a threshold READ OFF a measured
distribution -- 834 pages over 32 files, 818 at exactly 0.0 and 16 at 0.93 or
above, nothing in between. On the one corpus document with the failure: 95.07 %
cid to 0 %, 44 to 2561 words of four or more letters, 17 to 18 pages with text.
Its engine is an optional dependency group and never a runtime dependency; a
packaging test pins both halves, and without the group every affected file is a
coded rejection (`extractor_ocr_group_missing`) rather than a crash.
Also corrects two stale published facts found while measuring: the README still
said two segmentation rules were on by default after `f6fea13` made it three,
and CLAUDE.md's K2 digest named the round-3 default. The current default is
492 concepts / 944 files, `bdefa679...`.
Report: docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md
Co-Authored-By: Claude <claude-opus-5>
Arm E, capability only. A profile MAY name a renderer per suffix; no
domain-aware renderer is written here, that stays a Non-Goal, and `_RENDERERS`
is empty on purpose so the emptiness reads as a decision rather than an
omission.
THE LAYERING IS THE DESIGN, not an implementation detail. `extract.py` is the
extraction registry and must not import the contract layer, or the dependency
runs backwards and the registry stops standing on its own. So `extract_text`
gains a keyword-only `renderer: Callable[[str], str] | None`, knowing nothing
about profiles, and `inbox.py` -- which already holds the profile at that call
site -- resolves a NAME to a function. A test asserts extract.py still contains
no reference to the profile layer, because that constraint is the whole reason
the parameter is shaped this way.
The renderer runs AFTER extraction, never instead of it, so it never has to
re-implement a reader and the two cannot drift. The default is identity, which
is what keeps the five byte-pinned goldens byte-pinned -- asserted per suffix
rather than once.
An unknown renderer NAME is refused rather than falling back to identity: a
silent fallback would produce a bundle that looks rendered and is not, which is
the failure mode this arm exists to make visible. That needed a registered code
(`unknown_renderer`) and its test -- slightly beyond the step's named files,
but the capability cannot ship without defining what an unknown name does.
`tests/test_profile.py`'s exact-field-set assertion went red, as the plan's risk
table predicted. Updated deliberately with the reason recorded: that assertion
exists so a field cannot arrive without someone deciding it should, and its red
run is the mechanism working.
Suite 917 -> 926. All five goldens byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Registered before any step raises them, so no later step invents a spelling:
- `extractor_binary_missing` -- the converter binary is absent (distinct
from the extra not being installed)
- `extractor_binary_version` -- present, but not the pinned version
- `extractor_convert_error` -- the converter failed on this file
- `extractor_empty_conversion` -- the converter returned no text
Also widened the `extractor_extra_missing` type list, which still read
"pdf/docx/xlsx".
Denominators recounted after the change rather than carried from a note --
the stale 49/48 figure is what made the recount a step requirement:
code bullets in errors.py 50 -> 54
distinct codes 49 -> 53 (one code documented twice)
test definitions in this suite 56 -> 57 (one parametrized definition,
four cases -- one per code)
The four tests assert only what is true at this step: the code is documented
and an ExtractionError carries it. The resolver and the seam replace each with
a behavioural raise-site test; a code that never gains one stays visible here
as a test that still only reads a docstring.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both tests reached `extractor_extra_missing` through a `.docx`/`.xlsx`
filename, which works only while `_UNPARSED_OPTIONAL_EXTENSIONS` is non-empty.
Those types are about to gain a converter, which empties the set and makes the
membership branch unreachable -- the tests would have gone red for the right
reason at the worst moment, mid-series.
Repointed both at the import probe, the mechanism the pdf gate already uses and
the one path that stays reachable however many types gain parsers.
Measured negative control: without the probe the same call raises
`extractor_pdf_error`, so the probe is load-bearing and the test can still fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The registry file says one test per code and that this file IS the
conformance suite. During the 1-to-N voyage the byte-stability pin over
tests/ was the verification mechanism, so editing this file was
forbidden and six new codes were covered in the new segmentation
modules instead. That was right then. The end-to-end gate run released
the pin, so the convention applies again -- a conformance suite split
across two files stops being one quietly, which is why this was carried
as an obligation rather than a preference.
All seven segmentation codes now have a test here, including
segmentation_plan_unmatched from this session. The behavioural tests in
test_segmentation.py and test_segmented_inbox.py stay where they are:
they exercise the parser and the door, a different question from
whether every documented code has a raise site.
Measured after: 49 documented codes, 48 covered in this file. The one
gap, source_reference_unquotable, predates this work and is tested in
tests/test_okf_v0_2_profile.py:351 -- reported, not silently closed.
Also records both defects and this migration as closed in the decision
record's known-gaps list.
Co-Authored-By: Claude <claude-opus-5>
Order G2a. Populates the optional `[extract]` extra for the first time with
one parser, `pdfplumber>=0.11.10,<0.12` (MIT), and wires `pdf` through it.
The default install is untouched: exactly one runtime dependency, stdlib
otherwise, enforced by test_packaging.py.
The gate for `pdf` becomes an import probe rather than a frozenset membership
test, exactly as extract.py's docstring had promised. The rejection does not
change: without the extra, `pdf` still raises `extractor_extra_missing` with
the same message. That behaviour is asserted UNCONDITIONALLY via a sys.modules
monkeypatch, so it holds on machines where the parser is installed too — a
skip would have preserved nothing there. Verified in a clean venv without the
extra: 589 passed, 7 skipped; with it, 596 passed.
`docx`/`xlsx` are unchanged and still fail fast — the extra names exactly what
it ships.
The parser choice was forced by measurement, not preference (b73dd9d,
docs/2026-08-21-g2-pdf-extraction-measurement.md): on a real requirement table
pdfplumber keeps 4 of 4 rows with label and value on one line, where pypdf,
pdfminer.six and pymupdf each keep 0 of 4. pymupdf is additionally out on
licence (AGPL-3.0), which an MIT package must not push onto a consumer.
Three facts from that measurement are now carried in code rather than in a
report:
- Extracted text is pinned to an exact transitive parser version
(pdfplumber pins pdfminer.six==20260107; date-stamped, no stability
contract). tests/test_extract.py freezes the expected text of a committed
hand-written fixture so a parser upgrade breaks something visible instead of
drifting silently. Reasoning at the declaration site and in
tests/fixtures/README.md.
- Determinism within a version is now held by a test, not only measured once.
- Drawn content does not survive extraction. Every pdf extraction emits the
new `ExtractionWarning`: figures have no text to recover, so a bundle built
from drawn documents is incomplete by construction. Stated categorically
rather than detected — deciding "is there a figure here" is the layout
heuristic G2b declined.
Two new error codes, both mirroring existing patterns: `extractor_empty_pdf`
(a scanned/image-only PDF, refused rather than persisted as an empty concept)
and `extractor_pdf_error` (parser failure wrapped, never leaked).
Structured table recovery (G2b) is NOT implemented and is documented as out of
scope: two independent parsers return the same wrong shape, so the breakage is
document geometry, not a library choice. PDFs enter as prose.
Also corrects an install promise this change would otherwise have published:
the README no longer presents a bare `pip install 'llm-ingestion-okf[extract]'`
as working, because the package is not on an index.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HtNhsdHnMGtMi7U2mvMU8z
Reads an external OKF bundle as {bundle-relative path -> document text},
hands it WHOLE to an injected gate over the guard's okf.import_bundle (a
bundle-level call: it resolves the cross-link graph across concepts), and
merges only concepts clearing the non-blocking floor. Same injection pattern
as Door B, so the core stays dependency-free while the CI channel for the
real guard is settled.
Two constraints shaped the design and are pinned by tests:
- A merged concept is written VERBATIM. Stamping provenance into it would
require round-tripping its frontmatter through this library's line-oriented
parser, which cannot represent the block lists the guard's parser accepts --
silent data loss -- and would persist bytes the guard never screened.
- Ownership is therefore proven by content identity: identical bytes at the
target name are a no-op re-merge (re-import of an unchanged bundle is
idempotent), and anything else at the name is refused. Curated content and
an updated concept are refused alike; refusing is what never destroys.
The floor is fail-closed beyond the plan's "no error" wording: an error, an
unrecognised disposition, and a concept the gate returned no verdict for are
all refusals. quarantine_review stays its own bucket, as at Door B.
origin/channel are validated against the guard's pinned vocabulary -- it
derives trust from origin by enum identity, so an unrecognised string would be
silently downgraded rather than caught.
Three primitives promoted for reuse rather than duplicated:
reduce_to_id_grammar and check_filename_length to materialize.py, and
extract.decode_text. Door C slugs the WHOLE concept path, so tables/users.md
and views/users.md stay distinct. Concept discovery folds case explicitly
rather than globbing *.md, whose case-sensitivity follows the filesystem and
would import the same bundle differently on APFS and ext4.
README's "what is gated today" section corrected: it claimed nothing is gated,
which is no longer true, but the honest statement is narrower than "the doors
are gated" -- the library cannot verify that an injected adapter is a real
guard, and a permissive stub is believed.
405 tests green; ruff, ruff format and mypy --strict clean.
`process_inbox(inbox_dir, bundle_dir, ingested_at, *, okf_type, gate)`:
per dropped file, bytes -> extract_text -> guard gate -> render -> collision
gate -> write -> index link. Returns an InboxResult whose four buckets
(persisted / quarantined / rejected / failed) are disjoint and complete, so a
file that vanished shows up as a missing entry rather than as nothing.
The guard is INJECTED rather than imported. The library calls no guard
function and makes no security decision: the Gate adapter returns a
GateDecision carrying the guard's own disposition value and the sanitized
text, and the flow only branches on it. That keeps the core dependency-free
while B2 (the guard's CI channel) is still open, and lets the persist and
refuse branches run deterministically against a test double.
Decisions taken with the operator this session:
- What is persisted is the gate's SANITIZED text, not the extracted text.
Screening one string and writing another would make the verdict a statement
about bytes nobody kept. `sanitize` is exported by the guard precisely for
callers composing the checklist themselves, so this is sanctioned API, not
a reimplementation. Door B has no model call, so the fenced text the
bookends produce for a transform is never persisted.
- Quarantine is reported apart from rejection. QUARANTINE_REVIEW means "hold
for human review" — an operator queue — where FAIL_SECURE is a decision.
No quarantine directory in v1; that stays an extension point.
Fails closed by construction: only the guard's non-blocking floor (`warn`)
persists, so a renamed member, a future disposition or an adapter typo lands
in `rejected` rather than being guessed safe.
One bad file never aborts the run. Only three conditions fail the whole run,
and each is wrong for every file at once: an invalid `ingested_at`, a reserved
`okf_type`, and a missing inbox directory.
New code `inbox_slug_collision`: two dropped files reducing to one generated
name are BOTH refused. Persisting one would let iteration order decide the
winner, and overwriting would lose the other's content.
Phase 1 primitives are reused, never duplicated, so four helpers become
package-internal names (write_bytes, link_in_index, parse_frontmatter,
INDEX_NAME) and link_in_index learns to append to an empty index — Door A
always seeds its index with bundle_summary, but Door B has no summary to
invent and must not open with a blank line.
The slug inherited the whole filename stem with no length bound, so a long
dropped name produced a filename the filesystem cannot hold — surfacing as an
untyped OSError at the write, with an errno that differs per platform (63 on
macOS, 36 on Linux). The inbox contract promises a typed per-file outcome, so
the gate belongs in the pure function that forms the name.
Refusal, not truncation: truncating is lossy AND collision-prone — two long
names sharing a prefix would reduce to one filename and the second write would
silently claim the first file. Same posture as `inbox_slug_empty`: the library
never invents a filename the operator did not give it. The message carries the
actual size and the limit, since the fix is to rename the dropped file.
Limit verified empirically on APFS 2026-07-25: a 255-byte component writes, a
258-byte one raises errno 63. ext4 and NTFS cap at the same 255.
New MaterializationError code `inbox_slug_too_long`, registered in the errors
docstring and in the error-code conformance suite.
Second guard-independent step of Phase 2: the pure functions that turn a
dropped file's name and extracted text into an OKF concept file, with the
§7-analogous honesty marker. No runtime dependency, no guard call, no model
call — the persist gate is still steps 4–5.
- `inbox_slug(filename)` reduces a name to the Phase 1 id grammar
(`[a-z0-9][a-z0-9-]*`): drop the final extension, lowercase, collapse every
run of non-grammar characters to one `-`, strip the ends. A name that
reduces to nothing fails fast rather than inventing an `untitled` fallback
that would silently collide across unrelated files.
- `inbox_filename(slug)` namespaces it `inbox-{slug}.md`, disjoint from
`index.md`, Door A's `ingest-*`, and `promoted-verdict-*` for every slug the
grammar admits (asserted over a hostile-name table, not one example).
- `render_inbox_concept(...)` emits the six-key layer in fixed order — `type`,
`title`, `source_file`, `source_sha256`, `ingested_at`, `generated: true` —
with the digest taken over the ORIGINAL dropped bytes, never the extracted
text, so provenance stays re-verifiable against the operator's file. Body is
normalised to LF-only with exactly one trailing newline (dropped files
legitimately arrive with CRLF; Door A can validate instead because it renders
its own bodies).
Slugging normalises to NFC first. macOS (APFS) hands filenames over
decomposed, so `é` arrives as `e` + combining acute: without normalising, the
combining mark alone becomes a separator and the base letter survives
(`cafe`), where the composed form is one non-grammar character (`caf`) — one
visual filename, two slugs, depending on where the string came from. The test
pins both forms with explicit escapes so it cannot depend on the test file's
own encoding. No transliteration, deliberately: mapping non-ASCII to ASCII is
a semantic claim the slugger cannot make (Norwegian `møte`/meeting would
become `mote`/fashion). The readable name survives verbatim in `title`.
Fail-fast gates, all typed: reserved `verdict` layer refused at this door too
(the promotion gate stays the only path in); a title that is multi-line or
contains `[`/`]` refused, matching Door A's manifest rule, because it renders
verbatim into `- [title](target)`; a multi-line `source_file` refused because
line-oriented frontmatter would take the injected lines. Four new stable
MaterializationError codes, each with its entry in the test_error_codes.py
registry: `inbox_slug_empty`, `inbox_title_invalid`,
`inbox_source_file_invalid`, `okf_type_reserved`.
`ingested_at` validation moves to a shared `validate_ingested_at` in
materialize.py and is called by both doors — one rule in one place is what
keeps the determinism contract from drifting apart per door.
TDD: tests/test_inbox.py precedes the implementation. 325 tests green;
mypy --strict, ruff check/format, and the sanitize|quarantine|lexicon
boundary grep-gate clean; runtime dependencies still exactly none; the Phase 1
golden suite still passes byte-for-byte.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
First, guard-independent step of Phase 2: a stdlib-only registry mapping a
dropped file's extension to its text extractor, with the fail-fast gates that
keep binary parsing out of core. `extract_text(filename, data)` dispatches
(case-insensitively) to:
- `md`/`txt` — utf-8-sig passthrough (BOM never leaks, baseline parity with
Door A's read_csv);
- `csv` — the Phase 1 `render_table` (renderer reused, not duplicated);
- `json` — verbatim inside `render_fenced_block`;
- `html`/`htm` — text via `html.parser`, `script`/`style` stripped, tags as
word boundaries (spec B3: adequate for v1, richer is out of scope).
`pdf`/`docx`/`xlsx` are `[extract]`-gated; until that extra ships a parser they
fail fast with a typed error naming the extra — never a silent skip, never a
bundled parser in core. New `ExtractionError(IngestError)` carries four stable
codes (`extractor_unknown`, `extractor_extra_missing`, `extractor_decode_error`,
`extractor_empty_csv`); a non-UTF-8 file is a typed corrupt-input failure, never
a leaked UnicodeDecodeError. `extract_text` returns text content only — LF
framing and concept frontmatter are the materializer's job (step 2).
No runtime dependency and no guard call yet (the guard pin and 0.4.0 land with
the persist gate in steps 4–5). TDD: test_extract.py + the four codes in the
test_error_codes.py registry precede the implementation; mypy --strict, ruff,
and the `sanitize|quarantine|lexicon` boundary grep-gate all clean; the Phase 1
golden suite still passes byte-for-byte.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBbjgS5A55RVavoyjJC4FX
Ingest-spec §4 (ratified D1, commons bfa5a9b) requires a title to reject
`[` and `]` fail-fast at manifest load. The title is rendered verbatim into
the index link label and frontmatter (§5/§6), where a bracket would break
index-link and navigation parsing (method-spec §3 Step 1). The rule was
specced but unenforced: validation only checked single-line.
Satisfies the commons "Title link-safety" load-bearing §11 seam. Same
`manifest_schema` error code as the sibling single-line/max_rows rules;
parametrized cases added to test_manifest and test_error_codes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016oMpAhQcJZVGBW182stSPn
safe_resolve let an untyped ValueError escape when a target carried an
embedded NUL: os.path.realpath rejects the string before the boundary
check runs. That broke SourceError's documented contract ("always typed,
never a leaked OSError") and the v0.3.0 promise of stable machine-readable
codes on every failure.
A target that cannot be resolved at all now fails closed under the
existing `path_escape` code, with its own message. No new code is added —
`path_escape` already means "the boundary refused this target"; its
docstring is widened to say so. Consumers asserting on the code are
unaffected.
Found by consumer review: portfolio-optimiser's navigate_bundle leaks the
same ValueError from the same class of input, which makes an entire bundle
unreadable instead of skipping one link. That defect is theirs to fix; this
commit closes the library-side half.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WdVgowYC4LARgvNdNMiuvz
Consumer tests were binding message wording (pytest.raises(match=...)) to
distinguish sub-causes within one error type. Every raise site now carries
a documented, stable code attribute; the registry lives in the errors.py
class docstrings. Codes are stable API; message text is explicitly not.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
