An index ordering a profile names must be honoured wherever this library
writes an index. Door B and Door C have separate index writers, so an
ordering built on Door B's `_index_sort_key` seam alone would have been a
profile field Door B obeys and Door C ignores -- silently, because
nothing raises and both files still parse. That is
`IndexPolicy.per_directory` again: a field that reads as global and acts
on one path.
`IndexPolicy` gains `sort_key`, `sort_order` and `sort_missing`. Both
order fields draw from CLOSED sets, and `sort_order` is deliberately not
a caller-supplied callable: a callable cannot be serialised into the
bundle, reproduced from it, or audited by a reader, which is the whole of
what a deterministic bundle claims. A `sort_key` the facet policy does
not name is refused too -- every entry would be missing the key and the
ordering would silently do nothing, which is this row's own defect class.
`IndexPolicy.sort_entries` is the one helper. Four stable passes, so each
is the tie-break of the next: concept path, then the named key, then the
missing group partitioned to whichever end the policy says, then
navigation last. Passes 2 and 3 are separate on purpose -- folding them
into one reversible key tuple would flip the missing group along with the
order, so `sort_missing="last"` would mean "first" under `descending`.
The tie-break is the CONCEPT PATH, not the link target, and that is
measured rather than assumed: `notes-beta.md` precedes `notes/alpha.md`
by concept path and follows it by generated filename, so ordering Door C
on the target would have re-ordered every existing Door C bundle.
`IndexEntry` carries the path for that reason; `parse_entry` leaves it
`None` and the ordering falls back to the target, which costs nothing
because no caller sorts entries it read back off disk.
Door B's two reprojection writers and Door C's index emission all route
through the helper. Door B's unfaceted path is not routed and does not
need to be: `sort_key` requires a facet policy, and a faceted profile
never reaches that writer. Door C's guarantee is bounded and stated in
the code -- `link_in_index` appends what is absent and leaves what is
present, so the order holds within a run and never re-orders entries an
earlier run wrote.
Default ordering, unchanged and now stated: with no `sort_key`, concepts
before navigation, each group ascending by concept path.
TDD, and the red was watched twice. First behaviourally with the fields
inert (both doors emitted the exact reverse of the named order), then
again with Door B routed and Door C not -- the broken world reproduced,
where a Door-B-only test would have passed.
882 tests (868 before). The five byte-pinned goldens are untouched and
green; no shipped profile moved.
Co-Authored-By: Claude <claude-opus-5>
A segment's title comes from the plan, so a human adjudicated it. But
structure derivation runs over the segment body, finds no title key and
no usable heading, falls back to a stem, and adds "title" to derived.
The concept then emitted a stated fact under an inferred marker, and a
consumer that distrusts derived fields would distrust exactly the thing
a human decided. An over-marked field is the same defect class as an
unmarked heuristic: the marker is only worth something if it is
accurate in both directions.
Scoped to title alone, and pinned that way by test: number stays in
derived on a segment, because nothing about segmentation makes an
inferred document number declared. Without a segment a derived title is
still marked, so only a plan makes a title declared.
The SEGMENTED_V1 golden moves, which is the intended consequence and
the only golden that may. The four existing goldens are byte-identical
to baseline 770d8d4, measured against the sha rather than inspected.
Co-Authored-By: Claude <claude-opus-5>
A plan is selected by content hash, so a mistyped source_sha256 matched
nothing, every dropped file fell through to the one-concept rule, and
process_inbox returned an ordinary success over a flat bundle. The
operator asked for segmentation, got none, and had no error to read --
the silent skip this library refuses everywhere else. vegnormal-okf is
about to run an N500 corpus through this path, where a silent zero
would read as "the corpus has no concepts".
The refusal asks whether a covering plan was FOUND, not whether every
file was examined, so an unreadable drop cannot mask it; and coverage
is recorded at selection, not after path validation, so a matched plan
with a refused entry path still reports its own per-file code. The
first cut got that second question wrong and an existing collision test
caught it; the case is now pinned by its own test, verified red against
the earlier form.
New code segmentation_plan_unmatched, registered in the SegmentationError
docstring register in the same commit. Fail-fast before any disk
mutation. The four existing goldens are byte-identical to baseline.
Co-Authored-By: Claude <claude-opus-5>
Retiring a directory's last concept left its index.md standing, so the
directory survived a rebuild that never creates it -- diff -r reports it as
'Only in ...'. Found by S7 once its fixture was made to actually retire a
segment: with round 2 a superset of round 1, S7 stayed green with retirement
disabled entirely.
Door B takes a profile (keyword-only, DEFAULT) and, under a profile carrying
facets, derives each dropped document's title, number, hierarchy and
cross-references, writes them into the concept's own frontmatter, and projects
them into the index entry.
The additive requirement is answered by one decision rather than by an
algorithm: the index is a PROJECTION of the concept files, recomputed from the
whole bundle each round. Nothing is diffed, so the three invariants hold by
construction -- rebuild-from-scratch equals incremental byte for byte,
re-dropping a document replaces its entry instead of doubling it, and a
relation formed in round 3 UPDATES the round-1 entry it is about, which an
append-only index could never do.
An unresolved pointer is marked '?' in the entry rather than omitted: during
build-up, pointing at something not dropped yet is normal, and the dangerous
version is the one that leaves no trace. Facet values are validated per file
BEFORE the write, so a producer value that breaks the grammar fails that file
and not the run.
DEFAULT is byte-identical with and without the new parameter, and is asserted
so. Door B keeps writing the literal 'generated: true' rather than the
profile's ownership stamp -- routing it through the profile would move
DEFAULT's bytes and orphan every bundle this door has already written; that is
a separate question and answering it here would have answered it silently.
18 new tests; suite 677 -> 695.
The operator's condition for revisiting the pin was met (v1.2.0 contains
the flow-mapping frontmatter fix, commit 5870483) and the operator has now
approved the move itself, dispatched as its own order because it changes
consumer-visible Requires-Dist and shifts golden-fixture admission.
Floor 1.2: this library needs the flow-mapping support (`generated: { by:
x, at: y }`) that landed there — without it Door C fail-secures every
concept carrying that stamp. Ceiling <2.0, not narrower: the guard's own
1.0.0 release freezes its exported surface until a 2.0.0, and explicitly
keeps calibration (severities, dispositions) free to move within 1.x, so a
tighter ceiling here would claim a stability guarantee neither side needs.
Re-measured through Door C against the guard's own default
(allow_reserved=True, matching how the earlier recommendation measured
it), over the 9 concept documents across all four Door-A goldens:
4/9 admitted at 0.3.4 -> 8/9 admitted at 1.2.0, confirming the number
already reported. The ninth (a `sources` block-list carrying `resource`)
stays refused by design (G30) and is not expected to move.
Failing-test-first: test_guard_adapter.py::test_guard_version_is_inside_the_pin
and test_packaging.py::test_the_only_runtime_dependency_is_the_security_boundary
were updated to the new pin first and confirmed red against the
still-installed 0.3.4, then `uv sync --extra extract` installed 1.2.0 and
both went green. Full suite (615), ruff and mypy --strict clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RxcxzNwpX1kDP53n1rLhM5
Measure first, widen after. The 19-fixture guard-surface suite was re-run
against v0.3.4 in a scratch venv before the range moved, and reproduced the
three deltas measured against v0.3.3 exactly, with none added. v0.3.4 is the
tag pinned rather than v0.3.3 because it shipped first and repairs a quadratic
regex (okf._MD_LINK_RE) that sits on Door C's own call path.
Door C now passes allow_reserved=False explicitly. The guard added the keyword
in the 0.3 line and defaults it True for received bundles, which would merge a
sender's index.md / log.md instead of rejecting them. The override keeps the
unconditional reserved-name refusal committed to before the keyword existed,
and the reason is structural rather than a second opinion on the guard's scan:
Door C generates the merged bundle's index.md from what it merged and writes
every merged concept verbatim, so a sender's index.md would be a second and
irreconcilable claim on one path.
This is not a behaviour change for anyone on the previous pin: under v0.2.0
the keyword did not exist and reserved names were refused by construction.
The floor is >=0.3 and not >=0.2 for a measured reason. allow_reserved is
absent in v0.2.0 and present from v0.3.0 onward, checked across all five tags:
a >=0.2 floor would admit a version that raises TypeError on every Door C
import. That measurement also corrects a recorded premise -- the plan said the
keyword "shipped in v0.3.3", which read the first version we ran the suite
against as the version it was introduced in. The conclusion held; the reason
did not, and the reason is what a future bump would have relied on.
test_door_c_pins_allow_reserved_false_against_the_guards_default locks both
halves: that the guard still defaults True, without which the override is a
no-op that would pass forever over nothing, and that Door C overrides it.
586 tests, mypy --strict clean, goldens byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V2v1hrDhrff2H3y2TNJHkF
Phase 3 step 1. What Phases 1 and 2 hard-coded about a valid bundle now
lives on one frozen `BundleProfile`, and `DEFAULT` states exactly the
ingest-spec v1 + Phase 2 contract. Nothing observable changes: the 425
existing tests are unmodified and green, and `git diff --stat examples/`
is empty, which is assumption C1's whole proof. Both oracles are real
rather than nominal — the golden suite compares `read_bytes()`, and Door
B pins its frontmatter block as an exact string.
Moved onto the profile, each one previously a constant with a reader:
the index name and its managed-link shape (template plus pattern, kept
honest by a round-trip test), the three filename namespaces
(`ingest-`/`inbox-`/`import-` and `.md`), the reserved `verdict` layer
(duplicated in `manifest` and `inbox` before this), the frontmatter key
order, and the `source_query` whitespace collapse.
Two details are worth naming. The DEFAULT key order spans both doors:
Door A's seven keys and Door B's six are subsequences of one nine-key
order, so a single canonical order reproduces both doors byte-for-byte.
And emission follows commons decision D1 — ordered prefix, then any
remaining keys sorted — which is the mechanism the proving consumer's
hash registry needs; under DEFAULT the tail is always empty.
`TypePolicy` refuses the reserved layer at CONSTRUCTION (assumption C3),
so a profile admitting `verdict` cannot be built, let alone passed to a
door. It reports refusals rather than raising them, because Door A
refuses with `ManifestError` and Door B with `MaterializationError` for
the same type — the wording and the stable code come from the policy,
the exception class stays each door's own.
Deliberately NOT moved, each for a stated reason: the required and
allowlisted key sets the proving consumer needs (no code reads them
until the STRICT_V1 validator exists, and a field nothing reads is a
claim nothing tests), the id grammar (a pattern the slugger derives a
separator class from, not a flat name), `NAME_MAX_BYTES` (a filesystem
fact, not a contract choice), and every disposition/origin/channel
vocabulary (guard territory, always). The profile is also not exported
from the package root yet — the optional `profile` argument on the
flows is step 3's plumbing change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
Reads an external OKF bundle as {bundle-relative path -> document text},
hands it WHOLE to an injected gate over the guard's okf.import_bundle (a
bundle-level call: it resolves the cross-link graph across concepts), and
merges only concepts clearing the non-blocking floor. Same injection pattern
as Door B, so the core stays dependency-free while the CI channel for the
real guard is settled.
Two constraints shaped the design and are pinned by tests:
- A merged concept is written VERBATIM. Stamping provenance into it would
require round-tripping its frontmatter through this library's line-oriented
parser, which cannot represent the block lists the guard's parser accepts --
silent data loss -- and would persist bytes the guard never screened.
- Ownership is therefore proven by content identity: identical bytes at the
target name are a no-op re-merge (re-import of an unchanged bundle is
idempotent), and anything else at the name is refused. Curated content and
an updated concept are refused alike; refusing is what never destroys.
The floor is fail-closed beyond the plan's "no error" wording: an error, an
unrecognised disposition, and a concept the gate returned no verdict for are
all refusals. quarantine_review stays its own bucket, as at Door B.
origin/channel are validated against the guard's pinned vocabulary -- it
derives trust from origin by enum identity, so an unrecognised string would be
silently downgraded rather than caught.
Three primitives promoted for reuse rather than duplicated:
reduce_to_id_grammar and check_filename_length to materialize.py, and
extract.decode_text. Door C slugs the WHOLE concept path, so tables/users.md
and views/users.md stay distinct. Concept discovery folds case explicitly
rather than globbing *.md, whose case-sensitivity follows the filesystem and
would import the same bundle differently on APFS and ext4.
README's "what is gated today" section corrected: it claimed nothing is gated,
which is no longer true, but the honest statement is narrower than "the doors
are gated" -- the library cannot verify that an injected adapter is a real
guard, and a permissive stub is believed.
405 tests green; ruff, ruff format and mypy --strict clean.
`process_inbox(inbox_dir, bundle_dir, ingested_at, *, okf_type, gate)`:
per dropped file, bytes -> extract_text -> guard gate -> render -> collision
gate -> write -> index link. Returns an InboxResult whose four buckets
(persisted / quarantined / rejected / failed) are disjoint and complete, so a
file that vanished shows up as a missing entry rather than as nothing.
The guard is INJECTED rather than imported. The library calls no guard
function and makes no security decision: the Gate adapter returns a
GateDecision carrying the guard's own disposition value and the sanitized
text, and the flow only branches on it. That keeps the core dependency-free
while B2 (the guard's CI channel) is still open, and lets the persist and
refuse branches run deterministically against a test double.
Decisions taken with the operator this session:
- What is persisted is the gate's SANITIZED text, not the extracted text.
Screening one string and writing another would make the verdict a statement
about bytes nobody kept. `sanitize` is exported by the guard precisely for
callers composing the checklist themselves, so this is sanctioned API, not
a reimplementation. Door B has no model call, so the fenced text the
bookends produce for a transform is never persisted.
- Quarantine is reported apart from rejection. QUARANTINE_REVIEW means "hold
for human review" — an operator queue — where FAIL_SECURE is a decision.
No quarantine directory in v1; that stays an extension point.
Fails closed by construction: only the guard's non-blocking floor (`warn`)
persists, so a renamed member, a future disposition or an adapter typo lands
in `rejected` rather than being guessed safe.
One bad file never aborts the run. Only three conditions fail the whole run,
and each is wrong for every file at once: an invalid `ingested_at`, a reserved
`okf_type`, and a missing inbox directory.
New code `inbox_slug_collision`: two dropped files reducing to one generated
name are BOTH refused. Persisting one would let iteration order decide the
winner, and overwriting would lose the other's content.
Phase 1 primitives are reused, never duplicated, so four helpers become
package-internal names (write_bytes, link_in_index, parse_frontmatter,
INDEX_NAME) and link_in_index learns to append to an empty index — Door A
always seeds its index with bundle_summary, but Door B has no summary to
invent and must not open with a blank line.
The slug inherited the whole filename stem with no length bound, so a long
dropped name produced a filename the filesystem cannot hold — surfacing as an
untyped OSError at the write, with an errno that differs per platform (63 on
macOS, 36 on Linux). The inbox contract promises a typed per-file outcome, so
the gate belongs in the pure function that forms the name.
Refusal, not truncation: truncating is lossy AND collision-prone — two long
names sharing a prefix would reduce to one filename and the second write would
silently claim the first file. Same posture as `inbox_slug_empty`: the library
never invents a filename the operator did not give it. The message carries the
actual size and the limit, since the fix is to rename the dropped file.
Limit verified empirically on APFS 2026-07-25: a 255-byte component writes, a
258-byte one raises errno 63. ext4 and NTFS cap at the same 255.
New MaterializationError code `inbox_slug_too_long`, registered in the errors
docstring and in the error-code conformance suite.
Second guard-independent step of Phase 2: the pure functions that turn a
dropped file's name and extracted text into an OKF concept file, with the
§7-analogous honesty marker. No runtime dependency, no guard call, no model
call — the persist gate is still steps 4–5.
- `inbox_slug(filename)` reduces a name to the Phase 1 id grammar
(`[a-z0-9][a-z0-9-]*`): drop the final extension, lowercase, collapse every
run of non-grammar characters to one `-`, strip the ends. A name that
reduces to nothing fails fast rather than inventing an `untitled` fallback
that would silently collide across unrelated files.
- `inbox_filename(slug)` namespaces it `inbox-{slug}.md`, disjoint from
`index.md`, Door A's `ingest-*`, and `promoted-verdict-*` for every slug the
grammar admits (asserted over a hostile-name table, not one example).
- `render_inbox_concept(...)` emits the six-key layer in fixed order — `type`,
`title`, `source_file`, `source_sha256`, `ingested_at`, `generated: true` —
with the digest taken over the ORIGINAL dropped bytes, never the extracted
text, so provenance stays re-verifiable against the operator's file. Body is
normalised to LF-only with exactly one trailing newline (dropped files
legitimately arrive with CRLF; Door A can validate instead because it renders
its own bodies).
Slugging normalises to NFC first. macOS (APFS) hands filenames over
decomposed, so `é` arrives as `e` + combining acute: without normalising, the
combining mark alone becomes a separator and the base letter survives
(`cafe`), where the composed form is one non-grammar character (`caf`) — one
visual filename, two slugs, depending on where the string came from. The test
pins both forms with explicit escapes so it cannot depend on the test file's
own encoding. No transliteration, deliberately: mapping non-ASCII to ASCII is
a semantic claim the slugger cannot make (Norwegian `møte`/meeting would
become `mote`/fashion). The readable name survives verbatim in `title`.
Fail-fast gates, all typed: reserved `verdict` layer refused at this door too
(the promotion gate stays the only path in); a title that is multi-line or
contains `[`/`]` refused, matching Door A's manifest rule, because it renders
verbatim into `- [title](target)`; a multi-line `source_file` refused because
line-oriented frontmatter would take the injected lines. Four new stable
MaterializationError codes, each with its entry in the test_error_codes.py
registry: `inbox_slug_empty`, `inbox_title_invalid`,
`inbox_source_file_invalid`, `okf_type_reserved`.
`ingested_at` validation moves to a shared `validate_ingested_at` in
materialize.py and is called by both doors — one rule in one place is what
keeps the determinism contract from drifting apart per door.
TDD: tests/test_inbox.py precedes the implementation. 325 tests green;
mypy --strict, ruff check/format, and the sanitize|quarantine|lexicon
boundary grep-gate clean; runtime dependencies still exactly none; the Phase 1
golden suite still passes byte-for-byte.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
