Reads an external OKF bundle as {bundle-relative path -> document text},
hands it WHOLE to an injected gate over the guard's okf.import_bundle (a
bundle-level call: it resolves the cross-link graph across concepts), and
merges only concepts clearing the non-blocking floor. Same injection pattern
as Door B, so the core stays dependency-free while the CI channel for the
real guard is settled.
Two constraints shaped the design and are pinned by tests:
- A merged concept is written VERBATIM. Stamping provenance into it would
require round-tripping its frontmatter through this library's line-oriented
parser, which cannot represent the block lists the guard's parser accepts --
silent data loss -- and would persist bytes the guard never screened.
- Ownership is therefore proven by content identity: identical bytes at the
target name are a no-op re-merge (re-import of an unchanged bundle is
idempotent), and anything else at the name is refused. Curated content and
an updated concept are refused alike; refusing is what never destroys.
The floor is fail-closed beyond the plan's "no error" wording: an error, an
unrecognised disposition, and a concept the gate returned no verdict for are
all refusals. quarantine_review stays its own bucket, as at Door B.
origin/channel are validated against the guard's pinned vocabulary -- it
derives trust from origin by enum identity, so an unrecognised string would be
silently downgraded rather than caught.
Three primitives promoted for reuse rather than duplicated:
reduce_to_id_grammar and check_filename_length to materialize.py, and
extract.decode_text. Door C slugs the WHOLE concept path, so tables/users.md
and views/users.md stay distinct. Concept discovery folds case explicitly
rather than globbing *.md, whose case-sensitivity follows the filesystem and
would import the same bundle differently on APFS and ext4.
README's "what is gated today" section corrected: it claimed nothing is gated,
which is no longer true, but the honest statement is narrower than "the doors
are gated" -- the library cannot verify that an injected adapter is a real
guard, and a permissive stub is believed.
405 tests green; ruff, ruff format and mypy --strict clean.
`process_inbox(inbox_dir, bundle_dir, ingested_at, *, okf_type, gate)`:
per dropped file, bytes -> extract_text -> guard gate -> render -> collision
gate -> write -> index link. Returns an InboxResult whose four buckets
(persisted / quarantined / rejected / failed) are disjoint and complete, so a
file that vanished shows up as a missing entry rather than as nothing.
The guard is INJECTED rather than imported. The library calls no guard
function and makes no security decision: the Gate adapter returns a
GateDecision carrying the guard's own disposition value and the sanitized
text, and the flow only branches on it. That keeps the core dependency-free
while B2 (the guard's CI channel) is still open, and lets the persist and
refuse branches run deterministically against a test double.
Decisions taken with the operator this session:
- What is persisted is the gate's SANITIZED text, not the extracted text.
Screening one string and writing another would make the verdict a statement
about bytes nobody kept. `sanitize` is exported by the guard precisely for
callers composing the checklist themselves, so this is sanctioned API, not
a reimplementation. Door B has no model call, so the fenced text the
bookends produce for a transform is never persisted.
- Quarantine is reported apart from rejection. QUARANTINE_REVIEW means "hold
for human review" — an operator queue — where FAIL_SECURE is a decision.
No quarantine directory in v1; that stays an extension point.
Fails closed by construction: only the guard's non-blocking floor (`warn`)
persists, so a renamed member, a future disposition or an adapter typo lands
in `rejected` rather than being guessed safe.
One bad file never aborts the run. Only three conditions fail the whole run,
and each is wrong for every file at once: an invalid `ingested_at`, a reserved
`okf_type`, and a missing inbox directory.
New code `inbox_slug_collision`: two dropped files reducing to one generated
name are BOTH refused. Persisting one would let iteration order decide the
winner, and overwriting would lose the other's content.
Phase 1 primitives are reused, never duplicated, so four helpers become
package-internal names (write_bytes, link_in_index, parse_frontmatter,
INDEX_NAME) and link_in_index learns to append to an empty index — Door A
always seeds its index with bundle_summary, but Door B has no summary to
invent and must not open with a blank line.
The slug inherited the whole filename stem with no length bound, so a long
dropped name produced a filename the filesystem cannot hold — surfacing as an
untyped OSError at the write, with an errno that differs per platform (63 on
macOS, 36 on Linux). The inbox contract promises a typed per-file outcome, so
the gate belongs in the pure function that forms the name.
Refusal, not truncation: truncating is lossy AND collision-prone — two long
names sharing a prefix would reduce to one filename and the second write would
silently claim the first file. Same posture as `inbox_slug_empty`: the library
never invents a filename the operator did not give it. The message carries the
actual size and the limit, since the fix is to rename the dropped file.
Limit verified empirically on APFS 2026-07-25: a 255-byte component writes, a
258-byte one raises errno 63. ext4 and NTFS cap at the same 255.
New MaterializationError code `inbox_slug_too_long`, registered in the errors
docstring and in the error-code conformance suite.
Second guard-independent step of Phase 2: the pure functions that turn a
dropped file's name and extracted text into an OKF concept file, with the
§7-analogous honesty marker. No runtime dependency, no guard call, no model
call — the persist gate is still steps 4–5.
- `inbox_slug(filename)` reduces a name to the Phase 1 id grammar
(`[a-z0-9][a-z0-9-]*`): drop the final extension, lowercase, collapse every
run of non-grammar characters to one `-`, strip the ends. A name that
reduces to nothing fails fast rather than inventing an `untitled` fallback
that would silently collide across unrelated files.
- `inbox_filename(slug)` namespaces it `inbox-{slug}.md`, disjoint from
`index.md`, Door A's `ingest-*`, and `promoted-verdict-*` for every slug the
grammar admits (asserted over a hostile-name table, not one example).
- `render_inbox_concept(...)` emits the six-key layer in fixed order — `type`,
`title`, `source_file`, `source_sha256`, `ingested_at`, `generated: true` —
with the digest taken over the ORIGINAL dropped bytes, never the extracted
text, so provenance stays re-verifiable against the operator's file. Body is
normalised to LF-only with exactly one trailing newline (dropped files
legitimately arrive with CRLF; Door A can validate instead because it renders
its own bodies).
Slugging normalises to NFC first. macOS (APFS) hands filenames over
decomposed, so `é` arrives as `e` + combining acute: without normalising, the
combining mark alone becomes a separator and the base letter survives
(`cafe`), where the composed form is one non-grammar character (`caf`) — one
visual filename, two slugs, depending on where the string came from. The test
pins both forms with explicit escapes so it cannot depend on the test file's
own encoding. No transliteration, deliberately: mapping non-ASCII to ASCII is
a semantic claim the slugger cannot make (Norwegian `møte`/meeting would
become `mote`/fashion). The readable name survives verbatim in `title`.
Fail-fast gates, all typed: reserved `verdict` layer refused at this door too
(the promotion gate stays the only path in); a title that is multi-line or
contains `[`/`]` refused, matching Door A's manifest rule, because it renders
verbatim into `- [title](target)`; a multi-line `source_file` refused because
line-oriented frontmatter would take the injected lines. Four new stable
MaterializationError codes, each with its entry in the test_error_codes.py
registry: `inbox_slug_empty`, `inbox_title_invalid`,
`inbox_source_file_invalid`, `okf_type_reserved`.
`ingested_at` validation moves to a shared `validate_ingested_at` in
materialize.py and is called by both doors — one rule in one place is what
keeps the determinism contract from drifting apart per door.
TDD: tests/test_inbox.py precedes the implementation. 325 tests green;
mypy --strict, ruff check/format, and the sanitize|quarantine|lexicon
boundary grep-gate clean; runtime dependencies still exactly none; the Phase 1
golden suite still passes byte-for-byte.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
First, guard-independent step of Phase 2: a stdlib-only registry mapping a
dropped file's extension to its text extractor, with the fail-fast gates that
keep binary parsing out of core. `extract_text(filename, data)` dispatches
(case-insensitively) to:
- `md`/`txt` — utf-8-sig passthrough (BOM never leaks, baseline parity with
Door A's read_csv);
- `csv` — the Phase 1 `render_table` (renderer reused, not duplicated);
- `json` — verbatim inside `render_fenced_block`;
- `html`/`htm` — text via `html.parser`, `script`/`style` stripped, tags as
word boundaries (spec B3: adequate for v1, richer is out of scope).
`pdf`/`docx`/`xlsx` are `[extract]`-gated; until that extra ships a parser they
fail fast with a typed error naming the extra — never a silent skip, never a
bundled parser in core. New `ExtractionError(IngestError)` carries four stable
codes (`extractor_unknown`, `extractor_extra_missing`, `extractor_decode_error`,
`extractor_empty_csv`); a non-UTF-8 file is a typed corrupt-input failure, never
a leaked UnicodeDecodeError. `extract_text` returns text content only — LF
framing and concept frontmatter are the materializer's job (step 2).
No runtime dependency and no guard call yet (the guard pin and 0.4.0 land with
the persist gate in steps 4–5). TDD: test_extract.py + the four codes in the
test_error_codes.py registry precede the implementation; mypy --strict, ruff,
and the `sanitize|quarantine|lexicon` boundary grep-gate all clean; the Phase 1
golden suite still passes byte-for-byte.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBbjgS5A55RVavoyjJC4FX
Stage 0 guard-readiness check (this session) verified the real
`llm-ingestion-guard` v0.2.0 surface against ground truth: the Door C plan
assumed `okf.import_bundle(bundle, origin=…, channel=…, allow_reserved=False)`,
but the shipped signature is `import_bundle(bundle, *, origin, channel)` — no
`allow_reserved` parameter. Reserved-name rejection of `index.md`/`log.md` is
unconditional in the guard (OKFPathError, no toggle), so the intended default
semantics are unchanged; only the documented call was wrong and would have
raised TypeError if written verbatim. Correct the plan line before Phase 2
code is written against it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBbjgS5A55RVavoyjJC4FX
Sequences the remaining roadmap (Stage 0 guard gate -> Phase 2 -> Phase 3
-> Phase 4) with the quality rationale for the order: guard readiness
verified before any Phase 2 code, Phase 2 split guard-independent-first,
Phase 3's configurability extracted only after two concrete consumers
exist, Phase 4 code last on a frozen contract with its coordination
started early. Connective tissue over the four per-phase plans; owns the
order and reasons, not the phase detail.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBbjgS5A55RVavoyjJC4FX
Ingest-spec §4 (ratified D1, commons bfa5a9b) requires a title to reject
`[` and `]` fail-fast at manifest load. The title is rendered verbatim into
the index link label and frontmatter (§5/§6), where a bracket would break
index-link and navigation parsing (method-spec §3 Step 1). The rule was
specced but unenforced: validation only checked single-line.
Satisfies the commons "Title link-safety" load-bearing §11 seam. Same
`manifest_schema` error code as the sibling single-line/max_rows rules;
parametrized cases added to test_manifest and test_error_codes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016oMpAhQcJZVGBW182stSPn
The six `raise TypedError(...) from exc` sites (manifest.py x3, materialize.py,
connectors.py x2) turn leaked OSError / ValueError / sqlite3.Error / URLError
into typed IngestErrors while keeping the original under __cause__ — but no
test asserted the chain, so a stray bare `raise` could drop the cause
unnoticed. Add a conformance suite, one test per site, each asserting the
chained cause type. The module docstring documents the invariant. Companion
to test_error_codes.py, which pins the `.code` on the same sites.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBbjgS5A55RVavoyjJC4FX
The §3 ownership scan classified every ingest-stamped file as replaceable,
so materializing one manifest into a bundle deleted the ingest files another
manifest had stamped there — and removed their index links too.
Narrow ownership to files whose stamp names the running manifest by stem.
The stamp is `{stem}@{sha256[:16]}`; matching on the stem, not the whole
stamp, lets an edited manifest still reclaim the files a prior run of the
same manifest wrote (the stem is stable across content edits), while a
different manifest sharing the bundle keeps its own files. Implements the
trinn-e §10.2 decision. This does not close the operator-copy restriction
(a generated file copied into curated content), which stays documented in
ingest-spec.md:69-72, not enforced.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBbjgS5A55RVavoyjJC4FX
The Door A description said the §11 golden fixtures "are missing today".
They shipped in 9dd86b1 as ingest-golden-{file,sql,http}/. Correct the line
to state what is present.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBbjgS5A55RVavoyjJC4FX
a7e1bc8 unified title rendering at both sites and routed the open question
to commons, the spec owner: does §5's whitespace collapse apply to `title`,
or only to `source_query`? The ruling is source_query only. §5 mandates the
collapse for that one field — where a multi-line SQL SELECT must render on
one line — while every other value is validated single-line at manifest load
and emitted as-is: validation, not repair.
_single_line is renamed _collapse_whitespace and now runs for source_query
alone. Frontmatter values, the index link label, and the labels_by_target
map all pass through verbatim. A title carrying an internal whitespace run
(or any other operator-supplied bytes) now survives byte-for-byte at both
the frontmatter and the index-label site, instead of being silently altered.
No golden bytes change: no title in the shipped fixtures or either consumer
carries a collapsible whitespace run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBbjgS5A55RVavoyjJC4FX
ingest-spec §4 makes the extraction title BOTH the `title` frontmatter and
the index link label — one value, two sites. The frontmatter renderer
single-lined its values while the index label was written raw, so a title
carrying a whitespace run rendered two different ways from one input:
title: Energiforbruk 2024
- [Energiforbruk 2024](ingest-orders.md)
Single-lining is now one named rule applied at every site that renders the
title, instead of a detail of frontmatter rendering that the index path did
not share.
No golden bytes change: no title in this repo, portfolio-optimiser or
portfolio-optimiser-claude carries a whitespace run.
Open question routed to commons, the spec owner: §5 mandates collapsing
whitespace runs in `source_query` only, while `title` is separately required
to be single-line and is already validated as such at manifest load. So
collapsing the title may be over-application — the alternative reading is
that both sites should emit it verbatim. Either reading keeps the two sites
in agreement, which is what this commit establishes; which of the two is
correct is a spec question, not ours.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WdVgowYC4LARgvNdNMiuvz
safe_resolve let an untyped ValueError escape when a target carried an
embedded NUL: os.path.realpath rejects the string before the boundary
check runs. That broke SourceError's documented contract ("always typed,
never a leaked OSError") and the v0.3.0 promise of stable machine-readable
codes on every failure.
A target that cannot be resolved at all now fails closed under the
existing `path_escape` code, with its own message. No new code is added —
`path_escape` already means "the boundary refused this target"; its
docstring is widened to say so. Consumers asserting on the code are
unaffected.
Found by consumer review: portfolio-optimiser's navigate_bundle leaks the
same ValueError from the same class of input, which makes an entire bundle
unreadable instead of skipping one link. That defect is theirs to fix; this
commit closes the library-side half.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WdVgowYC4LARgvNdNMiuvz
Backfills CHANGELOG.md, which was empty despite two prior tags: entries for
0.1.0 (untagged, pinned by commit), 0.2.0, 0.3.0, and this release.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WdVgowYC4LARgvNdNMiuvz
The module docstring and README claimed the library "calls the guard at
every persist gate". That described the intended end state in the present
tense. Door A -- the only door shipped -- has zero runtime dependencies and
calls no guard function before writing to disk.
Both places now say so, and state that gating external or untrusted content
is the caller's responsibility (okf.import_bundle, or prepare_input /
screen_output) until the persist gates land with Doors B and C.
Reported as F2 in a dogfood review by claude-playlist-corpus, which read the
earlier wording as safe-by-default and had to wire the gating itself.
No behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WdVgowYC4LARgvNdNMiuvz
Consumer tests were binding message wording (pytest.raises(match=...)) to
distinguish sub-causes within one error type. Every raise site now carries
a documented, stable code attribute; the registry lives in the errors.py
class docstrings. Codes are stable API; message text is explicitly not.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stamp ({stem}@{sha256(raw)[:16]}) was computed internally but only
written into frontmatter, forcing consumers to recompute it from the
manifest bytes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Consumers run mypy --strict against the inline annotations; without the
marker every imported symbol degrades to Any.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TDD step 9: package-root exports (materialize_bundle, IngestResult, the
typed error hierarchy, manifest types) and the two missing named seam
tests — provenance stamping verified with an independent frontmatter
parser, and navigability verified by resolving every index link. The
test module header maps all six library-side §11 seams to their named
tests; spec integrity stays with commons.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
TDD step 8: three golden cases per the §11 format (manifest.json,
fixture/, ingested-at.txt, expected-bundle/) — ingest-golden-file and
ingest-golden-sql are the conformance MUSTs; ingest-golden-http
documents and exercises the optional extension against mock payloads.
The conformance test re-materializes each case into a fresh directory
and compares byte for byte; offline and credential-free (the sql
fixture is a committed database file, http never opens a socket).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
TDD step 7: read_http executes the optional http extension point behind
an injectable transport seam (urllib_get is the only socket path; the
suite runs socket-free against a mock). credential_ref resolves from the
environment at run time, fail-fast before any transport call; the secret
rides in the Authorization header only. Explicit single-slash URL join,
CRLF-to-LF normalization, LF line-count cap (missing trailing newline
still counts), and code-fence-marker rejection. materialize_bundle
renders the body as a verbatim fenced block; the §8 gate test asserts
zero transport calls without the opt-in.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
TDD step 6: on an existing index, managed lines whose target is an
ingest-owned file removed this run are dropped, a managed label is
refreshed in place when the extraction title changes, and every other
line — curated links, promoted-verdict links, foreign line endings — is
preserved byte for byte. Load-bearing test: a promoted verdict and its
index link survive re-ingest unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
TDD step 5: materialize_bundle takes three explicit inputs (manifest,
bundle dir, regex-validated ingested_at — no wall-clock default) and
stages every extraction in memory before the first disk mutation. The
§3 collision gate refuses to overwrite any file without the ingest
stamp, before any mutation; replacement removes exactly the stamped
set. Frontmatter is the seven §5 keys in spec order with whitespace
collapse; output is LF-only raw bytes with one trailing newline; the
provenance stamp hashes the same bytes that are parsed. Fresh index.md
gets bundle_summary plus idempotent links in extraction order (§6
creation path; preserve/removal rules come next). The §8 network gate
refuses http fail-fast without the per-run opt-in; the http transport
itself lands in step 7.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
TDD step 4: read_sql executes one read-only SELECT against the sqlite
database whose path is resolved at run time from the env var named by
connection_ref (credentials never in the manifest). Read-only enforced
by the connection mode (file:...?mode=ro), single statement via
Connection.execute, §5 cell text conversion, streaming max_rows cap,
and every sqlite failure wrapped in SourceError.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
TDD step 3: read_csv executes a file-source extraction — boundary-checked
path resolution via commonpath on canonical paths (rejects .. traversal,
absolute paths, symlink escapes, and prefix-collision siblings), utf-8-sig
decoding, streaming max_rows cap as a typed error (never silent
truncation), ragged-row rejection, and verbatim cells (escaping stays in
the §5 renderers). New SourceError in the typed hierarchy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
TDD step 2: markdown-table rendering with the load-bearing escape order
(backslash, then pipe, then newlines to spaces with CRLF as one unit),
SQL value-to-text conversion (NULL empty, plain-decimal integers,
shortest round-trip floats, explicit bool/BLOB rejection — never silent
coercion), and the verbatim fenced block for http bodies. Table shape
and fenced-block form match the reference implementation byte for byte
ahead of the shared §11 golden fixtures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
TDD step 1 of the phase 1 plan: typed error hierarchy rooted in
IngestError, manifest parsing with schema validation before any source
call — id grammar, polymorphic source types, extraction rules, verdict
reservation (okf_type and filename namespace), and the no-embedded-
credentials rule for base_url. Dev tooling (pytest/mypy/ruff) added as
a PEP 735 dependency group; runtime dependencies stay empty.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
Phase 1 (Door A): ingest-spec v1 implementation + §11 golden fixtures,
pinned to the current spec revision, with TDD order and byte-exact
verification criteria. Phase 2 (Doors B/C): guard-gated inbox and
external bundle import. Phase 3: configurable bundle contract via a
profile object, default profile locked to the golden suite. Phase 4:
zero-dep Node half, coordination-first with owner sign-offs.
Each plan carries explicit key assumptions with tests and a
verification section.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
Operator decision: the initial v1 boundary (portfolio pair only) left
five of eight OKF surfaces uncovered. CLAUDE.md/README now define a
four-phase roadmap: door A, doors B/C, configurable bundle contract
(wiki strict-v1), and a zero-dep Node/ESM half for the second-brain
plugin world. Halves share contract and fixtures, never code. Security
stays with llm-ingestion-guard in both runtimes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
