open/llm-ingestion-okf
1
0
Fork 0
Code Issues Pull requests Releases Activity

feat(import): Door C flow against an injected import gate (Phase 2 step 5)

Browse source 
Reads an external OKF bundle as {bundle-relative path -> document text},
hands it WHOLE to an injected gate over the guard's okf.import_bundle (a
bundle-level call: it resolves the cross-link graph across concepts), and
merges only concepts clearing the non-blocking floor. Same injection pattern
as Door B, so the core stays dependency-free while the CI channel for the
real guard is settled.

Two constraints shaped the design and are pinned by tests:

- A merged concept is written VERBATIM. Stamping provenance into it would
  require round-tripping its frontmatter through this library's line-oriented
  parser, which cannot represent the block lists the guard's parser accepts --
  silent data loss -- and would persist bytes the guard never screened.
- Ownership is therefore proven by content identity: identical bytes at the
  target name are a no-op re-merge (re-import of an unchanged bundle is
  idempotent), and anything else at the name is refused. Curated content and
  an updated concept are refused alike; refusing is what never destroys.

The floor is fail-closed beyond the plan's "no error" wording: an error, an
unrecognised disposition, and a concept the gate returned no verdict for are
all refusals. quarantine_review stays its own bucket, as at Door B.
origin/channel are validated against the guard's pinned vocabulary -- it
derives trust from origin by enum identity, so an unrecognised string would be
silently downgraded rather than caught.

Three primitives promoted for reuse rather than duplicated:
reduce_to_id_grammar and check_filename_length to materialize.py, and
extract.decode_text. Door C slugs the WHOLE concept path, so tables/users.md
and views/users.md stay distinct. Concept discovery folds case explicitly
rather than globbing *.md, whose case-sensitivity follows the filesystem and
would import the same bundle differently on APFS and ext4.

README's "what is gated today" section corrected: it claimed nothing is gated,
which is no longer true, but the honest statement is narrower than "the doors
are gated" -- the library cannot verify that an injected adapter is a real
guard, and a permissive stub is believed.

405 tests green; ruff, ruff format and mypy --strict clean.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-25 06:57:25 +02:00
commit f10fc60de2
11 changed files with 1250 additions and 61 deletions
Download patch file Download diff file Expand all files Collapse all files

10
CLAUDE.md
View file

@ -19,8 +19,14 @@ one boundary rule:
the optional `[extract]` extra; without it those types are rejected
fail-fast.
- **Door C — external bundle import:** third-party OKF bundles are assessed
per concept via the guard's `okf.import_bundle`; only non-rejected concepts
are merged/materialized/indexed here.
per concept via the guard's `okf.import_bundle`; only concepts clearing the
guard's non-blocking floor are merged/indexed here. Two invariants, both
load-bearing: a merged concept is written **verbatim** (this library's
line-oriented frontmatter parser cannot round-trip the block lists the
guard's parser accepts, so stamping an external concept would destroy sender
data and persist bytes the guard never screened), and ownership is therefore
proven by **content identity** — an occupied target name is re-used only
when the bytes there are already identical, never overwritten otherwise.
**Boundary rule (non-negotiable, zero overlap):** `llm-ingestion-guard`
(pinned `>=0.2,<0.3`) answers "is this content safe to persist?" —

32
README.md
View file

@ -4,7 +4,10 @@ Shared ingestion library for OKF (Open Knowledge Format) bundles.
Status: phase 1 (spec-based ingestion) is implemented — manifest validation,
the `file`/`sql`/`http` connectors, deterministic materialization, index
generation, and the golden fixture suite under `examples/`. Phases 2–4 are
generation, and the golden fixture suite under `examples/`. Phase 2 is in
progress: the bundle inbox (`process_inbox`) and external-bundle import
(`import_bundle`) are implemented against an **injected** persist gate; the
integration with the real guard is not done yet (see below). Phases 3–4 are
planned (see `docs/plan/`).
## Planned scope (v1)
@ -40,18 +43,27 @@ Security is owned by the sibling package
No security functionality is reimplemented here.
### What is gated today: nothing
### What is gated today: read this before trusting a door
Door A — the only door shipped so far — is **ungated**. The package has zero
runtime dependencies and calls no guard function before writing to disk.
`materialize_bundle` writes what it is given.
The package still has **zero runtime dependencies** and imports no guard
function anywhere. That has a direct consequence for what "gated" means here:
If you materialize external or otherwise untrusted content, gating is **your**
responsibility at the call site: `okf.import_bundle` for received bundles, or
`prepare_input`/`screen_output` around extracted text. The guard-calling
persist gates are part of Doors B and C (phase 2), not of Door A.
- **Door A (`materialize_bundle`) is ungated.** It calls nothing before
writing to disk and writes what it is given.
- **Doors B and C (`process_inbox`, `import_bundle`) gate through an adapter
you supply.** Each takes a `gate` argument; the flow hands it the content
and obeys the verdict, refusing to persist anything that does not clear the
guard's non-blocking floor — including a disposition it does not recognise,
and (at Door C) a concept the gate returned no verdict for. What it cannot
do is check that your adapter is a real guard. A permissive stub approves
everything, and the flow will believe it.
This is stated plainly because the earlier wording ("calls the guard at every
So gating remains **your** responsibility at the call site: pass an adapter
over `prepare_input`/`screen_output` (Door B) or `okf.import_bundle`
(Door C). Wiring the doors to the real guard, and pinning it as a dependency,
is step 4 of phase 2 and is not done yet.
This is stated plainly because earlier wording ("calls the guard at every
persist gate") described the intended end state in the present tense, and a
consumer reasonably read it as safe-by-default.

34
docs/plan/phase-2-doors-b-c.md
View file

@ -65,6 +65,40 @@ No scanning, sanitizing, or quarantine logic is implemented here.
(reserved-file policy differs per consumer and becomes configurable in
Phase 3).
### Settled during implementation (step 5)
- **Verbatim merge, no frontmatter of ours.** A merged concept is written
exactly as the gate saw it. This is a correctness constraint, not a
preference: the guard's frontmatter parser accepts block lists, which this
library's line-oriented `parse_frontmatter` cannot round-trip, so stamping
provenance into an external concept would silently drop sender data — and
would persist bytes the guard never screened.
- **Ownership by content identity.** With no stamp available, an occupied
target name is re-used only when the bytes already there are identical (a
no-op re-merge, so re-import of an unchanged bundle is idempotent).
Anything else at the name — curated content or an updated version of the
same concept — is refused under `collision_unstamped`; the operator removes
the file to accept an update. A configurable stamping/reserved-file policy
is Phase 3's.
- **The merge floor is fail-closed, not "no error".** Deliverable 1's wording
is the looser reading: a concept carrying no error but a FAIL_SECURE or
unrecognised disposition is refused, and a concept the gate returned no
verdict for at all is refused. Only the guard's non-blocking floor merges —
the same floor Door B applies, with `quarantine_review` reported as its own
bucket rather than folded into rejection.
- **Upstream has already moved past the pin (observed, not acted on).** The
guard's `main` carries a commit that adds `allow_reserved=True` to
`okf.import_bundle` and *scans* `index.md`/`log.md` in a mode-b import
instead of path-rejecting them. The pin is the `v0.2.0` tag, where the
kwarg does not exist and rejection is unconditional, so this door is built
against the tag. Whoever bumps the pin owns re-checking that branch: today
a reserved name in a received bundle arrives as a per-concept rejection.
- **`origin`/`channel` are validated against the guard's pinned vocabulary.**
The guard derives trust from `origin` by enum *identity*, so an unrecognised
string would be silently downgraded to untrusted. The library refuses to
carry a provenance declaration it cannot recognise rather than let a typo
decide trust — refusing to guess, not deciding trust itself.
## Design decisions fixed by this plan
- Trust follows `origin`, never `channel` (guard doctrine) — callers must supply

26
src/llm_ingestion_okf/__init__.py
View file

@ -20,6 +20,14 @@ Door B (bundle inbox) public surface: process_inbox plus its result types.
Its persist gate is INJECTED -- the caller supplies a Gate adapter over
llm-ingestion-guard and process_inbox obeys the verdict; the library imports
no guard function itself and makes no security decision of its own.
Door C (external bundle import) public surface: import_bundle plus its result
types. Its gate is injected the same way, over the guard's okf.import_bundle:
the caller declares origin and channel at the door, the gate assesses every
concept, and only concepts clearing the non-blocking floor are merged. Merged
concepts are written verbatim -- ownership is proven by content identity
rather than by a stamp, so an occupied name is only ever re-used when the
bytes already there are identical.
"""
from .errors import (
@ -41,6 +49,16 @@ from .inbox import (
PersistedFile,
process_inbox,
)
from .importer import (
BundleDecision,
FailedConcept,
ImportDecision,
ImportGate,
ImportResult,
MergedConcept,
RefusedConcept,
import_bundle,
)
from .manifest import (
Extraction,
FileSource,
@ -55,25 +73,33 @@ __version__ = "0.3.2"
__all__ = [
"BlockedFile",
"BundleDecision",
"Extraction",
"ExtractionError",
"FailedConcept",
"FailedFile",
"FileSource",
"Gate",
"GateDecision",
"HttpSource",
"ImportDecision",
"ImportGate",
"ImportResult",
"InboxResult",
"IngestError",
"IngestResult",
"Manifest",
"ManifestError",
"MaterializationError",
"MergedConcept",
"NetworkGateError",
"PersistedFile",
"RefusedConcept",
"RenderError",
"SourceError",
"SqlSource",
"extract_text",
"import_bundle",
"load_manifest",
"materialize_bundle",
"process_inbox",

13
src/llm_ingestion_okf/errors.py
View file

@ -110,6 +110,19 @@ class MaterializationError(IngestError):
would inject frontmatter lines
- `okf_type_reserved` — an inbox concept claims the reserved 'verdict'
layer (the same reservation ManifestError enforces at Door A)
- `import_path_empty` — an external concept path reduces to an empty slug
under the id grammar (Door C; never an invented fallback name)
- `import_path_too_long` — the generated import filename would exceed the
255-byte filesystem limit (Door C; never a truncated name)
- `import_slug_collision` — two concepts in one external bundle reduce to
one generated filename (Door C); both are refused rather than letting
iteration order decide which one survives
- `import_label_invalid` — an external concept path contains `[`/`]`, which
would break its index link (the guard's path gate permits them)
- `import_provenance_invalid` — an `origin`/`channel` outside the guard's
pinned vocabulary (Door C); refused rather than carried, because the
guard derives trust from `origin` by enum identity and an unrecognised
value would be silently downgraded
"""

10
src/llm_ingestion_okf/extract.py
View file

@ -33,7 +33,7 @@ _OPTIONAL_EXTENSIONS = frozenset({".pdf", ".docx", ".xlsx"})
_SKIP_TAGS = frozenset({"script", "style"})
def _decode(data: bytes) -> str:
def decode_text(data: bytes) -> str:
"""Decode file bytes as UTF-8 (BOM-stripping), typed on failure.
utf-8-sig so a byte-order mark never leaks into the first character
@ -50,12 +50,12 @@ def _decode(data: bytes) -> str:
def _extract_passthrough(data: bytes) -> str:
"""`md`/`txt`: the decoded text verbatim."""
return _decode(data)
return decode_text(data)
def _extract_csv(data: bytes) -> str:
"""`csv`: parse with the stdlib reader, render the Phase 1 markdown table."""
reader = csv.reader(io.StringIO(_decode(data)))
reader = csv.reader(io.StringIO(decode_text(data)))
header = next(reader, None)
if header is None:
raise ExtractionError("CSV has no header row", code="extractor_empty_csv")
@ -65,7 +65,7 @@ def _extract_csv(data: bytes) -> str:
def _extract_json(data: bytes) -> str:
"""`json`: the decoded text verbatim inside a fenced block (Phase 1 renderer)."""
return render_fenced_block(_decode(data))
return render_fenced_block(decode_text(data))
class _HTMLTextExtractor(HTMLParser):
@ -105,7 +105,7 @@ class _HTMLTextExtractor(HTMLParser):
def _extract_html(data: bytes) -> str:
"""`html`/`htm`: text via `html.parser`, script/style stripped (spec B3)."""
parser = _HTMLTextExtractor()
parser.feed(_decode(data))
parser.feed(decode_text(data))
parser.close()
return parser.text()

425
src/llm_ingestion_okf/importer.py Normal file
View file

@ -0,0 +1,425 @@
"""Door C: external bundle import — read, gate per concept, merge the accepted.
A third-party OKF bundle is read as `{bundle-relative path -> document text}`,
handed WHOLE to the guard's `okf.import_bundle` (a bundle-level call: it
resolves the cross-link graph across concepts, so gating them one at a time
would throw that half of the gate away), and only concepts whose verdict clears
the non-blocking floor are merged.
A merged concept is written VERBATIM — exactly the text the gate saw. Nothing
of this library's is merged into its frontmatter, and that is a correctness
constraint, not a preference: the guard's frontmatter parser accepts block
lists, which this library's line-oriented :func:`parse_frontmatter` cannot
round-trip, so re-rendering a concept would silently drop data the sender
supplied. It would also persist bytes the guard never screened.
That leaves ownership to be proven by content identity instead of by a stamp:
a target name held by byte-identical content is a no-op re-merge (so re-import
of an unchanged bundle is idempotent), and a target name held by anything else
is refused. Curated content and an UPDATED external concept are refused alike —
the library cannot tell them apart without a marker it has no safe place to
write, and refusing is the answer that never destroys. A configurable
reserved-file/frontmatter policy is Phase 3's; this is the v1 floor.
No security decision is taken here: the gate is injected, and this module only
obeys the verdict it returns.
"""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Protocol
from .errors import IngestError, MaterializationError, SourceError
from .extract import decode_text
from .materialize import (
INDEX_NAME,
check_filename_length,
link_in_index,
reduce_to_id_grammar,
validate_ingested_at,
write_bytes,
)
# An OKF concept is a `.md` document by definition — the guard's path gate
# rejects anything else outright — so nothing else in the source tree is a
# concept, and nothing else is this door's to merge.
_CONCEPT_SUFFIX = ".md"
_FILENAME_PREFIX = "import-"
# The guard's non-blocking floor and its review queue, by VALUE (`Disposition`
# is a `str, Enum`, so the value is the stable thing to compare against across
# the pinned `>=0.2,<0.3` range). Pinned as constants here rather than imported
# because the dependency is injected — deliberately restated independently of
# Door B's copy in `inbox.py`, so drift in either door is visible rather than
# silently shared. The step-4 adapter's signature smoke test is what catches a
# rename in the guard itself.
_DISPOSITION_MERGE = "warn"
_DISPOSITION_QUARANTINE = "quarantine_review"
# The guard's Origin/Channel vocabularies, likewise by value. Validated here
# because `trust_for` compares by enum IDENTITY: a value outside these sets
# would reach the guard as a plain string, miss the identity check, and be
# silently classified untrusted. That failure is safe but silent, and a
# provenance declaration the library cannot recognise is not one it should
# carry — refusing is not a trust decision, it is refusing to guess at one.
_ORIGINS = frozenset({"external", "internal"})
_CHANNELS = frozenset({"automatic", "manual"})
# --- the guard seam -------------------------------------------------------
@dataclass(frozen=True)
class ImportDecision:
"""One concept's verdict, carried verbatim from `okf.import_bundle`.
`disposition` is the guard's `Disposition` VALUE and `error` its
`ConceptResult.error` — non-`None` on a hard reject (bad path, unsafe
frontmatter, non-https `resource`), in which case the concept must not be
merged whatever the disposition says. `reasons` is the audit trail the
adapter flattened out of the concept's scan report.
"""
path: str
disposition: str
error: str | None = None
reasons: tuple[str, ...] = ()
@dataclass(frozen=True)
class BundleDecision:
"""The gate's verdict on a whole bundle: per-concept results plus the log.
`log` is the guard's `BundleResult.log()` body, returned to the caller and
never written here.
"""
concepts: tuple[ImportDecision, ...]
log: str = ""
class ImportGate(Protocol):
"""The persist gate, injected. The library never imports the guard itself.
`origin` and `channel` are keyword-only by design: both are plain strings
at this seam, and a positional call site that transposed them would move a
concept between trust tiers without any type error to catch it.
"""
def __call__(self, bundle: dict[str, str], *, origin: str, channel: str) -> BundleDecision: ...
# --- per-concept outcomes -------------------------------------------------
@dataclass(frozen=True)
class MergedConcept:
"""An external concept that cleared the gate and is present in the bundle.
Also covers the no-op re-merge: identical bytes already at the target name
are the concept being present, not a second write.
"""
concept_path: str
path: Path
reasons: tuple[str, ...] = ()
@dataclass(frozen=True)
class RefusedConcept:
"""A concept the guard did not clear. Quarantine and rejection are reported
separately: quarantine means "hold for human review" and is the operator's
queue, while a fail-secure verdict is a decision, not a queue."""
concept_path: str
disposition: str
error: str | None
reasons: tuple[str, ...]
@dataclass(frozen=True)
class FailedConcept:
"""A concept this library could not process — unreadable, unusable name, or
a collision. Always a typed error, never a leaked stdlib exception."""
concept_path: str
error: IngestError
@dataclass(frozen=True)
class ImportResult:
"""Every concept's outcome, in sorted concept-path order.
Four disjoint buckets, and every concept lands in exactly one: a run's
report is complete by construction, so a concept that silently vanished
would show up as a missing entry rather than as nothing at all. `log` is
the guard's log body and `ingested_at` the run's explicit timestamp — the
caller persists them if their reserved-file policy says to.
"""
merged: tuple[MergedConcept, ...]
quarantined: tuple[RefusedConcept, ...]
rejected: tuple[RefusedConcept, ...]
failed: tuple[FailedConcept, ...]
log: str
ingested_at: str
def import_slug(concept_path: str) -> str:
"""Reduce a bundle-relative concept path to the Phase 1 id grammar.
The whole path reduces, not just its final segment: `tables/users.md` and
`views/users.md` are distinct concepts, and slugging the stem alone would
collapse them onto one filename. A path that reduces to nothing fails fast
rather than being given an invented name.
"""
concept_id = concept_path[: -len(_CONCEPT_SUFFIX)]
slug = reduce_to_id_grammar(concept_id)
if not slug:
raise MaterializationError(
f"concept path {concept_path!r} reduces to an empty slug under the "
"id grammar ([a-z0-9][a-z0-9-]*) — refusing to invent a filename",
code="import_path_empty",
)
return slug
def import_filename(slug: str) -> str:
"""The bundle filename for an imported concept.
The `import-` prefix keeps the namespace disjoint from `index.md`, Door A's
`ingest-*`, Door B's `inbox-*`, and `promoted-verdict-*` for every slug the
grammar admits.
"""
return check_filename_length(
f"{_FILENAME_PREFIX}{slug}{_CONCEPT_SUFFIX}", code="import_path_too_long"
)
def _index_label(concept_path: str) -> str:
"""The concept-ID, validated as an index link label.
The guard's path gate permits brackets in a concept path; `- [label](target)`
does not. Fail-fast, never repair — the same rule Door A applies to a
manifest title and Door B to a dropped filename.
"""
label = concept_path[: -len(_CONCEPT_SUFFIX)]
if any(char in label for char in "\n\r[]"):
raise MaterializationError(
f"concept path {concept_path!r} contains '[' or ']', which would break "
"its index link — rename it at the sender",
code="import_label_invalid",
)
return label
def _read_bundle(source: Path) -> tuple[dict[str, str], list[FailedConcept]]:
"""Read every concept document under `source`, keyed by POSIX-relative path.
Unreadable and undecodable concepts never reach the gate — they are per-
concept failures, and a concept the gate never saw is never merged.
"""
documents: dict[str, str] = {}
failed: list[FailedConcept] = []
for path in sorted(source.rglob("*")):
# The suffix test is explicit and case-folded rather than a `*.md`
# glob: glob case-sensitivity follows the FILESYSTEM, so `NOTE.MD`
# would be a concept on APFS and not one on ext4 — the same bundle
# importing differently per platform. The guard folds case here too.
if not path.is_file() or path.suffix.lower() != _CONCEPT_SUFFIX:
continue
concept_path = path.relative_to(source).as_posix()
try:
documents[concept_path] = decode_text(path.read_bytes())
except OSError as exc:
failed.append(
FailedConcept(
concept_path=concept_path,
error=SourceError(
f"cannot read concept {concept_path}: {exc}", code="source_file_missing"
),
)
)
except IngestError as exc:
failed.append(FailedConcept(concept_path=concept_path, error=exc))
return documents, failed
def import_bundle(
source_dir: Path,
bundle_dir: Path,
ingested_at: str,
*,
origin: str,
channel: str,
gate: ImportGate,
) -> ImportResult:
"""Merge the accepted concepts of an external OKF bundle (Door C).
An explicit operator command, never a watcher or a scheduler. `origin` and
`channel` are required with no defaults — trust follows origin, never
channel, and the caller is the one who knows both. `gate` is the guard
adapter over `okf.import_bundle`: every concept is assessed before anything
is written, and only the guard's non-blocking floor merges — anything else,
INCLUDING a disposition this library does not recognise and a concept the
gate returned no verdict for, fails closed.
One bad concept never aborts the run. Unreadable concepts, unusable names
and collisions are reported per concept in :class:`ImportResult` while the
rest still merge. Only three conditions fail the whole run, and all three
are wrong for every concept at once: an invalid `ingested_at`, an
unrecognised `origin`/`channel`, and a missing source directory.
"""
validate_ingested_at(ingested_at)
if origin not in _ORIGINS or channel not in _CHANNELS:
raise MaterializationError(
f"origin must be one of {sorted(_ORIGINS)} and channel one of "
f"{sorted(_CHANNELS)}, got origin={origin!r} channel={channel!r} — "
"refusing to carry a provenance declaration the guard would not "
"recognise (it decides trust from these values)",
code="import_provenance_invalid",
)
source = Path(source_dir)
if not source.is_dir():
raise SourceError(
f"source bundle directory does not exist: {source}", code="source_root_missing"
)
documents, failed = _read_bundle(source)
merged: list[MergedConcept] = []
quarantined: list[RefusedConcept] = []
rejected: list[RefusedConcept] = []
log = ""
if documents:
decision = gate(dict(documents), origin=origin, channel=channel)
log = decision.log
by_path = {entry.path: entry for entry in decision.concepts}
accepted: list[tuple[str, str]] = []
for concept_path in sorted(documents):
verdict = by_path.get(concept_path)
if verdict is None:
# No verdict is not consent: a concept the gate dropped from
# its result is refused, never read as approval by omission.
rejected.append(
RefusedConcept(
concept_path=concept_path,
disposition="",
error="the gate returned no verdict for this concept",
reasons=(),
)
)
continue
# An error is a refusal on its own terms: the guard pairs one with
# FAIL_SECURE today, but the floor must not depend on that pairing.
if verdict.error is not None or verdict.disposition != _DISPOSITION_MERGE:
refused = RefusedConcept(
concept_path=concept_path,
disposition=verdict.disposition,
error=verdict.error,
reasons=verdict.reasons,
)
if verdict.error is None and verdict.disposition == _DISPOSITION_QUARANTINE:
quarantined.append(refused)
else:
rejected.append(refused)
continue
accepted.append((concept_path, documents[concept_path]))
# Name every accepted concept BEFORE any write, so an intra-run slug
# collision is caught while both concepts can still be refused together.
named: list[tuple[str, str, str]] = []
slug_owners: dict[str, list[str]] = {}
for concept_path, text in accepted:
try:
name = import_filename(import_slug(concept_path))
_index_label(concept_path)
except IngestError as exc:
failed.append(FailedConcept(concept_path=concept_path, error=exc))
continue
named.append((concept_path, name, text))
slug_owners.setdefault(name, []).append(concept_path)
colliding = {name for name, owners in slug_owners.items() if len(owners) > 1}
for name in sorted(colliding):
for concept_path in slug_owners[name]:
others = ", ".join(
repr(other) for other in slug_owners[name] if other != concept_path
)
failed.append(
FailedConcept(
concept_path=concept_path,
error=MaterializationError(
f"{concept_path!r} and {others} both reduce to {name!r} — "
"rename one at the sender; refusing to pick a winner",
code="import_slug_collision",
),
)
)
# The occupancy gate, evaluated against the bundle as it was BEFORE
# this run: a file written below must never be judged by a later
# concept's check. Byte-identity is the only ownership proof available
# at this door, so anything else at the name is curated content or an
# update — refused either way, never overwritten.
bundle = Path(bundle_dir)
# `None` marks a no-op re-merge — an explicit sentinel, because an
# empty concept document is legitimate and must still be written.
staged: list[tuple[str, str, str | None]] = []
for concept_path, name, text in named:
if name in colliding:
continue
content = text.encode("utf-8")
existing = bundle / name
if existing.is_file():
if existing.read_bytes() == content:
staged.append((concept_path, name, None))
continue
failed.append(
FailedConcept(
concept_path=concept_path,
error=MaterializationError(
f"generated filename {name!r} is occupied by different content — "
"it is either curated or an earlier version of this concept, and "
"without a stamp the two cannot be told apart; remove it to accept "
"the update (§3)",
code="collision_unstamped",
),
)
)
continue
staged.append((concept_path, name, text))
# Disk phase.
for concept_path, name, pending in staged:
if pending is None:
path = bundle / name
else:
bundle.mkdir(parents=True, exist_ok=True)
path = write_bytes(bundle, name, pending)
verdict = by_path[concept_path]
merged.append(
MergedConcept(concept_path=concept_path, path=path, reasons=verdict.reasons)
)
# §6 index — the last disk mutation, and only when something merged.
if merged:
index_path = bundle / INDEX_NAME
if not index_path.is_file():
write_bytes(bundle, INDEX_NAME, "")
for entry in merged:
link_in_index(bundle, entry.path.name, _index_label(entry.concept_path))
return ImportResult(
merged=tuple(merged),
quarantined=tuple(quarantined),
rejected=tuple(rejected),
failed=tuple(sorted(failed, key=lambda entry: entry.concept_path)),
log=log,
ingested_at=ingested_at,
)

54
src/llm_ingestion_okf/inbox.py
View file

@ -20,7 +20,6 @@ supplies the verdict and this module only obeys it.
from __future__ import annotations
import hashlib
import re
import unicodedata
from collections.abc import Callable
from dataclasses import dataclass
@ -30,28 +29,16 @@ from .errors import IngestError, MaterializationError, SourceError
from .extract import extract_text
from .materialize import (
INDEX_NAME,
check_filename_length,
link_in_index,
parse_frontmatter,
reduce_to_id_grammar,
validate_ingested_at,
write_bytes,
)
_RESERVED_OKF_TYPE = "verdict"
# Every character outside the Phase 1 id grammar (`[a-z0-9][a-z0-9-]*`) is a
# separator. Deliberately NOT a transliteration: mapping non-ASCII letters to
# ASCII ones would be a semantic claim the slugger cannot make — Norwegian
# `møte` (meeting) would become `mote` (fashion). The readable name survives
# verbatim in the `title` field; the slug is an identifier, not a label.
_SEPARATOR_RUN_RE = re.compile(r"[^a-z0-9]+")
# NAME_MAX: the per-component limit on every filesystem this library targets
# (APFS, ext4, NTFS all cap at 255). Checked here rather than caught at the
# write, because the OS signals it as an OSError whose errno differs per
# platform (63 on macOS, 36 on Linux) — an untyped, unportable failure at the
# very moment the caller needs a typed per-file outcome. Verified empirically
# on APFS 2026-07-25: a 255-byte name writes, a 258-byte one raises errno 63.
_MAX_FILENAME_BYTES = 255
_FILENAME_PREFIX = "inbox-"
_FILENAME_SUFFIX = ".md"
@ -64,15 +51,7 @@ def inbox_slug(source_filename: str) -> str:
A name that reduces to nothing fails fast: never an invented fallback like
`untitled`, which would silently collide across unrelated files.
"""
# NFC first: macOS (APFS/HFS+) hands filenames over DECOMPOSED, so an
# `é` arrives as `e` + combining acute. Without normalising, the same
# visual filename slugs differently depending on where it came from — the
# combining mark alone becomes a separator and the base letter survives
# (`cafe`), where a composed `é` is one non-grammar character (`caf`).
# Composing first makes the whole letter one unit, so non-ASCII is
# uniformly a separator and the slug is stable across both forms.
stem = unicodedata.normalize("NFC", Path(source_filename).stem)
slug = _SEPARATOR_RUN_RE.sub("-", stem.lower()).strip("-")
slug = reduce_to_id_grammar(Path(source_filename).stem)
if not slug:
raise MaterializationError(
f"inbox filename {source_filename!r} reduces to an empty slug under the "
@ -88,27 +67,14 @@ def inbox_filename(slug: str) -> str:
The `inbox-` prefix keeps the namespace disjoint from `index.md`, Door A's
`ingest-*`, and `promoted-verdict-*` for every slug the grammar admits.
A name the filesystem cannot hold fails fast rather than being truncated:
truncation is lossy AND collision-prone (two long names sharing a prefix
would reduce to one filename, and the second write would silently claim
the first file). Refusing keeps the same posture as `inbox_slug_empty` —
the library never invents a filename the operator did not give it. The
operator's fix is to rename the dropped file, so the message carries both
the actual size and the limit.
A name the filesystem cannot hold fails fast rather than being truncated
(see :func:`check_filename_length`). Refusing keeps the same posture as
`inbox_slug_empty` — the library never invents a filename the operator did
not give it.
"""
name = f"{_FILENAME_PREFIX}{slug}{_FILENAME_SUFFIX}"
# The slug is ASCII by construction (the id grammar admits nothing else),
# so len() in characters and in bytes agree — encoding here anyway keeps
# the check honest if the grammar is ever widened.
size = len(name.encode("utf-8"))
if size > _MAX_FILENAME_BYTES:
raise MaterializationError(
f"inbox filename for slug {slug!r} would be {size} bytes, over the "
f"{_MAX_FILENAME_BYTES}-byte filesystem limit — rename the dropped "
"file; refusing to truncate (lossy and collision-prone)",
code="inbox_slug_too_long",
)
return name
return check_filename_length(
f"{_FILENAME_PREFIX}{slug}{_FILENAME_SUFFIX}", code="inbox_slug_too_long"
)
def _normalize_body(text: str) -> str:

56
src/llm_ingestion_okf/materialize.py
View file

@ -12,6 +12,7 @@ from __future__ import annotations
import hashlib
import logging
import re
import unicodedata
from dataclasses import dataclass
from pathlib import Path
@ -64,6 +65,61 @@ def validate_ingested_at(ingested_at: str) -> None:
)
# --- generated-name primitives (shared by Doors B and C) ------------------
# Every character outside the Phase 1 id grammar (`[a-z0-9][a-z0-9-]*`) is a
# separator. Deliberately NOT a transliteration: mapping non-ASCII letters to
# ASCII ones would be a semantic claim the slugger cannot make — Norwegian
# `møte` (meeting) would become `mote` (fashion). The readable name survives
# verbatim in the concept's title or index label; the slug is an identifier,
# not a label.
_SEPARATOR_RUN_RE = re.compile(r"[^a-z0-9]+")
# NAME_MAX: the per-component limit on every filesystem this library targets
# (APFS, ext4, NTFS all cap at 255). Checked before the write rather than
# caught at it, because the OS signals it as an OSError whose errno differs per
# platform (63 on macOS, 36 on Linux) — an untyped, unportable failure at the
# very moment the caller needs a typed per-file outcome. Verified empirically
# on APFS 2026-07-25: a 255-byte name writes, a 258-byte one raises errno 63.
NAME_MAX_BYTES = 255
def reduce_to_id_grammar(text: str) -> str:
"""Reduce arbitrary text to the Phase 1 id grammar, or to the empty string.
Lowercased, with every run of non-grammar characters collapsed to a single
`-` and both ends stripped. The caller decides what an empty result means:
both doors refuse it rather than invent a fallback name.
"""
# NFC first: macOS (APFS/HFS+) hands filenames over DECOMPOSED, so an `é`
# arrives as `e` + combining acute. Without normalising, the same visual
# name reduces differently depending on where it came from — the combining
# mark alone becomes a separator and the base letter survives (`cafe`),
# where a composed `é` is one non-grammar character (`caf`). Composing
# first makes the whole letter one unit, so non-ASCII is uniformly a
# separator and the result is stable across both forms.
return _SEPARATOR_RUN_RE.sub("-", unicodedata.normalize("NFC", text).lower()).strip("-")
def check_filename_length(name: str, *, code: str) -> str:
"""Refuse a generated filename the filesystem cannot hold.
Never truncated: truncation is lossy AND collision-prone (two long names
sharing a prefix would reduce to one filename, and the second write would
silently claim the first file). The message carries both the actual size
and the limit, because the operator's fix is to shorten the source name.
"""
size = len(name.encode("utf-8"))
if size > NAME_MAX_BYTES:
raise MaterializationError(
f"the generated filename would be {size} bytes, over the "
f"{NAME_MAX_BYTES}-byte filesystem limit — shorten the source name; "
"refusing to truncate (lossy and collision-prone)",
code=code,
)
return name
def _collapse_whitespace(value: str) -> str:
# §5 mandates whitespace-run collapse for ONE field only: `source_query`
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT

75
tests/test_error_codes.py
View file

@ -28,6 +28,13 @@ from llm_ingestion_okf.errors import (
SourceError,
)
from llm_ingestion_okf.extract import extract_text
from llm_ingestion_okf.importer import (
BundleDecision,
ImportDecision,
import_bundle,
import_filename,
import_slug,
)
from llm_ingestion_okf.inbox import (
GateDecision,
inbox_filename,
@ -411,6 +418,74 @@ def test_okf_type_reserved_at_the_inbox_door() -> None:
assert code_of(excinfo) == "okf_type_reserved"
# --- MaterializationError codes at Door C ---
def approve_all(bundle: dict[str, str], *, origin: str, channel: str) -> BundleDecision:
"""A gate that clears every concept at the non-blocking floor."""
return BundleDecision(
concepts=tuple(ImportDecision(path=path, disposition="warn") for path in sorted(bundle))
)
def place(source: Path, relpath: str, content: str = "body\n") -> None:
path = source / relpath
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content, encoding="utf-8", newline="")
def run_import(tmp_path: Path, **overrides: Any) -> Any:
kwargs: dict[str, Any] = {"origin": "external", "channel": "automatic", "gate": approve_all}
kwargs.update(overrides)
return import_bundle(tmp_path / "source", tmp_path / "bundle", INGESTED_AT, **kwargs)
def test_import_path_empty() -> None:
with pytest.raises(MaterializationError) as excinfo:
import_slug("!!!.md")
assert code_of(excinfo) == "import_path_empty"
def test_import_path_too_long() -> None:
with pytest.raises(MaterializationError) as excinfo:
import_filename("a" * 250)
assert code_of(excinfo) == "import_path_too_long"
def test_import_slug_collision(tmp_path: Path) -> None:
# Two concept paths reducing to one generated filename: reported per
# concept, so this code surfaces in ImportResult.failed rather than as a raise.
place(tmp_path / "source", "a/b.md")
place(tmp_path / "source", "a-b.md")
result = run_import(tmp_path)
assert {entry.error.code for entry in result.failed} == {"import_slug_collision"}
def test_import_label_invalid(tmp_path: Path) -> None:
place(tmp_path / "source", "report [v2].md")
result = run_import(tmp_path)
assert [entry.error.code for entry in result.failed] == ["import_label_invalid"]
def test_collision_unstamped_at_the_import_door(tmp_path: Path) -> None:
# Same code as Door A's stamp collision: the generated name is occupied by
# content this library cannot prove is its own to replace.
bundle = tmp_path / "bundle"
bundle.mkdir()
(bundle / "import-note.md").write_text("curated\n", encoding="utf-8")
place(tmp_path / "source", "note.md")
result = run_import(tmp_path)
assert [entry.error.code for entry in result.failed] == ["collision_unstamped"]
@pytest.mark.parametrize(("origin", "channel"), [("nope", "automatic"), ("external", "nope")])
def test_import_provenance_invalid(tmp_path: Path, origin: str, channel: str) -> None:
place(tmp_path / "source", "a.md")
with pytest.raises(MaterializationError) as excinfo:
run_import(tmp_path, origin=origin, channel=channel)
assert code_of(excinfo) == "import_provenance_invalid"
# --- NetworkGateError codes ---

576
tests/test_import_flow.py Normal file
View file

@ -0,0 +1,576 @@
"""Door C import flow against a stub import gate (Phase 2 step 5).
The stub stands in for the pinned `llm-ingestion-guard` `okf.import_bundle`
surface so the merge, quarantine and reject branches run deterministically
without the dependency installed. It is a TEST DOUBLE and lives here on
purpose — never in `src/`.
What the flow owes the caller, and what this suite pins:
- the guard decides per concept; the library only branches on the verdict and
never re-derives one (an unrecognised disposition, and a concept the gate
returned no verdict for, must both fail CLOSED);
- a merged concept is written VERBATIM — exactly the text handed to the gate,
with no frontmatter of ours merged into it (round-tripping the concept's own
frontmatter through this library's line-oriented parser would silently
destroy the block lists the guard's parser accepts);
- ownership is proven by content identity, not by a stamp: a target name held
by identical bytes is a no-op re-merge, and a target name held by ANYTHING
else is refused — curated content and a changed external concept alike;
- one bad concept never aborts the run;
- the guard's log body is returned to the caller, never persisted (the
reserved-file policy is Phase 3's).
"""
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
import pytest
from llm_ingestion_okf.importer import BundleDecision, ImportDecision, ImportResult, import_bundle
INGESTED_AT = "2026-07-25T12:00:00Z"
# The guard's Origin/Channel/Disposition are `str, Enum`s; these are their
# VALUES, restated here independently of the library constants so a drift in
# either is visible.
EXTERNAL = "external"
INTERNAL = "internal"
AUTOMATIC = "automatic"
MANUAL = "manual"
WARN = "warn"
QUARANTINE_REVIEW = "quarantine_review"
FAIL_SECURE = "fail_secure"
# --- the stub gate --------------------------------------------------------
@dataclass
class StubImportGate:
"""A bundle gate whose per-concept verdict is scripted.
`by_marker` maps a substring to the disposition any concept containing it
gets; `errors` maps a concept path to a hard-reject reason (the guard's
`ConceptResult.error`). Everything else passes at the non-blocking floor.
"""
by_marker: dict[str, str] = field(default_factory=dict)
errors: dict[str, str] = field(default_factory=dict)
omit: frozenset[str] = frozenset()
log: str = ""
calls: list[dict[str, str]] = field(default_factory=list)
provenance: list[tuple[str, str]] = field(default_factory=list)
def __call__(self, bundle: dict[str, str], *, origin: str, channel: str) -> BundleDecision:
self.calls.append(dict(bundle))
self.provenance.append((origin, channel))
decisions = []
for path in sorted(bundle):
if path in self.omit:
continue
error = self.errors.get(path)
disposition = FAIL_SECURE if error is not None else WARN
for marker, scripted in self.by_marker.items():
if marker in bundle[path]:
disposition = scripted
decisions.append(
ImportDecision(
path=path,
disposition=disposition,
error=error,
reasons=(f"scanned {path}",),
)
)
return BundleDecision(concepts=tuple(decisions), log=self.log)
def place(source: Path, relpath: str, content: str) -> Path:
path = source / relpath
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content, encoding="utf-8", newline="")
return path
def run(
tmp_path: Path,
gate: StubImportGate,
*,
origin: str = EXTERNAL,
channel: str = AUTOMATIC,
ingested_at: str = INGESTED_AT,
) -> tuple[ImportResult, Path]:
bundle = tmp_path / "bundle"
result = import_bundle(
tmp_path / "source",
bundle,
ingested_at,
origin=origin,
channel=channel,
gate=gate,
)
return result, bundle
CONCEPT = "---\ntype: dataset\ntitle: Users\n---\n\nA table of users.\n"
# --- the merge branch -----------------------------------------------------
def test_an_accepted_concept_is_merged(tmp_path: Path) -> None:
place(tmp_path / "source", "tables/users.md", CONCEPT)
result, bundle = run(tmp_path, StubImportGate())
assert [entry.concept_path for entry in result.merged] == ["tables/users.md"]
assert (bundle / "import-tables-users.md").is_file()
def test_the_merged_bytes_are_exactly_the_bytes_the_gate_saw(tmp_path: Path) -> None:
"""Verbatim is the whole contract at this door: the guard validated a
document, so the document is what may be persisted. Nothing of ours is
merged into its frontmatter — this library's line-oriented parser cannot
round-trip the block lists the guard's parser accepts, so writing back a
re-rendered concept would silently drop data the operator sent us.
"""
document = "---\ntags:\n - alpha\n - beta\n---\n\nBody with a [link](other.md).\n"
place(tmp_path / "source", "notes/tagged.md", document)
result, bundle = run(tmp_path, gate := StubImportGate())
written = (bundle / "import-notes-tagged.md").read_bytes()
assert written == document.encode("utf-8")
assert gate.calls[0]["notes/tagged.md"] == document
assert result.failed == ()
def test_the_index_links_every_merged_concept_by_concept_id(tmp_path: Path) -> None:
place(tmp_path / "source", "tables/users.md", CONCEPT)
_, bundle = run(tmp_path, StubImportGate())
index = (bundle / "index.md").read_text(encoding="utf-8")
assert index == "- [tables/users](import-tables-users.md)\n"
def test_nested_paths_keep_their_directory_in_the_generated_name(tmp_path: Path) -> None:
"""Flattening on the stem alone would collide two distinct concepts; the
whole bundle-relative path is what reduces to the slug.
"""
place(tmp_path / "source", "tables/users.md", "one\n")
place(tmp_path / "source", "views/users.md", "two\n")
result, bundle = run(tmp_path, StubImportGate())
assert [entry.path.name for entry in result.merged] == [
"import-tables-users.md",
"import-views-users.md",
]
assert (bundle / "import-tables-users.md").read_text(encoding="utf-8") == "one\n"
assert (bundle / "import-views-users.md").read_text(encoding="utf-8") == "two\n"
def test_the_gate_is_called_once_with_the_whole_bundle(tmp_path: Path) -> None:
"""`okf.import_bundle` is a bundle-level call — it resolves the cross-link
graph across concepts — so the flow hands it the whole mapping at once,
keyed by POSIX-separated bundle-relative paths.
"""
place(tmp_path / "source", "b.md", "second\n")
place(tmp_path / "source", "nested/a.md", "first\n")
_, _ = run(tmp_path, gate := StubImportGate())
assert len(gate.calls) == 1
assert gate.calls[0] == {"b.md": "second\n", "nested/a.md": "first\n"}
def test_origin_and_channel_reach_the_gate_verbatim(tmp_path: Path) -> None:
"""Trust follows origin, never channel — the caller declares both at the
door and the library only carries them; it never defaults or derives them.
"""
place(tmp_path / "source", "a.md", "body\n")
run(tmp_path, gate := StubImportGate(), origin=INTERNAL, channel=MANUAL)
assert gate.provenance == [(INTERNAL, MANUAL)]
# --- the blocking branches ------------------------------------------------
def test_a_hard_rejected_concept_is_not_merged(tmp_path: Path) -> None:
place(tmp_path / "source", "bad.md", "body\n")
result, bundle = run(tmp_path, StubImportGate(errors={"bad.md": "reserved filename"}))
assert result.merged == ()
assert [entry.concept_path for entry in result.rejected] == ["bad.md"]
assert result.rejected[0].error == "reserved filename"
assert not (bundle / "import-bad.md").exists()
def test_quarantine_review_is_reported_separately_from_rejection(tmp_path: Path) -> None:
"""Quarantine means "hold for human review" and is the operator's queue; a
fail-secure verdict is a decision, not a queue. Same split as Door B.
"""
place(tmp_path / "source", "suspect.md", "hold me\n")
result, bundle = run(tmp_path, StubImportGate(by_marker={"hold me": QUARANTINE_REVIEW}))
assert result.merged == ()
assert [entry.concept_path for entry in result.quarantined] == ["suspect.md"]
assert result.rejected == ()
assert not (bundle / "import-suspect.md").exists()
@pytest.mark.parametrize("disposition", ["", "pass", "ok", "WARN", "warn "])
def test_an_unrecognised_disposition_fails_closed(tmp_path: Path, disposition: str) -> None:
place(tmp_path / "source", "a.md", "body\n")
result, bundle = run(tmp_path, StubImportGate(by_marker={"body": disposition}))
assert result.merged == ()
assert [entry.concept_path for entry in result.rejected] == ["a.md"]
assert list(bundle.glob("*.md")) == []
def test_an_error_with_a_non_blocking_disposition_still_fails_closed(tmp_path: Path) -> None:
"""The guard pairs a hard reject with FAIL_SECURE, but the library must not
depend on that pairing: an error is a refusal on its own terms.
"""
place(tmp_path / "source", "a.md", "body\n")
gate = StubImportGate(errors={"a.md": "unsafe resource"}, by_marker={"body": WARN})
result, bundle = run(tmp_path, gate)
assert result.merged == ()
assert [entry.concept_path for entry in result.rejected] == ["a.md"]
assert list(bundle.glob("*.md")) == []
def test_a_concept_the_gate_returned_no_verdict_for_is_not_merged(tmp_path: Path) -> None:
"""No verdict is not consent. A gate that drops a concept from its result —
an adapter bug, or a guard that stops reporting one — must not be read as
approval.
"""
place(tmp_path / "source", "a.md", "body\n")
place(tmp_path / "source", "b.md", "body\n")
result, bundle = run(tmp_path, StubImportGate(omit=frozenset({"a.md"})))
assert [entry.concept_path for entry in result.merged] == ["b.md"]
assert [entry.concept_path for entry in result.rejected] == ["a.md"]
assert not (bundle / "import-a.md").exists()
def test_a_rejected_concept_leaves_the_bundle_byte_identical(tmp_path: Path) -> None:
"""The persist-gate proof: a rejected concept produces ZERO new files and
does not touch the ones already there — not even the index.
"""
place(tmp_path / "source", "ok.md", "body\n")
result, bundle = run(tmp_path, StubImportGate())
assert result.merged != ()
before = sorted((path.name, path.read_bytes()) for path in bundle.iterdir())
(tmp_path / "source" / "ok.md").unlink()
place(tmp_path / "source", "poisoned.md", "body\n")
run(tmp_path, StubImportGate(errors={"poisoned.md": "injection"}))
after = sorted((path.name, path.read_bytes()) for path in bundle.iterdir())
assert after == before
def test_a_mixed_bundle_merges_only_the_accepted_concepts(tmp_path: Path) -> None:
"""Partial merge is the point of a per-concept gate: one poisoned concept
must not cost the operator the rest of the bundle, and must not sneak in.
"""
place(tmp_path / "source", "good.md", "harmless\n")
place(tmp_path / "source", "held.md", "hold me\n")
place(tmp_path / "source", "bad.md", "poison\n")
result, bundle = run(
tmp_path,
StubImportGate(by_marker={"poison": FAIL_SECURE, "hold me": QUARANTINE_REVIEW}),
)
assert [entry.concept_path for entry in result.merged] == ["good.md"]
assert [entry.concept_path for entry in result.quarantined] == ["held.md"]
assert [entry.concept_path for entry in result.rejected] == ["bad.md"]
assert sorted(path.name for path in bundle.glob("*.md")) == ["import-good.md", "index.md"]
assert (bundle / "index.md").read_text(encoding="utf-8") == "- [good](import-good.md)\n"
# --- ownership by content identity ----------------------------------------
def test_a_re_import_of_an_unchanged_bundle_is_a_no_op(tmp_path: Path) -> None:
"""Identical bytes at the target name are proof enough that the file is
ours: the re-import is reported as merged, writes nothing new, and does not
duplicate the index link.
"""
place(tmp_path / "source", "tables/users.md", CONCEPT)
result, bundle = run(tmp_path, StubImportGate())
first = sorted((path.name, path.read_bytes()) for path in bundle.iterdir())
result, bundle = run(tmp_path, StubImportGate())
second = sorted((path.name, path.read_bytes()) for path in bundle.iterdir())
assert [entry.concept_path for entry in result.merged] == ["tables/users.md"]
assert result.failed == ()
assert second == first
def test_a_changed_concept_is_refused_without_overwriting(tmp_path: Path) -> None:
"""Without a stamp the library cannot prove the existing file is a previous
import rather than an edit the operator made — so a changed concept is
refused, not merged. The operator resolves it by removing the old file.
"""
place(tmp_path / "source", "note.md", "first version\n")
run(tmp_path, StubImportGate())
place(tmp_path / "source", "note.md", "second version\n")
result, bundle = run(tmp_path, StubImportGate())
assert result.merged == ()
assert [entry.concept_path for entry in result.failed] == ["note.md"]
assert result.failed[0].error.code == "collision_unstamped"
assert (bundle / "import-note.md").read_text(encoding="utf-8") == "first version\n"
def test_curated_content_at_the_target_name_is_never_overwritten(tmp_path: Path) -> None:
bundle = tmp_path / "bundle"
bundle.mkdir()
curated = bundle / "import-note.md"
curated.write_text("curated by hand\n", encoding="utf-8")
place(tmp_path / "source", "note.md", "external\n")
place(tmp_path / "source", "other.md", "external\n")
result, _ = run(tmp_path, StubImportGate())
assert curated.read_text(encoding="utf-8") == "curated by hand\n"
assert [entry.error.code for entry in result.failed] == ["collision_unstamped"]
assert [entry.concept_path for entry in result.merged] == ["other.md"]
def test_the_collision_gate_is_evaluated_before_any_write(tmp_path: Path) -> None:
"""Pre-mutation, exactly as at Door A and Door B: occupancy is judged
against the bundle as it was BEFORE this run.
"""
bundle = tmp_path / "bundle"
bundle.mkdir()
(bundle / "import-zzz.md").write_text("curated\n", encoding="utf-8")
place(tmp_path / "source", "aaa.md", "body\n")
place(tmp_path / "source", "zzz.md", "body\n")
result, _ = run(tmp_path, StubImportGate())
assert [entry.error.code for entry in result.failed] == ["collision_unstamped"]
assert [entry.concept_path for entry in result.merged] == ["aaa.md"]
assert (bundle / "import-zzz.md").read_text(encoding="utf-8") == "curated\n"
def test_two_concept_paths_that_slug_alike_are_both_refused(tmp_path: Path) -> None:
"""`a/b.md` and `a-b.md` both reduce to `import-a-b.md`. The library will
not pick a winner: both are refused and the sender renames one.
"""
place(tmp_path / "source", "a/b.md", "one\n")
place(tmp_path / "source", "a-b.md", "two\n")
place(tmp_path / "source", "other.md", "three\n")
result, bundle = run(tmp_path, StubImportGate())
assert [entry.concept_path for entry in result.failed] == ["a-b.md", "a/b.md"]
assert {entry.error.code for entry in result.failed} == {"import_slug_collision"}
assert not (bundle / "import-a-b.md").exists()
assert [entry.concept_path for entry in result.merged] == ["other.md"]
# --- one bad concept never aborts the run ---------------------------------
def test_a_concept_path_that_reduces_to_an_empty_slug_is_a_per_concept_failure(
tmp_path: Path,
) -> None:
place(tmp_path / "source", "!!!.md", "body\n")
place(tmp_path / "source", "good.md", "body\n")
result, _ = run(tmp_path, StubImportGate())
assert [entry.concept_path for entry in result.merged] == ["good.md"]
assert [entry.error.code for entry in result.failed] == ["import_path_empty"]
def test_an_over_long_concept_path_is_a_per_concept_failure(tmp_path: Path) -> None:
"""A bundle-relative path has no 255-byte limit of its own — only its
individual segments do — so a deep enough tree reduces to a filename the
filesystem cannot hold. Refused, never truncated.
"""
deep = "/".join(["segment"] * 40) + ".md"
place(tmp_path / "source", deep, "body\n")
place(tmp_path / "source", "good.md", "body\n")
result, _ = run(tmp_path, StubImportGate())
assert [entry.concept_path for entry in result.merged] == ["good.md"]
assert [entry.error.code for entry in result.failed] == ["import_path_too_long"]
def test_a_concept_path_that_would_break_an_index_link_is_a_per_concept_failure(
tmp_path: Path,
) -> None:
"""The guard's path gate permits brackets; this library's index link does
not. The concept-ID is rendered verbatim as the link label, so it meets the
same fail-fast rule Door A and Door B apply to a title.
"""
place(tmp_path / "source", "report [v2].md", "body\n")
result, _ = run(tmp_path, StubImportGate())
assert result.merged == ()
assert [entry.error.code for entry in result.failed] == ["import_label_invalid"]
def test_a_non_utf8_concept_is_a_per_concept_failure(tmp_path: Path) -> None:
"""Read failures happen before the gate — an undecodable concept is never
handed to it, and never counted as approved.
"""
source = tmp_path / "source"
source.mkdir()
(source / "broken.md").write_bytes(b"\xffnot utf-8\n")
place(source, "good.md", "body\n")
result, _ = run(tmp_path, gate := StubImportGate())
assert [entry.concept_path for entry in result.merged] == ["good.md"]
assert [entry.error.code for entry in result.failed] == ["extractor_decode_error"]
assert "broken.md" not in gate.calls[0]
def test_an_uppercase_md_suffix_is_still_a_concept(tmp_path: Path) -> None:
"""Determinism across filesystems: a `*.md` glob is case-sensitive on ext4
and case-insensitive on APFS, so the same bundle would import differently
per platform. The suffix test is explicit and case-folded, as the guard's
own path gate is.
"""
place(tmp_path / "source", "NOTE.MD", "body\n")
result, bundle = run(tmp_path, gate := StubImportGate())
assert list(gate.calls[0]) == ["NOTE.MD"]
assert [entry.concept_path for entry in result.merged] == ["NOTE.MD"]
assert (bundle / "import-note.md").read_text(encoding="utf-8") == "body\n"
def test_non_markdown_files_are_not_concepts(tmp_path: Path) -> None:
"""An OKF concept is a `.md` document by definition (the guard's path gate
says so too), so anything else in the source tree is not this door's to
merge — it is neither gated nor written.
"""
place(tmp_path / "source", "a.md", "body\n")
place(tmp_path / "source", "data.csv", "a,b\n1,2\n")
result, bundle = run(tmp_path, gate := StubImportGate())
assert list(gate.calls[0]) == ["a.md"]
assert [entry.concept_path for entry in result.merged] == ["a.md"]
assert sorted(path.name for path in bundle.iterdir()) == ["import-a.md", "index.md"]
# --- the import report ----------------------------------------------------
def test_the_guards_log_is_returned_and_not_persisted(tmp_path: Path) -> None:
"""Deliverable 3: the caller gets the log body; writing it into the bundle
is a reserved-file decision that differs per consumer and lands in Phase 3.
"""
place(tmp_path / "source", "a.md", "body\n")
result, bundle = run(tmp_path, StubImportGate(log="a\texternal\tautomatic\tuntrusted\twarn"))
assert result.log == "a\texternal\tautomatic\tuntrusted\twarn"
assert not (bundle / "log.md").exists()
def test_the_result_carries_the_run_timestamp(tmp_path: Path) -> None:
"""`ingested_at` reaches no file at this door — a merged concept is written
verbatim — but the run is still timestamped explicitly rather than by wall
clock, so a caller persisting the log has a deterministic timestamp to use.
"""
place(tmp_path / "source", "a.md", "body\n")
result, _ = run(tmp_path, StubImportGate())
assert result.ingested_at == INGESTED_AT
def test_the_per_concept_reasons_are_carried_verbatim(tmp_path: Path) -> None:
place(tmp_path / "source", "a.md", "poison\n")
result, _ = run(tmp_path, StubImportGate(by_marker={"poison": FAIL_SECURE}))
assert result.rejected[0].reasons == ("scanned a.md",)
# --- run-level fail-fast --------------------------------------------------
def test_an_invalid_ingested_at_fails_the_whole_run(tmp_path: Path) -> None:
from llm_ingestion_okf import MaterializationError
place(tmp_path / "source", "a.md", "body\n")
with pytest.raises(MaterializationError) as excinfo:
run(tmp_path, StubImportGate(), ingested_at="2026-07-25")
assert excinfo.value.code == "ingested_at_invalid"
@pytest.mark.parametrize(
("origin", "channel"),
[("extenal", AUTOMATIC), ("", AUTOMATIC), (EXTERNAL, "auto"), (EXTERNAL, "")],
)
def test_an_unknown_origin_or_channel_fails_the_whole_run(
tmp_path: Path, origin: str, channel: str
) -> None:
"""A value outside the guard's pinned vocabulary would reach the guard as a
plain string, miss its enum identity check, and be silently downgraded to
untrusted. Fail-fast instead: the library refuses to carry a provenance
declaration it cannot recognise, rather than letting a typo decide trust.
"""
from llm_ingestion_okf import MaterializationError
place(tmp_path / "source", "a.md", "body\n")
with pytest.raises(MaterializationError) as excinfo:
run(tmp_path, StubImportGate(), origin=origin, channel=channel)
assert excinfo.value.code == "import_provenance_invalid"
def test_a_missing_source_directory_is_refused(tmp_path: Path) -> None:
from llm_ingestion_okf import SourceError
with pytest.raises(SourceError) as excinfo:
import_bundle(
tmp_path / "nope",
tmp_path / "bundle",
INGESTED_AT,
origin=EXTERNAL,
channel=AUTOMATIC,
gate=StubImportGate(),
)
assert excinfo.value.code == "source_root_missing"
def test_an_empty_source_bundle_writes_nothing_and_never_calls_the_gate(tmp_path: Path) -> None:
(tmp_path / "source").mkdir()
result, bundle = run(tmp_path, gate := StubImportGate())
assert result == ImportResult(
merged=(), quarantined=(), rejected=(), failed=(), log="", ingested_at=INGESTED_AT
)
assert gate.calls == []
assert not bundle.exists() or list(bundle.iterdir()) == []