feat(import): Door C flow against an injected import gate (Phase 2 step 5)
Reads an external OKF bundle as {bundle-relative path -> document text},
hands it WHOLE to an injected gate over the guard's okf.import_bundle (a
bundle-level call: it resolves the cross-link graph across concepts), and
merges only concepts clearing the non-blocking floor. Same injection pattern
as Door B, so the core stays dependency-free while the CI channel for the
real guard is settled.
Two constraints shaped the design and are pinned by tests:
- A merged concept is written VERBATIM. Stamping provenance into it would
require round-tripping its frontmatter through this library's line-oriented
parser, which cannot represent the block lists the guard's parser accepts --
silent data loss -- and would persist bytes the guard never screened.
- Ownership is therefore proven by content identity: identical bytes at the
target name are a no-op re-merge (re-import of an unchanged bundle is
idempotent), and anything else at the name is refused. Curated content and
an updated concept are refused alike; refusing is what never destroys.
The floor is fail-closed beyond the plan's "no error" wording: an error, an
unrecognised disposition, and a concept the gate returned no verdict for are
all refusals. quarantine_review stays its own bucket, as at Door B.
origin/channel are validated against the guard's pinned vocabulary -- it
derives trust from origin by enum identity, so an unrecognised string would be
silently downgraded rather than caught.
Three primitives promoted for reuse rather than duplicated:
reduce_to_id_grammar and check_filename_length to materialize.py, and
extract.decode_text. Door C slugs the WHOLE concept path, so tables/users.md
and views/users.md stay distinct. Concept discovery folds case explicitly
rather than globbing *.md, whose case-sensitivity follows the filesystem and
would import the same bundle differently on APFS and ext4.
README's "what is gated today" section corrected: it claimed nothing is gated,
which is no longer true, but the honest statement is narrower than "the doors
are gated" -- the library cannot verify that an injected adapter is a real
guard, and a permissive stub is believed.
405 tests green; ruff, ruff format and mypy --strict clean.
This commit is contained in:
parent
d812a839be
commit
f10fc60de2
11 changed files with 1250 additions and 61 deletions
10
CLAUDE.md
10
CLAUDE.md
|
|
@ -19,8 +19,14 @@ one boundary rule:
|
||||||
the optional `[extract]` extra; without it those types are rejected
|
the optional `[extract]` extra; without it those types are rejected
|
||||||
fail-fast.
|
fail-fast.
|
||||||
- **Door C — external bundle import:** third-party OKF bundles are assessed
|
- **Door C — external bundle import:** third-party OKF bundles are assessed
|
||||||
per concept via the guard's `okf.import_bundle`; only non-rejected concepts
|
per concept via the guard's `okf.import_bundle`; only concepts clearing the
|
||||||
are merged/materialized/indexed here.
|
guard's non-blocking floor are merged/indexed here. Two invariants, both
|
||||||
|
load-bearing: a merged concept is written **verbatim** (this library's
|
||||||
|
line-oriented frontmatter parser cannot round-trip the block lists the
|
||||||
|
guard's parser accepts, so stamping an external concept would destroy sender
|
||||||
|
data and persist bytes the guard never screened), and ownership is therefore
|
||||||
|
proven by **content identity** — an occupied target name is re-used only
|
||||||
|
when the bytes there are already identical, never overwritten otherwise.
|
||||||
|
|
||||||
**Boundary rule (non-negotiable, zero overlap):** `llm-ingestion-guard`
|
**Boundary rule (non-negotiable, zero overlap):** `llm-ingestion-guard`
|
||||||
(pinned `>=0.2,<0.3`) answers "is this content safe to persist?" —
|
(pinned `>=0.2,<0.3`) answers "is this content safe to persist?" —
|
||||||
|
|
|
||||||
32
README.md
32
README.md
|
|
@ -4,7 +4,10 @@ Shared ingestion library for OKF (Open Knowledge Format) bundles.
|
||||||
|
|
||||||
Status: phase 1 (spec-based ingestion) is implemented — manifest validation,
|
Status: phase 1 (spec-based ingestion) is implemented — manifest validation,
|
||||||
the `file`/`sql`/`http` connectors, deterministic materialization, index
|
the `file`/`sql`/`http` connectors, deterministic materialization, index
|
||||||
generation, and the golden fixture suite under `examples/`. Phases 2–4 are
|
generation, and the golden fixture suite under `examples/`. Phase 2 is in
|
||||||
|
progress: the bundle inbox (`process_inbox`) and external-bundle import
|
||||||
|
(`import_bundle`) are implemented against an **injected** persist gate; the
|
||||||
|
integration with the real guard is not done yet (see below). Phases 3–4 are
|
||||||
planned (see `docs/plan/`).
|
planned (see `docs/plan/`).
|
||||||
|
|
||||||
## Planned scope (v1)
|
## Planned scope (v1)
|
||||||
|
|
@ -40,18 +43,27 @@ Security is owned by the sibling package
|
||||||
|
|
||||||
No security functionality is reimplemented here.
|
No security functionality is reimplemented here.
|
||||||
|
|
||||||
### What is gated today: nothing
|
### What is gated today: read this before trusting a door
|
||||||
|
|
||||||
Door A — the only door shipped so far — is **ungated**. The package has zero
|
The package still has **zero runtime dependencies** and imports no guard
|
||||||
runtime dependencies and calls no guard function before writing to disk.
|
function anywhere. That has a direct consequence for what "gated" means here:
|
||||||
`materialize_bundle` writes what it is given.
|
|
||||||
|
|
||||||
If you materialize external or otherwise untrusted content, gating is **your**
|
- **Door A (`materialize_bundle`) is ungated.** It calls nothing before
|
||||||
responsibility at the call site: `okf.import_bundle` for received bundles, or
|
writing to disk and writes what it is given.
|
||||||
`prepare_input`/`screen_output` around extracted text. The guard-calling
|
- **Doors B and C (`process_inbox`, `import_bundle`) gate through an adapter
|
||||||
persist gates are part of Doors B and C (phase 2), not of Door A.
|
you supply.** Each takes a `gate` argument; the flow hands it the content
|
||||||
|
and obeys the verdict, refusing to persist anything that does not clear the
|
||||||
|
guard's non-blocking floor — including a disposition it does not recognise,
|
||||||
|
and (at Door C) a concept the gate returned no verdict for. What it cannot
|
||||||
|
do is check that your adapter is a real guard. A permissive stub approves
|
||||||
|
everything, and the flow will believe it.
|
||||||
|
|
||||||
This is stated plainly because the earlier wording ("calls the guard at every
|
So gating remains **your** responsibility at the call site: pass an adapter
|
||||||
|
over `prepare_input`/`screen_output` (Door B) or `okf.import_bundle`
|
||||||
|
(Door C). Wiring the doors to the real guard, and pinning it as a dependency,
|
||||||
|
is step 4 of phase 2 and is not done yet.
|
||||||
|
|
||||||
|
This is stated plainly because earlier wording ("calls the guard at every
|
||||||
persist gate") described the intended end state in the present tense, and a
|
persist gate") described the intended end state in the present tense, and a
|
||||||
consumer reasonably read it as safe-by-default.
|
consumer reasonably read it as safe-by-default.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -65,6 +65,40 @@ No scanning, sanitizing, or quarantine logic is implemented here.
|
||||||
(reserved-file policy differs per consumer and becomes configurable in
|
(reserved-file policy differs per consumer and becomes configurable in
|
||||||
Phase 3).
|
Phase 3).
|
||||||
|
|
||||||
|
### Settled during implementation (step 5)
|
||||||
|
|
||||||
|
- **Verbatim merge, no frontmatter of ours.** A merged concept is written
|
||||||
|
exactly as the gate saw it. This is a correctness constraint, not a
|
||||||
|
preference: the guard's frontmatter parser accepts block lists, which this
|
||||||
|
library's line-oriented `parse_frontmatter` cannot round-trip, so stamping
|
||||||
|
provenance into an external concept would silently drop sender data — and
|
||||||
|
would persist bytes the guard never screened.
|
||||||
|
- **Ownership by content identity.** With no stamp available, an occupied
|
||||||
|
target name is re-used only when the bytes already there are identical (a
|
||||||
|
no-op re-merge, so re-import of an unchanged bundle is idempotent).
|
||||||
|
Anything else at the name — curated content or an updated version of the
|
||||||
|
same concept — is refused under `collision_unstamped`; the operator removes
|
||||||
|
the file to accept an update. A configurable stamping/reserved-file policy
|
||||||
|
is Phase 3's.
|
||||||
|
- **The merge floor is fail-closed, not "no error".** Deliverable 1's wording
|
||||||
|
is the looser reading: a concept carrying no error but a FAIL_SECURE or
|
||||||
|
unrecognised disposition is refused, and a concept the gate returned no
|
||||||
|
verdict for at all is refused. Only the guard's non-blocking floor merges —
|
||||||
|
the same floor Door B applies, with `quarantine_review` reported as its own
|
||||||
|
bucket rather than folded into rejection.
|
||||||
|
- **Upstream has already moved past the pin (observed, not acted on).** The
|
||||||
|
guard's `main` carries a commit that adds `allow_reserved=True` to
|
||||||
|
`okf.import_bundle` and *scans* `index.md`/`log.md` in a mode-b import
|
||||||
|
instead of path-rejecting them. The pin is the `v0.2.0` tag, where the
|
||||||
|
kwarg does not exist and rejection is unconditional, so this door is built
|
||||||
|
against the tag. Whoever bumps the pin owns re-checking that branch: today
|
||||||
|
a reserved name in a received bundle arrives as a per-concept rejection.
|
||||||
|
- **`origin`/`channel` are validated against the guard's pinned vocabulary.**
|
||||||
|
The guard derives trust from `origin` by enum *identity*, so an unrecognised
|
||||||
|
string would be silently downgraded to untrusted. The library refuses to
|
||||||
|
carry a provenance declaration it cannot recognise rather than let a typo
|
||||||
|
decide trust — refusing to guess, not deciding trust itself.
|
||||||
|
|
||||||
## Design decisions fixed by this plan
|
## Design decisions fixed by this plan
|
||||||
|
|
||||||
- Trust follows `origin`, never `channel` (guard doctrine) — callers must supply
|
- Trust follows `origin`, never `channel` (guard doctrine) — callers must supply
|
||||||
|
|
|
||||||
|
|
@ -20,6 +20,14 @@ Door B (bundle inbox) public surface: process_inbox plus its result types.
|
||||||
Its persist gate is INJECTED -- the caller supplies a Gate adapter over
|
Its persist gate is INJECTED -- the caller supplies a Gate adapter over
|
||||||
llm-ingestion-guard and process_inbox obeys the verdict; the library imports
|
llm-ingestion-guard and process_inbox obeys the verdict; the library imports
|
||||||
no guard function itself and makes no security decision of its own.
|
no guard function itself and makes no security decision of its own.
|
||||||
|
|
||||||
|
Door C (external bundle import) public surface: import_bundle plus its result
|
||||||
|
types. Its gate is injected the same way, over the guard's okf.import_bundle:
|
||||||
|
the caller declares origin and channel at the door, the gate assesses every
|
||||||
|
concept, and only concepts clearing the non-blocking floor are merged. Merged
|
||||||
|
concepts are written verbatim -- ownership is proven by content identity
|
||||||
|
rather than by a stamp, so an occupied name is only ever re-used when the
|
||||||
|
bytes already there are identical.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from .errors import (
|
from .errors import (
|
||||||
|
|
@ -41,6 +49,16 @@ from .inbox import (
|
||||||
PersistedFile,
|
PersistedFile,
|
||||||
process_inbox,
|
process_inbox,
|
||||||
)
|
)
|
||||||
|
from .importer import (
|
||||||
|
BundleDecision,
|
||||||
|
FailedConcept,
|
||||||
|
ImportDecision,
|
||||||
|
ImportGate,
|
||||||
|
ImportResult,
|
||||||
|
MergedConcept,
|
||||||
|
RefusedConcept,
|
||||||
|
import_bundle,
|
||||||
|
)
|
||||||
from .manifest import (
|
from .manifest import (
|
||||||
Extraction,
|
Extraction,
|
||||||
FileSource,
|
FileSource,
|
||||||
|
|
@ -55,25 +73,33 @@ __version__ = "0.3.2"
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"BlockedFile",
|
"BlockedFile",
|
||||||
|
"BundleDecision",
|
||||||
"Extraction",
|
"Extraction",
|
||||||
"ExtractionError",
|
"ExtractionError",
|
||||||
|
"FailedConcept",
|
||||||
"FailedFile",
|
"FailedFile",
|
||||||
"FileSource",
|
"FileSource",
|
||||||
"Gate",
|
"Gate",
|
||||||
"GateDecision",
|
"GateDecision",
|
||||||
"HttpSource",
|
"HttpSource",
|
||||||
|
"ImportDecision",
|
||||||
|
"ImportGate",
|
||||||
|
"ImportResult",
|
||||||
"InboxResult",
|
"InboxResult",
|
||||||
"IngestError",
|
"IngestError",
|
||||||
"IngestResult",
|
"IngestResult",
|
||||||
"Manifest",
|
"Manifest",
|
||||||
"ManifestError",
|
"ManifestError",
|
||||||
"MaterializationError",
|
"MaterializationError",
|
||||||
|
"MergedConcept",
|
||||||
"NetworkGateError",
|
"NetworkGateError",
|
||||||
"PersistedFile",
|
"PersistedFile",
|
||||||
|
"RefusedConcept",
|
||||||
"RenderError",
|
"RenderError",
|
||||||
"SourceError",
|
"SourceError",
|
||||||
"SqlSource",
|
"SqlSource",
|
||||||
"extract_text",
|
"extract_text",
|
||||||
|
"import_bundle",
|
||||||
"load_manifest",
|
"load_manifest",
|
||||||
"materialize_bundle",
|
"materialize_bundle",
|
||||||
"process_inbox",
|
"process_inbox",
|
||||||
|
|
|
||||||
|
|
@ -110,6 +110,19 @@ class MaterializationError(IngestError):
|
||||||
would inject frontmatter lines
|
would inject frontmatter lines
|
||||||
- `okf_type_reserved` — an inbox concept claims the reserved 'verdict'
|
- `okf_type_reserved` — an inbox concept claims the reserved 'verdict'
|
||||||
layer (the same reservation ManifestError enforces at Door A)
|
layer (the same reservation ManifestError enforces at Door A)
|
||||||
|
- `import_path_empty` — an external concept path reduces to an empty slug
|
||||||
|
under the id grammar (Door C; never an invented fallback name)
|
||||||
|
- `import_path_too_long` — the generated import filename would exceed the
|
||||||
|
255-byte filesystem limit (Door C; never a truncated name)
|
||||||
|
- `import_slug_collision` — two concepts in one external bundle reduce to
|
||||||
|
one generated filename (Door C); both are refused rather than letting
|
||||||
|
iteration order decide which one survives
|
||||||
|
- `import_label_invalid` — an external concept path contains `[`/`]`, which
|
||||||
|
would break its index link (the guard's path gate permits them)
|
||||||
|
- `import_provenance_invalid` — an `origin`/`channel` outside the guard's
|
||||||
|
pinned vocabulary (Door C); refused rather than carried, because the
|
||||||
|
guard derives trust from `origin` by enum identity and an unrecognised
|
||||||
|
value would be silently downgraded
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -33,7 +33,7 @@ _OPTIONAL_EXTENSIONS = frozenset({".pdf", ".docx", ".xlsx"})
|
||||||
_SKIP_TAGS = frozenset({"script", "style"})
|
_SKIP_TAGS = frozenset({"script", "style"})
|
||||||
|
|
||||||
|
|
||||||
def _decode(data: bytes) -> str:
|
def decode_text(data: bytes) -> str:
|
||||||
"""Decode file bytes as UTF-8 (BOM-stripping), typed on failure.
|
"""Decode file bytes as UTF-8 (BOM-stripping), typed on failure.
|
||||||
|
|
||||||
utf-8-sig so a byte-order mark never leaks into the first character
|
utf-8-sig so a byte-order mark never leaks into the first character
|
||||||
|
|
@ -50,12 +50,12 @@ def _decode(data: bytes) -> str:
|
||||||
|
|
||||||
def _extract_passthrough(data: bytes) -> str:
|
def _extract_passthrough(data: bytes) -> str:
|
||||||
"""`md`/`txt`: the decoded text verbatim."""
|
"""`md`/`txt`: the decoded text verbatim."""
|
||||||
return _decode(data)
|
return decode_text(data)
|
||||||
|
|
||||||
|
|
||||||
def _extract_csv(data: bytes) -> str:
|
def _extract_csv(data: bytes) -> str:
|
||||||
"""`csv`: parse with the stdlib reader, render the Phase 1 markdown table."""
|
"""`csv`: parse with the stdlib reader, render the Phase 1 markdown table."""
|
||||||
reader = csv.reader(io.StringIO(_decode(data)))
|
reader = csv.reader(io.StringIO(decode_text(data)))
|
||||||
header = next(reader, None)
|
header = next(reader, None)
|
||||||
if header is None:
|
if header is None:
|
||||||
raise ExtractionError("CSV has no header row", code="extractor_empty_csv")
|
raise ExtractionError("CSV has no header row", code="extractor_empty_csv")
|
||||||
|
|
@ -65,7 +65,7 @@ def _extract_csv(data: bytes) -> str:
|
||||||
|
|
||||||
def _extract_json(data: bytes) -> str:
|
def _extract_json(data: bytes) -> str:
|
||||||
"""`json`: the decoded text verbatim inside a fenced block (Phase 1 renderer)."""
|
"""`json`: the decoded text verbatim inside a fenced block (Phase 1 renderer)."""
|
||||||
return render_fenced_block(_decode(data))
|
return render_fenced_block(decode_text(data))
|
||||||
|
|
||||||
|
|
||||||
class _HTMLTextExtractor(HTMLParser):
|
class _HTMLTextExtractor(HTMLParser):
|
||||||
|
|
@ -105,7 +105,7 @@ class _HTMLTextExtractor(HTMLParser):
|
||||||
def _extract_html(data: bytes) -> str:
|
def _extract_html(data: bytes) -> str:
|
||||||
"""`html`/`htm`: text via `html.parser`, script/style stripped (spec B3)."""
|
"""`html`/`htm`: text via `html.parser`, script/style stripped (spec B3)."""
|
||||||
parser = _HTMLTextExtractor()
|
parser = _HTMLTextExtractor()
|
||||||
parser.feed(_decode(data))
|
parser.feed(decode_text(data))
|
||||||
parser.close()
|
parser.close()
|
||||||
return parser.text()
|
return parser.text()
|
||||||
|
|
||||||
|
|
|
||||||
425
src/llm_ingestion_okf/importer.py
Normal file
425
src/llm_ingestion_okf/importer.py
Normal file
|
|
@ -0,0 +1,425 @@
|
||||||
|
"""Door C: external bundle import — read, gate per concept, merge the accepted.
|
||||||
|
|
||||||
|
A third-party OKF bundle is read as `{bundle-relative path -> document text}`,
|
||||||
|
handed WHOLE to the guard's `okf.import_bundle` (a bundle-level call: it
|
||||||
|
resolves the cross-link graph across concepts, so gating them one at a time
|
||||||
|
would throw that half of the gate away), and only concepts whose verdict clears
|
||||||
|
the non-blocking floor are merged.
|
||||||
|
|
||||||
|
A merged concept is written VERBATIM — exactly the text the gate saw. Nothing
|
||||||
|
of this library's is merged into its frontmatter, and that is a correctness
|
||||||
|
constraint, not a preference: the guard's frontmatter parser accepts block
|
||||||
|
lists, which this library's line-oriented :func:`parse_frontmatter` cannot
|
||||||
|
round-trip, so re-rendering a concept would silently drop data the sender
|
||||||
|
supplied. It would also persist bytes the guard never screened.
|
||||||
|
|
||||||
|
That leaves ownership to be proven by content identity instead of by a stamp:
|
||||||
|
a target name held by byte-identical content is a no-op re-merge (so re-import
|
||||||
|
of an unchanged bundle is idempotent), and a target name held by anything else
|
||||||
|
is refused. Curated content and an UPDATED external concept are refused alike —
|
||||||
|
the library cannot tell them apart without a marker it has no safe place to
|
||||||
|
write, and refusing is the answer that never destroys. A configurable
|
||||||
|
reserved-file/frontmatter policy is Phase 3's; this is the v1 floor.
|
||||||
|
|
||||||
|
No security decision is taken here: the gate is injected, and this module only
|
||||||
|
obeys the verdict it returns.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Protocol
|
||||||
|
|
||||||
|
from .errors import IngestError, MaterializationError, SourceError
|
||||||
|
from .extract import decode_text
|
||||||
|
from .materialize import (
|
||||||
|
INDEX_NAME,
|
||||||
|
check_filename_length,
|
||||||
|
link_in_index,
|
||||||
|
reduce_to_id_grammar,
|
||||||
|
validate_ingested_at,
|
||||||
|
write_bytes,
|
||||||
|
)
|
||||||
|
|
||||||
|
# An OKF concept is a `.md` document by definition — the guard's path gate
|
||||||
|
# rejects anything else outright — so nothing else in the source tree is a
|
||||||
|
# concept, and nothing else is this door's to merge.
|
||||||
|
_CONCEPT_SUFFIX = ".md"
|
||||||
|
|
||||||
|
_FILENAME_PREFIX = "import-"
|
||||||
|
|
||||||
|
# The guard's non-blocking floor and its review queue, by VALUE (`Disposition`
|
||||||
|
# is a `str, Enum`, so the value is the stable thing to compare against across
|
||||||
|
# the pinned `>=0.2,<0.3` range). Pinned as constants here rather than imported
|
||||||
|
# because the dependency is injected — deliberately restated independently of
|
||||||
|
# Door B's copy in `inbox.py`, so drift in either door is visible rather than
|
||||||
|
# silently shared. The step-4 adapter's signature smoke test is what catches a
|
||||||
|
# rename in the guard itself.
|
||||||
|
_DISPOSITION_MERGE = "warn"
|
||||||
|
_DISPOSITION_QUARANTINE = "quarantine_review"
|
||||||
|
|
||||||
|
# The guard's Origin/Channel vocabularies, likewise by value. Validated here
|
||||||
|
# because `trust_for` compares by enum IDENTITY: a value outside these sets
|
||||||
|
# would reach the guard as a plain string, miss the identity check, and be
|
||||||
|
# silently classified untrusted. That failure is safe but silent, and a
|
||||||
|
# provenance declaration the library cannot recognise is not one it should
|
||||||
|
# carry — refusing is not a trust decision, it is refusing to guess at one.
|
||||||
|
_ORIGINS = frozenset({"external", "internal"})
|
||||||
|
_CHANNELS = frozenset({"automatic", "manual"})
|
||||||
|
|
||||||
|
|
||||||
|
# --- the guard seam -------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ImportDecision:
|
||||||
|
"""One concept's verdict, carried verbatim from `okf.import_bundle`.
|
||||||
|
|
||||||
|
`disposition` is the guard's `Disposition` VALUE and `error` its
|
||||||
|
`ConceptResult.error` — non-`None` on a hard reject (bad path, unsafe
|
||||||
|
frontmatter, non-https `resource`), in which case the concept must not be
|
||||||
|
merged whatever the disposition says. `reasons` is the audit trail the
|
||||||
|
adapter flattened out of the concept's scan report.
|
||||||
|
"""
|
||||||
|
|
||||||
|
path: str
|
||||||
|
disposition: str
|
||||||
|
error: str | None = None
|
||||||
|
reasons: tuple[str, ...] = ()
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class BundleDecision:
|
||||||
|
"""The gate's verdict on a whole bundle: per-concept results plus the log.
|
||||||
|
|
||||||
|
`log` is the guard's `BundleResult.log()` body, returned to the caller and
|
||||||
|
never written here.
|
||||||
|
"""
|
||||||
|
|
||||||
|
concepts: tuple[ImportDecision, ...]
|
||||||
|
log: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
class ImportGate(Protocol):
|
||||||
|
"""The persist gate, injected. The library never imports the guard itself.
|
||||||
|
|
||||||
|
`origin` and `channel` are keyword-only by design: both are plain strings
|
||||||
|
at this seam, and a positional call site that transposed them would move a
|
||||||
|
concept between trust tiers without any type error to catch it.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __call__(self, bundle: dict[str, str], *, origin: str, channel: str) -> BundleDecision: ...
|
||||||
|
|
||||||
|
|
||||||
|
# --- per-concept outcomes -------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class MergedConcept:
|
||||||
|
"""An external concept that cleared the gate and is present in the bundle.
|
||||||
|
|
||||||
|
Also covers the no-op re-merge: identical bytes already at the target name
|
||||||
|
are the concept being present, not a second write.
|
||||||
|
"""
|
||||||
|
|
||||||
|
concept_path: str
|
||||||
|
path: Path
|
||||||
|
reasons: tuple[str, ...] = ()
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class RefusedConcept:
|
||||||
|
"""A concept the guard did not clear. Quarantine and rejection are reported
|
||||||
|
separately: quarantine means "hold for human review" and is the operator's
|
||||||
|
queue, while a fail-secure verdict is a decision, not a queue."""
|
||||||
|
|
||||||
|
concept_path: str
|
||||||
|
disposition: str
|
||||||
|
error: str | None
|
||||||
|
reasons: tuple[str, ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class FailedConcept:
|
||||||
|
"""A concept this library could not process — unreadable, unusable name, or
|
||||||
|
a collision. Always a typed error, never a leaked stdlib exception."""
|
||||||
|
|
||||||
|
concept_path: str
|
||||||
|
error: IngestError
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ImportResult:
|
||||||
|
"""Every concept's outcome, in sorted concept-path order.
|
||||||
|
|
||||||
|
Four disjoint buckets, and every concept lands in exactly one: a run's
|
||||||
|
report is complete by construction, so a concept that silently vanished
|
||||||
|
would show up as a missing entry rather than as nothing at all. `log` is
|
||||||
|
the guard's log body and `ingested_at` the run's explicit timestamp — the
|
||||||
|
caller persists them if their reserved-file policy says to.
|
||||||
|
"""
|
||||||
|
|
||||||
|
merged: tuple[MergedConcept, ...]
|
||||||
|
quarantined: tuple[RefusedConcept, ...]
|
||||||
|
rejected: tuple[RefusedConcept, ...]
|
||||||
|
failed: tuple[FailedConcept, ...]
|
||||||
|
log: str
|
||||||
|
ingested_at: str
|
||||||
|
|
||||||
|
|
||||||
|
def import_slug(concept_path: str) -> str:
|
||||||
|
"""Reduce a bundle-relative concept path to the Phase 1 id grammar.
|
||||||
|
|
||||||
|
The whole path reduces, not just its final segment: `tables/users.md` and
|
||||||
|
`views/users.md` are distinct concepts, and slugging the stem alone would
|
||||||
|
collapse them onto one filename. A path that reduces to nothing fails fast
|
||||||
|
rather than being given an invented name.
|
||||||
|
"""
|
||||||
|
concept_id = concept_path[: -len(_CONCEPT_SUFFIX)]
|
||||||
|
slug = reduce_to_id_grammar(concept_id)
|
||||||
|
if not slug:
|
||||||
|
raise MaterializationError(
|
||||||
|
f"concept path {concept_path!r} reduces to an empty slug under the "
|
||||||
|
"id grammar ([a-z0-9][a-z0-9-]*) — refusing to invent a filename",
|
||||||
|
code="import_path_empty",
|
||||||
|
)
|
||||||
|
return slug
|
||||||
|
|
||||||
|
|
||||||
|
def import_filename(slug: str) -> str:
|
||||||
|
"""The bundle filename for an imported concept.
|
||||||
|
|
||||||
|
The `import-` prefix keeps the namespace disjoint from `index.md`, Door A's
|
||||||
|
`ingest-*`, Door B's `inbox-*`, and `promoted-verdict-*` for every slug the
|
||||||
|
grammar admits.
|
||||||
|
"""
|
||||||
|
return check_filename_length(
|
||||||
|
f"{_FILENAME_PREFIX}{slug}{_CONCEPT_SUFFIX}", code="import_path_too_long"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _index_label(concept_path: str) -> str:
|
||||||
|
"""The concept-ID, validated as an index link label.
|
||||||
|
|
||||||
|
The guard's path gate permits brackets in a concept path; `- [label](target)`
|
||||||
|
does not. Fail-fast, never repair — the same rule Door A applies to a
|
||||||
|
manifest title and Door B to a dropped filename.
|
||||||
|
"""
|
||||||
|
label = concept_path[: -len(_CONCEPT_SUFFIX)]
|
||||||
|
if any(char in label for char in "\n\r[]"):
|
||||||
|
raise MaterializationError(
|
||||||
|
f"concept path {concept_path!r} contains '[' or ']', which would break "
|
||||||
|
"its index link — rename it at the sender",
|
||||||
|
code="import_label_invalid",
|
||||||
|
)
|
||||||
|
return label
|
||||||
|
|
||||||
|
|
||||||
|
def _read_bundle(source: Path) -> tuple[dict[str, str], list[FailedConcept]]:
|
||||||
|
"""Read every concept document under `source`, keyed by POSIX-relative path.
|
||||||
|
|
||||||
|
Unreadable and undecodable concepts never reach the gate — they are per-
|
||||||
|
concept failures, and a concept the gate never saw is never merged.
|
||||||
|
"""
|
||||||
|
documents: dict[str, str] = {}
|
||||||
|
failed: list[FailedConcept] = []
|
||||||
|
for path in sorted(source.rglob("*")):
|
||||||
|
# The suffix test is explicit and case-folded rather than a `*.md`
|
||||||
|
# glob: glob case-sensitivity follows the FILESYSTEM, so `NOTE.MD`
|
||||||
|
# would be a concept on APFS and not one on ext4 — the same bundle
|
||||||
|
# importing differently per platform. The guard folds case here too.
|
||||||
|
if not path.is_file() or path.suffix.lower() != _CONCEPT_SUFFIX:
|
||||||
|
continue
|
||||||
|
concept_path = path.relative_to(source).as_posix()
|
||||||
|
try:
|
||||||
|
documents[concept_path] = decode_text(path.read_bytes())
|
||||||
|
except OSError as exc:
|
||||||
|
failed.append(
|
||||||
|
FailedConcept(
|
||||||
|
concept_path=concept_path,
|
||||||
|
error=SourceError(
|
||||||
|
f"cannot read concept {concept_path}: {exc}", code="source_file_missing"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
except IngestError as exc:
|
||||||
|
failed.append(FailedConcept(concept_path=concept_path, error=exc))
|
||||||
|
return documents, failed
|
||||||
|
|
||||||
|
|
||||||
|
def import_bundle(
|
||||||
|
source_dir: Path,
|
||||||
|
bundle_dir: Path,
|
||||||
|
ingested_at: str,
|
||||||
|
*,
|
||||||
|
origin: str,
|
||||||
|
channel: str,
|
||||||
|
gate: ImportGate,
|
||||||
|
) -> ImportResult:
|
||||||
|
"""Merge the accepted concepts of an external OKF bundle (Door C).
|
||||||
|
|
||||||
|
An explicit operator command, never a watcher or a scheduler. `origin` and
|
||||||
|
`channel` are required with no defaults — trust follows origin, never
|
||||||
|
channel, and the caller is the one who knows both. `gate` is the guard
|
||||||
|
adapter over `okf.import_bundle`: every concept is assessed before anything
|
||||||
|
is written, and only the guard's non-blocking floor merges — anything else,
|
||||||
|
INCLUDING a disposition this library does not recognise and a concept the
|
||||||
|
gate returned no verdict for, fails closed.
|
||||||
|
|
||||||
|
One bad concept never aborts the run. Unreadable concepts, unusable names
|
||||||
|
and collisions are reported per concept in :class:`ImportResult` while the
|
||||||
|
rest still merge. Only three conditions fail the whole run, and all three
|
||||||
|
are wrong for every concept at once: an invalid `ingested_at`, an
|
||||||
|
unrecognised `origin`/`channel`, and a missing source directory.
|
||||||
|
"""
|
||||||
|
validate_ingested_at(ingested_at)
|
||||||
|
if origin not in _ORIGINS or channel not in _CHANNELS:
|
||||||
|
raise MaterializationError(
|
||||||
|
f"origin must be one of {sorted(_ORIGINS)} and channel one of "
|
||||||
|
f"{sorted(_CHANNELS)}, got origin={origin!r} channel={channel!r} — "
|
||||||
|
"refusing to carry a provenance declaration the guard would not "
|
||||||
|
"recognise (it decides trust from these values)",
|
||||||
|
code="import_provenance_invalid",
|
||||||
|
)
|
||||||
|
source = Path(source_dir)
|
||||||
|
if not source.is_dir():
|
||||||
|
raise SourceError(
|
||||||
|
f"source bundle directory does not exist: {source}", code="source_root_missing"
|
||||||
|
)
|
||||||
|
|
||||||
|
documents, failed = _read_bundle(source)
|
||||||
|
|
||||||
|
merged: list[MergedConcept] = []
|
||||||
|
quarantined: list[RefusedConcept] = []
|
||||||
|
rejected: list[RefusedConcept] = []
|
||||||
|
log = ""
|
||||||
|
|
||||||
|
if documents:
|
||||||
|
decision = gate(dict(documents), origin=origin, channel=channel)
|
||||||
|
log = decision.log
|
||||||
|
by_path = {entry.path: entry for entry in decision.concepts}
|
||||||
|
|
||||||
|
accepted: list[tuple[str, str]] = []
|
||||||
|
for concept_path in sorted(documents):
|
||||||
|
verdict = by_path.get(concept_path)
|
||||||
|
if verdict is None:
|
||||||
|
# No verdict is not consent: a concept the gate dropped from
|
||||||
|
# its result is refused, never read as approval by omission.
|
||||||
|
rejected.append(
|
||||||
|
RefusedConcept(
|
||||||
|
concept_path=concept_path,
|
||||||
|
disposition="",
|
||||||
|
error="the gate returned no verdict for this concept",
|
||||||
|
reasons=(),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
# An error is a refusal on its own terms: the guard pairs one with
|
||||||
|
# FAIL_SECURE today, but the floor must not depend on that pairing.
|
||||||
|
if verdict.error is not None or verdict.disposition != _DISPOSITION_MERGE:
|
||||||
|
refused = RefusedConcept(
|
||||||
|
concept_path=concept_path,
|
||||||
|
disposition=verdict.disposition,
|
||||||
|
error=verdict.error,
|
||||||
|
reasons=verdict.reasons,
|
||||||
|
)
|
||||||
|
if verdict.error is None and verdict.disposition == _DISPOSITION_QUARANTINE:
|
||||||
|
quarantined.append(refused)
|
||||||
|
else:
|
||||||
|
rejected.append(refused)
|
||||||
|
continue
|
||||||
|
accepted.append((concept_path, documents[concept_path]))
|
||||||
|
|
||||||
|
# Name every accepted concept BEFORE any write, so an intra-run slug
|
||||||
|
# collision is caught while both concepts can still be refused together.
|
||||||
|
named: list[tuple[str, str, str]] = []
|
||||||
|
slug_owners: dict[str, list[str]] = {}
|
||||||
|
for concept_path, text in accepted:
|
||||||
|
try:
|
||||||
|
name = import_filename(import_slug(concept_path))
|
||||||
|
_index_label(concept_path)
|
||||||
|
except IngestError as exc:
|
||||||
|
failed.append(FailedConcept(concept_path=concept_path, error=exc))
|
||||||
|
continue
|
||||||
|
named.append((concept_path, name, text))
|
||||||
|
slug_owners.setdefault(name, []).append(concept_path)
|
||||||
|
|
||||||
|
colliding = {name for name, owners in slug_owners.items() if len(owners) > 1}
|
||||||
|
for name in sorted(colliding):
|
||||||
|
for concept_path in slug_owners[name]:
|
||||||
|
others = ", ".join(
|
||||||
|
repr(other) for other in slug_owners[name] if other != concept_path
|
||||||
|
)
|
||||||
|
failed.append(
|
||||||
|
FailedConcept(
|
||||||
|
concept_path=concept_path,
|
||||||
|
error=MaterializationError(
|
||||||
|
f"{concept_path!r} and {others} both reduce to {name!r} — "
|
||||||
|
"rename one at the sender; refusing to pick a winner",
|
||||||
|
code="import_slug_collision",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# The occupancy gate, evaluated against the bundle as it was BEFORE
|
||||||
|
# this run: a file written below must never be judged by a later
|
||||||
|
# concept's check. Byte-identity is the only ownership proof available
|
||||||
|
# at this door, so anything else at the name is curated content or an
|
||||||
|
# update — refused either way, never overwritten.
|
||||||
|
bundle = Path(bundle_dir)
|
||||||
|
# `None` marks a no-op re-merge — an explicit sentinel, because an
|
||||||
|
# empty concept document is legitimate and must still be written.
|
||||||
|
staged: list[tuple[str, str, str | None]] = []
|
||||||
|
for concept_path, name, text in named:
|
||||||
|
if name in colliding:
|
||||||
|
continue
|
||||||
|
content = text.encode("utf-8")
|
||||||
|
existing = bundle / name
|
||||||
|
if existing.is_file():
|
||||||
|
if existing.read_bytes() == content:
|
||||||
|
staged.append((concept_path, name, None))
|
||||||
|
continue
|
||||||
|
failed.append(
|
||||||
|
FailedConcept(
|
||||||
|
concept_path=concept_path,
|
||||||
|
error=MaterializationError(
|
||||||
|
f"generated filename {name!r} is occupied by different content — "
|
||||||
|
"it is either curated or an earlier version of this concept, and "
|
||||||
|
"without a stamp the two cannot be told apart; remove it to accept "
|
||||||
|
"the update (§3)",
|
||||||
|
code="collision_unstamped",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
staged.append((concept_path, name, text))
|
||||||
|
|
||||||
|
# Disk phase.
|
||||||
|
for concept_path, name, pending in staged:
|
||||||
|
if pending is None:
|
||||||
|
path = bundle / name
|
||||||
|
else:
|
||||||
|
bundle.mkdir(parents=True, exist_ok=True)
|
||||||
|
path = write_bytes(bundle, name, pending)
|
||||||
|
verdict = by_path[concept_path]
|
||||||
|
merged.append(
|
||||||
|
MergedConcept(concept_path=concept_path, path=path, reasons=verdict.reasons)
|
||||||
|
)
|
||||||
|
|
||||||
|
# §6 index — the last disk mutation, and only when something merged.
|
||||||
|
if merged:
|
||||||
|
index_path = bundle / INDEX_NAME
|
||||||
|
if not index_path.is_file():
|
||||||
|
write_bytes(bundle, INDEX_NAME, "")
|
||||||
|
for entry in merged:
|
||||||
|
link_in_index(bundle, entry.path.name, _index_label(entry.concept_path))
|
||||||
|
|
||||||
|
return ImportResult(
|
||||||
|
merged=tuple(merged),
|
||||||
|
quarantined=tuple(quarantined),
|
||||||
|
rejected=tuple(rejected),
|
||||||
|
failed=tuple(sorted(failed, key=lambda entry: entry.concept_path)),
|
||||||
|
log=log,
|
||||||
|
ingested_at=ingested_at,
|
||||||
|
)
|
||||||
|
|
@ -20,7 +20,6 @@ supplies the verdict and this module only obeys it.
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import hashlib
|
import hashlib
|
||||||
import re
|
|
||||||
import unicodedata
|
import unicodedata
|
||||||
from collections.abc import Callable
|
from collections.abc import Callable
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
|
|
@ -30,28 +29,16 @@ from .errors import IngestError, MaterializationError, SourceError
|
||||||
from .extract import extract_text
|
from .extract import extract_text
|
||||||
from .materialize import (
|
from .materialize import (
|
||||||
INDEX_NAME,
|
INDEX_NAME,
|
||||||
|
check_filename_length,
|
||||||
link_in_index,
|
link_in_index,
|
||||||
parse_frontmatter,
|
parse_frontmatter,
|
||||||
|
reduce_to_id_grammar,
|
||||||
validate_ingested_at,
|
validate_ingested_at,
|
||||||
write_bytes,
|
write_bytes,
|
||||||
)
|
)
|
||||||
|
|
||||||
_RESERVED_OKF_TYPE = "verdict"
|
_RESERVED_OKF_TYPE = "verdict"
|
||||||
|
|
||||||
# Every character outside the Phase 1 id grammar (`[a-z0-9][a-z0-9-]*`) is a
|
|
||||||
# separator. Deliberately NOT a transliteration: mapping non-ASCII letters to
|
|
||||||
# ASCII ones would be a semantic claim the slugger cannot make — Norwegian
|
|
||||||
# `møte` (meeting) would become `mote` (fashion). The readable name survives
|
|
||||||
# verbatim in the `title` field; the slug is an identifier, not a label.
|
|
||||||
_SEPARATOR_RUN_RE = re.compile(r"[^a-z0-9]+")
|
|
||||||
|
|
||||||
# NAME_MAX: the per-component limit on every filesystem this library targets
|
|
||||||
# (APFS, ext4, NTFS all cap at 255). Checked here rather than caught at the
|
|
||||||
# write, because the OS signals it as an OSError whose errno differs per
|
|
||||||
# platform (63 on macOS, 36 on Linux) — an untyped, unportable failure at the
|
|
||||||
# very moment the caller needs a typed per-file outcome. Verified empirically
|
|
||||||
# on APFS 2026-07-25: a 255-byte name writes, a 258-byte one raises errno 63.
|
|
||||||
_MAX_FILENAME_BYTES = 255
|
|
||||||
_FILENAME_PREFIX = "inbox-"
|
_FILENAME_PREFIX = "inbox-"
|
||||||
_FILENAME_SUFFIX = ".md"
|
_FILENAME_SUFFIX = ".md"
|
||||||
|
|
||||||
|
|
@ -64,15 +51,7 @@ def inbox_slug(source_filename: str) -> str:
|
||||||
A name that reduces to nothing fails fast: never an invented fallback like
|
A name that reduces to nothing fails fast: never an invented fallback like
|
||||||
`untitled`, which would silently collide across unrelated files.
|
`untitled`, which would silently collide across unrelated files.
|
||||||
"""
|
"""
|
||||||
# NFC first: macOS (APFS/HFS+) hands filenames over DECOMPOSED, so an
|
slug = reduce_to_id_grammar(Path(source_filename).stem)
|
||||||
# `é` arrives as `e` + combining acute. Without normalising, the same
|
|
||||||
# visual filename slugs differently depending on where it came from — the
|
|
||||||
# combining mark alone becomes a separator and the base letter survives
|
|
||||||
# (`cafe`), where a composed `é` is one non-grammar character (`caf`).
|
|
||||||
# Composing first makes the whole letter one unit, so non-ASCII is
|
|
||||||
# uniformly a separator and the slug is stable across both forms.
|
|
||||||
stem = unicodedata.normalize("NFC", Path(source_filename).stem)
|
|
||||||
slug = _SEPARATOR_RUN_RE.sub("-", stem.lower()).strip("-")
|
|
||||||
if not slug:
|
if not slug:
|
||||||
raise MaterializationError(
|
raise MaterializationError(
|
||||||
f"inbox filename {source_filename!r} reduces to an empty slug under the "
|
f"inbox filename {source_filename!r} reduces to an empty slug under the "
|
||||||
|
|
@ -88,27 +67,14 @@ def inbox_filename(slug: str) -> str:
|
||||||
The `inbox-` prefix keeps the namespace disjoint from `index.md`, Door A's
|
The `inbox-` prefix keeps the namespace disjoint from `index.md`, Door A's
|
||||||
`ingest-*`, and `promoted-verdict-*` for every slug the grammar admits.
|
`ingest-*`, and `promoted-verdict-*` for every slug the grammar admits.
|
||||||
|
|
||||||
A name the filesystem cannot hold fails fast rather than being truncated:
|
A name the filesystem cannot hold fails fast rather than being truncated
|
||||||
truncation is lossy AND collision-prone (two long names sharing a prefix
|
(see :func:`check_filename_length`). Refusing keeps the same posture as
|
||||||
would reduce to one filename, and the second write would silently claim
|
`inbox_slug_empty` — the library never invents a filename the operator did
|
||||||
the first file). Refusing keeps the same posture as `inbox_slug_empty` —
|
not give it.
|
||||||
the library never invents a filename the operator did not give it. The
|
|
||||||
operator's fix is to rename the dropped file, so the message carries both
|
|
||||||
the actual size and the limit.
|
|
||||||
"""
|
"""
|
||||||
name = f"{_FILENAME_PREFIX}{slug}{_FILENAME_SUFFIX}"
|
return check_filename_length(
|
||||||
# The slug is ASCII by construction (the id grammar admits nothing else),
|
f"{_FILENAME_PREFIX}{slug}{_FILENAME_SUFFIX}", code="inbox_slug_too_long"
|
||||||
# so len() in characters and in bytes agree — encoding here anyway keeps
|
)
|
||||||
# the check honest if the grammar is ever widened.
|
|
||||||
size = len(name.encode("utf-8"))
|
|
||||||
if size > _MAX_FILENAME_BYTES:
|
|
||||||
raise MaterializationError(
|
|
||||||
f"inbox filename for slug {slug!r} would be {size} bytes, over the "
|
|
||||||
f"{_MAX_FILENAME_BYTES}-byte filesystem limit — rename the dropped "
|
|
||||||
"file; refusing to truncate (lossy and collision-prone)",
|
|
||||||
code="inbox_slug_too_long",
|
|
||||||
)
|
|
||||||
return name
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_body(text: str) -> str:
|
def _normalize_body(text: str) -> str:
|
||||||
|
|
|
||||||
|
|
@ -12,6 +12,7 @@ from __future__ import annotations
|
||||||
import hashlib
|
import hashlib
|
||||||
import logging
|
import logging
|
||||||
import re
|
import re
|
||||||
|
import unicodedata
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
@ -64,6 +65,61 @@ def validate_ingested_at(ingested_at: str) -> None:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- generated-name primitives (shared by Doors B and C) ------------------
|
||||||
|
|
||||||
|
# Every character outside the Phase 1 id grammar (`[a-z0-9][a-z0-9-]*`) is a
|
||||||
|
# separator. Deliberately NOT a transliteration: mapping non-ASCII letters to
|
||||||
|
# ASCII ones would be a semantic claim the slugger cannot make — Norwegian
|
||||||
|
# `møte` (meeting) would become `mote` (fashion). The readable name survives
|
||||||
|
# verbatim in the concept's title or index label; the slug is an identifier,
|
||||||
|
# not a label.
|
||||||
|
_SEPARATOR_RUN_RE = re.compile(r"[^a-z0-9]+")
|
||||||
|
|
||||||
|
# NAME_MAX: the per-component limit on every filesystem this library targets
|
||||||
|
# (APFS, ext4, NTFS all cap at 255). Checked before the write rather than
|
||||||
|
# caught at it, because the OS signals it as an OSError whose errno differs per
|
||||||
|
# platform (63 on macOS, 36 on Linux) — an untyped, unportable failure at the
|
||||||
|
# very moment the caller needs a typed per-file outcome. Verified empirically
|
||||||
|
# on APFS 2026-07-25: a 255-byte name writes, a 258-byte one raises errno 63.
|
||||||
|
NAME_MAX_BYTES = 255
|
||||||
|
|
||||||
|
|
||||||
|
def reduce_to_id_grammar(text: str) -> str:
|
||||||
|
"""Reduce arbitrary text to the Phase 1 id grammar, or to the empty string.
|
||||||
|
|
||||||
|
Lowercased, with every run of non-grammar characters collapsed to a single
|
||||||
|
`-` and both ends stripped. The caller decides what an empty result means:
|
||||||
|
both doors refuse it rather than invent a fallback name.
|
||||||
|
"""
|
||||||
|
# NFC first: macOS (APFS/HFS+) hands filenames over DECOMPOSED, so an `é`
|
||||||
|
# arrives as `e` + combining acute. Without normalising, the same visual
|
||||||
|
# name reduces differently depending on where it came from — the combining
|
||||||
|
# mark alone becomes a separator and the base letter survives (`cafe`),
|
||||||
|
# where a composed `é` is one non-grammar character (`caf`). Composing
|
||||||
|
# first makes the whole letter one unit, so non-ASCII is uniformly a
|
||||||
|
# separator and the result is stable across both forms.
|
||||||
|
return _SEPARATOR_RUN_RE.sub("-", unicodedata.normalize("NFC", text).lower()).strip("-")
|
||||||
|
|
||||||
|
|
||||||
|
def check_filename_length(name: str, *, code: str) -> str:
|
||||||
|
"""Refuse a generated filename the filesystem cannot hold.
|
||||||
|
|
||||||
|
Never truncated: truncation is lossy AND collision-prone (two long names
|
||||||
|
sharing a prefix would reduce to one filename, and the second write would
|
||||||
|
silently claim the first file). The message carries both the actual size
|
||||||
|
and the limit, because the operator's fix is to shorten the source name.
|
||||||
|
"""
|
||||||
|
size = len(name.encode("utf-8"))
|
||||||
|
if size > NAME_MAX_BYTES:
|
||||||
|
raise MaterializationError(
|
||||||
|
f"the generated filename would be {size} bytes, over the "
|
||||||
|
f"{NAME_MAX_BYTES}-byte filesystem limit — shorten the source name; "
|
||||||
|
"refusing to truncate (lossy and collision-prone)",
|
||||||
|
code=code,
|
||||||
|
)
|
||||||
|
return name
|
||||||
|
|
||||||
|
|
||||||
def _collapse_whitespace(value: str) -> str:
|
def _collapse_whitespace(value: str) -> str:
|
||||||
# §5 mandates whitespace-run collapse for ONE field only: `source_query`
|
# §5 mandates whitespace-run collapse for ONE field only: `source_query`
|
||||||
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT
|
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT
|
||||||
|
|
|
||||||
|
|
@ -28,6 +28,13 @@ from llm_ingestion_okf.errors import (
|
||||||
SourceError,
|
SourceError,
|
||||||
)
|
)
|
||||||
from llm_ingestion_okf.extract import extract_text
|
from llm_ingestion_okf.extract import extract_text
|
||||||
|
from llm_ingestion_okf.importer import (
|
||||||
|
BundleDecision,
|
||||||
|
ImportDecision,
|
||||||
|
import_bundle,
|
||||||
|
import_filename,
|
||||||
|
import_slug,
|
||||||
|
)
|
||||||
from llm_ingestion_okf.inbox import (
|
from llm_ingestion_okf.inbox import (
|
||||||
GateDecision,
|
GateDecision,
|
||||||
inbox_filename,
|
inbox_filename,
|
||||||
|
|
@ -411,6 +418,74 @@ def test_okf_type_reserved_at_the_inbox_door() -> None:
|
||||||
assert code_of(excinfo) == "okf_type_reserved"
|
assert code_of(excinfo) == "okf_type_reserved"
|
||||||
|
|
||||||
|
|
||||||
|
# --- MaterializationError codes at Door C ---
|
||||||
|
|
||||||
|
|
||||||
|
def approve_all(bundle: dict[str, str], *, origin: str, channel: str) -> BundleDecision:
|
||||||
|
"""A gate that clears every concept at the non-blocking floor."""
|
||||||
|
return BundleDecision(
|
||||||
|
concepts=tuple(ImportDecision(path=path, disposition="warn") for path in sorted(bundle))
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def place(source: Path, relpath: str, content: str = "body\n") -> None:
|
||||||
|
path = source / relpath
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(content, encoding="utf-8", newline="")
|
||||||
|
|
||||||
|
|
||||||
|
def run_import(tmp_path: Path, **overrides: Any) -> Any:
|
||||||
|
kwargs: dict[str, Any] = {"origin": "external", "channel": "automatic", "gate": approve_all}
|
||||||
|
kwargs.update(overrides)
|
||||||
|
return import_bundle(tmp_path / "source", tmp_path / "bundle", INGESTED_AT, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def test_import_path_empty() -> None:
|
||||||
|
with pytest.raises(MaterializationError) as excinfo:
|
||||||
|
import_slug("!!!.md")
|
||||||
|
assert code_of(excinfo) == "import_path_empty"
|
||||||
|
|
||||||
|
|
||||||
|
def test_import_path_too_long() -> None:
|
||||||
|
with pytest.raises(MaterializationError) as excinfo:
|
||||||
|
import_filename("a" * 250)
|
||||||
|
assert code_of(excinfo) == "import_path_too_long"
|
||||||
|
|
||||||
|
|
||||||
|
def test_import_slug_collision(tmp_path: Path) -> None:
|
||||||
|
# Two concept paths reducing to one generated filename: reported per
|
||||||
|
# concept, so this code surfaces in ImportResult.failed rather than as a raise.
|
||||||
|
place(tmp_path / "source", "a/b.md")
|
||||||
|
place(tmp_path / "source", "a-b.md")
|
||||||
|
result = run_import(tmp_path)
|
||||||
|
assert {entry.error.code for entry in result.failed} == {"import_slug_collision"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_import_label_invalid(tmp_path: Path) -> None:
|
||||||
|
place(tmp_path / "source", "report [v2].md")
|
||||||
|
result = run_import(tmp_path)
|
||||||
|
assert [entry.error.code for entry in result.failed] == ["import_label_invalid"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_collision_unstamped_at_the_import_door(tmp_path: Path) -> None:
|
||||||
|
# Same code as Door A's stamp collision: the generated name is occupied by
|
||||||
|
# content this library cannot prove is its own to replace.
|
||||||
|
bundle = tmp_path / "bundle"
|
||||||
|
bundle.mkdir()
|
||||||
|
(bundle / "import-note.md").write_text("curated\n", encoding="utf-8")
|
||||||
|
place(tmp_path / "source", "note.md")
|
||||||
|
result = run_import(tmp_path)
|
||||||
|
assert [entry.error.code for entry in result.failed] == ["collision_unstamped"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(("origin", "channel"), [("nope", "automatic"), ("external", "nope")])
|
||||||
|
def test_import_provenance_invalid(tmp_path: Path, origin: str, channel: str) -> None:
|
||||||
|
place(tmp_path / "source", "a.md")
|
||||||
|
with pytest.raises(MaterializationError) as excinfo:
|
||||||
|
run_import(tmp_path, origin=origin, channel=channel)
|
||||||
|
assert code_of(excinfo) == "import_provenance_invalid"
|
||||||
|
|
||||||
|
|
||||||
# --- NetworkGateError codes ---
|
# --- NetworkGateError codes ---
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
576
tests/test_import_flow.py
Normal file
576
tests/test_import_flow.py
Normal file
|
|
@ -0,0 +1,576 @@
|
||||||
|
"""Door C import flow against a stub import gate (Phase 2 step 5).
|
||||||
|
|
||||||
|
The stub stands in for the pinned `llm-ingestion-guard` `okf.import_bundle`
|
||||||
|
surface so the merge, quarantine and reject branches run deterministically
|
||||||
|
without the dependency installed. It is a TEST DOUBLE and lives here on
|
||||||
|
purpose — never in `src/`.
|
||||||
|
|
||||||
|
What the flow owes the caller, and what this suite pins:
|
||||||
|
|
||||||
|
- the guard decides per concept; the library only branches on the verdict and
|
||||||
|
never re-derives one (an unrecognised disposition, and a concept the gate
|
||||||
|
returned no verdict for, must both fail CLOSED);
|
||||||
|
- a merged concept is written VERBATIM — exactly the text handed to the gate,
|
||||||
|
with no frontmatter of ours merged into it (round-tripping the concept's own
|
||||||
|
frontmatter through this library's line-oriented parser would silently
|
||||||
|
destroy the block lists the guard's parser accepts);
|
||||||
|
- ownership is proven by content identity, not by a stamp: a target name held
|
||||||
|
by identical bytes is a no-op re-merge, and a target name held by ANYTHING
|
||||||
|
else is refused — curated content and a changed external concept alike;
|
||||||
|
- one bad concept never aborts the run;
|
||||||
|
- the guard's log body is returned to the caller, never persisted (the
|
||||||
|
reserved-file policy is Phase 3's).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from llm_ingestion_okf.importer import BundleDecision, ImportDecision, ImportResult, import_bundle
|
||||||
|
|
||||||
|
INGESTED_AT = "2026-07-25T12:00:00Z"
|
||||||
|
|
||||||
|
# The guard's Origin/Channel/Disposition are `str, Enum`s; these are their
|
||||||
|
# VALUES, restated here independently of the library constants so a drift in
|
||||||
|
# either is visible.
|
||||||
|
EXTERNAL = "external"
|
||||||
|
INTERNAL = "internal"
|
||||||
|
AUTOMATIC = "automatic"
|
||||||
|
MANUAL = "manual"
|
||||||
|
WARN = "warn"
|
||||||
|
QUARANTINE_REVIEW = "quarantine_review"
|
||||||
|
FAIL_SECURE = "fail_secure"
|
||||||
|
|
||||||
|
|
||||||
|
# --- the stub gate --------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class StubImportGate:
|
||||||
|
"""A bundle gate whose per-concept verdict is scripted.
|
||||||
|
|
||||||
|
`by_marker` maps a substring to the disposition any concept containing it
|
||||||
|
gets; `errors` maps a concept path to a hard-reject reason (the guard's
|
||||||
|
`ConceptResult.error`). Everything else passes at the non-blocking floor.
|
||||||
|
"""
|
||||||
|
|
||||||
|
by_marker: dict[str, str] = field(default_factory=dict)
|
||||||
|
errors: dict[str, str] = field(default_factory=dict)
|
||||||
|
omit: frozenset[str] = frozenset()
|
||||||
|
log: str = ""
|
||||||
|
calls: list[dict[str, str]] = field(default_factory=list)
|
||||||
|
provenance: list[tuple[str, str]] = field(default_factory=list)
|
||||||
|
|
||||||
|
def __call__(self, bundle: dict[str, str], *, origin: str, channel: str) -> BundleDecision:
|
||||||
|
self.calls.append(dict(bundle))
|
||||||
|
self.provenance.append((origin, channel))
|
||||||
|
decisions = []
|
||||||
|
for path in sorted(bundle):
|
||||||
|
if path in self.omit:
|
||||||
|
continue
|
||||||
|
error = self.errors.get(path)
|
||||||
|
disposition = FAIL_SECURE if error is not None else WARN
|
||||||
|
for marker, scripted in self.by_marker.items():
|
||||||
|
if marker in bundle[path]:
|
||||||
|
disposition = scripted
|
||||||
|
decisions.append(
|
||||||
|
ImportDecision(
|
||||||
|
path=path,
|
||||||
|
disposition=disposition,
|
||||||
|
error=error,
|
||||||
|
reasons=(f"scanned {path}",),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return BundleDecision(concepts=tuple(decisions), log=self.log)
|
||||||
|
|
||||||
|
|
||||||
|
def place(source: Path, relpath: str, content: str) -> Path:
|
||||||
|
path = source / relpath
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(content, encoding="utf-8", newline="")
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def run(
|
||||||
|
tmp_path: Path,
|
||||||
|
gate: StubImportGate,
|
||||||
|
*,
|
||||||
|
origin: str = EXTERNAL,
|
||||||
|
channel: str = AUTOMATIC,
|
||||||
|
ingested_at: str = INGESTED_AT,
|
||||||
|
) -> tuple[ImportResult, Path]:
|
||||||
|
bundle = tmp_path / "bundle"
|
||||||
|
result = import_bundle(
|
||||||
|
tmp_path / "source",
|
||||||
|
bundle,
|
||||||
|
ingested_at,
|
||||||
|
origin=origin,
|
||||||
|
channel=channel,
|
||||||
|
gate=gate,
|
||||||
|
)
|
||||||
|
return result, bundle
|
||||||
|
|
||||||
|
|
||||||
|
CONCEPT = "---\ntype: dataset\ntitle: Users\n---\n\nA table of users.\n"
|
||||||
|
|
||||||
|
|
||||||
|
# --- the merge branch -----------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_accepted_concept_is_merged(tmp_path: Path) -> None:
|
||||||
|
place(tmp_path / "source", "tables/users.md", CONCEPT)
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, StubImportGate())
|
||||||
|
|
||||||
|
assert [entry.concept_path for entry in result.merged] == ["tables/users.md"]
|
||||||
|
assert (bundle / "import-tables-users.md").is_file()
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_merged_bytes_are_exactly_the_bytes_the_gate_saw(tmp_path: Path) -> None:
|
||||||
|
"""Verbatim is the whole contract at this door: the guard validated a
|
||||||
|
document, so the document is what may be persisted. Nothing of ours is
|
||||||
|
merged into its frontmatter — this library's line-oriented parser cannot
|
||||||
|
round-trip the block lists the guard's parser accepts, so writing back a
|
||||||
|
re-rendered concept would silently drop data the operator sent us.
|
||||||
|
"""
|
||||||
|
document = "---\ntags:\n - alpha\n - beta\n---\n\nBody with a [link](other.md).\n"
|
||||||
|
place(tmp_path / "source", "notes/tagged.md", document)
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, gate := StubImportGate())
|
||||||
|
|
||||||
|
written = (bundle / "import-notes-tagged.md").read_bytes()
|
||||||
|
assert written == document.encode("utf-8")
|
||||||
|
assert gate.calls[0]["notes/tagged.md"] == document
|
||||||
|
assert result.failed == ()
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_index_links_every_merged_concept_by_concept_id(tmp_path: Path) -> None:
|
||||||
|
place(tmp_path / "source", "tables/users.md", CONCEPT)
|
||||||
|
|
||||||
|
_, bundle = run(tmp_path, StubImportGate())
|
||||||
|
|
||||||
|
index = (bundle / "index.md").read_text(encoding="utf-8")
|
||||||
|
assert index == "- [tables/users](import-tables-users.md)\n"
|
||||||
|
|
||||||
|
|
||||||
|
def test_nested_paths_keep_their_directory_in_the_generated_name(tmp_path: Path) -> None:
|
||||||
|
"""Flattening on the stem alone would collide two distinct concepts; the
|
||||||
|
whole bundle-relative path is what reduces to the slug.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "tables/users.md", "one\n")
|
||||||
|
place(tmp_path / "source", "views/users.md", "two\n")
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, StubImportGate())
|
||||||
|
|
||||||
|
assert [entry.path.name for entry in result.merged] == [
|
||||||
|
"import-tables-users.md",
|
||||||
|
"import-views-users.md",
|
||||||
|
]
|
||||||
|
assert (bundle / "import-tables-users.md").read_text(encoding="utf-8") == "one\n"
|
||||||
|
assert (bundle / "import-views-users.md").read_text(encoding="utf-8") == "two\n"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_gate_is_called_once_with_the_whole_bundle(tmp_path: Path) -> None:
|
||||||
|
"""`okf.import_bundle` is a bundle-level call — it resolves the cross-link
|
||||||
|
graph across concepts — so the flow hands it the whole mapping at once,
|
||||||
|
keyed by POSIX-separated bundle-relative paths.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "b.md", "second\n")
|
||||||
|
place(tmp_path / "source", "nested/a.md", "first\n")
|
||||||
|
|
||||||
|
_, _ = run(tmp_path, gate := StubImportGate())
|
||||||
|
|
||||||
|
assert len(gate.calls) == 1
|
||||||
|
assert gate.calls[0] == {"b.md": "second\n", "nested/a.md": "first\n"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_origin_and_channel_reach_the_gate_verbatim(tmp_path: Path) -> None:
|
||||||
|
"""Trust follows origin, never channel — the caller declares both at the
|
||||||
|
door and the library only carries them; it never defaults or derives them.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "a.md", "body\n")
|
||||||
|
|
||||||
|
run(tmp_path, gate := StubImportGate(), origin=INTERNAL, channel=MANUAL)
|
||||||
|
|
||||||
|
assert gate.provenance == [(INTERNAL, MANUAL)]
|
||||||
|
|
||||||
|
|
||||||
|
# --- the blocking branches ------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_hard_rejected_concept_is_not_merged(tmp_path: Path) -> None:
|
||||||
|
place(tmp_path / "source", "bad.md", "body\n")
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, StubImportGate(errors={"bad.md": "reserved filename"}))
|
||||||
|
|
||||||
|
assert result.merged == ()
|
||||||
|
assert [entry.concept_path for entry in result.rejected] == ["bad.md"]
|
||||||
|
assert result.rejected[0].error == "reserved filename"
|
||||||
|
assert not (bundle / "import-bad.md").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_quarantine_review_is_reported_separately_from_rejection(tmp_path: Path) -> None:
|
||||||
|
"""Quarantine means "hold for human review" and is the operator's queue; a
|
||||||
|
fail-secure verdict is a decision, not a queue. Same split as Door B.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "suspect.md", "hold me\n")
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, StubImportGate(by_marker={"hold me": QUARANTINE_REVIEW}))
|
||||||
|
|
||||||
|
assert result.merged == ()
|
||||||
|
assert [entry.concept_path for entry in result.quarantined] == ["suspect.md"]
|
||||||
|
assert result.rejected == ()
|
||||||
|
assert not (bundle / "import-suspect.md").exists()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("disposition", ["", "pass", "ok", "WARN", "warn "])
|
||||||
|
def test_an_unrecognised_disposition_fails_closed(tmp_path: Path, disposition: str) -> None:
|
||||||
|
place(tmp_path / "source", "a.md", "body\n")
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, StubImportGate(by_marker={"body": disposition}))
|
||||||
|
|
||||||
|
assert result.merged == ()
|
||||||
|
assert [entry.concept_path for entry in result.rejected] == ["a.md"]
|
||||||
|
assert list(bundle.glob("*.md")) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_error_with_a_non_blocking_disposition_still_fails_closed(tmp_path: Path) -> None:
|
||||||
|
"""The guard pairs a hard reject with FAIL_SECURE, but the library must not
|
||||||
|
depend on that pairing: an error is a refusal on its own terms.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "a.md", "body\n")
|
||||||
|
gate = StubImportGate(errors={"a.md": "unsafe resource"}, by_marker={"body": WARN})
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, gate)
|
||||||
|
|
||||||
|
assert result.merged == ()
|
||||||
|
assert [entry.concept_path for entry in result.rejected] == ["a.md"]
|
||||||
|
assert list(bundle.glob("*.md")) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_concept_the_gate_returned_no_verdict_for_is_not_merged(tmp_path: Path) -> None:
|
||||||
|
"""No verdict is not consent. A gate that drops a concept from its result —
|
||||||
|
an adapter bug, or a guard that stops reporting one — must not be read as
|
||||||
|
approval.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "a.md", "body\n")
|
||||||
|
place(tmp_path / "source", "b.md", "body\n")
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, StubImportGate(omit=frozenset({"a.md"})))
|
||||||
|
|
||||||
|
assert [entry.concept_path for entry in result.merged] == ["b.md"]
|
||||||
|
assert [entry.concept_path for entry in result.rejected] == ["a.md"]
|
||||||
|
assert not (bundle / "import-a.md").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_rejected_concept_leaves_the_bundle_byte_identical(tmp_path: Path) -> None:
|
||||||
|
"""The persist-gate proof: a rejected concept produces ZERO new files and
|
||||||
|
does not touch the ones already there — not even the index.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "ok.md", "body\n")
|
||||||
|
result, bundle = run(tmp_path, StubImportGate())
|
||||||
|
assert result.merged != ()
|
||||||
|
before = sorted((path.name, path.read_bytes()) for path in bundle.iterdir())
|
||||||
|
|
||||||
|
(tmp_path / "source" / "ok.md").unlink()
|
||||||
|
place(tmp_path / "source", "poisoned.md", "body\n")
|
||||||
|
run(tmp_path, StubImportGate(errors={"poisoned.md": "injection"}))
|
||||||
|
|
||||||
|
after = sorted((path.name, path.read_bytes()) for path in bundle.iterdir())
|
||||||
|
assert after == before
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_mixed_bundle_merges_only_the_accepted_concepts(tmp_path: Path) -> None:
|
||||||
|
"""Partial merge is the point of a per-concept gate: one poisoned concept
|
||||||
|
must not cost the operator the rest of the bundle, and must not sneak in.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "good.md", "harmless\n")
|
||||||
|
place(tmp_path / "source", "held.md", "hold me\n")
|
||||||
|
place(tmp_path / "source", "bad.md", "poison\n")
|
||||||
|
|
||||||
|
result, bundle = run(
|
||||||
|
tmp_path,
|
||||||
|
StubImportGate(by_marker={"poison": FAIL_SECURE, "hold me": QUARANTINE_REVIEW}),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert [entry.concept_path for entry in result.merged] == ["good.md"]
|
||||||
|
assert [entry.concept_path for entry in result.quarantined] == ["held.md"]
|
||||||
|
assert [entry.concept_path for entry in result.rejected] == ["bad.md"]
|
||||||
|
assert sorted(path.name for path in bundle.glob("*.md")) == ["import-good.md", "index.md"]
|
||||||
|
assert (bundle / "index.md").read_text(encoding="utf-8") == "- [good](import-good.md)\n"
|
||||||
|
|
||||||
|
|
||||||
|
# --- ownership by content identity ----------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_re_import_of_an_unchanged_bundle_is_a_no_op(tmp_path: Path) -> None:
|
||||||
|
"""Identical bytes at the target name are proof enough that the file is
|
||||||
|
ours: the re-import is reported as merged, writes nothing new, and does not
|
||||||
|
duplicate the index link.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "tables/users.md", CONCEPT)
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, StubImportGate())
|
||||||
|
first = sorted((path.name, path.read_bytes()) for path in bundle.iterdir())
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, StubImportGate())
|
||||||
|
second = sorted((path.name, path.read_bytes()) for path in bundle.iterdir())
|
||||||
|
|
||||||
|
assert [entry.concept_path for entry in result.merged] == ["tables/users.md"]
|
||||||
|
assert result.failed == ()
|
||||||
|
assert second == first
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_changed_concept_is_refused_without_overwriting(tmp_path: Path) -> None:
|
||||||
|
"""Without a stamp the library cannot prove the existing file is a previous
|
||||||
|
import rather than an edit the operator made — so a changed concept is
|
||||||
|
refused, not merged. The operator resolves it by removing the old file.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "note.md", "first version\n")
|
||||||
|
run(tmp_path, StubImportGate())
|
||||||
|
place(tmp_path / "source", "note.md", "second version\n")
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, StubImportGate())
|
||||||
|
|
||||||
|
assert result.merged == ()
|
||||||
|
assert [entry.concept_path for entry in result.failed] == ["note.md"]
|
||||||
|
assert result.failed[0].error.code == "collision_unstamped"
|
||||||
|
assert (bundle / "import-note.md").read_text(encoding="utf-8") == "first version\n"
|
||||||
|
|
||||||
|
|
||||||
|
def test_curated_content_at_the_target_name_is_never_overwritten(tmp_path: Path) -> None:
|
||||||
|
bundle = tmp_path / "bundle"
|
||||||
|
bundle.mkdir()
|
||||||
|
curated = bundle / "import-note.md"
|
||||||
|
curated.write_text("curated by hand\n", encoding="utf-8")
|
||||||
|
place(tmp_path / "source", "note.md", "external\n")
|
||||||
|
place(tmp_path / "source", "other.md", "external\n")
|
||||||
|
|
||||||
|
result, _ = run(tmp_path, StubImportGate())
|
||||||
|
|
||||||
|
assert curated.read_text(encoding="utf-8") == "curated by hand\n"
|
||||||
|
assert [entry.error.code for entry in result.failed] == ["collision_unstamped"]
|
||||||
|
assert [entry.concept_path for entry in result.merged] == ["other.md"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_collision_gate_is_evaluated_before_any_write(tmp_path: Path) -> None:
|
||||||
|
"""Pre-mutation, exactly as at Door A and Door B: occupancy is judged
|
||||||
|
against the bundle as it was BEFORE this run.
|
||||||
|
"""
|
||||||
|
bundle = tmp_path / "bundle"
|
||||||
|
bundle.mkdir()
|
||||||
|
(bundle / "import-zzz.md").write_text("curated\n", encoding="utf-8")
|
||||||
|
place(tmp_path / "source", "aaa.md", "body\n")
|
||||||
|
place(tmp_path / "source", "zzz.md", "body\n")
|
||||||
|
|
||||||
|
result, _ = run(tmp_path, StubImportGate())
|
||||||
|
|
||||||
|
assert [entry.error.code for entry in result.failed] == ["collision_unstamped"]
|
||||||
|
assert [entry.concept_path for entry in result.merged] == ["aaa.md"]
|
||||||
|
assert (bundle / "import-zzz.md").read_text(encoding="utf-8") == "curated\n"
|
||||||
|
|
||||||
|
|
||||||
|
def test_two_concept_paths_that_slug_alike_are_both_refused(tmp_path: Path) -> None:
|
||||||
|
"""`a/b.md` and `a-b.md` both reduce to `import-a-b.md`. The library will
|
||||||
|
not pick a winner: both are refused and the sender renames one.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "a/b.md", "one\n")
|
||||||
|
place(tmp_path / "source", "a-b.md", "two\n")
|
||||||
|
place(tmp_path / "source", "other.md", "three\n")
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, StubImportGate())
|
||||||
|
|
||||||
|
assert [entry.concept_path for entry in result.failed] == ["a-b.md", "a/b.md"]
|
||||||
|
assert {entry.error.code for entry in result.failed} == {"import_slug_collision"}
|
||||||
|
assert not (bundle / "import-a-b.md").exists()
|
||||||
|
assert [entry.concept_path for entry in result.merged] == ["other.md"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- one bad concept never aborts the run ---------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_concept_path_that_reduces_to_an_empty_slug_is_a_per_concept_failure(
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
place(tmp_path / "source", "!!!.md", "body\n")
|
||||||
|
place(tmp_path / "source", "good.md", "body\n")
|
||||||
|
|
||||||
|
result, _ = run(tmp_path, StubImportGate())
|
||||||
|
|
||||||
|
assert [entry.concept_path for entry in result.merged] == ["good.md"]
|
||||||
|
assert [entry.error.code for entry in result.failed] == ["import_path_empty"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_over_long_concept_path_is_a_per_concept_failure(tmp_path: Path) -> None:
|
||||||
|
"""A bundle-relative path has no 255-byte limit of its own — only its
|
||||||
|
individual segments do — so a deep enough tree reduces to a filename the
|
||||||
|
filesystem cannot hold. Refused, never truncated.
|
||||||
|
"""
|
||||||
|
deep = "/".join(["segment"] * 40) + ".md"
|
||||||
|
place(tmp_path / "source", deep, "body\n")
|
||||||
|
place(tmp_path / "source", "good.md", "body\n")
|
||||||
|
|
||||||
|
result, _ = run(tmp_path, StubImportGate())
|
||||||
|
|
||||||
|
assert [entry.concept_path for entry in result.merged] == ["good.md"]
|
||||||
|
assert [entry.error.code for entry in result.failed] == ["import_path_too_long"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_concept_path_that_would_break_an_index_link_is_a_per_concept_failure(
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
"""The guard's path gate permits brackets; this library's index link does
|
||||||
|
not. The concept-ID is rendered verbatim as the link label, so it meets the
|
||||||
|
same fail-fast rule Door A and Door B apply to a title.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "report [v2].md", "body\n")
|
||||||
|
|
||||||
|
result, _ = run(tmp_path, StubImportGate())
|
||||||
|
|
||||||
|
assert result.merged == ()
|
||||||
|
assert [entry.error.code for entry in result.failed] == ["import_label_invalid"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_non_utf8_concept_is_a_per_concept_failure(tmp_path: Path) -> None:
|
||||||
|
"""Read failures happen before the gate — an undecodable concept is never
|
||||||
|
handed to it, and never counted as approved.
|
||||||
|
"""
|
||||||
|
source = tmp_path / "source"
|
||||||
|
source.mkdir()
|
||||||
|
(source / "broken.md").write_bytes(b"\xffnot utf-8\n")
|
||||||
|
place(source, "good.md", "body\n")
|
||||||
|
|
||||||
|
result, _ = run(tmp_path, gate := StubImportGate())
|
||||||
|
|
||||||
|
assert [entry.concept_path for entry in result.merged] == ["good.md"]
|
||||||
|
assert [entry.error.code for entry in result.failed] == ["extractor_decode_error"]
|
||||||
|
assert "broken.md" not in gate.calls[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_uppercase_md_suffix_is_still_a_concept(tmp_path: Path) -> None:
|
||||||
|
"""Determinism across filesystems: a `*.md` glob is case-sensitive on ext4
|
||||||
|
and case-insensitive on APFS, so the same bundle would import differently
|
||||||
|
per platform. The suffix test is explicit and case-folded, as the guard's
|
||||||
|
own path gate is.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "NOTE.MD", "body\n")
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, gate := StubImportGate())
|
||||||
|
|
||||||
|
assert list(gate.calls[0]) == ["NOTE.MD"]
|
||||||
|
assert [entry.concept_path for entry in result.merged] == ["NOTE.MD"]
|
||||||
|
assert (bundle / "import-note.md").read_text(encoding="utf-8") == "body\n"
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_markdown_files_are_not_concepts(tmp_path: Path) -> None:
|
||||||
|
"""An OKF concept is a `.md` document by definition (the guard's path gate
|
||||||
|
says so too), so anything else in the source tree is not this door's to
|
||||||
|
merge — it is neither gated nor written.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "a.md", "body\n")
|
||||||
|
place(tmp_path / "source", "data.csv", "a,b\n1,2\n")
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, gate := StubImportGate())
|
||||||
|
|
||||||
|
assert list(gate.calls[0]) == ["a.md"]
|
||||||
|
assert [entry.concept_path for entry in result.merged] == ["a.md"]
|
||||||
|
assert sorted(path.name for path in bundle.iterdir()) == ["import-a.md", "index.md"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- the import report ----------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_guards_log_is_returned_and_not_persisted(tmp_path: Path) -> None:
|
||||||
|
"""Deliverable 3: the caller gets the log body; writing it into the bundle
|
||||||
|
is a reserved-file decision that differs per consumer and lands in Phase 3.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "a.md", "body\n")
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, StubImportGate(log="a\texternal\tautomatic\tuntrusted\twarn"))
|
||||||
|
|
||||||
|
assert result.log == "a\texternal\tautomatic\tuntrusted\twarn"
|
||||||
|
assert not (bundle / "log.md").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_result_carries_the_run_timestamp(tmp_path: Path) -> None:
|
||||||
|
"""`ingested_at` reaches no file at this door — a merged concept is written
|
||||||
|
verbatim — but the run is still timestamped explicitly rather than by wall
|
||||||
|
clock, so a caller persisting the log has a deterministic timestamp to use.
|
||||||
|
"""
|
||||||
|
place(tmp_path / "source", "a.md", "body\n")
|
||||||
|
|
||||||
|
result, _ = run(tmp_path, StubImportGate())
|
||||||
|
|
||||||
|
assert result.ingested_at == INGESTED_AT
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_per_concept_reasons_are_carried_verbatim(tmp_path: Path) -> None:
|
||||||
|
place(tmp_path / "source", "a.md", "poison\n")
|
||||||
|
|
||||||
|
result, _ = run(tmp_path, StubImportGate(by_marker={"poison": FAIL_SECURE}))
|
||||||
|
|
||||||
|
assert result.rejected[0].reasons == ("scanned a.md",)
|
||||||
|
|
||||||
|
|
||||||
|
# --- run-level fail-fast --------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_invalid_ingested_at_fails_the_whole_run(tmp_path: Path) -> None:
|
||||||
|
from llm_ingestion_okf import MaterializationError
|
||||||
|
|
||||||
|
place(tmp_path / "source", "a.md", "body\n")
|
||||||
|
|
||||||
|
with pytest.raises(MaterializationError) as excinfo:
|
||||||
|
run(tmp_path, StubImportGate(), ingested_at="2026-07-25")
|
||||||
|
assert excinfo.value.code == "ingested_at_invalid"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("origin", "channel"),
|
||||||
|
[("extenal", AUTOMATIC), ("", AUTOMATIC), (EXTERNAL, "auto"), (EXTERNAL, "")],
|
||||||
|
)
|
||||||
|
def test_an_unknown_origin_or_channel_fails_the_whole_run(
|
||||||
|
tmp_path: Path, origin: str, channel: str
|
||||||
|
) -> None:
|
||||||
|
"""A value outside the guard's pinned vocabulary would reach the guard as a
|
||||||
|
plain string, miss its enum identity check, and be silently downgraded to
|
||||||
|
untrusted. Fail-fast instead: the library refuses to carry a provenance
|
||||||
|
declaration it cannot recognise, rather than letting a typo decide trust.
|
||||||
|
"""
|
||||||
|
from llm_ingestion_okf import MaterializationError
|
||||||
|
|
||||||
|
place(tmp_path / "source", "a.md", "body\n")
|
||||||
|
|
||||||
|
with pytest.raises(MaterializationError) as excinfo:
|
||||||
|
run(tmp_path, StubImportGate(), origin=origin, channel=channel)
|
||||||
|
assert excinfo.value.code == "import_provenance_invalid"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_missing_source_directory_is_refused(tmp_path: Path) -> None:
|
||||||
|
from llm_ingestion_okf import SourceError
|
||||||
|
|
||||||
|
with pytest.raises(SourceError) as excinfo:
|
||||||
|
import_bundle(
|
||||||
|
tmp_path / "nope",
|
||||||
|
tmp_path / "bundle",
|
||||||
|
INGESTED_AT,
|
||||||
|
origin=EXTERNAL,
|
||||||
|
channel=AUTOMATIC,
|
||||||
|
gate=StubImportGate(),
|
||||||
|
)
|
||||||
|
assert excinfo.value.code == "source_root_missing"
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_empty_source_bundle_writes_nothing_and_never_calls_the_gate(tmp_path: Path) -> None:
|
||||||
|
(tmp_path / "source").mkdir()
|
||||||
|
|
||||||
|
result, bundle = run(tmp_path, gate := StubImportGate())
|
||||||
|
|
||||||
|
assert result == ImportResult(
|
||||||
|
merged=(), quarantined=(), rejected=(), failed=(), log="", ingested_at=INGESTED_AT
|
||||||
|
)
|
||||||
|
assert gate.calls == []
|
||||||
|
assert not bundle.exists() or list(bundle.iterdir()) == []
|
||||||
Loading…
Add table
Add a link
Reference in a new issue