feat(inbox): Door B flow against an injected guard gate (Phase 2 step 3)
`process_inbox(inbox_dir, bundle_dir, ingested_at, *, okf_type, gate)`: per dropped file, bytes -> extract_text -> guard gate -> render -> collision gate -> write -> index link. Returns an InboxResult whose four buckets (persisted / quarantined / rejected / failed) are disjoint and complete, so a file that vanished shows up as a missing entry rather than as nothing. The guard is INJECTED rather than imported. The library calls no guard function and makes no security decision: the Gate adapter returns a GateDecision carrying the guard's own disposition value and the sanitized text, and the flow only branches on it. That keeps the core dependency-free while B2 (the guard's CI channel) is still open, and lets the persist and refuse branches run deterministically against a test double. Decisions taken with the operator this session: - What is persisted is the gate's SANITIZED text, not the extracted text. Screening one string and writing another would make the verdict a statement about bytes nobody kept. `sanitize` is exported by the guard precisely for callers composing the checklist themselves, so this is sanctioned API, not a reimplementation. Door B has no model call, so the fenced text the bookends produce for a transform is never persisted. - Quarantine is reported apart from rejection. QUARANTINE_REVIEW means "hold for human review" — an operator queue — where FAIL_SECURE is a decision. No quarantine directory in v1; that stays an extension point. Fails closed by construction: only the guard's non-blocking floor (`warn`) persists, so a renamed member, a future disposition or an adapter typo lands in `rejected` rather than being guessed safe. One bad file never aborts the run. Only three conditions fail the whole run, and each is wrong for every file at once: an invalid `ingested_at`, a reserved `okf_type`, and a missing inbox directory. New code `inbox_slug_collision`: two dropped files reducing to one generated name are BOTH refused. Persisting one would let iteration order decide the winner, and overwriting would lose the other's content. Phase 1 primitives are reused, never duplicated, so four helpers become package-internal names (write_bytes, link_in_index, parse_frontmatter, INDEX_NAME) and link_in_index learns to append to an empty index — Door A always seeds its index with bundle_summary, but Door B has no summary to invent and must not open with a blank line.
This commit is contained in:
parent
b7f5ce3800
commit
d812a839be
6 changed files with 763 additions and 18 deletions
|
|
@ -1,4 +1,4 @@
|
|||
"""Door B inbox provenance rendering and filename slugging (Phase 2 step 2).
|
||||
"""Door B: the bundle inbox — provenance rendering, slugging, and the flow.
|
||||
|
||||
Pure functions on top of the Phase 1 primitives: a dropped file's name is
|
||||
reduced to the Phase 1 id grammar and namespaced `inbox-{slug}.md` — disjoint
|
||||
|
|
@ -11,8 +11,10 @@ extracted text, so provenance stays re-verifiable against the operator's file.
|
|||
`ingested_at` is explicit and validated by the same rule as Door A: no
|
||||
wall-clock anywhere. Output is LF-only with exactly one trailing newline.
|
||||
|
||||
No guard call and no model call in this module — the persist gate arrives with
|
||||
the inbox flow (step 3).
|
||||
`process_inbox` is the flow: per file, extracted text goes through the guard
|
||||
gate before anything is written, and only the gate's non-blocking floor
|
||||
persists. No model call anywhere, and no security decision here — the gate
|
||||
supplies the verdict and this module only obeys it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -20,10 +22,19 @@ from __future__ import annotations
|
|||
import hashlib
|
||||
import re
|
||||
import unicodedata
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from .errors import MaterializationError
|
||||
from .materialize import validate_ingested_at
|
||||
from .errors import IngestError, MaterializationError, SourceError
|
||||
from .extract import extract_text
|
||||
from .materialize import (
|
||||
INDEX_NAME,
|
||||
link_in_index,
|
||||
parse_frontmatter,
|
||||
validate_ingested_at,
|
||||
write_bytes,
|
||||
)
|
||||
|
||||
_RESERVED_OKF_TYPE = "verdict"
|
||||
|
||||
|
|
@ -156,3 +167,253 @@ def render_inbox_concept(
|
|||
}
|
||||
rendered = "\n".join(f"{key}: {value}" for key, value in frontmatter.items())
|
||||
return f"---\n{rendered}\n---\n\n{_normalize_body(text)}"
|
||||
|
||||
|
||||
# --- the guard seam -------------------------------------------------------
|
||||
|
||||
# The guard's non-blocking floor. `Disposition` is a `str, Enum` in
|
||||
# llm-ingestion-guard, so its VALUE is the stable thing to compare against
|
||||
# across the pinned `>=0.2,<0.3` range. Pinned as a constant here rather than
|
||||
# imported, because the dependency is injected (see `Gate`): the step-4
|
||||
# adapter's signature smoke test is what catches a rename in the guard.
|
||||
_DISPOSITION_PERSIST = "warn"
|
||||
_DISPOSITION_QUARANTINE = "quarantine_review"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GateDecision:
|
||||
"""One guard verdict, carried verbatim from `llm-ingestion-guard`.
|
||||
|
||||
`sanitized_text` is what the gate actually screened, and therefore the only
|
||||
text that may be persisted — screening one string and writing another would
|
||||
make the verdict a statement about bytes nobody kept. `disposition` is the
|
||||
guard's `Disposition` VALUE and `reasons` its audit trail; neither is
|
||||
interpreted here beyond the single persist/do-not-persist branch.
|
||||
"""
|
||||
|
||||
sanitized_text: str
|
||||
disposition: str
|
||||
reasons: tuple[str, ...] = ()
|
||||
|
||||
|
||||
# The persist gate, injected. The library never imports the guard directly:
|
||||
# the caller supplies the adapter, which keeps this module dependency-free and
|
||||
# lets the flow's branches be exercised deterministically by a test double.
|
||||
Gate = Callable[[str], GateDecision]
|
||||
|
||||
|
||||
# --- per-file outcomes ----------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PersistedFile:
|
||||
"""A dropped file that cleared the gate and was written."""
|
||||
|
||||
source_file: str
|
||||
path: Path
|
||||
reasons: tuple[str, ...] = ()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BlockedFile:
|
||||
"""A dropped file the guard refused. Quarantine and rejection are reported
|
||||
separately: quarantine means "hold for human review" and is the operator's
|
||||
queue, while a fail-secure verdict is a decision, not a queue."""
|
||||
|
||||
source_file: str
|
||||
disposition: str
|
||||
reasons: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FailedFile:
|
||||
"""A dropped file this library could not process — extraction, filename or
|
||||
collision. Always a typed error, never a leaked stdlib exception."""
|
||||
|
||||
source_file: str
|
||||
error: IngestError
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InboxResult:
|
||||
"""Every dropped file's outcome, in sorted filename order.
|
||||
|
||||
Four disjoint buckets, and every file lands in exactly one: a run's report
|
||||
is complete by construction, so a file that silently vanished would show up
|
||||
as a missing entry rather than as nothing at all.
|
||||
"""
|
||||
|
||||
persisted: tuple[PersistedFile, ...]
|
||||
quarantined: tuple[BlockedFile, ...]
|
||||
rejected: tuple[BlockedFile, ...]
|
||||
failed: tuple[FailedFile, ...]
|
||||
|
||||
|
||||
def _is_inbox_owned(path: Path) -> bool:
|
||||
# Door B's ownership marker: `generated: true` AND a `source_file`
|
||||
# reference. Deliberately disjoint from Door A's test, which keys on
|
||||
# `ingest_manifest` — a Door A concept is never this door's to replace,
|
||||
# and curated content carries neither key.
|
||||
frontmatter = parse_frontmatter(path)
|
||||
return frontmatter.get("generated") == "true" and "source_file" in frontmatter
|
||||
|
||||
|
||||
def process_inbox(
|
||||
inbox_dir: Path,
|
||||
bundle_dir: Path,
|
||||
ingested_at: str,
|
||||
*,
|
||||
okf_type: str,
|
||||
gate: Gate,
|
||||
) -> InboxResult:
|
||||
"""Convert every file dropped in `inbox_dir` into an OKF concept.
|
||||
|
||||
An explicit operator command, never a watcher or a scheduler (spec §9's
|
||||
human-in-the-loop rule). `ingested_at` is required and stamped verbatim, as
|
||||
at Door A. `gate` is the guard adapter: extracted text is screened before
|
||||
anything is written, and only the guard's non-blocking floor persists —
|
||||
anything else, INCLUDING a disposition this library does not recognise,
|
||||
fails closed.
|
||||
|
||||
One bad file never aborts the run. Extraction failures, unusable filenames
|
||||
and collisions are reported per file in :class:`InboxResult` while the
|
||||
remaining files still process. Only three conditions fail the whole run,
|
||||
and all three are wrong for every file at once: an invalid `ingested_at`,
|
||||
a reserved `okf_type`, and a missing inbox directory.
|
||||
"""
|
||||
validate_ingested_at(ingested_at)
|
||||
if okf_type.lower() == _RESERVED_OKF_TYPE:
|
||||
raise MaterializationError(
|
||||
f"okf_type must not be {_RESERVED_OKF_TYPE!r} (reserved layer)",
|
||||
code="okf_type_reserved",
|
||||
)
|
||||
inbox = Path(inbox_dir)
|
||||
if not inbox.is_dir():
|
||||
raise SourceError(f"inbox directory does not exist: {inbox}", code="source_root_missing")
|
||||
|
||||
# Top-level only, sorted: the operator's nested structure is theirs, and a
|
||||
# deterministic order is what makes a re-run comparable.
|
||||
dropped = sorted((path for path in inbox.iterdir() if path.is_file()), key=lambda p: p.name)
|
||||
|
||||
persisted: list[PersistedFile] = []
|
||||
quarantined: list[BlockedFile] = []
|
||||
rejected: list[BlockedFile] = []
|
||||
failed: list[FailedFile] = []
|
||||
|
||||
# Phase 1: name every file BEFORE any gate call or write, so an intra-run
|
||||
# slug collision is caught while both files can still be refused together.
|
||||
named: list[tuple[Path, str]] = []
|
||||
slug_owners: dict[str, list[Path]] = {}
|
||||
for path in dropped:
|
||||
try:
|
||||
name = inbox_filename(inbox_slug(path.name))
|
||||
except IngestError as exc:
|
||||
failed.append(FailedFile(source_file=path.name, error=exc))
|
||||
continue
|
||||
named.append((path, name))
|
||||
slug_owners.setdefault(name, []).append(path)
|
||||
|
||||
colliding = {name for name, owners in slug_owners.items() if len(owners) > 1}
|
||||
for name in sorted(colliding):
|
||||
for path in slug_owners[name]:
|
||||
failed.append(
|
||||
FailedFile(
|
||||
source_file=path.name,
|
||||
error=MaterializationError(
|
||||
f"{path.name!r} and "
|
||||
f"{', '.join(repr(other.name) for other in slug_owners[name] if other != path)}"
|
||||
f" both reduce to {name!r} — rename one; refusing to pick a winner",
|
||||
code="inbox_slug_collision",
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
# Phase 2: the §3 ownership scan, evaluated against the bundle as it was
|
||||
# BEFORE this run — a file written below must never be mistaken for
|
||||
# pre-existing curated content by a later file's check.
|
||||
bundle = Path(bundle_dir)
|
||||
pre_existing = (
|
||||
{path.name for path in bundle.glob("*.md") if path.name != INDEX_NAME}
|
||||
if bundle.is_dir()
|
||||
else set()
|
||||
)
|
||||
owned = {name for name in pre_existing if _is_inbox_owned(bundle / name)}
|
||||
|
||||
for path, name in named:
|
||||
if name in colliding:
|
||||
continue
|
||||
if name in pre_existing and name not in owned:
|
||||
failed.append(
|
||||
FailedFile(
|
||||
source_file=path.name,
|
||||
error=MaterializationError(
|
||||
f"generated filename {name!r} collides with an existing file that does "
|
||||
"not carry the inbox marker — refusing to overwrite curated content (§3)",
|
||||
code="collision_unstamped",
|
||||
),
|
||||
)
|
||||
)
|
||||
continue
|
||||
try:
|
||||
source_bytes = path.read_bytes()
|
||||
text = extract_text(path.name, source_bytes)
|
||||
decision = gate(text)
|
||||
if decision.disposition != _DISPOSITION_PERSIST:
|
||||
blocked = BlockedFile(
|
||||
source_file=path.name,
|
||||
disposition=decision.disposition,
|
||||
reasons=decision.reasons,
|
||||
)
|
||||
# Quarantine is a queue for the operator; everything else —
|
||||
# fail-secure, or a disposition from outside the pinned range —
|
||||
# is a refusal. Unknown values land here by construction.
|
||||
if decision.disposition == _DISPOSITION_QUARANTINE:
|
||||
quarantined.append(blocked)
|
||||
else:
|
||||
rejected.append(blocked)
|
||||
continue
|
||||
title = unicodedata.normalize("NFC", path.stem)
|
||||
content = render_inbox_concept(
|
||||
decision.sanitized_text,
|
||||
okf_type=okf_type,
|
||||
title=title,
|
||||
source_file=path.name,
|
||||
source_bytes=source_bytes,
|
||||
ingested_at=ingested_at,
|
||||
)
|
||||
except OSError as exc:
|
||||
failed.append(
|
||||
FailedFile(
|
||||
source_file=path.name,
|
||||
error=SourceError(
|
||||
f"cannot read dropped file {path.name}: {exc}", code="source_file_missing"
|
||||
),
|
||||
)
|
||||
)
|
||||
continue
|
||||
except IngestError as exc:
|
||||
failed.append(FailedFile(source_file=path.name, error=exc))
|
||||
continue
|
||||
|
||||
bundle.mkdir(parents=True, exist_ok=True)
|
||||
written = write_bytes(bundle, name, content)
|
||||
persisted.append(
|
||||
PersistedFile(source_file=path.name, path=written, reasons=decision.reasons)
|
||||
)
|
||||
|
||||
# §6 index — the last disk mutation, and only when something was written.
|
||||
if persisted:
|
||||
index_path = bundle / INDEX_NAME
|
||||
if not index_path.is_file():
|
||||
write_bytes(bundle, INDEX_NAME, "")
|
||||
for entry in persisted:
|
||||
link_in_index(
|
||||
bundle, entry.path.name, unicodedata.normalize("NFC", Path(entry.source_file).stem)
|
||||
)
|
||||
|
||||
return InboxResult(
|
||||
persisted=tuple(persisted),
|
||||
quarantined=tuple(quarantined),
|
||||
rejected=tuple(rejected),
|
||||
failed=tuple(sorted(failed, key=lambda entry: entry.source_file)),
|
||||
)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue