feat(ingest): P2/S1.b — innholdsgaten står, rundt materialize og ikke i den
Planens §3 sa at `ingest.materialize` er repoets ENE skrivepunkt på Door A, og det premisset ble felt av måling FØR bygging: `materialize` er en ren delegasjon til pinnet llm_ingestion_okf v0.3.2s `materialize_bundle`, som stager i minnet og utfører sin egen disk-fase. Det finnes ingen callback mellom de to, så en gate plassert der kunne bare kjørt ETTER at bytene landet — en opprydding, ikke en gate. Sømmen ble i stedet kopier bundelen → materialiser inn i kopien → skann det som ble generert → publiser eller forkast. Kopien er bærende, ikke bekvemmelighet: bibliotekets §3 eierskaps-skann, kollisjonsgaten mot kuratert innhold og §6 index-merge leser alle den EKSISTERENDE bundelen. Staging i tom katalog mister alle tre og publiserer en bundle uten kuraterte naboer — datatap forkledd som sikkerhetsfiks. De fire §4-beslutningene, tatt og målt: (1) ingen av guardens to preset — Origin.EXTERNAL/AUTOMATIC, fordi trust_for utleder policy fra origin alene og PRESET_USER_UPLOAD bærer en quarantine-semantikk Door A ikke har; (2) utfall per BUNDLE, diagnostikk per DOKUMENT — delvis publisering ville etterlatt bundle + index som svarer til intet manifest, men import_bundle itererer forbi første avvisning; (3) Report til log.md, aldri konsept-frontmatter, der fire golden-suiter pinner bytene; (4) mypy-override OG adapter, siden override alene gjør sømmen type-blind i stedet for type-sikker. `materialize` forblir ugatet med vilje — goldenene pinner den, og en kaller som vil ha gaten ber om den ved navn. Fem mutasjoner alle røde + grønn kontroll (hele suiten, ~120 s hver): detach gaten · la den fyre ETTER publisering · Origin.INTERNAL · tom staging-katalog · rapporter kun første avvisning. Målingen felte en VAKUØS test først: en hard injeksjon scorer fail_secure under BEGGE trust-tierene, så Origin.INTERNAL-mutasjonen lot alle tre avvisningstestene stå grønne — beslutning 1 så dekket ut uten å være testet. Båndet der tieren faktisk avgjør er høy-entropi-innhold (quarantine_review vs warn), og testen ble skrevet mot nøyaktig det før mutasjonen ble re-målt. Mutasjon 4 ble på sin side felt av KUN én test; 809 andre merket ikke at bundle-kopien forsvant. Laveste disposition er `warn`, ikke `allow` — `allow` finnes ikke i guarden. En gate skrevet mot == allow ville avvist hvert dokument som noensinne ingestes. Kriterium 5 står: demo-stdout er byte-identisk med tests/golden/demo-transcript.stdout, målt både i suiten og ved eksplisitt kjøring. shared/ er urørt. 801 -> 810 passed / 4 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DDu94KoyxAmhJsG2n63X8Q
This commit is contained in:
parent
50232fb88d
commit
c255662802
8 changed files with 575 additions and 9 deletions
|
|
@ -17,13 +17,17 @@ it re-exports the library's typed surface and keeps the historical ``materialize
|
|||
repo-local name rather than to the library's evolving one.
|
||||
|
||||
**Gating is the CALL SITE's responsibility (library README, "What is gated today: nothing").**
|
||||
Door A calls no guard function before writing to disk — ``materialize_bundle`` writes what it
|
||||
is given. The repo's own local-only posture still holds at this seam via ``allow_network``
|
||||
(default ``False``): an ``http`` source is refused fail-fast at the library's network gate
|
||||
unless a run explicitly opts in — the manifest can never grant itself network (§8, no silent
|
||||
egress). Untrusted-content scanning remains the separate, still-planned ``llm-ingestion-guard``
|
||||
wiring (see ``docs/plan/2026-07-16-llm-ingestion-guard-inclusion.md``); adopting this library
|
||||
does NOT provide it.
|
||||
The library writes what it is given, so this module owns both gates Door A has:
|
||||
|
||||
- NETWORK (§8, no silent egress) — ``allow_network`` (default ``False``): an ``http`` source is
|
||||
refused fail-fast unless a run explicitly opts in, so the manifest can never grant itself
|
||||
network access.
|
||||
- CONTENT (P2/S1.b) — :func:`materialize_gated` scans every concept an ingest run generates,
|
||||
with ``llm-ingestion-guard`` (git-pinned to ``v0.3.4``), BEFORE any of it reaches the bundle.
|
||||
|
||||
:func:`materialize` is the UNGATED form and stays that way: the four golden suites pin its bytes,
|
||||
and a caller who wants the gate asks for it by name. Nothing on the run path calls either — Door
|
||||
A is not on the 8-step loop's path (``run.py`` / ``simulation.py`` do not import this module).
|
||||
|
||||
MAF-free (D7-portable), like the rest of the context seam: the library has zero runtime
|
||||
dependencies and imports no ``agent_framework`` / ``mcp``. Guarded by
|
||||
|
|
@ -33,8 +37,11 @@ dependencies and imports no ``agent_framework`` / ``mcp``. Guarded by
|
|||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import shutil
|
||||
import socket
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from llm_ingestion_okf import (
|
||||
Extraction,
|
||||
|
|
@ -62,6 +69,12 @@ from llm_ingestion_okf.connectors import (
|
|||
)
|
||||
from llm_ingestion_okf.manifest import generated_filename
|
||||
from llm_ingestion_okf.render import render_fenced_block, render_table
|
||||
from llm_ingestion_guard.okf import (
|
||||
Channel,
|
||||
Origin,
|
||||
format_log_entry,
|
||||
import_bundle,
|
||||
)
|
||||
|
||||
# Two historical PRIVATE names the repo's existing tests bind to, re-exported so those bindings
|
||||
# survive the adoption unchanged. `_urllib_get` backs an identity assertion that the default http
|
||||
|
|
@ -85,6 +98,8 @@ HTTP_TIMEOUT_SECONDS = 30.0
|
|||
|
||||
__all__ = [
|
||||
"HTTP_TIMEOUT_SECONDS",
|
||||
"Channel",
|
||||
"ContentGateRefused",
|
||||
"Extraction",
|
||||
"FileSource",
|
||||
"HttpGet",
|
||||
|
|
@ -96,6 +111,7 @@ __all__ = [
|
|||
"ManifestV1",
|
||||
"MaterializationError",
|
||||
"NetworkGateError",
|
||||
"Origin",
|
||||
"RenderError",
|
||||
"SourceError",
|
||||
"SqlSource",
|
||||
|
|
@ -103,6 +119,7 @@ __all__ = [
|
|||
"load_manifest",
|
||||
"materialize",
|
||||
"materialize_bundle",
|
||||
"materialize_gated",
|
||||
"read_csv",
|
||||
"read_http",
|
||||
"read_sql",
|
||||
|
|
@ -200,3 +217,143 @@ def materialize(
|
|||
http_get=transport,
|
||||
).written
|
||||
)
|
||||
|
||||
|
||||
class ContentGateRefused(IngestError):
|
||||
"""The Door A content gate refused an ingest run; NOTHING was written (P2/S1.b).
|
||||
|
||||
Code: ``content_gate_refused``. ``rejected`` names every generated concept file that failed
|
||||
the gate, in sorted order — the outcome is per BUNDLE, but the diagnostics are per DOCUMENT,
|
||||
so one run surfaces every poisoned source rather than one per re-run.
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, *, rejected: tuple[str, ...]) -> None:
|
||||
super().__init__(message, code="content_gate_refused")
|
||||
self.rejected = rejected
|
||||
|
||||
|
||||
#: The lowest disposition the guard emits — a CLEAN concept scores `warn`, not `allow`
|
||||
#: (measured against v0.3.4; `Disposition` = warn < quarantine_review < fail_secure). The gate
|
||||
#: therefore accepts `warn` and refuses everything above it. Written against an `allow` tier
|
||||
#: that does not exist, the gate would refuse every document ever ingested.
|
||||
_ACCEPTED_DISPOSITION = "warn"
|
||||
|
||||
|
||||
def _stamp_line(concept: Any, ingested_at: str) -> str | None:
|
||||
"""The concept's ``log.md`` line, or ``None`` when the guard produced no stamp.
|
||||
|
||||
A hard-rejected concept (bad path, unsafe frontmatter, non-https ``resource``) carries no
|
||||
stamp — it never got far enough to be judged on content — so there is nothing to log; such a
|
||||
run is refused anyway. Isolated into its own function so the coercion to ``str`` is a typed
|
||||
boundary the type checker can see, rather than an ``Any`` flowing through a comprehension.
|
||||
"""
|
||||
stamp = concept.stamp
|
||||
if stamp is None:
|
||||
return None
|
||||
return str(format_log_entry(stamp, timestamp=ingested_at))
|
||||
|
||||
|
||||
def materialize_gated(
|
||||
manifest_path: str | Path,
|
||||
bundle_dir: str | Path,
|
||||
*,
|
||||
ingested_at: str,
|
||||
allow_network: bool = False,
|
||||
http_get: HttpGet | None = None,
|
||||
origin: Origin = Origin.EXTERNAL,
|
||||
channel: Channel = Channel.AUTOMATIC,
|
||||
) -> list[Path]:
|
||||
""":func:`materialize`, with every generated concept scanned BEFORE it reaches the bundle.
|
||||
|
||||
The Door A content gate. Same signature and same return as :func:`materialize`, plus the
|
||||
guard's provenance pair. On refusal nothing is written and ``ContentGateRefused`` carries
|
||||
the offending filenames.
|
||||
|
||||
**Why the gate is not inside** :func:`materialize` **(measured, and the reason the plan's
|
||||
premise was wrong):** ``materialize`` delegates wholly to the pinned library's
|
||||
``materialize_bundle``, which stages in memory and performs its own disk phase. No callback
|
||||
exists between the two, so a gate placed there could only run after the bytes had landed —
|
||||
a cleanup, not a gate. The seam is instead: stage into a COPY of the live bundle, scan what
|
||||
was generated, then publish the copy or discard it.
|
||||
|
||||
**The copy is load-bearing, not a convenience.** The library's §3 ownership scan, its
|
||||
collision gate against curated files, and its §6 index merge all read the EXISTING bundle
|
||||
contents. Materializing into an empty temp directory would lose all three, and publishing
|
||||
that on top of the live bundle would drop curated files and their index links — a data-loss
|
||||
bug wearing a security fix's clothing.
|
||||
|
||||
**Trust follows origin, never channel** (the guard's own rule): Door A pulls external
|
||||
sources named in a manifest, automatically, so the defaults are ``EXTERNAL`` / ``AUTOMATIC``
|
||||
— ``UNTRUSTED``. This is deliberately NOT one of the guard's two ``Policy`` presets:
|
||||
``PRESET_USER_UPLOAD`` additionally carries ``quarantine_default=True``, an upload
|
||||
semantics Door A does not have, and ``PRESET_TRUSTED_SOURCE`` would grant a trust tier a
|
||||
manifest-named external source has not earned.
|
||||
|
||||
**Only what this run generated is scanned.** Curated files already in the bundle are
|
||||
human-authored and are not re-judged here; gating the bundle READ path is a separate
|
||||
decision with a separate rationale, and it has not been taken.
|
||||
|
||||
Validation, ALWAYS — repair, NEVER: a refused document is not sanitised into the bundle,
|
||||
it stays out of it (the ``write_concept_file`` / ``promote_verdict`` precedent).
|
||||
"""
|
||||
target = Path(bundle_dir)
|
||||
with tempfile.TemporaryDirectory(prefix="po-ingest-gate-") as tmp:
|
||||
# `resolve()` because the library returns resolved paths and macOS hands out `/var/...`
|
||||
# temp dirs that are symlinks to `/private/var/...` — `relative_to` below compares the
|
||||
# two literally, so an unresolved base raises ValueError on every macOS run.
|
||||
staging = Path(tmp).resolve() / "bundle"
|
||||
if target.exists():
|
||||
shutil.copytree(target, staging)
|
||||
written = materialize(
|
||||
manifest_path,
|
||||
staging,
|
||||
ingested_at=ingested_at,
|
||||
allow_network=allow_network,
|
||||
http_get=http_get,
|
||||
)
|
||||
|
||||
# Scan ONLY this run's output, keyed by the bundle-relative path the guard expects.
|
||||
generated = {
|
||||
path.relative_to(staging).as_posix(): path.read_text(encoding="utf-8")
|
||||
for path in written
|
||||
}
|
||||
# ADAPTER at the untyped boundary (§4.4). The guard ships no `py.typed`, so everything
|
||||
# below arrives as `Any`; the mypy override alone would make this seam type-BLIND, not
|
||||
# type-safe. Each value read off a guard result object is therefore coerced to a
|
||||
# concrete type HERE, so `Any` stops at this line instead of propagating into the
|
||||
# module — and an upstream field rename fails loudly rather than type-checking happily.
|
||||
verdicts: list[tuple[str, str, str | None]] = [
|
||||
(str(concept.path), str(concept.disposition.value), _stamp_line(concept, ingested_at))
|
||||
for concept in import_bundle(generated, origin=origin, channel=channel).concepts
|
||||
]
|
||||
rejected = tuple(
|
||||
sorted(
|
||||
path for path, disposition, _ in verdicts if disposition != _ACCEPTED_DISPOSITION
|
||||
)
|
||||
)
|
||||
if rejected:
|
||||
raise ContentGateRefused(
|
||||
"Door A content gate refused "
|
||||
f"{len(rejected)} of {len(verdicts)} generated concept(s): "
|
||||
f"{', '.join(rejected)} — nothing was written to {target}",
|
||||
rejected=rejected,
|
||||
)
|
||||
|
||||
# Decision 3: the findings are recorded, never discarded — in `log.md` (OKF §7), the
|
||||
# structural update log, NEVER in the concept frontmatter. The concept bytes are the
|
||||
# pinned library's, and four golden suites pin them; a gate field injected there would
|
||||
# break all four. `ingested_at` is stamped verbatim, mirroring the rest of Door A —
|
||||
# `format_log_entry` keeps wall-clock out of the stamp itself.
|
||||
log_lines = [line for _, _, line in verdicts if line is not None]
|
||||
log_path = staging / "log.md"
|
||||
existing = log_path.read_text(encoding="utf-8") if log_path.is_file() else ""
|
||||
log_path.write_text(existing + "".join(f"{line}\n" for line in log_lines), "utf-8")
|
||||
|
||||
# Publish: the staged bundle IS the live bundle now. Replacing wholesale keeps the
|
||||
# library's index merge intact — re-deriving it here would be a second copy of §6.
|
||||
if target.exists():
|
||||
shutil.rmtree(target)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copytree(staging, target)
|
||||
|
||||
return [target / path.relative_to(staging) for path in written]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue