open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(materialize): add §5 materialization with collision gate and replacement

Browse source 
TDD step 5: materialize_bundle takes three explicit inputs (manifest,
bundle dir, regex-validated ingested_at — no wall-clock default) and
stages every extraction in memory before the first disk mutation. The
§3 collision gate refuses to overwrite any file without the ingest
stamp, before any mutation; replacement removes exactly the stamped
set. Frontmatter is the seven §5 keys in spec order with whitespace
collapse; output is LF-only raw bytes with one trailing newline; the
provenance stamp hashes the same bytes that are parsed. Fresh index.md
gets bundle_summary plus idempotent links in extraction order (§6
creation path; preserve/removal rules come next). The §8 network gate
refuses http fail-fast without the per-run opt-in; the http transport
itself lands in step 7.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
This commit is contained in:
Kjell Tore Guttormsen 2026-07-16 20:02:35 +02:00
commit 0ff946696c
5 changed files with 593 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

8
src/llm_ingestion_okf/connectors.py
View file

@ -17,11 +17,11 @@ from .errors import SourceError
from .render import sql_value_to_text
def _safe_resolve(root: Path, relative: str) -> Path:
def safe_resolve(root: Path, relative: str) -> Path:
"""Resolve `relative` against `root`, fail-closed (the OKF path rule).
Raises SourceError if the resolved path escapes `root` — `..` traversal,
an absolute query path, a symlink escape, or a prefix-collision sibling
an absolute path, a symlink escape, or a prefix-collision sibling
(`/a/data-evil` vs `/a/data`; commonpath on canonical paths catches what
a naive startswith would not).
"""
@ -33,7 +33,7 @@ def _safe_resolve(root: Path, relative: str) -> Path:
# Different drives / mixed absolute-relative -> not within.
within = False
if not within:
raise SourceError(f"extraction query escapes the source root: {relative!r}")
raise SourceError(f"path escapes its root directory: {relative!r}")
return Path(candidate)
@ -48,7 +48,7 @@ def read_csv(root: str | Path, query: str, *, max_rows: int) -> tuple[list[str],
root_path = Path(root)
if not root_path.is_dir():
raise SourceError(f"file-source root is not a directory: {root_path}")
resolved = _safe_resolve(root_path, query)
resolved = safe_resolve(root_path, query)
if not resolved.is_file():
raise SourceError(f"extraction query does not resolve to a file: {query!r}")
with resolved.open(encoding="utf-8-sig", newline="") as handle: