feat(ingest): I3 — D7-speil av ingest (filkatalog/CSV), bygget fra commons-spec alene

Speiler MAF I2 fra shared/ingest-spec.md alene: manifest → CSV-konnektor →
materialisert OKF-bundle, byte-identisk med den delte golden-fasiten.

- ingest.py: ManifestContract (pydantic, fail-fast, file-kilde, verdict-reservasjon
  §3, id-grammatikk, max_rows), CSV-konnektor (boundary-checked fail-closed),
  materialisering (§5-frontmatter eksakt rekkefølge, markdown-tabell m/ escaping,
  LF-only, SHA-256 manifest-stamp), index-generering (§6), replacement §3/§5.
- okf.py: _parse_index_entry — tolererer frontmatterløs index (method-spec §3:
  index rendres via body = summary, ikke som typet concept-fil). Golden var
  spec-konform; D7-okf var strengere enn standarden. Scoped: non-index concept-
  filer krever fortsatt type (honesty-test).
- examples/ingest-golden-file/: repo-lokal golden (byte-frossen kopi av I2s fasit).
- Speiltester (I2s load-bearing-sett, alle detach-bevist røde): golden byte-fasit
  + mutasjonskontroller · provenance/navigability/verdict-reservasjon/re-ingest-safety
  · kontrakt fail-fast/max_rows/boundary/kollisjon · spec-integritet §11.
- docs/2026-07-04-I3-brief.md: brief + de to operatør-avgjorte beslutningene.

Suite 239 passed uten nøkkel/nettverk (189 + 50 nye) · ruff + mypy --strict rene.

[skip-docs] README + docs/extending.md er bevisst utsatt til I7 per sesjonsplan
(programmet batcher ingest-doc der, avgrenset til det D7 faktisk har — CSV nå,
SQL/HTTP senere). Endringen er dokumentert i docs/2026-07-04-I3-brief.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017MM6BWb1hWmJZuXFZ7rjxT

src/portfolio_optimiser_claude/ingest.py

@@ -0,0 +1,299 @@
+"""Deterministic ingest layer: manifest → file/CSV connector → OKF bundle (ingest-spec).
+
+An addition IN FRONT of the loop (ingest-spec §1): a deterministic step that couples the
+framework to a real data source and materializes the extract as an OKF knowledge bundle,
+which the existing 8-step loop then consumes UNCHANGED. Zero model calls; no network.
+
+D7 scope (I3, mirror of MAF I2): the ``file`` source type (a local CSV catalogue). The
+``sql`` and ``http`` source types are later work — a manifest naming them is rejected
+fail-fast at validation, never silently accepted.
+
+Contract discipline mirrors ``contracts.py``: the manifest is schema-validated fail-fast
+BEFORE any source call (§4), and queries are declarative configuration, never evaluated as
+code. The provenance frontmatter layer (§7) is ITS OWN contract — separate from the method
+spec's §9 proposal provenance, never mixed.
+"""
+
+from __future__ import annotations
+
+import csv
+import hashlib
+import json
+import logging
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Literal
+
+from pydantic import BaseModel, Field, field_validator
+
+_logger = logging.getLogger(__name__)
+
+_ID_RE = re.compile(r"^[a-z0-9][a-z0-9-]*$")
+_VERDICT_TYPE = "verdict"
+_INGEST_PREFIX = "ingest-"
+_STAMP_HASH_LEN = 16
+# An index cross-link owned by ingest: ``- [label](ingest-{id}.md)`` (§6). Only these
+# are managed on re-materialization; curated and promoted links are preserved verbatim.
+_INGEST_LINK_RE = re.compile(r"\]\((ingest-[a-z0-9][a-z0-9-]*\.md)\)")
+_CROSSLINK_RE = re.compile(r"\]\(([^)]+\.md)\)")
+
+# Frontmatter keys, in the exact §5 order.
+_FRONTMATTER_ORDER = (
+    "type",
+    "title",
+    "source_system",
+    "source_query",
+    "ingested_at",
+    "ingest_manifest",
+    "generated",
+)
+
+
+class FileSource(BaseModel):
+    """A local file catalogue (``source.type == "file"``, ingest-spec §4)."""
+
+    type: Literal["file"]
+    id: str
+    root: str
+
+    @field_validator("id")
+    @classmethod
+    def _id_grammar(cls, value: str) -> str:
+        if not _ID_RE.match(value):
+            raise ValueError(f"source.id {value!r} must match [a-z0-9][a-z0-9-]*")
+        return value
+
+
+class Extraction(BaseModel):
+    """One extraction description (ingest-spec §4). ``max_rows`` is a required cap (§8)."""
+
+    id: str
+    title: str
+    query: str
+    okf_type: str
+    max_rows: int = Field(gt=0)
+
+    @field_validator("id")
+    @classmethod
+    def _id_grammar(cls, value: str) -> str:
+        if not _ID_RE.match(value):
+            raise ValueError(f"extraction.id {value!r} must match [a-z0-9][a-z0-9-]*")
+        return value
+
+    @field_validator("title")
+    @classmethod
+    def _title_single_line(cls, value: str) -> str:
+        if "\n" in value or "\r" in value:
+            raise ValueError("extraction.title must be single-line")
+        return value
+
+    @field_validator("okf_type")
+    @classmethod
+    def _okf_type_not_verdict(cls, value: str) -> str:
+        # §3 verdict-layer reservation: the promotion gate is the ONLY path into the
+        # verdict layer — an ingest mapping to type: verdict would inject machine-made
+        # "approved" verdicts around the gate. Case-insensitive, fail-fast.
+        if value.lower() == _VERDICT_TYPE:
+            raise ValueError("okf_type MUST NOT be 'verdict' (verdict layer is reserved, §3)")
+        return value
+
+
+class ManifestContract(BaseModel):
+    """The ingest manifest (§4) — schema-validated fail-fast before any source call."""
+
+    manifest_version: Literal[1]
+    source: FileSource
+    bundle_summary: str
+    extractions: list[Extraction] = Field(min_length=1)
+
+    @field_validator("extractions")
+    @classmethod
+    def _unique_extraction_ids(cls, value: list[Extraction]) -> list[Extraction]:
+        ids = [e.id for e in value]
+        if len(ids) != len(set(ids)):
+            raise ValueError("extraction ids must be unique within the manifest")
+        return value
+
+
+@dataclass(frozen=True)
+class LoadedManifest:
+    """A validated manifest plus its provenance stamp and on-disk location."""
+
+    contract: ManifestContract
+    stamp: str  # ``{manifest stem}@{sha256(raw bytes)[:16]}`` (§5)
+    path: Path
+
+
+def load_manifest(manifest_path: Path) -> LoadedManifest:
+    """Validate the manifest fail-fast (§4) and compute its provenance stamp (§5).
+
+    Raises ``pydantic.ValidationError`` on a malformed manifest — never starts a run on a
+    bad contract. The stamp is over the manifest's RAW bytes, so any edit re-stamps.
+    """
+    raw = manifest_path.read_bytes()
+    contract = ManifestContract(**json.loads(raw))
+    digest = hashlib.sha256(raw).hexdigest()[:_STAMP_HASH_LEN]
+    stamp = f"{manifest_path.stem}@{digest}"
+    return LoadedManifest(contract=contract, stamp=stamp, path=manifest_path)
+
+
+def _escape_cell(value: str) -> str:
+    """Render a cell value: text verbatim with §5 escaping.
+
+    Backslash FIRST (so the pipe escape it introduces is not re-escaped), then pipe,
+    then any newline collapsed to a single space. For the ``file`` source every cell is
+    a ``str`` — the integer/float/NULL typing rules (§5) belong to the ``sql`` source.
+    """
+    escaped = value.replace("\\", "\\\\").replace("|", "\\|")
+    escaped = escaped.replace("\r\n", " ").replace("\r", " ").replace("\n", " ")
+    return escaped
+
+
+def _render_table(rows: list[list[str]]) -> str:
+    """Header row, separator, data rows — GitHub-flavoured markdown table (§5)."""
+    header, *data = rows
+    lines = [
+        "| " + " | ".join(_escape_cell(c) for c in header) + " |",
+        "| " + " | ".join("---" for _ in header) + " |",
+    ]
+    for row in data:
+        lines.append("| " + " | ".join(_escape_cell(c) for c in row) + " |")
+    return "\n".join(lines)
+
+
+def _read_csv(csv_path: Path, max_rows: int) -> list[list[str]]:
+    """Read a CSV (first row = header). Enforce ``max_rows`` fail-fast on DATA rows (§8)."""
+    with csv_path.open(encoding="utf-8", newline="") as handle:
+        rows = list(csv.reader(handle))
+    data_rows = len(rows) - 1 if rows else 0
+    if data_rows > max_rows:
+        raise ValueError(
+            f"extraction exceeds max_rows: {data_rows} > {max_rows} for {csv_path.name}"
+        )
+    return rows
+
+
+def _resolve_within(root: Path, relative: str) -> Path:
+    """Resolve ``relative`` under ``root``, boundary-checked fail-closed (the OKF path rule)."""
+    resolved = (root / relative).resolve()
+    if not resolved.is_relative_to(root.resolve()):
+        raise ValueError(f"extraction path {relative!r} escapes the source root")
+    return resolved
+
+
+def _frontmatter_block(values: dict[str, str]) -> str:
+    return "---\n" + "".join(f"{k}: {values[k]}\n" for k in _FRONTMATTER_ORDER) + "---"
+
+
+def _concept_file(
+    extraction: Extraction, source_id: str, ingested_at: str, stamp: str, body: str
+) -> str:
+    frontmatter = _frontmatter_block(
+        {
+            "type": extraction.okf_type,
+            "title": extraction.title,
+            "source_system": source_id,
+            "source_query": _escape_ws(extraction.query),
+            "ingested_at": ingested_at,
+            "ingest_manifest": stamp,
+            "generated": "true",
+        }
+    )
+    return f"{frontmatter}\n\n{body}\n"
+
+
+def _escape_ws(value: str) -> str:
+    """Collapse whitespace runs (incl. newlines) to single spaces (§5) — single-line values."""
+    return " ".join(value.split())
+
+
+def _is_ingest_stamped(md_path: Path) -> bool:
+    """True iff the file carries the ingest stamp: ``generated: true`` + an ``ingest_manifest``."""
+    frontmatter: dict[str, str] = {}
+    lines = md_path.read_text(encoding="utf-8").splitlines()
+    if not lines or lines[0].strip() != "---":
+        return False
+    for line in lines[1:]:
+        if line.strip() == "---":
+            break
+        key, sep, val = line.partition(":")
+        if sep:
+            frontmatter[key.strip()] = val.strip()
+    return frontmatter.get("generated") == "true" and "ingest_manifest" in frontmatter
+
+
+def _update_index(bundle_dir: Path, bundle_summary: str, extractions: list[Extraction]) -> None:
+    """Create or update ``index.md`` (§6).
+
+    A fresh index carries ``bundle_summary`` as its body; an existing index keeps every
+    line it does not manage byte for byte. Ingest cross-links (targets ``ingest-*.md``)
+    are the ONLY managed lines: stale ones are dropped and the current set re-appended in
+    extraction order, idempotently by target — curated and promoted links are preserved
+    (so a promoted verdict's index link survives re-ingest, load-bearing §11).
+    """
+    index_path = bundle_dir / "index.md"
+    if index_path.is_file():
+        kept = [
+            line
+            for line in index_path.read_text(encoding="utf-8").splitlines()
+            if not _INGEST_LINK_RE.search(line)
+        ]
+    else:
+        kept = [bundle_summary]
+    present = {t for line in kept for t in _CROSSLINK_RE.findall(line)}
+    lines = list(kept)
+    for extraction in extractions:
+        target = f"{_INGEST_PREFIX}{extraction.id}.md"
+        if target not in present:
+            lines.append(f"- [{extraction.title}]({target})")
+            present.add(target)
+    index_path.write_text("\n".join(lines) + "\n", encoding="utf-8")
+
+
+def materialize(manifest_path: Path, bundle_dir: Path, ingested_at: str) -> list[Path]:
+    """Materialize the manifest's extractions into ``bundle_dir`` (deterministic, §5).
+
+    ``ingested_at`` is an EXPLICIT required argument (no wall-clock default, §5) — the ISO
+    timestamp stamped verbatim into every generated file, which is what makes extractions
+    bit-deterministic. Returns the generated concept-file paths in extraction order.
+
+    Replacement semantics (§3, §5): first every ingest-stamped file is removed, then the
+    new set is written (a name colliding with a NON-stamped file fails — curated content is
+    never overwritten), then the index is updated. Curated and promoted files always survive.
+    """
+    loaded = load_manifest(manifest_path)
+    source = loaded.contract.source
+    root = manifest_path.parent / source.root
+
+    bundle_dir.mkdir(parents=True, exist_ok=True)
+
+    # 1. Remove exactly the files carrying the ingest stamp (§5) — never curated/promoted.
+    for md_path in sorted(bundle_dir.glob("*.md")):
+        if md_path.name != "index.md" and _is_ingest_stamped(md_path):
+            md_path.unlink()
+
+    # 2. Write the new set.
+    generated: list[Path] = []
+    for extraction in loaded.contract.extractions:
+        csv_path = _resolve_within(root, extraction.query)
+        rows = _read_csv(csv_path, extraction.max_rows)
+        _logger.info(
+            "ingest source=%s extraction=%s rows=%d",
+            source.id,
+            extraction.id,
+            max(len(rows) - 1, 0),
+        )
+        target = bundle_dir / f"{_INGEST_PREFIX}{extraction.id}.md"
+        if target.exists():
+            # Post-removal, an existing target is a NON-stamped collision (§3): never overwrite.
+            raise ValueError(f"generated name {target.name} collides with a non-ingest file")
+        target.write_text(
+            _concept_file(extraction, source.id, ingested_at, loaded.stamp, _render_table(rows)),
+            encoding="utf-8",
+        )
+        generated.append(target)
+
+    # 3. Update the index (§6).
+    _update_index(bundle_dir, loaded.contract.bundle_summary, loaded.contract.extractions)
+    return generated

src/portfolio_optimiser_claude/okf.py

@@ -18,6 +18,7 @@ from dataclasses import dataclass
 from pathlib import Path
 
 _INDEX_FILENAME = "index.md"
+_INDEX_TYPE = "index"
 _VERDICT_TYPE = "verdict"
 # (reference) the intra-bundle cross-link pattern, per §3 Step 1.
 _CROSSLINK_PATTERN = re.compile(r"\]\(([^)]+\.md)\)")
@@ -70,19 +71,40 @@ def parse_concept_file(path: Path) -> ConceptFile:
     )
 
 
+def _parse_index_entry(index_path: Path) -> ConceptFile:
+    """Parse the bundle entry point, tolerating a frontmatter-less index (§3 Step 1).
+
+    method-spec §3 Step 1 renders the index as "the index body (the summary)" and
+    treats every OTHER file as a "non-index concept file" (`## {type}: {title}`
+    section) — so the index is NOT a concept file, and it MAY omit frontmatter: a
+    generated index carrying only ``bundle_summary`` + cross-links (ingest spec §6,
+    frozen in the ingest golden) is valid. A frontmatter-ful index (the curated
+    convention, ``type: index``) still parses normally, its extra fields preserved.
+    When frontmatter is absent, the whole file is the body and ``type`` defaults to
+    ``index`` so downstream ``.type`` reads never raise.
+    """
+    text = index_path.read_text(encoding="utf-8")
+    lines = text.splitlines()
+    if lines and lines[0].strip() == "---":
+        return parse_concept_file(index_path)
+    return ConceptFile(path=index_path, frontmatter={"type": _INDEX_TYPE}, body=text.strip())
+
+
 def navigate_bundle(bundle_dir: Path) -> list[ConceptFile]:
     """Navigate from ``index.md`` — deterministic order: index first, links first-seen.
 
     Targets containing a path separator are out-of-bundle and skipped; resolution is
     boundary-checked against the bundle directory (fail-closed); broken links are
-    skipped, never raised. Repeated links are de-duplicated.
+    skipped, never raised. Repeated links are de-duplicated. The index entry point
+    may omit frontmatter (``_parse_index_entry``); non-index concept files still
+    require ``type``.
     """
     index_path = bundle_dir / _INDEX_FILENAME
     if not index_path.is_file():
         raise FileNotFoundError(
             f"bundle has no entry point: missing {_INDEX_FILENAME} in {bundle_dir}"
         )
-    index = parse_concept_file(index_path)
+    index = _parse_index_entry(index_path)
     bundle_root = bundle_dir.resolve()
     concepts = [index]
     seen = {_INDEX_FILENAME}