feat(ingest): I3 — D7-speil av ingest (filkatalog/CSV), bygget fra commons-spec alene

Speiler MAF I2 fra shared/ingest-spec.md alene: manifest → CSV-konnektor →
materialisert OKF-bundle, byte-identisk med den delte golden-fasiten.

- ingest.py: ManifestContract (pydantic, fail-fast, file-kilde, verdict-reservasjon
  §3, id-grammatikk, max_rows), CSV-konnektor (boundary-checked fail-closed),
  materialisering (§5-frontmatter eksakt rekkefølge, markdown-tabell m/ escaping,
  LF-only, SHA-256 manifest-stamp), index-generering (§6), replacement §3/§5.
- okf.py: _parse_index_entry — tolererer frontmatterløs index (method-spec §3:
  index rendres via body = summary, ikke som typet concept-fil). Golden var
  spec-konform; D7-okf var strengere enn standarden. Scoped: non-index concept-
  filer krever fortsatt type (honesty-test).
- examples/ingest-golden-file/: repo-lokal golden (byte-frossen kopi av I2s fasit).
- Speiltester (I2s load-bearing-sett, alle detach-bevist røde): golden byte-fasit
  + mutasjonskontroller · provenance/navigability/verdict-reservasjon/re-ingest-safety
  · kontrakt fail-fast/max_rows/boundary/kollisjon · spec-integritet §11.
- docs/2026-07-04-I3-brief.md: brief + de to operatør-avgjorte beslutningene.

Suite 239 passed uten nøkkel/nettverk (189 + 50 nye) · ruff + mypy --strict rene.

[skip-docs] README + docs/extending.md er bevisst utsatt til I7 per sesjonsplan
(programmet batcher ingest-doc der, avgrenset til det D7 faktisk har — CSV nå,
SQL/HTTP senere). Endringen er dokumentert i docs/2026-07-04-I3-brief.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017MM6BWb1hWmJZuXFZ7rjxT

docs/2026-07-04-I3-brief.md

@@ -0,0 +1,69 @@
+# I3 — D7-speil av ingest (filkatalog/CSV) — brief
+
+> Forankret i hovedrepoets sesjonsplan
+> `~/repos/portfolio-optimiser/docs/plan/2026-07-03-sesjonsplan-ingest.md` §I3 og dette
+> repoets STATE. Bygget **fra `shared/ingest-spec.md` alene** (subtree-pull fra commons) —
+> aldri ved reverse-engineering av MAF-koden. Golden-**bytene** (den delte fasiten) er lest,
+> siden spec §11 gjør dem til konformanskravet begge stacker reproduserer.
+
+## Mål (§I3)
+
+Samme ingest-kontrakt konsumert **uendret** fra commons i dette søskenrepoet — beviser at
+ingest-spec-en faktisk er delt: D7s uavhengige impl reproduserer MAF I2s golden **byte-identisk**
+fra spec alene. Scope = `file`-kildetypen (lokal CSV-katalog). `sql` = senere (I5), `http` =
+ikke i D7 (målbildet: HTTP/MCP demonstreres MAF-only, I6).
+
+## To beslutninger (operatør-avgjort denne økta)
+
+1. **Golden-fasit: repo-lokal kopi.** I2 la ingest-golden repo-lokalt (`examples/ingest-golden-file`,
+   ulikt method-golden i `shared/examples/`). D7 speiler mønsteret: kopi i dette repoets
+   `examples/ingest-golden-file/`, speiltest peker dit. Holder I3 som én lukket økt, ingen
+   commons-runde. (Alternativ — løfte golden til commons — avvist: egen GATET runde + avvik fra
+   I2-mønsteret.)
+
+2. **okf relaksert for frontmatterløs index (følg standarden).** Funn: den delte golden-ens
+   `index.md` har **ingen** frontmatter; D7s `okf.navigate_bundle` krevde det → kunne ikke både
+   reprodusere golden byte-identisk OG navigere den med uendret okf. **method-spec §3 Step 1 er
+   avgjørende:** «the rendered read-context is *the index body (the summary)* followed by each
+   *non-index concept file* as a `## {type}: {title}` section» — standarden behandler index via
+   sin *body*, ikke som en typet concept-fil. Golden-en er altså spec-konform; D7s okf var
+   **strengere enn standarden**. Fix: `okf` tolererer nå frontmatterløs index (hele fila = body,
+   `type` defaulter til `index`); frontmatter-ful kuratert index parses uendret; non-index
+   concept-filer krever fortsatt `type` (relaksasjonen er scoped, bevist av egen honesty-test).
+   (Alternativ — «typet index overalt» via commons-RFC + MAF golden-regen — avvist: omskriver
+   standarden for å romme en D7-quirk, endrer frossen fasit, flerrepo GATET.)
+
+## Leveranse
+
+- `examples/ingest-golden-file/` — repo-lokal golden (kopi av I2s, byte-frossen).
+- `src/portfolio_optimiser_claude/ingest.py` — `ManifestContract` (pydantic, fail-fast,
+  `file`-kilde, verdict-reservasjon §3, id-grammatikk, `max_rows`>0), CSV-konnektor
+  (boundary-checked fail-closed), materialisering (§5-frontmatter i eksakt rekkefølge,
+  markdown-tabell m/ §5-escaping, LF-only, SHA-256 manifest-stamp), index-generering (§6),
+  replacement-semantikk (§3/§5).
+- `src/…/okf.py` — `_parse_index_entry` (frontmatterløs index-toleranse, method-spec §3).
+- Speiltester (I2s load-bearing-sett): `test_ingest_golden.py` (byte-fasit + 3 mutasjonskontroller),
+  `test_ingest_loadbearing.py` (provenance / navigability / verdict-reservasjon / re-ingest-safety),
+  `test_ingest.py` (kontrakt fail-fast, max_rows, boundary, kollisjon, escaping),
+  `test_ingest_spec_loadbearing.py` (spec-integritet §11: fil finnes · navngir ingen toolkit ·
+  dokumenterer hvert kontrakt-felt) — D7-analogen til MAFs I1-guard.
+
+## SDK-kroker (STATE steg 4 — søk-først)
+
+I3 (filkatalog/CSV) introduserer **ingen SDK-flate**: ingest er deterministisk, null modellkall.
+Søk-først SDK-krok-reverifisering (API-drift på `create_sdk_mcp_server`) er relevant for I6
+(HTTP/MCP), ikke I3 — utsatt dit kroken faktisk brukes. Eksisterende SDK-klient (`sdk_client.py`,
+`test_sdk_isolation.py`) uendret og grønn i suiten.
+
+## Verifisering (kjørt, grønn)
+
+- Golden **byte-identisk** konsumert (5/5, inkl. idempotens + mutasjonskontroller).
+- Speiltester **røde ved detach** — alle fire sømmer bevist load-bearing (detach-bevis-skript).
+- Søskensuite **grønn uten nøkkel/nettverk**: 239 passed (189 + 50 nye).
+- `ruff check` + `ruff format` rene · `mypy --strict src`: no issues (16 filer).
+
+## Åpent (forslag, ikke utført — utenfor I3-scope)
+
+- **Standard-evolusjon:** vil operatøren senere ha «alle bundle-filer er typede concept-filer»
+  (typet index overalt) som bevisst OKF-endring, er det en egen koordinert commons-RFC (begge
+  specs, begge golden, begge okf). Dagens permissive okf er forover-kompatibel med det.

examples/ingest-golden-file/expected-bundle/index.md

@@ -0,0 +1,3 @@
+Golden extraction case: two CSV extracts from a local project archive (file source type).
+- [Project costs](ingest-costs.md)
+- [Escaping edge cases](ingest-edge.md)

examples/ingest-golden-file/expected-bundle/ingest-costs.md

@@ -0,0 +1,14 @@
+---
+type: dataset
+title: Project costs
+source_system: prosjekt-arkiv
+source_query: costs.csv
+ingested_at: 2026-07-03T12:00:00Z
+ingest_manifest: manifest@24109f486b9ef708
+generated: true
+---
+
+| item | cost_nok | note |
+| --- | --- | --- |
+| led-retrofit | 120000 | ROI under 2 years |
+| hvac-tuning | 80000 | requires vendor quote |

examples/ingest-golden-file/expected-bundle/ingest-edge.md

@@ -0,0 +1,18 @@
+---
+type: reference
+title: Escaping edge cases
+source_system: prosjekt-arkiv
+source_query: edge.csv
+ingested_at: 2026-07-03T12:00:00Z
+ingest_manifest: manifest@24109f486b9ef708
+generated: true
+---
+
+| id | value |
+| --- | --- |
+| backslash | a\\b |
+| pipe | a\|b |
+| newline | x y |
+| both | \\\| |
+| zeros | 007 |
+| decimal | 1.50 |

examples/ingest-golden-file/fixture/costs.csv

@@ -0,0 +1,3 @@
+item,cost_nok,note
+led-retrofit,120000,ROI under 2 years
+hvac-tuning,80000,requires vendor quote

examples/ingest-golden-file/fixture/edge.csv

@@ -0,0 +1,8 @@
+id,value
+backslash,a\b
+pipe,a|b
+newline,"x
+y"
+both,\|
+zeros,007
+decimal,1.50

examples/ingest-golden-file/ingested-at.txt

@@ -0,0 +1 @@
+2026-07-03T12:00:00Z

examples/ingest-golden-file/manifest.json

@@ -0,0 +1,21 @@
+{
+  "manifest_version": 1,
+  "source": {"type": "file", "id": "prosjekt-arkiv", "root": "fixture"},
+  "bundle_summary": "Golden extraction case: two CSV extracts from a local project archive (file source type).",
+  "extractions": [
+    {
+      "id": "costs",
+      "title": "Project costs",
+      "query": "costs.csv",
+      "okf_type": "dataset",
+      "max_rows": 10
+    },
+    {
+      "id": "edge",
+      "title": "Escaping edge cases",
+      "query": "edge.csv",
+      "okf_type": "reference",
+      "max_rows": 10
+    }
+  ]
+}

src/portfolio_optimiser_claude/ingest.py

@@ -0,0 +1,299 @@
+"""Deterministic ingest layer: manifest → file/CSV connector → OKF bundle (ingest-spec).
+
+An addition IN FRONT of the loop (ingest-spec §1): a deterministic step that couples the
+framework to a real data source and materializes the extract as an OKF knowledge bundle,
+which the existing 8-step loop then consumes UNCHANGED. Zero model calls; no network.
+
+D7 scope (I3, mirror of MAF I2): the ``file`` source type (a local CSV catalogue). The
+``sql`` and ``http`` source types are later work — a manifest naming them is rejected
+fail-fast at validation, never silently accepted.
+
+Contract discipline mirrors ``contracts.py``: the manifest is schema-validated fail-fast
+BEFORE any source call (§4), and queries are declarative configuration, never evaluated as
+code. The provenance frontmatter layer (§7) is ITS OWN contract — separate from the method
+spec's §9 proposal provenance, never mixed.
+"""
+
+from __future__ import annotations
+
+import csv
+import hashlib
+import json
+import logging
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Literal
+
+from pydantic import BaseModel, Field, field_validator
+
+_logger = logging.getLogger(__name__)
+
+_ID_RE = re.compile(r"^[a-z0-9][a-z0-9-]*$")
+_VERDICT_TYPE = "verdict"
+_INGEST_PREFIX = "ingest-"
+_STAMP_HASH_LEN = 16
+# An index cross-link owned by ingest: ``- [label](ingest-{id}.md)`` (§6). Only these
+# are managed on re-materialization; curated and promoted links are preserved verbatim.
+_INGEST_LINK_RE = re.compile(r"\]\((ingest-[a-z0-9][a-z0-9-]*\.md)\)")
+_CROSSLINK_RE = re.compile(r"\]\(([^)]+\.md)\)")
+
+# Frontmatter keys, in the exact §5 order.
+_FRONTMATTER_ORDER = (
+    "type",
+    "title",
+    "source_system",
+    "source_query",
+    "ingested_at",
+    "ingest_manifest",
+    "generated",
+)
+
+
+class FileSource(BaseModel):
+    """A local file catalogue (``source.type == "file"``, ingest-spec §4)."""
+
+    type: Literal["file"]
+    id: str
+    root: str
+
+    @field_validator("id")
+    @classmethod
+    def _id_grammar(cls, value: str) -> str:
+        if not _ID_RE.match(value):
+            raise ValueError(f"source.id {value!r} must match [a-z0-9][a-z0-9-]*")
+        return value
+
+
+class Extraction(BaseModel):
+    """One extraction description (ingest-spec §4). ``max_rows`` is a required cap (§8)."""
+
+    id: str
+    title: str
+    query: str
+    okf_type: str
+    max_rows: int = Field(gt=0)
+
+    @field_validator("id")
+    @classmethod
+    def _id_grammar(cls, value: str) -> str:
+        if not _ID_RE.match(value):
+            raise ValueError(f"extraction.id {value!r} must match [a-z0-9][a-z0-9-]*")
+        return value
+
+    @field_validator("title")
+    @classmethod
+    def _title_single_line(cls, value: str) -> str:
+        if "\n" in value or "\r" in value:
+            raise ValueError("extraction.title must be single-line")
+        return value
+
+    @field_validator("okf_type")
+    @classmethod
+    def _okf_type_not_verdict(cls, value: str) -> str:
+        # §3 verdict-layer reservation: the promotion gate is the ONLY path into the
+        # verdict layer — an ingest mapping to type: verdict would inject machine-made
+        # "approved" verdicts around the gate. Case-insensitive, fail-fast.
+        if value.lower() == _VERDICT_TYPE:
+            raise ValueError("okf_type MUST NOT be 'verdict' (verdict layer is reserved, §3)")
+        return value
+
+
+class ManifestContract(BaseModel):
+    """The ingest manifest (§4) — schema-validated fail-fast before any source call."""
+
+    manifest_version: Literal[1]
+    source: FileSource
+    bundle_summary: str
+    extractions: list[Extraction] = Field(min_length=1)
+
+    @field_validator("extractions")
+    @classmethod
+    def _unique_extraction_ids(cls, value: list[Extraction]) -> list[Extraction]:
+        ids = [e.id for e in value]
+        if len(ids) != len(set(ids)):
+            raise ValueError("extraction ids must be unique within the manifest")
+        return value
+
+
+@dataclass(frozen=True)
+class LoadedManifest:
+    """A validated manifest plus its provenance stamp and on-disk location."""
+
+    contract: ManifestContract
+    stamp: str  # ``{manifest stem}@{sha256(raw bytes)[:16]}`` (§5)
+    path: Path
+
+
+def load_manifest(manifest_path: Path) -> LoadedManifest:
+    """Validate the manifest fail-fast (§4) and compute its provenance stamp (§5).
+
+    Raises ``pydantic.ValidationError`` on a malformed manifest — never starts a run on a
+    bad contract. The stamp is over the manifest's RAW bytes, so any edit re-stamps.
+    """
+    raw = manifest_path.read_bytes()
+    contract = ManifestContract(**json.loads(raw))
+    digest = hashlib.sha256(raw).hexdigest()[:_STAMP_HASH_LEN]
+    stamp = f"{manifest_path.stem}@{digest}"
+    return LoadedManifest(contract=contract, stamp=stamp, path=manifest_path)
+
+
+def _escape_cell(value: str) -> str:
+    """Render a cell value: text verbatim with §5 escaping.
+
+    Backslash FIRST (so the pipe escape it introduces is not re-escaped), then pipe,
+    then any newline collapsed to a single space. For the ``file`` source every cell is
+    a ``str`` — the integer/float/NULL typing rules (§5) belong to the ``sql`` source.
+    """
+    escaped = value.replace("\\", "\\\\").replace("|", "\\|")
+    escaped = escaped.replace("\r\n", " ").replace("\r", " ").replace("\n", " ")
+    return escaped
+
+
+def _render_table(rows: list[list[str]]) -> str:
+    """Header row, separator, data rows — GitHub-flavoured markdown table (§5)."""
+    header, *data = rows
+    lines = [
+        "| " + " | ".join(_escape_cell(c) for c in header) + " |",
+        "| " + " | ".join("---" for _ in header) + " |",
+    ]
+    for row in data:
+        lines.append("| " + " | ".join(_escape_cell(c) for c in row) + " |")
+    return "\n".join(lines)
+
+
+def _read_csv(csv_path: Path, max_rows: int) -> list[list[str]]:
+    """Read a CSV (first row = header). Enforce ``max_rows`` fail-fast on DATA rows (§8)."""
+    with csv_path.open(encoding="utf-8", newline="") as handle:
+        rows = list(csv.reader(handle))
+    data_rows = len(rows) - 1 if rows else 0
+    if data_rows > max_rows:
+        raise ValueError(
+            f"extraction exceeds max_rows: {data_rows} > {max_rows} for {csv_path.name}"
+        )
+    return rows
+
+
+def _resolve_within(root: Path, relative: str) -> Path:
+    """Resolve ``relative`` under ``root``, boundary-checked fail-closed (the OKF path rule)."""
+    resolved = (root / relative).resolve()
+    if not resolved.is_relative_to(root.resolve()):
+        raise ValueError(f"extraction path {relative!r} escapes the source root")
+    return resolved
+
+
+def _frontmatter_block(values: dict[str, str]) -> str:
+    return "---\n" + "".join(f"{k}: {values[k]}\n" for k in _FRONTMATTER_ORDER) + "---"
+
+
+def _concept_file(
+    extraction: Extraction, source_id: str, ingested_at: str, stamp: str, body: str
+) -> str:
+    frontmatter = _frontmatter_block(
+        {
+            "type": extraction.okf_type,
+            "title": extraction.title,
+            "source_system": source_id,
+            "source_query": _escape_ws(extraction.query),
+            "ingested_at": ingested_at,
+            "ingest_manifest": stamp,
+            "generated": "true",
+        }
+    )
+    return f"{frontmatter}\n\n{body}\n"
+
+
+def _escape_ws(value: str) -> str:
+    """Collapse whitespace runs (incl. newlines) to single spaces (§5) — single-line values."""
+    return " ".join(value.split())
+
+
+def _is_ingest_stamped(md_path: Path) -> bool:
+    """True iff the file carries the ingest stamp: ``generated: true`` + an ``ingest_manifest``."""
+    frontmatter: dict[str, str] = {}
+    lines = md_path.read_text(encoding="utf-8").splitlines()
+    if not lines or lines[0].strip() != "---":
+        return False
+    for line in lines[1:]:
+        if line.strip() == "---":
+            break
+        key, sep, val = line.partition(":")
+        if sep:
+            frontmatter[key.strip()] = val.strip()
+    return frontmatter.get("generated") == "true" and "ingest_manifest" in frontmatter
+
+
+def _update_index(bundle_dir: Path, bundle_summary: str, extractions: list[Extraction]) -> None:
+    """Create or update ``index.md`` (§6).
+
+    A fresh index carries ``bundle_summary`` as its body; an existing index keeps every
+    line it does not manage byte for byte. Ingest cross-links (targets ``ingest-*.md``)
+    are the ONLY managed lines: stale ones are dropped and the current set re-appended in
+    extraction order, idempotently by target — curated and promoted links are preserved
+    (so a promoted verdict's index link survives re-ingest, load-bearing §11).
+    """
+    index_path = bundle_dir / "index.md"
+    if index_path.is_file():
+        kept = [
+            line
+            for line in index_path.read_text(encoding="utf-8").splitlines()
+            if not _INGEST_LINK_RE.search(line)
+        ]
+    else:
+        kept = [bundle_summary]
+    present = {t for line in kept for t in _CROSSLINK_RE.findall(line)}
+    lines = list(kept)
+    for extraction in extractions:
+        target = f"{_INGEST_PREFIX}{extraction.id}.md"
+        if target not in present:
+            lines.append(f"- [{extraction.title}]({target})")
+            present.add(target)
+    index_path.write_text("\n".join(lines) + "\n", encoding="utf-8")
+
+
+def materialize(manifest_path: Path, bundle_dir: Path, ingested_at: str) -> list[Path]:
+    """Materialize the manifest's extractions into ``bundle_dir`` (deterministic, §5).
+
+    ``ingested_at`` is an EXPLICIT required argument (no wall-clock default, §5) — the ISO
+    timestamp stamped verbatim into every generated file, which is what makes extractions
+    bit-deterministic. Returns the generated concept-file paths in extraction order.
+
+    Replacement semantics (§3, §5): first every ingest-stamped file is removed, then the
+    new set is written (a name colliding with a NON-stamped file fails — curated content is
+    never overwritten), then the index is updated. Curated and promoted files always survive.
+    """
+    loaded = load_manifest(manifest_path)
+    source = loaded.contract.source
+    root = manifest_path.parent / source.root
+
+    bundle_dir.mkdir(parents=True, exist_ok=True)
+
+    # 1. Remove exactly the files carrying the ingest stamp (§5) — never curated/promoted.
+    for md_path in sorted(bundle_dir.glob("*.md")):
+        if md_path.name != "index.md" and _is_ingest_stamped(md_path):
+            md_path.unlink()
+
+    # 2. Write the new set.
+    generated: list[Path] = []
+    for extraction in loaded.contract.extractions:
+        csv_path = _resolve_within(root, extraction.query)
+        rows = _read_csv(csv_path, extraction.max_rows)
+        _logger.info(
+            "ingest source=%s extraction=%s rows=%d",
+            source.id,
+            extraction.id,
+            max(len(rows) - 1, 0),
+        )
+        target = bundle_dir / f"{_INGEST_PREFIX}{extraction.id}.md"
+        if target.exists():
+            # Post-removal, an existing target is a NON-stamped collision (§3): never overwrite.
+            raise ValueError(f"generated name {target.name} collides with a non-ingest file")
+        target.write_text(
+            _concept_file(extraction, source.id, ingested_at, loaded.stamp, _render_table(rows)),
+            encoding="utf-8",
+        )
+        generated.append(target)
+
+    # 3. Update the index (§6).
+    _update_index(bundle_dir, loaded.contract.bundle_summary, loaded.contract.extractions)
+    return generated

src/portfolio_optimiser_claude/okf.py

@@ -18,6 +18,7 @@ from dataclasses import dataclass
 from pathlib import Path
 
 _INDEX_FILENAME = "index.md"
+_INDEX_TYPE = "index"
 _VERDICT_TYPE = "verdict"
 # (reference) the intra-bundle cross-link pattern, per §3 Step 1.
 _CROSSLINK_PATTERN = re.compile(r"\]\(([^)]+\.md)\)")
@@ -70,19 +71,40 @@ def parse_concept_file(path: Path) -> ConceptFile:
     )
 
 
+def _parse_index_entry(index_path: Path) -> ConceptFile:
+    """Parse the bundle entry point, tolerating a frontmatter-less index (§3 Step 1).
+
+    method-spec §3 Step 1 renders the index as "the index body (the summary)" and
+    treats every OTHER file as a "non-index concept file" (`## {type}: {title}`
+    section) — so the index is NOT a concept file, and it MAY omit frontmatter: a
+    generated index carrying only ``bundle_summary`` + cross-links (ingest spec §6,
+    frozen in the ingest golden) is valid. A frontmatter-ful index (the curated
+    convention, ``type: index``) still parses normally, its extra fields preserved.
+    When frontmatter is absent, the whole file is the body and ``type`` defaults to
+    ``index`` so downstream ``.type`` reads never raise.
+    """
+    text = index_path.read_text(encoding="utf-8")
+    lines = text.splitlines()
+    if lines and lines[0].strip() == "---":
+        return parse_concept_file(index_path)
+    return ConceptFile(path=index_path, frontmatter={"type": _INDEX_TYPE}, body=text.strip())
+
+
 def navigate_bundle(bundle_dir: Path) -> list[ConceptFile]:
     """Navigate from ``index.md`` — deterministic order: index first, links first-seen.
 
     Targets containing a path separator are out-of-bundle and skipped; resolution is
     boundary-checked against the bundle directory (fail-closed); broken links are
-    skipped, never raised. Repeated links are de-duplicated.
+    skipped, never raised. Repeated links are de-duplicated. The index entry point
+    may omit frontmatter (``_parse_index_entry``); non-index concept files still
+    require ``type``.
     """
     index_path = bundle_dir / _INDEX_FILENAME
     if not index_path.is_file():
         raise FileNotFoundError(
             f"bundle has no entry point: missing {_INDEX_FILENAME} in {bundle_dir}"
         )
-    index = parse_concept_file(index_path)
+    index = _parse_index_entry(index_path)
     bundle_root = bundle_dir.resolve()
     concepts = [index]
     seen = {_INDEX_FILENAME}

tests/test_ingest.py

@@ -0,0 +1,138 @@
+"""Ingest unit + fail-fast contract tests (ingest-spec §4, §5, §8).
+
+The manifest is schema-validated fail-fast BEFORE any source call (§4, the startup-contract
+discipline). These tests pin the malformed-manifest rejections, the §5 cell-escaping rules,
+and the §8 size cap / fail-closed path resolution / no-overwrite collision.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+from pydantic import ValidationError
+
+from portfolio_optimiser_claude.ingest import (
+    ManifestContract,
+    _escape_cell,
+    load_manifest,
+    materialize,
+)
+
+GOLDEN = Path(__file__).resolve().parents[1] / "examples" / "ingest-golden-file"
+INGESTED_AT = "2026-07-03T12:00:00Z"
+
+
+def _valid() -> dict:
+    return {
+        "manifest_version": 1,
+        "source": {"type": "file", "id": "arkiv", "root": "fixture"},
+        "bundle_summary": "s",
+        "extractions": [
+            {"id": "e", "title": "T", "query": "e.csv", "okf_type": "dataset", "max_rows": 5}
+        ],
+    }
+
+
+def _write_case(tmp_path: Path, manifest: dict, csvs: dict[str, str]) -> Path:
+    case = tmp_path / "case"
+    (case / "fixture").mkdir(parents=True)
+    (case / "manifest.json").write_text(json.dumps(manifest), encoding="utf-8")
+    for name, text in csvs.items():
+        (case / "fixture" / name).write_text(text, encoding="utf-8")
+    return case
+
+
+class TestManifestValidation:
+    """§4: a malformed manifest never starts a run (fail-fast)."""
+
+    def test_valid_manifest_loads_and_stamps(self) -> None:
+        contract = ManifestContract(**_valid())
+        assert contract.manifest_version == 1
+        assert contract.source.id == "arkiv"
+
+    def test_stamp_is_stem_at_sha256_16(self, tmp_path: Path) -> None:
+        case = _write_case(tmp_path, _valid(), {"e.csv": "a\n1\n"})
+        loaded = load_manifest(case / "manifest.json")
+        stem, _, digest = loaded.stamp.partition("@")
+        assert stem == "manifest"
+        assert len(digest) == 16 and all(c in "0123456789abcdef" for c in digest)
+
+    @pytest.mark.parametrize(
+        "mutate",
+        [
+            lambda m: m.pop("manifest_version"),
+            lambda m: m.__setitem__("manifest_version", 2),
+            lambda m: m.pop("source"),
+            lambda m: m.pop("bundle_summary"),
+            lambda m: m.__setitem__("extractions", []),
+            lambda m: m["source"].__setitem__("id", "Bad_Id"),
+            lambda m: m["source"].__setitem__("type", "sql"),
+            lambda m: m["extractions"][0].__setitem__("id", "Bad Id"),
+            lambda m: m["extractions"][0].__setitem__("title", "two\nlines"),
+            lambda m: m["extractions"][0].__setitem__("max_rows", 0),
+            lambda m: m["extractions"][0].__setitem__("max_rows", -1),
+        ],
+    )
+    def test_malformed_manifest_is_rejected(self, mutate) -> None:
+        manifest = _valid()
+        mutate(manifest)
+        with pytest.raises(ValidationError):
+            ManifestContract(**manifest)
+
+    def test_duplicate_extraction_ids_are_rejected(self) -> None:
+        manifest = _valid()
+        manifest["extractions"].append(dict(manifest["extractions"][0]))
+        with pytest.raises(ValidationError):
+            ManifestContract(**manifest)
+
+
+class TestCellEscaping:
+    """§5: text verbatim with backslash → \\\\, pipe → \\|, newline → single space."""
+
+    def test_backslash_then_pipe_order(self) -> None:
+        assert _escape_cell("\\|") == "\\\\\\|"
+
+    def test_pipe_escaped(self) -> None:
+        assert _escape_cell("a|b") == "a\\|b"
+
+    def test_newline_becomes_single_space(self) -> None:
+        assert _escape_cell("x\ny") == "x y"
+        assert _escape_cell("x\r\ny") == "x y"
+
+    def test_plain_text_verbatim(self) -> None:
+        assert _escape_cell("007") == "007"
+        assert _escape_cell("1.50") == "1.50"
+
+
+class TestSecurityFrame:
+    """§8: size cap fail-fast, path resolution fail-closed, curated never overwritten."""
+
+    def test_extraction_exceeding_max_rows_fails(self, tmp_path: Path) -> None:
+        manifest = _valid()
+        manifest["extractions"][0]["max_rows"] = 1
+        case = _write_case(tmp_path, manifest, {"e.csv": "col\n1\n2\n"})  # 2 data rows > 1
+        with pytest.raises(ValueError, match="max_rows"):
+            materialize(case / "manifest.json", tmp_path / "bundle", INGESTED_AT)
+
+    def test_query_escaping_root_is_refused(self, tmp_path: Path) -> None:
+        manifest = _valid()
+        manifest["extractions"][0]["query"] = "../secret.csv"
+        case = _write_case(tmp_path, manifest, {"e.csv": "col\n1\n"})
+        (case / "secret.csv").write_text("col\nx\n", encoding="utf-8")
+        with pytest.raises(ValueError, match="escapes"):
+            materialize(case / "manifest.json", tmp_path / "bundle", INGESTED_AT)
+
+    def test_collision_with_non_ingest_file_fails(self, tmp_path: Path) -> None:
+        case = _write_case(tmp_path, _valid(), {"e.csv": "col\n1\n"})
+        bundle = tmp_path / "bundle"
+        bundle.mkdir()
+        # A curated (non-stamped) file already occupies the generated name.
+        (bundle / "ingest-e.md").write_text(
+            "---\ntype: reference\ntitle: hand\n---\n\nCurated.\n", encoding="utf-8"
+        )
+        with pytest.raises(ValueError, match="collides"):
+            materialize(case / "manifest.json", bundle, INGESTED_AT)
+        # The curated file is untouched — never overwritten.
+        assert "Curated." in (bundle / "ingest-e.md").read_text(encoding="utf-8")

tests/test_ingest_golden.py

@@ -0,0 +1,86 @@
+"""Ingest golden regression (ingest-spec §11) — the shared byte-for-byte fasit.
+
+D7 mirror of MAF I2. Consumes the repo-local ``examples/ingest-golden-file/``
+(a byte-frozen copy of MAF I2's golden — the SAME shared extraction, so this test
+proves D7's independent implementation reproduces MAF's exact bytes from the shared
+spec alone). ``expected-bundle/`` is compared file by file, byte for byte.
+
+The mutation controls prove the net is taut: one changed determinism input (a CSV
+cell, the timestamp, the manifest bytes) must diverge from the golden bytes.
+"""
+
+from __future__ import annotations
+
+import shutil
+from pathlib import Path
+
+from portfolio_optimiser_claude.ingest import materialize
+
+GOLDEN = Path(__file__).resolve().parents[1] / "examples" / "ingest-golden-file"
+
+
+def _ingested_at() -> str:
+    return (GOLDEN / "ingested-at.txt").read_text(encoding="utf-8").strip()
+
+
+def test_materializes_golden_byte_for_byte(tmp_path: Path) -> None:
+    bundle = tmp_path / "bundle"
+    bundle.mkdir()
+    materialize(GOLDEN / "manifest.json", bundle, _ingested_at())
+
+    expected = GOLDEN / "expected-bundle"
+    expected_names = sorted(p.name for p in expected.iterdir())
+    produced_names = sorted(p.name for p in bundle.iterdir())
+    assert produced_names == expected_names  # no missing, no extra files
+    for name in expected_names:
+        assert (bundle / name).read_bytes() == (expected / name).read_bytes(), name
+
+
+def test_reingest_is_idempotent(tmp_path: Path) -> None:
+    # ingest-spec §10: same source content, manifest, ingested_at → byte-identical,
+    # repeated runs idempotent.
+    bundle = tmp_path / "bundle"
+    bundle.mkdir()
+    materialize(GOLDEN / "manifest.json", bundle, _ingested_at())
+    first = {p.name: p.read_bytes() for p in bundle.iterdir()}
+    materialize(GOLDEN / "manifest.json", bundle, _ingested_at())
+    second = {p.name: p.read_bytes() for p in bundle.iterdir()}
+    assert first == second
+
+
+class TestMutationControl:
+    """One changed determinism input → divergence from the golden bytes."""
+
+    def _case(self, tmp_path: Path) -> Path:
+        case = tmp_path / "case"
+        shutil.copytree(GOLDEN, case)
+        return case
+
+    def test_changed_cell_diverges(self, tmp_path: Path) -> None:
+        case = self._case(tmp_path)
+        csv = case / "fixture" / "costs.csv"
+        csv.write_bytes(csv.read_bytes().replace(b"120000", b"999999"))
+        bundle = tmp_path / "bundle"
+        bundle.mkdir()
+        materialize(case / "manifest.json", bundle, _ingested_at())
+        golden_bytes = (GOLDEN / "expected-bundle" / "ingest-costs.md").read_bytes()
+        assert (bundle / "ingest-costs.md").read_bytes() != golden_bytes
+
+    def test_changed_timestamp_diverges(self, tmp_path: Path) -> None:
+        bundle = tmp_path / "bundle"
+        bundle.mkdir()
+        materialize(GOLDEN / "manifest.json", bundle, "2099-01-01T00:00:00Z")
+        golden_bytes = (GOLDEN / "expected-bundle" / "ingest-costs.md").read_bytes()
+        assert (bundle / "ingest-costs.md").read_bytes() != golden_bytes
+
+    def test_changed_manifest_bytes_change_the_stamp(self, tmp_path: Path) -> None:
+        # The ingest_manifest stamp is SHA-256 of the manifest's raw bytes (§5) —
+        # a whitespace-only edit still re-stamps every generated file.
+        case = self._case(tmp_path)
+        manifest = case / "manifest.json"
+        manifest.write_bytes(manifest.read_bytes() + b"\n")
+        bundle = tmp_path / "bundle"
+        bundle.mkdir()
+        materialize(manifest, bundle, _ingested_at())
+        golden_bytes = (GOLDEN / "expected-bundle" / "ingest-costs.md").read_bytes()
+        assert (bundle / "ingest-costs.md").read_bytes() != golden_bytes

tests/test_ingest_loadbearing.py

@@ -0,0 +1,178 @@
+"""Load-bearing ingest seams (ingest-spec §11) — D7 mirror of MAF I2's set.
+
+Each test must go RED when its seam is detached (the method-spec §11 regime): a
+grønn-men-død test is the failure mode the rule exists for. The seams mirrored here:
+
+- Provenance stamping — a generated file carries the §7 provenance layer, in order.
+- Navigability — the generated bundle is consumable by the UNCHANGED ``okf`` navigation
+  (index links included), incl. the frontmatter-less generated index.
+- Verdict reservation — a manifest mapping to ``type: verdict`` is rejected fail-fast,
+  before any source call.
+- Re-ingest layer safety — re-materialization over a bundle carrying a promoted verdict
+  preserves the verdict file AND its index link (curated content always survives).
+"""
+
+from __future__ import annotations
+
+import json
+import shutil
+from pathlib import Path
+
+import pytest
+from pydantic import ValidationError
+
+from portfolio_optimiser_claude import okf
+from portfolio_optimiser_claude.ingest import ManifestContract, load_manifest, materialize
+
+GOLDEN = Path(__file__).resolve().parents[1] / "examples" / "ingest-golden-file"
+INGESTED_AT = (GOLDEN / "ingested-at.txt").read_text(encoding="utf-8").strip()
+
+_PROVENANCE_KEYS = ("source_system", "source_query", "ingested_at", "ingest_manifest", "generated")
+
+
+def _materialized(tmp_path: Path) -> Path:
+    bundle = tmp_path / "bundle"
+    bundle.mkdir()
+    materialize(GOLDEN / "manifest.json", bundle, INGESTED_AT)
+    return bundle
+
+
+class TestProvenanceStamping:
+    """Seam: a generated file carries the §7 provenance layer (RED if it stops)."""
+
+    def test_generated_file_carries_the_provenance_layer(self, tmp_path: Path) -> None:
+        bundle = _materialized(tmp_path)
+        concept = okf.parse_concept_file(bundle / "ingest-costs.md")
+        for key in _PROVENANCE_KEYS:
+            assert key in concept.frontmatter, f"provenance key {key} detached"
+        assert concept.frontmatter["source_system"] == "prosjekt-arkiv"
+        assert concept.frontmatter["ingested_at"] == INGESTED_AT
+        assert concept.frontmatter["generated"] == "true"
+        stamp = load_manifest(GOLDEN / "manifest.json").stamp
+        assert concept.frontmatter["ingest_manifest"] == stamp
+
+    def test_provenance_keys_are_in_the_spec_order(self, tmp_path: Path) -> None:
+        # §5: exactly these keys, in exactly this order — the chain is a contract.
+        bundle = _materialized(tmp_path)
+        lines = (bundle / "ingest-costs.md").read_text(encoding="utf-8").splitlines()
+        keys = [ln.split(":", 1)[0] for ln in lines[1 : lines.index("---", 1)]]
+        assert keys == [
+            "type",
+            "title",
+            "source_system",
+            "source_query",
+            "ingested_at",
+            "ingest_manifest",
+            "generated",
+        ]
+
+
+class TestNavigability:
+    """Seam: the generated bundle is consumable by the UNCHANGED okf navigation."""
+
+    def test_generated_bundle_navigates_via_unchanged_okf(self, tmp_path: Path) -> None:
+        bundle = _materialized(tmp_path)
+        names = [c.path.name for c in okf.navigate_bundle(bundle)]
+        # Every generated ingest file must be REACHABLE via index cross-links — an
+        # unlinked generated file is unreachable (RED if index linking detaches).
+        assert names == ["index.md", "ingest-costs.md", "ingest-edge.md"]
+
+    def test_generated_context_renders_the_extracted_rows(self, tmp_path: Path) -> None:
+        bundle = _materialized(tmp_path)
+        context = okf.bundle_context(bundle)
+        assert "led-retrofit" in context and "requires vendor quote" in context
+
+    def test_generated_index_is_frontmatterless_and_still_navigates(self, tmp_path: Path) -> None:
+        # The generated index carries no frontmatter (§6 shape) — navigation must
+        # still work, exercising the okf entry-point relaxation end to end.
+        bundle = _materialized(tmp_path)
+        assert not (bundle / "index.md").read_text(encoding="utf-8").startswith("---")
+        assert okf.navigate_bundle(bundle)[0].path.name == "index.md"
+
+
+class TestVerdictReservation:
+    """Seam: an ingest mapping to the verdict layer is rejected fail-fast (§3)."""
+
+    def _manifest(self, okf_type: str) -> dict:
+        return {
+            "manifest_version": 1,
+            "source": {"type": "file", "id": "x", "root": "fixture"},
+            "bundle_summary": "s",
+            "extractions": [
+                {"id": "e", "title": "T", "query": "e.csv", "okf_type": okf_type, "max_rows": 1}
+            ],
+        }
+
+    def test_verdict_okf_type_is_rejected(self) -> None:
+        with pytest.raises(ValidationError):
+            ManifestContract(**self._manifest("verdict"))
+
+    def test_verdict_reservation_is_case_insensitive(self) -> None:
+        with pytest.raises(ValidationError):
+            ManifestContract(**self._manifest("Verdict"))
+
+    def test_rejection_is_fail_fast_before_any_source_call(self, tmp_path: Path) -> None:
+        # A verdict manifest never touches the source: no bundle is written.
+        manifest = tmp_path / "manifest.json"
+        manifest.write_text(json.dumps(self._manifest("verdict")), encoding="utf-8")
+        bundle = tmp_path / "bundle"
+        with pytest.raises(ValidationError):
+            materialize(manifest, bundle, INGESTED_AT)
+        assert not bundle.exists() or not any(bundle.iterdir())
+
+
+class TestReingestLayerSafety:
+    """Seam: re-ingest preserves a promoted verdict AND its index link (§3, §6)."""
+
+    def _promote(self, bundle: Path) -> None:
+        # Simulate the promotion gate writing a verdict file + its (neutral) index link,
+        # plus a curated file + link. Neither carries the ingest stamp.
+        (bundle / "promoted-verdict-led.md").write_text(
+            "---\ntype: verdict\ndecision: approved\ndescription: LED holds up.\n---\n\nBody.\n",
+            encoding="utf-8",
+        )
+        (bundle / "curated-note.md").write_text(
+            "---\ntype: reference\ntitle: Curated\n---\n\nHand-written.\n", encoding="utf-8"
+        )
+        index = bundle / "index.md"
+        index.write_text(
+            index.read_text(encoding="utf-8")
+            + "- [promoted](promoted-verdict-led.md)\n"
+            + "- [Curated](curated-note.md)\n",
+            encoding="utf-8",
+        )
+
+    def test_promoted_verdict_and_curated_survive_reingest(self, tmp_path: Path) -> None:
+        bundle = _materialized(tmp_path)
+        self._promote(bundle)
+        materialize(GOLDEN / "manifest.json", bundle, INGESTED_AT)  # re-ingest
+
+        assert (bundle / "promoted-verdict-led.md").is_file()  # verdict file survives
+        assert (bundle / "curated-note.md").is_file()  # curated file survives
+        index_text = (bundle / "index.md").read_text(encoding="utf-8")
+        assert "- [promoted](promoted-verdict-led.md)" in index_text  # verdict link survives
+        assert "- [Curated](curated-note.md)" in index_text  # curated link survives
+        # ...and the ingest files were still refreshed.
+        assert (bundle / "ingest-costs.md").is_file()
+        assert "](ingest-costs.md)" in index_text
+
+    def test_reingest_drops_a_stale_ingest_file_and_its_link(self, tmp_path: Path) -> None:
+        # A manifest that no longer generates ingest-edge → the stale file AND its index
+        # link are removed; the promoted/curated content is untouched (RED if replacement
+        # over-reaches or under-reaches).
+        bundle = _materialized(tmp_path)
+        self._promote(bundle)
+        # The shrunk manifest must sit beside its own fixture/ (root resolves relative to
+        # the manifest dir), so copy the golden case and rewrite the manifest there.
+        case = tmp_path / "case"
+        shutil.copytree(GOLDEN, case)
+        shrunk = json.loads((case / "manifest.json").read_text(encoding="utf-8"))
+        shrunk["extractions"] = [e for e in shrunk["extractions"] if e["id"] == "costs"]
+        (case / "manifest.json").write_text(json.dumps(shrunk), encoding="utf-8")
+        materialize(case / "manifest.json", bundle, INGESTED_AT)
+
+        assert not (bundle / "ingest-edge.md").exists()  # stale ingest file removed
+        index_text = (bundle / "index.md").read_text(encoding="utf-8")
+        assert "](ingest-edge.md)" not in index_text  # stale ingest link removed
+        assert "- [promoted](promoted-verdict-led.md)" in index_text  # promoted survives
+        assert (bundle / "ingest-costs.md").is_file()  # kept extraction survives

tests/test_ingest_spec_loadbearing.py

@@ -0,0 +1,70 @@
+"""Spec-integrity seam for the ingest spec (ingest-spec §11).
+
+The D7 analog of MAF's I1 framework-guard: this repo consumes ``shared/ingest-spec.md``
+UNCHANGED from commons, and this test keeps the contract honest — it goes RED when the
+spec goes missing, names a concrete agent toolkit (the framework-neutrality rule), or
+stops documenting a contract field. It is the load-bearing guard the ingest layer relies
+on to keep being implementable "from this spec alone".
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+SPEC = Path(__file__).resolve().parents[1] / "shared" / "ingest-spec.md"
+
+# Concrete agent toolkits / vendor stacks the framework-neutral spec MUST NOT name.
+_FORBIDDEN_TOOLKITS = (
+    "claude",
+    "anthropic",
+    "openai",
+    "gpt",
+    "gemini",
+    "llama",
+    "langchain",
+    "autogen",
+    "crewai",
+    "semantic kernel",
+    "microsoft agent framework",
+    "agent sdk",
+    "bedrock",
+    "vertex",
+    "foundry",
+    "maf",
+)
+
+# Every field of the machine-readable contracts the D7 implementation depends on — the
+# spec's §12 cross-check table must keep documenting each (spec-integrity).
+_CONTRACT_FIELDS = (
+    "manifest_version",
+    "source",
+    "bundle_summary",
+    "extractions",
+    "source_system",
+    "source_query",
+    "ingested_at",
+    "ingest_manifest",
+    "generated",
+    "okf_type",
+    "max_rows",
+    "root",
+)
+
+
+def test_spec_is_present() -> None:
+    # RED if the spec goes missing (the layer stops being implementable from spec alone).
+    assert SPEC.is_file(), "ingest-spec.md missing — subtree pull the commons contract"
+
+
+def test_spec_names_no_agent_toolkit() -> None:
+    text = SPEC.read_text(encoding="utf-8").lower()
+    present = [tok for tok in _FORBIDDEN_TOOLKITS if tok in text]
+    assert not present, f"framework-neutral spec names a concrete toolkit: {present}"
+
+
+@pytest.mark.parametrize("field", _CONTRACT_FIELDS)
+def test_spec_documents_contract_field(field: str) -> None:
+    text = SPEC.read_text(encoding="utf-8")
+    assert field in text, f"contract field {field!r} is no longer documented in the spec"

tests/test_okf.py

@@ -54,6 +54,32 @@ class TestNavigation:
         names = [c.path.name for c in navigate_bundle(bundle)]
         assert names == ["index.md", "b.md", "a.md"]
 
+    def test_frontmatterless_index_navigates(self, tmp_path: Path) -> None:
+        # method-spec §3 Step 1: the index contributes "the index body (the summary)"
+        # and is NOT a "non-index concept file" rendered as a `## {type}: {title}`
+        # section — so a generated index carrying only the summary + links (ingest
+        # spec §6 shape, frozen in the ingest golden) is valid. The index MAY omit
+        # frontmatter; okf must navigate it via its body, never raise.
+        _write(tmp_path / "index.md", "Summary line.\n- [A](a.md)\n")
+        _write(tmp_path / "a.md", "---\ntype: project\ntitle: A\n---\nBody A.")
+        concepts = navigate_bundle(tmp_path)
+        assert [c.path.name for c in concepts] == ["index.md", "a.md"]
+        index = concepts[0]
+        assert index.type == "index"  # default type, no KeyError for downstream .type reads
+        assert index.body == "Summary line.\n- [A](a.md)"  # body = the whole file
+        # The summary flows into the read-context as the leading section.
+        assert bundle_context(tmp_path).startswith("Summary line.")
+
+    def test_frontmatterless_tolerance_is_scoped_to_the_index(self, tmp_path: Path) -> None:
+        # LOAD-BEARING (honesty): the relaxation is for the index ENTRY POINT only.
+        # A NON-index concept file without frontmatter is still malformed and MUST
+        # raise — else the relaxation has silently weakened the `type`-required rule
+        # for concept files (method-spec §3 Step 1). RED if the tolerance leaks.
+        _write(tmp_path / "index.md", "Summary.\n- [A](a.md)\n")
+        _write(tmp_path / "a.md", "no frontmatter here\n")
+        with pytest.raises(ValueError, match="frontmatter"):
+            navigate_bundle(tmp_path)
+
     def test_out_of_bundle_and_escaping_targets_are_skipped(self, tmp_path: Path) -> None:
         # A target containing a path separator is out-of-bundle — skipped, never
         # raised; ../-escapes never resolve outside the bundle (fail-closed).