docs(ingest): I7 — programavslutning, D7-docs avgrenset til CSV + SQL

Docs-only program-avslutning for ingest-laget (D7-stacken). Bygget fra
commons-spec + eget repo alene — MAF-koden kun eksistens-bekreftet (I6-gate),
aldri lest/reverse-engineert.

- README: ingest-seksjon (file/CSV + sql/sqlite RO) som post-S10-tillegg;
  testtall 187 -> 265; HTTP/MCP kun peker (D7 har ingen HTTP-konnektor).
- docs/extending.md (ny): hvordan ingest-laget virker + hvordan legge til en
  kildetype; HTTP/MCP som ærlig extension-point-peker (spec §4, MAF I6-demo,
  create_sdk_mcp_server som ubygd vehikkel — brukt ingen steder i src/).
- docs/2026-07-04-I7-brief.md + -statusrapport.md: I7-brief + D7-lokal
  statusrapport m/ verifiseringslogg. Kryss-stack-kriteriet peker til
  MAF-programrapporten.

Ærlighetsregelen (method-spec §1): ingen artefakt påstår HTTP/MCP-støtte i D7
eller live-kilde-integrasjon. Grep-sjekk mot overpåstand ren.

Verifisering: 265 passed uten nøkkel/nettverk · ruff/mypy rene.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7nuUMUyWB9viiBjz8Bo7R

README.md

@@ -6,11 +6,12 @@ project in a portfolio of independent projects: agents generate candidate measur
 mandatory deterministic validator gates the numbers, domain experts judge via
 human-in-the-loop, and the system learns from the verdicts.
 
-> **Status:** the D7 build (S5–S10) is complete. The deterministic backbone, the agentic
-> loop, and the learning loop are wired seam by seam, each proven by load-bearing tests
-> (187 tests, all running offline without an API key). The programme's single budgeted
-> **live model run has been executed and validated** — its artifacts are committed under
-> [`runs/s10/`](runs/s10/) (see below).
+> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer**
+> (CSV and SQL source types) has since been added in front of the loop. The deterministic
+> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
+> seam, each proven by load-bearing tests (265 tests, all running offline without an API
+> key). The programme's single budgeted **live model run has been executed and validated** —
+> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
 
 > **Disclaimer — technical framework only.** The deployer owns DPIA, risk assessment, and
 > the legal basis for any processing. The framework ships only the technical
@@ -87,6 +88,36 @@ empty-inbox control), `test_step8_promotion_loadbearing.py` (the gate refuses no
 verdicts; the promoted signal stays out of the read-context), and
 `test_sdk_isolation.py` (local config cannot capture the checker).
 
+## The ingest layer — CSV and SQL, in front of the loop
+
+The method spec forbids query-time retrieval against the bundle (§3 Step 1), so **data
+reaches the model only via OKF bundles.** The ingest layer is the deterministic step that
+satisfies that: a connector reads a real source, and the extract is materialized as an OKF
+bundle the existing 8-step loop then consumes **unchanged**. It makes **zero model calls**,
+touches no network, and `ingest.py` imports nothing from the SDK — it is pure standard
+library. Built from the shared [`ingest-spec.md`](shared/ingest-spec.md) alone.
+
+D7 implements the two conformance-required source types:
+
+- **`file`** — a local CSV catalogue; extraction paths are boundary-checked fail-closed
+  against the source `root` (the OKF path rule).
+- **`sql`** — a local SQLite database, opened **read-only** (`mode=ro`), one SELECT per
+  extraction; the connection location is resolved at run time from a named environment
+  variable (`connection_ref`), never stored in the manifest.
+
+Both are frozen by byte-identical golden extractions
+([`examples/ingest-golden-file/`](examples/ingest-golden-file/),
+[`examples/ingest-golden-sql/`](examples/ingest-golden-sql/)) and by the load-bearing seam
+tests (provenance stamping, navigability through the unchanged `okf.py`, the reserved
+verdict layer, and re-ingest safety over a promoted verdict).
+
+**Honesty rule (§1):** the `http` source type is an **optional extension point, not built in
+D7** — a manifest naming it is rejected fail-fast at validation
+(`test_malformed_manifest_is_rejected`), never silently accepted. The HTTP/MCP extension
+point is demonstrated only in the MAF sibling (against a local mock, behind an opt-in
+network flag); this repo ships no network connector and no live-source integration. How the
+layer works and how one would extend it is documented in [`docs/extending.md`](docs/extending.md).
+
 ## The live run — S10, executed and validated
 
 Where the MAF sibling proves its loop end-to-end with a scripted offline simulation, this
@@ -115,7 +146,7 @@ Python ≥3.10 · [`claude-agent-sdk`](https://pypi.org/project/claude-agent-sdk
 
 ```bash
 uv sync          # install dependencies
-uv run pytest    # 187 tests — run without any API key and without network
+uv run pytest    # 265 tests — run without any API key and without network
 uv run ruff check . && uv run ruff format --check .
 uv run mypy src  # strict
 ```

docs/2026-07-04-I7-brief.md

@@ -0,0 +1,47 @@
+# I7-brief (D7) — program-avslutning: dokumentasjon + statusrapport
+
+> **Type:** docs-avslutning, **ingen ny kode**. Komprimert pipeline (ingen
+> research/plan/execute-runder — deterministisk docs-oppgave). Forankret i sesjonsplan
+> [§I7](/Users/ktg/repos/portfolio-optimiser/docs/plan/2026-07-03-sesjonsplan-ingest.md) og
+> STATE-ens «når I7 åpnes»-oppskrift.
+
+## Kontekst / gate
+
+I7 var GATET på at MAF lander **I6** (HTTP/MCP demonstrert MAF-only mot lokal mock, bak
+opt-in-nettverksflagget). Gaten er **oppfylt**: MAF I6 landet (commits `58dda46`→`696f19a`,
+review `72cb15f`, WARN akseptert). Bekreftet ved `git -C …/portfolio-optimiser log` —
+**kun eksistens**, ingen lesing/reverse-engineering av MAF-koden.
+
+## Scope (denne økta rører KUN dette repoet)
+
+1. **README.md** — flett inn ingest-laget (CSV + SQL) som post-S10-tillegg; oppdater
+   testtall (187 → 265); HTTP/MCP kun som peker (D7 har ingen HTTP-konnektor).
+2. **`docs/extending.md`** (ny) — avgrenset til det D7 faktisk har: `file` (CSV) + `sql`
+   (sqlite RO). Hvordan legge til en kildetype. HTTP/MCP KUN som peker: spec §4-skjema +
+   MAF-søskenets I6-demonstrasjon + SDK-primitivet `create_sdk_mcp_server` som *ubygd*
+   vehikkel (D7-`src/` bruker det ingen steder; `ingest.py` er ren stdlib).
+3. **Grep-sjekk mot overpåstand** før commit.
+4. **Norsk statusrapport** med verifiseringslogg (hver påstand → kommando/kilde).
+
+## Ærlighetsregelen (ufravikelig, method-spec §1 / ingest-spec §1)
+
+Ingen artefakt påstår:
+- HTTP/MCP-**støtte i D7** (D7 avviser `http`-manifest fail-fast — testbevist
+  `test_malformed_manifest_is_rejected`).
+- **Live-kilde-integrasjon** (ingest gjør null modellkall, null nettverk; testene kjører
+  uten nøkkel/nettverk).
+- Mer live-atferd enn den ENE dokumenterte S10-kjøringen (uendret påstand).
+
+## Verifisering (øktas suksesskriterier)
+
+- `uv run pytest -q` → **265 passed** uten nøkkel/nettverk.
+- Grep: ingen artefakt påstår HTTP/MCP-støtte i D7 eller live-kilde-integrasjon.
+- `uv run ruff check .` + `uv run mypy src` grønne (docs-only, men suite + lint holdes grønn).
+- Statusrapport med verifiseringslogg levert.
+
+## Ikke i scope (gated på operatør)
+
+- Åpent-speil-synk (`open/…`) av I3+I5+I7-koden — egen outward-facing operatør-handling.
+- LICENSE / S12-release.
+- Kryss-stack-sluttkriteriet (begge stacker identiske golden) eies av MAF-programrapporten;
+  denne D7-rapporten peker dit i stedet for å duplisere.

docs/2026-07-04-I7-statusrapport.md

@@ -0,0 +1,57 @@
+# I7 — statusrapport (D7): ingest-programmet, D7-stacken
+
+> **Scope:** denne rapporten dekker **D7-stacken alene** (dette repoet). Kryss-stack-
+> sluttkriteriet (begge stacker materialiserer identiske golden-uttrekk) eies av
+> MAF-programrapporten — se [Kryss-stack](#kryss-stack-eies-av-maf-programrapporten). Dato:
+> 2026-07-04. Ingen ny kode i I7 (docs-avslutning).
+
+## Hva D7 har bevist
+
+| # | Påstand | Bevis |
+|---|---------|-------|
+| 1 | D7 implementerer `file` (CSV) + `sql` (sqlite RO) mot delt spec | `src/…/ingest.py` (`FileSource`, `SqlSource`, diskriminert union); `test_ingest_golden.py`, `test_ingest_sql_golden.py` |
+| 2 | Golden-uttrekk er byte-identisk (CSV + SQL) | `examples/ingest-golden-file/`, `examples/ingest-golden-sql/` mot `expected-bundle/` — golden-regresjonstester grønne |
+| 3 | Alle sømmer load-bearing (rød ved detach) | `test_ingest_loadbearing.py`, `test_ingest_sql_loadbearing.py`, `test_ingest_spec_loadbearing.py` (provenance, navigerbarhet via UENDRET `okf.py`, verdict-reservasjon, re-ingest-sikkerhet) |
+| 4 | `http`-manifest avvises fail-fast (ingen stille aksept) | `test_ingest.py::test_malformed_manifest_is_rejected` (parametriserer `http` OG ukjent diskriminator → `ValidationError`) |
+| 5 | Ingest gjør null modellkall / null nettverk | `ingest.py` er ren stdlib (ingen SDK/`requests`-import); hele suiten kjører uten nøkkel/nettverk |
+| 6 | Full suite grønn uten nøkkel/nettverk | `uv run pytest -q` → **265 passed** |
+| 7 | Lint + type rene | `uv run ruff check .` → All checks passed · `uv run mypy src` → no issues (16 filer) |
+| 8 | Dokumentasjonen er sannferdig avgrenset (CSV+SQL; HTTP/MCP kun peker) | `README.md` (ingest-seksjon + honesty-note), `docs/extending.md`; grep-sjekk mot overpåstand ren |
+
+## Hva D7 IKKE har (ærlighetsregelen, method-spec §1 / ingest-spec §1)
+
+- **Ingen HTTP-konnektor, ingen MCP-konnektor.** `http` er et OPTIONAL extension point (spec
+  §1) som D7 bevisst ikke bygger; et `http`-manifest avvises fail-fast (bevis #4).
+- **Ingen live-kilde-integrasjon.** Ingest er deterministisk, offline, null egress.
+- **`create_sdk_mcp_server`/`@tool` brukes ingen steder i `src/`** — omtales i `docs/extending.md`
+  kun som det ubygde vehikkelet man *ville* grepet til for en MCP-konnektor.
+- **Ingen ny live-atferd utover S10.** Programmets ENE ekte API-kjøring forblir S10 (D6);
+  ingest la ingenting til på den fronten.
+
+## Verifiseringslogg (hver påstand → kommando/kilde)
+
+| Påstand | Kommando / kilde | Resultat |
+|---------|------------------|----------|
+| Suite grønn uten nøkkel/nettverk | `uv run pytest -q` | `265 passed in 1.18s` |
+| Lint rent | `uv run ruff check .` | `All checks passed!` |
+| Type rent | `uv run mypy src` | `Success: no issues found in 16 source files` |
+| `http` avvist fail-fast | `tests/test_ingest.py:71-72` (parametrisert `http`+`unknown` → `ValidationError`) | testbevist |
+| Ingest rører ikke SDK/nett | `grep -n "sdk\|claude_agent\|create_sdk" src/…/ingest.py` | ingen treff (ren stdlib) |
+| `create_sdk_mcp_server` ubrukt i src | `grep -rn "create_sdk_mcp_server\|@tool" src/` | ingen treff |
+| Ingen overpåstand i docs | grep `http\|mcp\|live\|egress` over `README.md` + `docs/extending.md` + briefer | hvert treff = negasjon eller peker; ingen påstand om D7-HTTP/MCP-støtte eller live-kilde |
+| MAF I6 landet (I7-gaten) | `git -C /Users/ktg/repos/portfolio-optimiser log --oneline` | `58dda46`→`696f19a`, review `72cb15f` — kun eksistens bekreftet |
+| Golden byte-identisk | `test_ingest_golden.py`, `test_ingest_sql_golden.py` | grønne |
+
+## Kryss-stack (eies av MAF-programrapporten)
+
+Program-sluttkriteriet «begge stacker materialiserer identiske golden-uttrekk fra delt spec
+(CSV + SQL)» er en **kryss-stack**-påstand. Den eies av MAF-programrapporten (hovedrepoet),
+som ser begge stackene; denne D7-rapporten dupliserer den ikke. D7-siden av kontrakten er
+oppfylt: D7 konsumerer den delte spec-en uendret og reproduserer golden-bytene (bevis #1, #2).
+HTTP/MCP-extension-pointet er demonstrert **MAF-only** (I6, lokal mock, bak opt-in-flagg);
+D7 har det bevisst ikke.
+
+## Utestående (gated på operatør — ikke I7-scope)
+
+- Åpent-speil-synk (`open/…`) av I3+I5+I7-koden — outward-facing operatør-handling.
+- LICENSE-valg / S12-release.

docs/extending.md

@@ -0,0 +1,160 @@
+# Extending the ingest layer
+
+This document explains the ingest layer as **implemented in this repo (D7)** and how to
+extend it. It is scoped to what D7 actually ships: the `file` (CSV) and `sql` (SQLite)
+source types. The `http`/MCP source type is an optional extension point that D7 does **not**
+implement — see [HTTP/MCP — an extension point D7 does not build](#httpmcp--an-extension-point-d7-does-not-build).
+
+The layer is built from the framework-neutral [`ingest-spec.md`](../shared/ingest-spec.md)
+alone (section references below, `§n`, point into it). Everything here is deterministic and
+runs offline: ingest makes **zero model calls**, touches no network, and
+[`ingest.py`](../src/portfolio_optimiser_claude/ingest.py) imports nothing from the Agent SDK
+— it is pure standard library.
+
+## Where ingest sits
+
+```
+manifest → connector → materialization → OKF bundle (+ index) → [existing 8-step loop, unchanged]
+```
+
+The method spec forbids query-time retrieval against the bundle (method-spec §3 Step 1), so
+**data reaches the model only via OKF bundles.** Ingest is the deterministic step in front of
+the loop that makes that true: a connector reads a real source, and the extract is
+materialized as an OKF bundle the loop then reads exactly as it reads a hand-curated one —
+by navigation, never retrieval (§1–§2). This is not RAG and not a live lookup inside the
+loop.
+
+## The manifest — the whole coupling to a source
+
+One JSON file couples an expert to a source (§4). It is schema-validated **fail-fast before
+any source call** ([`ManifestContract`](../src/portfolio_optimiser_claude/ingest.py)); a
+malformed manifest never starts a run. Queries are **configuration, not code** — declarative
+strings the connector interprets, never evaluated as program code.
+
+A `file` manifest (from [`examples/ingest-golden-file/manifest.json`](../examples/ingest-golden-file/manifest.json)):
+
+```json
+{
+  "manifest_version": 1,
+  "source": {"type": "file", "id": "prosjekt-arkiv", "root": "fixture"},
+  "bundle_summary": "Two CSV extracts from a local project archive (file source type).",
+  "extractions": [
+    {"id": "costs", "title": "Project costs", "query": "costs.csv",
+     "okf_type": "dataset", "max_rows": 10}
+  ]
+}
+```
+
+A `sql` manifest differs only in `source`; the `query` becomes a single read-only SELECT
+(from [`examples/ingest-golden-sql/manifest.json`](../examples/ingest-golden-sql/manifest.json)):
+
+```json
+{
+  "source": {"type": "sql", "id": "portefolje-db", "connection_ref": "PORTEFOLJE_SQL_DSN"},
+  "extractions": [
+    {"id": "costs", "title": "Project costs",
+     "query": "SELECT id, item, amount, note FROM costs ORDER BY id",
+     "okf_type": "dataset", "max_rows": 10}
+  ]
+}
+```
+
+`source` is **polymorphic on `type`** — a Pydantic discriminated union
+(`FileSource | SqlSource`). A manifest whose `source.type` is anything other than `file` or
+`sql` (including `http`, or an unknown tag) fails validation fail-fast; it is never silently
+accepted.
+
+## The two source types D7 implements
+
+### `file` — a local CSV catalogue
+
+`source.root` is the directory extraction paths resolve against; `query` is a relative path
+to a CSV whose first row is the header. Path resolution is **boundary-checked fail-closed**
+against `root` (`_resolve_within`) — an extraction can never read outside the catalogue.
+
+### `sql` — a local SQLite database
+
+`source.connection_ref` is the **name** of an environment variable whose value is the
+database path — the location/credential never lives in the manifest (§4, §8), so a manifest
+is versionable and shareable without secrets. An unset reference fails fast. The connection
+is opened **read-only** (`sqlite3` URI with `?mode=ro`), and SQLite executes exactly one
+statement per `execute`, so the single-statement rule is enforced by the driver. Give the
+`query` an explicit `ORDER BY`: golden conformance requires a stable row order (§4).
+
+Both connectors enforce `max_rows` fail-fast — an extraction over its cap is an error, never
+a silent truncation (§8).
+
+## Materialization — what a connector's rows become
+
+Each extraction becomes one concept file `ingest-{id}.md` (§5) with:
+
+- a **provenance frontmatter layer** (§7) — `type`, `title`, `source_system`,
+  `source_query`, `ingested_at`, `ingest_manifest`, `generated: true`, in exactly that
+  order. This is a *separate* contract from the method spec's §9 proposal provenance — the
+  two are never mixed. `generated: true` plus the manifest reference is the honesty marker:
+  a machine-made bundle is always labelled as such (§1).
+- a **markdown table** body — header row = column names, data rows in source order, with §5
+  cell escaping. Cell typing is the connector's contract: `file` cells are all `str`; `sql`
+  cells are typed (`int` → decimal, `float` → shortest round-trip form, `NULL` → empty
+  string, **any other type → fail**, never a silent coercion). See `_render_sql_cell`.
+
+`ingested_at` is an **explicit required argument** to `materialize` — there is no wall-clock
+default. That is what makes golden extractions bit-deterministic. Re-materialization removes
+exactly the ingest-stamped files, writes the new set, and updates `index.md` — curated and
+promoted files (and their index links) always survive (§3, §5, §6).
+
+## How to add a new source type
+
+Adding `sql` on top of `file` touched four places; a new local source type follows the same
+shape (all offline, no spec change if the type already exists in §4):
+
+1. **A source model.** Add a Pydantic variant (like `SqlSource`) with `type:
+   Literal["yourtype"]`, its own fields, and the shared `id` grammar validator. Add it to
+   the discriminated union in `ManifestContract.source`.
+2. **A connector.** Add a `_read_yourtype(...) -> list[list[str]]` returning the header row
+   followed by data rows in source order, and enforcing `max_rows` fail-fast. Route to it
+   from `_read_extraction` on `source.type`. If cells can be non-`str`, add a
+   `_render_*_cell` that types them explicitly and **fails on any unsupported type**.
+3. **A golden case.** Add `examples/ingest-golden-{type}/` with `manifest.json`, `fixture/`,
+   `ingested-at.txt`, and the byte-exact `expected-bundle/` (§11). This is the only fasit.
+4. **Load-bearing tests.** Mirror the existing seam tests so each goes **red when its seam is
+   detached** (§11): golden regression, provenance stamping, navigability through the
+   unchanged `okf.py`, verdict-layer reservation, and re-ingest safety. Green-but-dead is the
+   failure mode the rule exists for.
+
+Keep it deterministic and offline: connectors are tested only against fixtures and golden
+extractions, and the suite must run without credentials or network (§11).
+
+## HTTP/MCP — an extension point D7 does not build
+
+The spec names `http` as an **optional** source type (§1, §4): a remote endpoint with a
+`base_url` and an optional named `credential_ref`, and "an MCP-based connector is an
+extension of this family." Implementing it does not require a spec change, and **not**
+implementing it does not break conformance (§1).
+
+**D7 does not implement it.** There is no HTTP connector and no MCP connector in this repo. A
+manifest naming `type: "http"` is rejected fail-fast at validation (proven by
+`test_malformed_manifest_is_rejected`, which parametrizes both `http` and an unknown
+discriminator). This repo ships **no** network egress path and **no** live-source
+integration.
+
+If you were to build it, three honest pointers — none of which exist in D7 today:
+
+- **The spec's `http` schema (§4, §5, §8).** `base_url` must not embed credentials;
+  `credential_ref` names a runtime-resolved secret; the network opt-in flag (§8) is a **run
+  argument, never a manifest field** — without it the connector MUST refuse fail-fast; the
+  response body materializes verbatim inside a fenced code block (§5).
+- **The MAF sibling's demonstration (I6).** The sibling implementation
+  ([`open/portfolio-optimiser`](https://git.fromaitochitta.com/open/portfolio-optimiser))
+  demonstrates the HTTP/MCP extension point against a **local mock**, behind the opt-in
+  network flag — no live source. It is referenced here as a pointer only; D7 is built from
+  the shared spec, not from the sibling's code.
+- **The SDK vehicle for an MCP connector.** An MCP-based connector would be built on the
+  Agent SDK's `create_sdk_mcp_server` / `@tool` primitives. These are **not used anywhere in
+  this repo** — `ingest.py` is pure standard library. They are named here only as the
+  mechanism you would reach for, not as something D7 wires up.
+
+Whatever the transport, an `http`/MCP connector MUST honour the same manifest,
+materialization, gate, and golden contracts as the local connectors (§4) — and the network
+gate itself is a load-bearing seam (§11): the connector must refuse fail-fast without the
+opt-in flag.