portfolio-optimiser/.claude/projects/2026-07-03-i2-ingest-csv-maf/brief.md

---
type: trekbrief
brief_version: "2.2"
created: 2026-07-03
task: "I2 — MAF reference implementation of the ingest layer: file catalogue/CSV (offline)"
slug: i2-ingest-csv-maf
project_dir: .claude/projects/2026-07-03-i2-ingest-csv-maf/
research_topics: 0
research_status: skipped
auto_research: false
interview_turns: 0
source: manual
framing: preserve
phase_signals:
  - phase: research
    effort: high
    model: fable
  - phase: plan
    effort: high
    model: fable
  - phase: execute
    effort: high
    model: fable
  - phase: review
    effort: high
    model: fable
---

# Task: I2 — MAF reference implementation: ingest file catalogue/CSV (offline)

> Generated by `/trekbrief` on 2026-07-03.
> This brief is the contract between requirements and planning. `/trekplan`
> reads it to produce the implementation plan. Every decision in the plan must
> trace back to content in this brief.
>
> **Provenance note (framing defense):** this brief was derived autonomously from
> two FROZEN, operator-approved sources — the I2 section of
> `docs/plan/2026-07-03-sesjonsplan-ingest.md` and the normative
> `shared/ingest-spec.md` — under the STATE.md operating model ("Les STATE.md og
> følg instruksjonene" = go-signal for the NEXT step; I2 is not gated). No
> interview questions were asked because every answer already exists in written,
> approved form; re-asking would violate the operator's recorded
> plan-over-menu preference. `framing: preserve` is documented, not guessed:
> I2 continues the frozen plan arc unchanged (STATE.md: "I2 [NESTE]").

## TL;DR

Implement the ingest layer's first source type on the MAF side, exactly as frozen in
`shared/ingest-spec.md`: fail-fast pydantic manifest contract, CSV/file-catalogue
connector, deterministic materialization into an OKF bundle with provenance
frontmatter + index generation — offline, zero model calls, TDD. Framing: **preserve**
— the "I2 [NESTE]" step of the operator-approved, frozen ingest session plan.

## Intent

The 8-step optimiser loop consumes hand-curated OKF bundles today; the frozen ingest
målbilde closes the gap to real data sources while preserving the architecture-defining
rule that **data reaches the model ONLY via OKF bundles** (no RAG, no query-time
retrieval — method spec §3). I2 is the first implementation session of that program:
it proves the frozen contract (`shared/ingest-spec.md`, authored and guarded in I1) is
actually implementable from the spec alone, on the reference (MAF) side, using the
offline `file` source type so the whole session stays at zero model calls and zero
network (cost discipline D6). It also closes the plan's key assumption 2 — that an
ingest manifest can be validated fail-fast without network — which is currently
marked RISK with only pattern-level evidence (`contracts.py` is in-process). Finally,
I2 turns the planning session's ephemeral ad-hoc proof (a synthetic generated bundle
passing `navigate_bundle`/`bundle_context`) into a durable load-bearing test, so the
seam can never silently detach.

## Goal

`uv run pytest` green (without network or credentials) on a repo where: a pydantic
manifest contract validates ingest manifests fail-fast per spec §4 (polymorphic on
`source.type`, verdict-layer reservation enforced before any source call); a
file-catalogue/CSV connector executes extractions per §4 (boundary-checked path
resolution, `max_rows` fail-fast); materialization per §5–§7 writes
`ingest-{id}.md` concept files with the exact 7-key provenance frontmatter and
generates/updates `index.md` per §6 (idempotent by target, curated/promoted links
preserved); and a golden extraction case at `examples/ingest-golden-file/` (spec §11
layout) reproduces byte for byte from fixture + fixed `ingested_at`. The existing
loop consumes the generated bundle through UNCHANGED `okf.py` navigation code. Every
seam in the I2 load-bearing set has a test proven RED at detach.

## Non-Goals

- **SQL connector** — I4 (the pydantic manifest schema MAY validate the `sql`/`http`
  variants per §4, but no `sql` connector execution in I2; see Open Questions).
- **HTTP/MCP connector and the network opt-in gate test** — I6 (gated: network
  capability). The `file` source type requires no network flag (§5, §8).
- **Sibling repo (D7) work** — I3/I5 (gated: other repo). No files outside this repo.
- **Commons/spec changes** — any needed spec change means STOP and a separate gated
  commons round (session-plan frame rule); never folded into I2. The golden case is
  authored LOCALLY in this repo in I2; sharing it via commons is a later gated round.
- **Wiring ingest into `run_project`** — ingest is an explicit operator-run step
  BEFORE the loop (spec §2, §9); the run path is untouched.
- **Extension points explicitly deferred by the spec:** per-row file splitting,
  multiple manifests per bundle, incremental re-ingest, approval registry /
  signed `approved_by` (spec §5, §9, §10).
- **S12 release work** — independently gated (LICENSE is an operator-owned choice).

Note on spec conformance: full §1 conformance requires both `file` and `sql`
connectors — I2 is deliberately a partial-conformance milestone; §1 conformance
completes at I4. Reviewers must not read §1 as an I2 failure.

## Constraints

- **The spec is normative and frozen.** Implement from `shared/ingest-spec.md` alone;
  MUST/MUST NOT clauses are binding. On any discovered spec defect or needed change:
  stop, report — separate gated commons round.
- **Offline, deterministic, zero model calls** (D6 + spec §1, §11): tests run without
  credentials and without network; no live sources anywhere.
- **TDD Iron Law:** no production code without a failing test first. Load-bearing
  tests must be proven RED at detach (detach → observe red → reattach), the method
  spec §11 regime.
- **`okf.py` navigation code stays UNCHANGED** — the navigability criterion is
  explicitly "consumable via UNCHANGED `okf.py`". Reusing its existing primitives
  (e.g. the index-linking primitive, whose idempotent-by-target semantics §6
  references) is expected; modifying them is not in scope.
- **Context-layer neutrality invariant:** `okf.py` and `shared/` remain pure stdlib,
  MAF-free (`test_okf_is_maf_free`). The new ingest module makes zero model calls and
  MUST NOT import `agent_framework`/`mcp` (it is a deterministic pre-loop step);
  pydantic is fine (already a core dependency, `contracts.py` pattern).
- **Commons is PULL-ONLY**; nothing in I2 writes to commons or `shared/`.
- **One session = one closed delivery:** green suite, STATE.md overwritten,
  commit + push to Forgejo; never a broken state across the session boundary.
- **Surgical scope:** new ingest module + tests + golden example; no refactoring of
  adjacent code.

## Preferences

- Follow the existing `contracts.py` pydantic style for the manifest contract
  (fail-fast, explicit models, type hints everywhere).
- Mirror the existing load-bearing test idiom (e.g.
  `tests/test_step8_promotion_loadbearing.py`): each test names its seam and fails
  when the seam is detached.
- Reuse `okf.py`'s existing index-linking primitive for §6 index updates rather than
  reimplementing link semantics.
- Golden case directory per spec §11 convention: `examples/ingest-golden-file/`
  with `manifest.json`, `fixture/`, `ingested-at.txt`, `expected-bundle/`.
- Small synthetic fixture data (cost discipline — no large test payloads).
- Code/comments/docs in English; Conventional Commits.

## Non-Functional Requirements

- **Bit-determinism:** same source content + manifest + `ingested_at` → byte-identical
  bundle; repeated runs idempotent (spec §10, §11).
- **Zero new runtime dependencies** (pydantic and stdlib suffice; CSV via stdlib `csv`).
- **Fail-fast, never silent:** malformed manifest, verdict-typed mapping, path escape,
  `max_rows` overflow, non-CSV-representable cell value, curated-file collision — all
  ERRORS, never warnings/truncation/coercion (spec §3, §4, §5, §8).
- **File hygiene per spec §5:** LF-only, exactly one trailing newline, single-line
  frontmatter values, whitespace-collapsed `source_query`.
- **Source calls are logged** (which source, when, row count — spec §8); test-visible
  via stdlib `logging`, no new dependency.
- **Contract completeness:** the plan traces spec §12's cross-check table
  field-by-field (`manifest_version` … `generated`, golden-case entries) so no
  contract field is silently dropped — including `bundle_summary` (§4/§6) and the
  `ingest_manifest` stamp format `{stem}@{hash16}` (§5).

## Success Criteria

All commands run from the repo root; "without network/credentials" is the standing
test condition (spec §11).

1. **Golden extraction, bit-deterministic:** a test materializes
   `examples/ingest-golden-file/` from `manifest.json` + `fixture/` +
   `ingested-at.txt` and compares against `expected-bundle/` file by file, byte for
   byte — `uv run pytest tests/ -k ingest` exits 0. Running materialization twice
   over the same target produces identical bytes (idempotence, §10).
2. **Load-bearing: provenance detach** — with the §7 stamping detached (e.g.
   provenance keys not written), the suite goes RED. Proof recorded (detach → red →
   reattach) in the session log.
3. **Load-bearing: navigability via UNCHANGED `okf.py`** — the generated bundle is
   consumed by the existing navigation code (`navigate_bundle`/`bundle_context`
   seam): generated files reachable via index cross-links, unknown provenance fields
   preserved. Test goes RED if index generation/linking is detached. `git diff` for
   the delivery shows no modification to `okf.py`.
4. **Load-bearing: verdict reservation** — a manifest with `okf_type: verdict`
   (case-insensitive) or a generated-filename collision with `index.md`/the
   `promoted-verdict-*` namespace is REJECTED at validation, before any source call;
   test goes RED if the reservation is removed.
5. **Load-bearing: re-ingest layer safety** — re-materialization over a bundle
   containing a promoted verdict (created via the real `promote_verdict`) preserves
   the verdict file AND its index link; test goes RED if replacement semantics stop
   honouring the ingest-stamp ownership rule (§3, §6).
6. **Fail-fast manifest validation without network (closes key assumption 2):**
   malformed manifests (missing fields, bad `id` grammar, empty `extractions`,
   non-positive `max_rows`, unknown `source.type`) raise before any source access —
   asserted by tests that run with no network available.
7. **Full suite + gates green:** `uv run pytest` exits 0 — the pre-existing suite
   passes unchanged (baseline captured at session start before any I2 change) plus
   the new ingest tests; `uv run ruff check .` exits 0, `uv run ruff format --check .`
   clean, `uv run mypy src` exits 0.
8. **Honesty of artefacts:** no artefact delivered in I2 claims SQL/HTTP support,
   live-source integration, or commons sharing of the golden case. Concretely:
   case-insensitive grep for `sql`, `http`, `live`, `commons` over the new module's
   docstrings, new test files' docstrings, and any touched README/docs sections
   yields no support-claiming hits (mentions that explicitly DEFER — "I4", "I6",
   "extension point" — are fine); the plan fixes the exact grep invocation.

## Research Plan

No external research needed — the codebase and this brief contain sufficient context
for planning. Rationale (recorded in lieu of an interview): the task is fully
specified by the frozen `shared/ingest-spec.md`; the implementation surface is
stdlib + pydantic patterns already established in this repo (`contracts.py`,
`okf.py`, `verdicts.py`); the session is offline by rule, so there is no external
API/library/version decision to research. Key assumption 4 (D7 SDK hooks) belongs to
I3, not I2.

## Open Questions / Assumptions

- **[ASSUMPTION] Manifest schema breadth:** the pydantic contract is built
  polymorphic per spec §4 (validating `file`, `sql`, `http` source variants), while
  I2 implements ONLY the `file` connector execution path. This is the cheapest
  spec-complete reading; the plan may narrow it to `file`-only validation with
  justification (spec conformance for `sql` arrives in I4 either way).
- **[ASSUMPTION] Invocation surface:** spec §9 requires ingest to be an explicit
  operator command. For I2 a public programmatic API (module function) satisfies the
  procedural gate for tests; whether to also add a `python -m` CLI entry in I2 is a
  plan decision (cheap, but not demanded by the I2 verification set).
- **[ASSUMPTION] Module placement:** new module `src/portfolio_optimiser/ingest.py`
  (name/path is a plan decision; the constraint is only that `okf.py` stays unchanged
  and the module is MAF-free).
- **[ASSUMPTION] Expected-bundle authoring:** `expected-bundle/` in the golden case is
  generated ONCE by the implementation after review of its content (then frozen in
  git), rather than hand-typed — acceptable because the golden test's job is
  regression pinning (spec §11), while correctness is carried by the unit/load-bearing
  tests. The plan should sequence this so the golden freeze happens LAST, after all
  behaviour tests are green.

## Prior Attempts

None as implementation — this is the first ingest implementation session. Two prior
results are load-bearing context, not failures: (1) key assumption 1 was proven ad hoc
on 2026-07-03 (a synthetic bundle with provenance frontmatter passed
`navigate_bundle`/`bundle_context`; unknown fields preserved, context rendered) — I2
makes that proof durable; (2) I1 delivered the frozen spec into `shared/` with the
framework guard + structure test extended
(`tests/test_method_spec_loadbearing.py`, commits `4df2140` + `7ba0fae`), so the
contract I2 implements is already guarded in this repo.

## Metadata

- **Created:** 2026-07-03
- **Interview turns:** 0 (brief derived from frozen, operator-approved plan + spec;
  see provenance note at top)
- **Auto-research opted in:** no (0 topics)
- **Source:** manual

---

## How to continue

Manual (default):

```bash
# No research topics — go straight to plan:
/trekplan --project .claude/projects/2026-07-03-i2-ingest-csv-maf/

# Then execute:
/trekexecute --project .claude/projects/2026-07-03-i2-ingest-csv-maf/
```