portfolio-optimiser/.claude/projects/2026-07-03-i2-ingest-csv-maf/brief.md

type	brief_version	created	task	slug	project_dir	research_topics	research_status	auto_research	interview_turns	source	framing	phase_signals
trekbrief	2.2	2026-07-03	I2 — MAF reference implementation of the ingest layer: file catalogue/CSV (offline)	i2-ingest-csv-maf	.claude/projects/2026-07-03-i2-ingest-csv-maf/	0	skipped	false	0	manual	preserve	phase effort model research high fable phase effort model plan high fable phase effort model execute high fable phase effort model review high fable

Task: I2 — MAF reference implementation: ingest file catalogue/CSV (offline)

Generated by /trekbrief on 2026-07-03. This brief is the contract between requirements and planning. /trekplan reads it to produce the implementation plan. Every decision in the plan must trace back to content in this brief.

Provenance note (framing defense): this brief was derived autonomously from two FROZEN, operator-approved sources — the I2 section of docs/plan/2026-07-03-sesjonsplan-ingest.md and the normative shared/ingest-spec.md — under the STATE.md operating model ("Les STATE.md og følg instruksjonene" = go-signal for the NEXT step; I2 is not gated). No interview questions were asked because every answer already exists in written, approved form; re-asking would violate the operator's recorded plan-over-menu preference. framing: preserve is documented, not guessed: I2 continues the frozen plan arc unchanged (STATE.md: "I2 [NESTE]").

TL;DR

Implement the ingest layer's first source type on the MAF side, exactly as frozen in shared/ingest-spec.md: fail-fast pydantic manifest contract, CSV/file-catalogue connector, deterministic materialization into an OKF bundle with provenance frontmatter + index generation — offline, zero model calls, TDD. Framing: preserve — the "I2 [NESTE]" step of the operator-approved, frozen ingest session plan.

Intent

The 8-step optimiser loop consumes hand-curated OKF bundles today; the frozen ingest målbilde closes the gap to real data sources while preserving the architecture-defining rule that data reaches the model ONLY via OKF bundles (no RAG, no query-time retrieval — method spec §3). I2 is the first implementation session of that program: it proves the frozen contract (shared/ingest-spec.md, authored and guarded in I1) is actually implementable from the spec alone, on the reference (MAF) side, using the offline file source type so the whole session stays at zero model calls and zero network (cost discipline D6). It also closes the plan's key assumption 2 — that an ingest manifest can be validated fail-fast without network — which is currently marked RISK with only pattern-level evidence (contracts.py is in-process). Finally, I2 turns the planning session's ephemeral ad-hoc proof (a synthetic generated bundle passing navigate_bundle/bundle_context) into a durable load-bearing test, so the seam can never silently detach.

Goal

uv run pytest green (without network or credentials) on a repo where: a pydantic manifest contract validates ingest manifests fail-fast per spec §4 (polymorphic on source.type, verdict-layer reservation enforced before any source call); a file-catalogue/CSV connector executes extractions per §4 (boundary-checked path resolution, max_rows fail-fast); materialization per §5–§7 writes ingest-{id}.md concept files with the exact 7-key provenance frontmatter and generates/updates index.md per §6 (idempotent by target, curated/promoted links preserved); and a golden extraction case at examples/ingest-golden-file/ (spec §11 layout) reproduces byte for byte from fixture + fixed ingested_at. The existing loop consumes the generated bundle through UNCHANGED okf.py navigation code. Every seam in the I2 load-bearing set has a test proven RED at detach.

Non-Goals

• SQL connector — I4 (the pydantic manifest schema MAY validate the sql/http variants per §4, but no sql connector execution in I2; see Open Questions).
• HTTP/MCP connector and the network opt-in gate test — I6 (gated: network capability). The file source type requires no network flag (§5, §8).
• Sibling repo (D7) work — I3/I5 (gated: other repo). No files outside this repo.
• Commons/spec changes — any needed spec change means STOP and a separate gated commons round (session-plan frame rule); never folded into I2. The golden case is authored LOCALLY in this repo in I2; sharing it via commons is a later gated round.
• Wiring ingest into run_project — ingest is an explicit operator-run step BEFORE the loop (spec §2, §9); the run path is untouched.
• Extension points explicitly deferred by the spec: per-row file splitting, multiple manifests per bundle, incremental re-ingest, approval registry / signed approved_by (spec §5, §9, §10).
• S12 release work — independently gated (LICENSE is an operator-owned choice).

Note on spec conformance: full §1 conformance requires both file and sql connectors — I2 is deliberately a partial-conformance milestone; §1 conformance completes at I4. Reviewers must not read §1 as an I2 failure.

Constraints

• The spec is normative and frozen. Implement from shared/ingest-spec.md alone; MUST/MUST NOT clauses are binding. On any discovered spec defect or needed change: stop, report — separate gated commons round.
• Offline, deterministic, zero model calls (D6 + spec §1, §11): tests run without credentials and without network; no live sources anywhere.
• TDD Iron Law: no production code without a failing test first. Load-bearing tests must be proven RED at detach (detach → observe red → reattach), the method spec §11 regime.
• okf.py navigation code stays UNCHANGED — the navigability criterion is explicitly "consumable via UNCHANGED okf.py". Reusing its existing primitives (e.g. the index-linking primitive, whose idempotent-by-target semantics §6 references) is expected; modifying them is not in scope.
• Context-layer neutrality invariant: okf.py and shared/ remain pure stdlib, MAF-free (test_okf_is_maf_free). The new ingest module makes zero model calls and MUST NOT import agent_framework/mcp (it is a deterministic pre-loop step); pydantic is fine (already a core dependency, contracts.py pattern).
• Commons is PULL-ONLY; nothing in I2 writes to commons or shared/.
• One session = one closed delivery: green suite, STATE.md overwritten, commit + push to Forgejo; never a broken state across the session boundary.
• Surgical scope: new ingest module + tests + golden example; no refactoring of adjacent code.

Preferences

• Follow the existing contracts.py pydantic style for the manifest contract (fail-fast, explicit models, type hints everywhere).
• Mirror the existing load-bearing test idiom (e.g. tests/test_step8_promotion_loadbearing.py): each test names its seam and fails when the seam is detached.
• Reuse okf.py's existing index-linking primitive for §6 index updates rather than reimplementing link semantics.
• Golden case directory per spec §11 convention: examples/ingest-golden-file/ with manifest.json, fixture/, ingested-at.txt, expected-bundle/.
• Small synthetic fixture data (cost discipline — no large test payloads).
• Code/comments/docs in English; Conventional Commits.

Non-Functional Requirements

• Bit-determinism: same source content + manifest + ingested_at → byte-identical bundle; repeated runs idempotent (spec §10, §11).
• Zero new runtime dependencies (pydantic and stdlib suffice; CSV via stdlib csv).
• Fail-fast, never silent: malformed manifest, verdict-typed mapping, path escape, max_rows overflow, non-CSV-representable cell value, curated-file collision — all ERRORS, never warnings/truncation/coercion (spec §3, §4, §5, §8).
• File hygiene per spec §5: LF-only, exactly one trailing newline, single-line frontmatter values, whitespace-collapsed source_query.
• Source calls are logged (which source, when, row count — spec §8); test-visible via stdlib logging, no new dependency.
• Contract completeness: the plan traces spec §12's cross-check table field-by-field (manifest_version … generated, golden-case entries) so no contract field is silently dropped — including bundle_summary (§4/§6) and the ingest_manifest stamp format {stem}@{hash16} (§5).

Success Criteria

All commands run from the repo root; "without network/credentials" is the standing test condition (spec §11).

1. Golden extraction, bit-deterministic: a test materializes examples/ingest-golden-file/ from manifest.json + fixture/ + ingested-at.txt and compares against expected-bundle/ file by file, byte for byte — uv run pytest tests/ -k ingest exits 0. Running materialization twice over the same target produces identical bytes (idempotence, §10).
2. Load-bearing: provenance detach — with the §7 stamping detached (e.g. provenance keys not written), the suite goes RED. Proof recorded (detach → red → reattach) in the session log.
3. Load-bearing: navigability via UNCHANGED okf.py — the generated bundle is consumed by the existing navigation code (navigate_bundle/bundle_context seam): generated files reachable via index cross-links, unknown provenance fields preserved. Test goes RED if index generation/linking is detached. git diff for the delivery shows no modification to okf.py.
4. Load-bearing: verdict reservation — a manifest with okf_type: verdict (case-insensitive) or a generated-filename collision with index.md/the promoted-verdict-* namespace is REJECTED at validation, before any source call; test goes RED if the reservation is removed.
5. Load-bearing: re-ingest layer safety — re-materialization over a bundle containing a promoted verdict (created via the real promote_verdict) preserves the verdict file AND its index link; test goes RED if replacement semantics stop honouring the ingest-stamp ownership rule (§3, §6).
6. Fail-fast manifest validation without network (closes key assumption 2): malformed manifests (missing fields, bad id grammar, empty extractions, non-positive max_rows, unknown source.type) raise before any source access — asserted by tests that run with no network available.
7. Full suite + gates green: uv run pytest exits 0 — the pre-existing suite passes unchanged (baseline captured at session start before any I2 change) plus the new ingest tests; uv run ruff check . exits 0, uv run ruff format --check . clean, uv run mypy src exits 0.
8. Honesty of artefacts: no artefact delivered in I2 claims SQL/HTTP support, live-source integration, or commons sharing of the golden case. Concretely: case-insensitive grep for sql, http, live, commons over the new module's docstrings, new test files' docstrings, and any touched README/docs sections yields no support-claiming hits (mentions that explicitly DEFER — "I4", "I6", "extension point" — are fine); the plan fixes the exact grep invocation.

Research Plan

No external research needed — the codebase and this brief contain sufficient context for planning. Rationale (recorded in lieu of an interview): the task is fully specified by the frozen shared/ingest-spec.md; the implementation surface is stdlib + pydantic patterns already established in this repo (contracts.py, okf.py, verdicts.py); the session is offline by rule, so there is no external API/library/version decision to research. Key assumption 4 (D7 SDK hooks) belongs to I3, not I2.

Open Questions / Assumptions

• [ASSUMPTION] Manifest schema breadth: the pydantic contract is built polymorphic per spec §4 (validating file, sql, http source variants), while I2 implements ONLY the file connector execution path. This is the cheapest spec-complete reading; the plan may narrow it to file-only validation with justification (spec conformance for sql arrives in I4 either way).
• [ASSUMPTION] Invocation surface: spec §9 requires ingest to be an explicit operator command. For I2 a public programmatic API (module function) satisfies the procedural gate for tests; whether to also add a python -m CLI entry in I2 is a plan decision (cheap, but not demanded by the I2 verification set).
• [ASSUMPTION] Module placement: new module src/portfolio_optimiser/ingest.py (name/path is a plan decision; the constraint is only that okf.py stays unchanged and the module is MAF-free).
• [ASSUMPTION] Expected-bundle authoring: expected-bundle/ in the golden case is generated ONCE by the implementation after review of its content (then frozen in git), rather than hand-typed — acceptable because the golden test's job is regression pinning (spec §11), while correctness is carried by the unit/load-bearing tests. The plan should sequence this so the golden freeze happens LAST, after all behaviour tests are green.

Prior Attempts

None as implementation — this is the first ingest implementation session. Two prior results are load-bearing context, not failures: (1) key assumption 1 was proven ad hoc on 2026-07-03 (a synthetic bundle with provenance frontmatter passed navigate_bundle/bundle_context; unknown fields preserved, context rendered) — I2 makes that proof durable; (2) I1 delivered the frozen spec into shared/ with the framework guard + structure test extended (tests/test_method_spec_loadbearing.py, commits 4df2140 + 7ba0fae), so the contract I2 implements is already guarded in this repo.

Metadata

• Created: 2026-07-03
• Interview turns: 0 (brief derived from frozen, operator-approved plan + spec; see provenance note at top)
• Auto-research opted in: no (0 topics)
• Source: manual

---

How to continue

Manual (default):

# No research topics — go straight to plan:
/trekplan --project .claude/projects/2026-07-03-i2-ingest-csv-maf/

# Then execute:
/trekexecute --project .claude/projects/2026-07-03-i2-ingest-csv-maf/