From fdcbaffc88965ace73f22ce92fb4b1e5f2898288 Mon Sep 17 00:00:00 2001 From: Kjell Tore Guttormsen Date: Fri, 3 Jul 2026 18:44:37 +0200 Subject: [PATCH] docs(i2): brief + plan for ingest file/CSV session (I2) --- .../2026-07-03-i2-ingest-csv-maf/brief.md | 263 +++++++ .../2026-07-03-i2-ingest-csv-maf/plan.md | 732 ++++++++++++++++++ 2 files changed, 995 insertions(+) create mode 100644 .claude/projects/2026-07-03-i2-ingest-csv-maf/brief.md create mode 100644 .claude/projects/2026-07-03-i2-ingest-csv-maf/plan.md diff --git a/.claude/projects/2026-07-03-i2-ingest-csv-maf/brief.md b/.claude/projects/2026-07-03-i2-ingest-csv-maf/brief.md new file mode 100644 index 0000000..856dff8 --- /dev/null +++ b/.claude/projects/2026-07-03-i2-ingest-csv-maf/brief.md @@ -0,0 +1,263 @@ +--- +type: trekbrief +brief_version: "2.2" +created: 2026-07-03 +task: "I2 — MAF reference implementation of the ingest layer: file catalogue/CSV (offline)" +slug: i2-ingest-csv-maf +project_dir: .claude/projects/2026-07-03-i2-ingest-csv-maf/ +research_topics: 0 +research_status: skipped +auto_research: false +interview_turns: 0 +source: manual +framing: preserve +phase_signals: + - phase: research + effort: high + model: fable + - phase: plan + effort: high + model: fable + - phase: execute + effort: high + model: fable + - phase: review + effort: high + model: fable +--- + +# Task: I2 — MAF reference implementation: ingest file catalogue/CSV (offline) + +> Generated by `/trekbrief` on 2026-07-03. +> This brief is the contract between requirements and planning. `/trekplan` +> reads it to produce the implementation plan. Every decision in the plan must +> trace back to content in this brief. +> +> **Provenance note (framing defense):** this brief was derived autonomously from +> two FROZEN, operator-approved sources — the I2 section of +> `docs/plan/2026-07-03-sesjonsplan-ingest.md` and the normative +> `shared/ingest-spec.md` — under the STATE.md operating model ("Les STATE.md og +> følg instruksjonene" = go-signal for the NEXT step; I2 is not gated). No +> interview questions were asked because every answer already exists in written, +> approved form; re-asking would violate the operator's recorded +> plan-over-menu preference. `framing: preserve` is documented, not guessed: +> I2 continues the frozen plan arc unchanged (STATE.md: "I2 [NESTE]"). + +## TL;DR + +Implement the ingest layer's first source type on the MAF side, exactly as frozen in +`shared/ingest-spec.md`: fail-fast pydantic manifest contract, CSV/file-catalogue +connector, deterministic materialization into an OKF bundle with provenance +frontmatter + index generation — offline, zero model calls, TDD. Framing: **preserve** +— the "I2 [NESTE]" step of the operator-approved, frozen ingest session plan. + +## Intent + +The 8-step optimiser loop consumes hand-curated OKF bundles today; the frozen ingest +målbilde closes the gap to real data sources while preserving the architecture-defining +rule that **data reaches the model ONLY via OKF bundles** (no RAG, no query-time +retrieval — method spec §3). I2 is the first implementation session of that program: +it proves the frozen contract (`shared/ingest-spec.md`, authored and guarded in I1) is +actually implementable from the spec alone, on the reference (MAF) side, using the +offline `file` source type so the whole session stays at zero model calls and zero +network (cost discipline D6). It also closes the plan's key assumption 2 — that an +ingest manifest can be validated fail-fast without network — which is currently +marked RISK with only pattern-level evidence (`contracts.py` is in-process). Finally, +I2 turns the planning session's ephemeral ad-hoc proof (a synthetic generated bundle +passing `navigate_bundle`/`bundle_context`) into a durable load-bearing test, so the +seam can never silently detach. + +## Goal + +`uv run pytest` green (without network or credentials) on a repo where: a pydantic +manifest contract validates ingest manifests fail-fast per spec §4 (polymorphic on +`source.type`, verdict-layer reservation enforced before any source call); a +file-catalogue/CSV connector executes extractions per §4 (boundary-checked path +resolution, `max_rows` fail-fast); materialization per §5–§7 writes +`ingest-{id}.md` concept files with the exact 7-key provenance frontmatter and +generates/updates `index.md` per §6 (idempotent by target, curated/promoted links +preserved); and a golden extraction case at `examples/ingest-golden-file/` (spec §11 +layout) reproduces byte for byte from fixture + fixed `ingested_at`. The existing +loop consumes the generated bundle through UNCHANGED `okf.py` navigation code. Every +seam in the I2 load-bearing set has a test proven RED at detach. + +## Non-Goals + +- **SQL connector** — I4 (the pydantic manifest schema MAY validate the `sql`/`http` + variants per §4, but no `sql` connector execution in I2; see Open Questions). +- **HTTP/MCP connector and the network opt-in gate test** — I6 (gated: network + capability). The `file` source type requires no network flag (§5, §8). +- **Sibling repo (D7) work** — I3/I5 (gated: other repo). No files outside this repo. +- **Commons/spec changes** — any needed spec change means STOP and a separate gated + commons round (session-plan frame rule); never folded into I2. The golden case is + authored LOCALLY in this repo in I2; sharing it via commons is a later gated round. +- **Wiring ingest into `run_project`** — ingest is an explicit operator-run step + BEFORE the loop (spec §2, §9); the run path is untouched. +- **Extension points explicitly deferred by the spec:** per-row file splitting, + multiple manifests per bundle, incremental re-ingest, approval registry / + signed `approved_by` (spec §5, §9, §10). +- **S12 release work** — independently gated (LICENSE is an operator-owned choice). + +Note on spec conformance: full §1 conformance requires both `file` and `sql` +connectors — I2 is deliberately a partial-conformance milestone; §1 conformance +completes at I4. Reviewers must not read §1 as an I2 failure. + +## Constraints + +- **The spec is normative and frozen.** Implement from `shared/ingest-spec.md` alone; + MUST/MUST NOT clauses are binding. On any discovered spec defect or needed change: + stop, report — separate gated commons round. +- **Offline, deterministic, zero model calls** (D6 + spec §1, §11): tests run without + credentials and without network; no live sources anywhere. +- **TDD Iron Law:** no production code without a failing test first. Load-bearing + tests must be proven RED at detach (detach → observe red → reattach), the method + spec §11 regime. +- **`okf.py` navigation code stays UNCHANGED** — the navigability criterion is + explicitly "consumable via UNCHANGED `okf.py`". Reusing its existing primitives + (e.g. the index-linking primitive, whose idempotent-by-target semantics §6 + references) is expected; modifying them is not in scope. +- **Context-layer neutrality invariant:** `okf.py` and `shared/` remain pure stdlib, + MAF-free (`test_okf_is_maf_free`). The new ingest module makes zero model calls and + MUST NOT import `agent_framework`/`mcp` (it is a deterministic pre-loop step); + pydantic is fine (already a core dependency, `contracts.py` pattern). +- **Commons is PULL-ONLY**; nothing in I2 writes to commons or `shared/`. +- **One session = one closed delivery:** green suite, STATE.md overwritten, + commit + push to Forgejo; never a broken state across the session boundary. +- **Surgical scope:** new ingest module + tests + golden example; no refactoring of + adjacent code. + +## Preferences + +- Follow the existing `contracts.py` pydantic style for the manifest contract + (fail-fast, explicit models, type hints everywhere). +- Mirror the existing load-bearing test idiom (e.g. + `tests/test_step8_promotion_loadbearing.py`): each test names its seam and fails + when the seam is detached. +- Reuse `okf.py`'s existing index-linking primitive for §6 index updates rather than + reimplementing link semantics. +- Golden case directory per spec §11 convention: `examples/ingest-golden-file/` + with `manifest.json`, `fixture/`, `ingested-at.txt`, `expected-bundle/`. +- Small synthetic fixture data (cost discipline — no large test payloads). +- Code/comments/docs in English; Conventional Commits. + +## Non-Functional Requirements + +- **Bit-determinism:** same source content + manifest + `ingested_at` → byte-identical + bundle; repeated runs idempotent (spec §10, §11). +- **Zero new runtime dependencies** (pydantic and stdlib suffice; CSV via stdlib `csv`). +- **Fail-fast, never silent:** malformed manifest, verdict-typed mapping, path escape, + `max_rows` overflow, non-CSV-representable cell value, curated-file collision — all + ERRORS, never warnings/truncation/coercion (spec §3, §4, §5, §8). +- **File hygiene per spec §5:** LF-only, exactly one trailing newline, single-line + frontmatter values, whitespace-collapsed `source_query`. +- **Source calls are logged** (which source, when, row count — spec §8); test-visible + via stdlib `logging`, no new dependency. +- **Contract completeness:** the plan traces spec §12's cross-check table + field-by-field (`manifest_version` … `generated`, golden-case entries) so no + contract field is silently dropped — including `bundle_summary` (§4/§6) and the + `ingest_manifest` stamp format `{stem}@{hash16}` (§5). + +## Success Criteria + +All commands run from the repo root; "without network/credentials" is the standing +test condition (spec §11). + +1. **Golden extraction, bit-deterministic:** a test materializes + `examples/ingest-golden-file/` from `manifest.json` + `fixture/` + + `ingested-at.txt` and compares against `expected-bundle/` file by file, byte for + byte — `uv run pytest tests/ -k ingest` exits 0. Running materialization twice + over the same target produces identical bytes (idempotence, §10). +2. **Load-bearing: provenance detach** — with the §7 stamping detached (e.g. + provenance keys not written), the suite goes RED. Proof recorded (detach → red → + reattach) in the session log. +3. **Load-bearing: navigability via UNCHANGED `okf.py`** — the generated bundle is + consumed by the existing navigation code (`navigate_bundle`/`bundle_context` + seam): generated files reachable via index cross-links, unknown provenance fields + preserved. Test goes RED if index generation/linking is detached. `git diff` for + the delivery shows no modification to `okf.py`. +4. **Load-bearing: verdict reservation** — a manifest with `okf_type: verdict` + (case-insensitive) or a generated-filename collision with `index.md`/the + `promoted-verdict-*` namespace is REJECTED at validation, before any source call; + test goes RED if the reservation is removed. +5. **Load-bearing: re-ingest layer safety** — re-materialization over a bundle + containing a promoted verdict (created via the real `promote_verdict`) preserves + the verdict file AND its index link; test goes RED if replacement semantics stop + honouring the ingest-stamp ownership rule (§3, §6). +6. **Fail-fast manifest validation without network (closes key assumption 2):** + malformed manifests (missing fields, bad `id` grammar, empty `extractions`, + non-positive `max_rows`, unknown `source.type`) raise before any source access — + asserted by tests that run with no network available. +7. **Full suite + gates green:** `uv run pytest` exits 0 — the pre-existing suite + passes unchanged (baseline captured at session start before any I2 change) plus + the new ingest tests; `uv run ruff check .` exits 0, `uv run ruff format --check .` + clean, `uv run mypy src` exits 0. +8. **Honesty of artefacts:** no artefact delivered in I2 claims SQL/HTTP support, + live-source integration, or commons sharing of the golden case. Concretely: + case-insensitive grep for `sql`, `http`, `live`, `commons` over the new module's + docstrings, new test files' docstrings, and any touched README/docs sections + yields no support-claiming hits (mentions that explicitly DEFER — "I4", "I6", + "extension point" — are fine); the plan fixes the exact grep invocation. + +## Research Plan + +No external research needed — the codebase and this brief contain sufficient context +for planning. Rationale (recorded in lieu of an interview): the task is fully +specified by the frozen `shared/ingest-spec.md`; the implementation surface is +stdlib + pydantic patterns already established in this repo (`contracts.py`, +`okf.py`, `verdicts.py`); the session is offline by rule, so there is no external +API/library/version decision to research. Key assumption 4 (D7 SDK hooks) belongs to +I3, not I2. + +## Open Questions / Assumptions + +- **[ASSUMPTION] Manifest schema breadth:** the pydantic contract is built + polymorphic per spec §4 (validating `file`, `sql`, `http` source variants), while + I2 implements ONLY the `file` connector execution path. This is the cheapest + spec-complete reading; the plan may narrow it to `file`-only validation with + justification (spec conformance for `sql` arrives in I4 either way). +- **[ASSUMPTION] Invocation surface:** spec §9 requires ingest to be an explicit + operator command. For I2 a public programmatic API (module function) satisfies the + procedural gate for tests; whether to also add a `python -m` CLI entry in I2 is a + plan decision (cheap, but not demanded by the I2 verification set). +- **[ASSUMPTION] Module placement:** new module `src/portfolio_optimiser/ingest.py` + (name/path is a plan decision; the constraint is only that `okf.py` stays unchanged + and the module is MAF-free). +- **[ASSUMPTION] Expected-bundle authoring:** `expected-bundle/` in the golden case is + generated ONCE by the implementation after review of its content (then frozen in + git), rather than hand-typed — acceptable because the golden test's job is + regression pinning (spec §11), while correctness is carried by the unit/load-bearing + tests. The plan should sequence this so the golden freeze happens LAST, after all + behaviour tests are green. + +## Prior Attempts + +None as implementation — this is the first ingest implementation session. Two prior +results are load-bearing context, not failures: (1) key assumption 1 was proven ad hoc +on 2026-07-03 (a synthetic bundle with provenance frontmatter passed +`navigate_bundle`/`bundle_context`; unknown fields preserved, context rendered) — I2 +makes that proof durable; (2) I1 delivered the frozen spec into `shared/` with the +framework guard + structure test extended +(`tests/test_method_spec_loadbearing.py`, commits `4df2140` + `7ba0fae`), so the +contract I2 implements is already guarded in this repo. + +## Metadata + +- **Created:** 2026-07-03 +- **Interview turns:** 0 (brief derived from frozen, operator-approved plan + spec; + see provenance note at top) +- **Auto-research opted in:** no (0 topics) +- **Source:** manual + +--- + +## How to continue + +Manual (default): + +```bash +# No research topics — go straight to plan: +/trekplan --project .claude/projects/2026-07-03-i2-ingest-csv-maf/ + +# Then execute: +/trekexecute --project .claude/projects/2026-07-03-i2-ingest-csv-maf/ +``` diff --git a/.claude/projects/2026-07-03-i2-ingest-csv-maf/plan.md b/.claude/projects/2026-07-03-i2-ingest-csv-maf/plan.md new file mode 100644 index 0000000..87b3a17 --- /dev/null +++ b/.claude/projects/2026-07-03-i2-ingest-csv-maf/plan.md @@ -0,0 +1,732 @@ +# I2 — MAF reference implementation: ingest file catalogue/CSV (offline) + +> **Plan quality: B+** (88/100) — APPROVE_WITH_NOTES +> +> Generated by trekplan v5.9 on 2026-07-03 — `plan_version: 1.7` + +## Context + +The 8-step optimiser loop consumes hand-curated OKF bundles today; the frozen ingest +målbilde closes the gap to real data sources while preserving the architecture-defining +rule that data reaches the model ONLY via OKF bundles (no RAG, no query-time retrieval — +method spec §3). I2 is the first implementation session of that program: it proves the +frozen contract (`shared/ingest-spec.md`, authored and guarded in I1) is implementable +from the spec alone, on the reference (MAF) side, using the offline `file` source type so +the session stays at zero model calls and zero network (cost discipline D6). It closes the +session plan's key assumption 2 (fail-fast manifest validation without network, currently +RISK-marked) and turns the planning session's ephemeral ad-hoc navigability proof into a +durable load-bearing test. Source: brief Intent, verbatim intent trace. + +Scope note (brief, Non-Goals): I2 is deliberately a **partial-conformance milestone** — +spec §1 full conformance (`file` + `sql`) completes at I4. The manifest contract validates +all three source variants; only the `file` connector executes in I2. + +## Architecture Diagram + +```mermaid +graph TD + subgraph "New in this plan" + M[manifest.json] -->|"load_manifest() — pydantic fail-fast (§4), verdict reservation (§3)"| C[ingest.py contract models] + C -->|"file connector: safe_resolve boundary + csv.reader + max_rows cap"| X[extraction rows in memory] + X -->|"render: escaped markdown table + §5/§7 provenance frontmatter"| S[staged files in memory] + S -->|"replacement semantics: delete stamped only, collision fail (§3)"| B[OKF bundle dir] + S -->|"index create/update: bundle_summary body, managed-line removal, link_in_index append (§6)"| I[index.md] + end + subgraph "Existing, UNCHANGED" + B --> N["okf.navigate_bundle / bundle_context"] + I --> N + V["verdicts.promote_verdict"] -->|"promoted-verdict-*.md + index link — must survive re-ingest"| B + R["retrieval.safe_resolve / PathSecurityError"] -.reused by.-> C + L["okf.link_in_index / render_frontmatter / parse_frontmatter"] -.reused by.-> S + end +``` + +## Codebase Analysis + +- **Tech stack:** Python ≥3.10, pydantic ≥2.11, `uv`, pytest ≥8 (`asyncio_mode=auto`; ingest + tests are plain sync), ruff (line-length 100), mypy (py3.10 target). No new runtime + dependencies needed: stdlib `csv`, `hashlib`, `logging`, `pathlib` + pydantic suffice. +- **Key patterns:** fail-fast pydantic startup contracts (`contracts.py:30-96`); frozen + dataclasses for value objects, `BaseModel` for validated external input; PEP 604 unions, + keyword-only options via bare `*`; condition-noun exceptions (`PromotionRefused` + `verdicts.py:447`, `PathSecurityError` `retrieval.py:24`, dominant base `RuntimeError`); + narrative module docstrings citing målbilde/spec sections; load-bearing tests with + detach-RED docstrings (`tests/test_step8_promotion_loadbearing.py`). +- **Relevant files:** `src/portfolio_optimiser/okf.py` (UNCHANGED — reuse + `parse_frontmatter:31`, `render_frontmatter:146`, `link_in_index:168`, + `navigate_bundle:113`, `bundle_context:133`), `src/portfolio_optimiser/retrieval.py` + (`safe_resolve:63`, `is_within_dir:48`, `PathSecurityError:24`), + `src/portfolio_optimiser/verdicts.py` (`promote_verdict:465` — structural analogue and + the SC5 test's real promotion; SHA-256 `[:16]` idiom at `verdicts.py:95,461`), + `tests/test_method_spec_loadbearing.py:156-174` (recorded I2 obligation: field-level + cross-check), `tests/conftest.py` (no ingest-relevant fixtures — none needed), + `shared/examples/bygg-energi-mikro/` (copytree fixture for the re-ingest safety test). +- **Reusable code:** `safe_resolve` for the §4 root boundary (fail-closed); + `render_frontmatter` for §5 key order + single-lining (collapses whitespace in ALL + values — acceptable: §5 requires single-line values everywhere; golden pins it); + `link_in_index` for §6 appends (idempotent by `](target)` — exactly the primitive §6 + references); `parse_frontmatter` for stamp detection (`generated == "true"` string + compare — it returns strings, never booleans); `promote_verdict` as the shape template + (explicit required `timestamp`, no wall-clock default). +- **Deliberately NOT reused:** `okf.write_concept_file` for the final write — it delegates + to `Path.write_text` (platform newline translation risk) and does not guarantee exactly + one trailing newline (`okf.py:164`). The ingest module builds each file's full content as + a string and writes bytes (`encode("utf-8")`) for the §5 LF-only guarantee on CONCEPT + files. Honest limit: index APPENDS reuse the frozen `link_in_index` (`Path.write_text`, + `okf.py:180`), so `index.md` LF-ness is platform-scoped — nil on the LF-only + development/target platform; recorded as Assumption 6. + `contracts.py` is NOT imported: it imports `backends` → `agent_framework` + (`contracts.py:25`), which would break the ingest module's MAF-free guard. Manifest + models live in `ingest.py` itself. +- **External tech (researched):** none — offline task, frozen spec, established repo + patterns (brief Research Plan: 0 topics). +- **Recent git activity:** I1 landed the spec + guards 1h before planning (`4df2140`, + `7ba0fae`); `tests/test_method_spec_loadbearing.py` is freshly touched — extend, don't + conflict. All work is sequential on `main`, clean tree. Commons subtree is PULL-ONLY + (violation observed + cleaned 2026-07-03); **nothing in this plan writes under `shared/`**. +- **Codebase-analysis correction (honesty):** the brief's Research Plan rationale mentions + JSON-Schema validation; exploration verified `datasource.py` contains no schema + validation — the fail-fast validation pattern lives in `contracts.py` as pydantic. The + plan follows the pydantic pattern (which the brief's Preferences already mandate). + +## Implementation Plan + +Every step is TDD (Iron Law): the step's tests are written and observed RED before the +production code that turns them green. `okf.py` and everything under `shared/` are +untouchable in every step (see per-step `forbidden_paths`). All commands run offline. + +### Step 1: Fail-fast manifest contract (pydantic, polymorphic, verdict reservation) + +- **Files:** `src/portfolio_optimiser/ingest.py` (new), `tests/test_ingest_manifest.py` (new) +- **Changes:** Create the ingest module with its narrative docstring (purpose, spec § + references, MAF-free invariant, D7-portability) and the manifest contract per spec §4: + `FileSource` (`type: Literal["file"]`, `root: str`), `SqlSource` (`type: Literal["sql"]`, + `connection_ref: str`), `HttpSource` (`type: Literal["http"]`, `base_url: str` — reject + embedded credentials (userinfo `@`) via validator, optional `credential_ref: str`), all + with common `id: str` constrained to `^[a-z0-9][a-z0-9-]*$`; `source` as a discriminated + union on `type` (`Field(discriminator="type")`). `Extraction`: `id` (same grammar), + `title` (non-empty, single-line — reject `\n`/`\r`), `query: str`, `okf_type` (non-empty, + single-line, `@field_validator` rejecting `verdict` case-insensitively — spec §3, before + any source call), `max_rows: int = Field(gt=0)`. `ManifestV1`: `manifest_version: + Literal[1]`, `source`, `bundle_summary: str` (non-empty), `extractions` (`min_length=1`, + `@model_validator` rejecting duplicate extraction ids). Public loader + `load_manifest(path: str | Path) -> tuple[ManifestV1, str]` that reads raw bytes ONCE, + computes the `{stem}@{sha256(raw)[:16]}` stamp (spec §5), `json.loads`, validates — + raising `pydantic.ValidationError` / `json.JSONDecodeError` before ANY source access. + Define `IngestError(RuntimeError)` (condition-noun family; used by later steps for + materialization-time refusals). `title` is whitespace-normalized at validation + (`" ".join(title.split())` after the single-line check) so the frontmatter rendering + (which collapses runs, `okf.py:154`) and the index label are guaranteed identical. + The `HttpSource` credential validator is defined precisely: reject when + `urllib.parse.urlsplit(base_url)` yields a non-`None` `username` or `password` + (userinfo is THE credential-embedding URL mechanism §4 targets; query-param token + heuristics are out of scope — documented in the validator docstring). + **Baseline capture (SC7):** BEFORE writing anything in this step, run `uv run pytest -q` + on the clean tree and record the summary line in the session log — this is the + pre-existing-suite baseline SC7's "unchanged-green" is checked against. (new file) +- **Reuses:** pydantic contract idiom from `contracts.py:30-96` (`Field` constraints, + `Literal` enums, `@model_validator(mode="after")` returning `self`); SHA-256 `[:16]` + idiom from `verdicts.py:95`; docstring style from `okf.py:1-12`. +- **Test first:** + - File: `tests/test_ingest_manifest.py` (new) + - Verifies: valid `file` manifest loads with correct stamp (stem + 16-hex of raw bytes); + each missing top-level field raises `ValidationError`; `manifest_version != 1` rejected; + empty `extractions` rejected; bad `id` grammar (uppercase, leading `-`, empty) rejected + for both source and extraction ids; duplicate extraction ids rejected; `max_rows <= 0` + rejected; unknown `source.type` rejected; `file` source missing `root` rejected; + `okf_type` of `verdict`/`Verdict`/`VERDICT` rejected; multi-line `title` rejected; + `base_url` with embedded credentials rejected; **fail-fast ordering proof**: a manifest + that is malformed AND whose `source.root` does not exist raises `ValidationError` + without touching the missing root (the ingest analogue of + `test_no_chat_client_call_on_malformed_contract` in `tests/test_contracts.py`); + `sql`/`http` variants VALIDATE (schema breadth per brief assumption) though no + connector executes them in I2. + - Pattern: `tests/test_contracts.py` (inline dict constants, one `pytest.raises` per + malformation) +- **Verify:** `uv run pytest tests/test_ingest_manifest.py -q` → expected: all pass (after + observed RED on first run without implementation) +- **On failure:** revert — `rm -f src/portfolio_optimiser/ingest.py tests/test_ingest_manifest.py` + (both files are NEW and untracked until this step's checkpoint — `git checkout --` cannot + remove untracked files) +- **Checkpoint:** `git commit -m "feat(ingest): fail-fast manifest contract with verdict reservation (I2)"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - src/portfolio_optimiser/ingest.py + - tests/test_ingest_manifest.py + min_file_count: 2 + commit_message_pattern: "^feat\\(ingest\\): fail-fast manifest contract with verdict reservation \\(I2\\)$" + bash_syntax_check: [] + forbidden_paths: + - src/portfolio_optimiser/okf.py + - shared/ingest-spec.md + must_contain: + - path: src/portfolio_optimiser/ingest.py + pattern: "load_manifest" + - path: src/portfolio_optimiser/ingest.py + pattern: "discriminator" + ``` + +### Step 2: CSV connector — boundary check, streaming row cap, cell escaping, table body + +- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_materialize.py` (new) +- **Changes:** Add the `file` connector + renderer. Connector: resolve the manifest's + `root` against the **manifest file's parent directory** when relative (pinned decision — + extraction must not depend on cwd), then `safe_resolve(root, extraction.query)` for the + fail-closed §4 boundary check; open with `encoding="utf-8-sig"` (BOM never leaks into + the first header cell) and `newline=""`; parse with `csv.reader`. First row = header; + fail (`IngestError`) on empty file (no header). Stream data rows counting against + `max_rows` — raise `IngestError` the moment the cap is exceeded (§8: error, never silent + truncation; no fetch-all-then-count). Fail on ragged rows (data row width ≠ header + width) — silent padding/truncation is coercion. Renderer: cell text **verbatim** with + escaping in this exact order: `\` → `\\` FIRST, then `|` → `\|`, then any newline (CR, + LF, CRLF) → single space (pinned decision: on the `file`/CSV path every cell is a + string, so §5's integer/float/NULL clauses bite typed `sql` values in I4 — a spec-text + reading recorded as Assumption 1). Newline replacement treats CRLF as ONE unit: replace + `\r\n` first, then lone `\r`, then lone `\n` — each with a single space (a per-character + replace would turn CRLF into two spaces). A well-formed manifest whose `root` directory + or resolved `query` CSV does not exist raises `IngestError` with a clear message (never a + bare `FileNotFoundError` leaking from `open()`). Body = markdown table: `| h1 | h2 |` + header row, `| --- | --- |` separator, data rows in source order, LF line endings, ending + in exactly one `\n`. **Header cells are escaped identically to data cells** (spec §5 + separates "column names" from "cell values" without giving headers their own rule — + pinned decision: same escaping/newline-collapse, or a header containing `|` or an + embedded newline breaks the table). The connector itself is PURE (returns header + + rows); the §8 source-call logging lives in `materialize` (Step 3), which owns + `ingested_at`. +- **Reuses:** `retrieval.safe_resolve`/`PathSecurityError` (`retrieval.py:63,24`) — + identical fail-closed semantics `okf.py` itself relies on. +- **Test first:** + - File: `tests/test_ingest_materialize.py` (new) + - Verifies: table rendering (header/separator/rows, source order); escaping of `\`, + `|`, embedded quoted newline (one assertion each, plus one cell containing BOTH `\` + and `|` proving escape order, plus an explicit embedded-CRLF cell → exactly ONE + space); a header cell containing `|` and one containing an embedded newline → + escaped/collapsed identically to data cells; header-only CSV → header + separator, + zero data rows; empty CSV file → `IngestError`; ragged row → `IngestError`; `max_rows` fixture with cap+1 rows → + `IngestError`; `../escape` query → `PathSecurityError`; missing `root` dir and + missing `query` file → `IngestError` (not bare `FileNotFoundError`); BOM fixture → + first header cell clean. + - Pattern: local `tmp_path` CSV-builder helper (conftest's LLM fixtures are irrelevant + to ingest — no conftest changes) +- **Verify:** `uv run pytest tests/test_ingest_materialize.py -q` → expected: all pass +- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py && rm -f tests/test_ingest_materialize.py` + (`ingest.py` is tracked after Step 1's checkpoint; the test file is new in this step) +- **Checkpoint:** `git commit -m "feat(ingest): CSV connector with boundary check, row cap and escaped table body (I2)"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - src/portfolio_optimiser/ingest.py + - tests/test_ingest_materialize.py + min_file_count: 2 + commit_message_pattern: "^feat\\(ingest\\): CSV connector with boundary check, row cap and escaped table body \\(I2\\)$" + bash_syntax_check: [] + forbidden_paths: + - src/portfolio_optimiser/okf.py + - tests/conftest.py + must_contain: + - path: src/portfolio_optimiser/ingest.py + pattern: "safe_resolve" + - path: src/portfolio_optimiser/ingest.py + pattern: "utf-8-sig" + ``` + +### Step 3: Materialization — provenance frontmatter, LF byte-writer, in-memory staging + +- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_materialize.py` +- **Changes:** Add the per-extraction concept-file builder and the public entry point + `materialize(manifest_path: str | Path, bundle_dir: str | Path, *, ingested_at: str) -> + list[Path]` (explicit required `ingested_at` keyword, NO wall-clock default — mirrors + `promote_verdict`; validate against the regex `^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$` — + NOT `datetime.fromisoformat`, which rejects the `Z` suffix on Python 3.10, the repo's + version floor — then stamp the string verbatim). This programmatic function IS the I2 + invocation surface (recorded decision, resolving the brief's open question: a `python -m` + CLI is not demanded by the I2 verification set and is deferred). `materialize` performs + the §8 source-call log per extraction on logger `portfolio_optimiser.ingest`: source id, + the `ingested_at` argument (deterministic "when" — never wall-clock), row count; never + log cell contents or (in later source types) resolved secrets. It creates the bundle + directory first: `Path(bundle_dir).mkdir(parents=True, exist_ok=True)` — `safe_resolve` + never creates directories, and unlike `okf.write_concept_file` the byte-writer has no + implicit mkdir. Frontmatter: ordered dict with EXACTLY the §5 keys in §5 order — `type` + (= `okf_type`), `title`, `source_system` (= `source.id`), `source_query` (the query, + whitespace-collapsed), `ingested_at`, `ingest_manifest` (the Step-1 stamp), `generated` + (`true`) — rendered via `okf.render_frontmatter` (insertion-ordered, single-lining). + Filenames: `ingest-{extraction.id}.md` (§5; grammar keeps the namespace disjoint from + `index.md`/`promoted-verdict-*` by construction). File content assembled as ONE string + `---\n{fm}\n---\n\n{body}` (body already ends in exactly one `\n`) and written as bytes + (`content.encode("utf-8")`) to a `safe_resolve`d path — LF-only and exactly one trailing + newline guaranteed on every platform (§5). **All extractions execute and render in + memory BEFORE the first disk mutation** (crash-window mitigation for §5's non-atomic + replace sequence; recovery = idempotent re-run, §10). +- **Reuses:** `okf.render_frontmatter` (`okf.py:146`), `retrieval.safe_resolve`; + `promote_verdict`'s explicit-timestamp shape (`verdicts.py:465-472`). +- **Test first:** + - File: `tests/test_ingest_materialize.py` (extend) + - Verifies: generated file's RAW TEXT has the 7 frontmatter keys in exact §5 order (read + the file text, not just the parsed dict); `okf.parse_frontmatter` round-trips the + values (`generated == "true"` as string; `ingest_manifest == {stem}@{hash16}` computed + independently in the test from the manifest's raw bytes; `ingested_at` verbatim; + `source_query` whitespace-collapsed); file bytes are LF-only with exactly one trailing + `\n` (`read_bytes()` assertions: no `\r`, endswith exactly one `b"\n"`); + missing/invalid `ingested_at` (no argument has no default — non-`Z`/non-ISO string + raises `ValueError`; the golden timestamp `2026-07-03T12:00:00Z` passes the regex); + materializing into a NON-EXISTENT nested `bundle_dir` succeeds (mkdir path); §8 log + record (via `caplog`) carries source id, the explicit `ingested_at` string, and row + count; two `materialize` runs with identical inputs produce byte-identical files + (§10 idempotence at file level). + - Pattern: `tests/test_okf.py` frontmatter round-trip assertions +- **Verify:** `uv run pytest tests/test_ingest_materialize.py -q` → expected: all pass +- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_materialize.py` +- **Checkpoint:** `git commit -m "feat(ingest): deterministic materialization with §5/§7 provenance stamp (I2)"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - src/portfolio_optimiser/ingest.py + - tests/test_ingest_materialize.py + min_file_count: 2 + commit_message_pattern: "^feat\\(ingest\\): deterministic materialization with .5/.7 provenance stamp \\(I2\\)$" + bash_syntax_check: [] + forbidden_paths: + - src/portfolio_optimiser/okf.py + must_contain: + - path: src/portfolio_optimiser/ingest.py + pattern: "ingested_at" + - path: src/portfolio_optimiser/ingest.py + pattern: "ingest_manifest" + ``` + +### Step 4: Index generation + stamped-replacement semantics + +- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_materialize.py` +- **Changes:** Complete `materialize` with §3/§5/§6 bundle semantics. (a) **Ownership + scan:** classify a bundle file as ingest-owned iff `okf.parse_frontmatter` yields + `generated == "true"` AND an `ingest_manifest` key (string comparisons; sorted directory + scan for determinism). (b) **Collision gate:** if a target filename `ingest-{id}.md` + exists WITHOUT the stamp → `IngestError`, nothing written (§3: never overwrite curated + content). (c) **Replace:** delete every stamped file, then write the new staged set. + (d) **Index:** if `index.md` missing → create it with `bundle_summary` as the body (no + frontmatter — spec-minimal, pinned decision; content = `{bundle_summary}\n` written as + bytes). If present → preserve every unmanaged line byte-for-byte; remove ONLY + whole lines matching the managed-link form `- [