docs(i2): brief + plan for ingest file/CSV session (I2)
This commit is contained in:
parent
994f5be11c
commit
fdcbaffc88
2 changed files with 995 additions and 0 deletions
263
.claude/projects/2026-07-03-i2-ingest-csv-maf/brief.md
Normal file
263
.claude/projects/2026-07-03-i2-ingest-csv-maf/brief.md
Normal file
|
|
@ -0,0 +1,263 @@
|
|||
---
|
||||
type: trekbrief
|
||||
brief_version: "2.2"
|
||||
created: 2026-07-03
|
||||
task: "I2 — MAF reference implementation of the ingest layer: file catalogue/CSV (offline)"
|
||||
slug: i2-ingest-csv-maf
|
||||
project_dir: .claude/projects/2026-07-03-i2-ingest-csv-maf/
|
||||
research_topics: 0
|
||||
research_status: skipped
|
||||
auto_research: false
|
||||
interview_turns: 0
|
||||
source: manual
|
||||
framing: preserve
|
||||
phase_signals:
|
||||
- phase: research
|
||||
effort: high
|
||||
model: fable
|
||||
- phase: plan
|
||||
effort: high
|
||||
model: fable
|
||||
- phase: execute
|
||||
effort: high
|
||||
model: fable
|
||||
- phase: review
|
||||
effort: high
|
||||
model: fable
|
||||
---
|
||||
|
||||
# Task: I2 — MAF reference implementation: ingest file catalogue/CSV (offline)
|
||||
|
||||
> Generated by `/trekbrief` on 2026-07-03.
|
||||
> This brief is the contract between requirements and planning. `/trekplan`
|
||||
> reads it to produce the implementation plan. Every decision in the plan must
|
||||
> trace back to content in this brief.
|
||||
>
|
||||
> **Provenance note (framing defense):** this brief was derived autonomously from
|
||||
> two FROZEN, operator-approved sources — the I2 section of
|
||||
> `docs/plan/2026-07-03-sesjonsplan-ingest.md` and the normative
|
||||
> `shared/ingest-spec.md` — under the STATE.md operating model ("Les STATE.md og
|
||||
> følg instruksjonene" = go-signal for the NEXT step; I2 is not gated). No
|
||||
> interview questions were asked because every answer already exists in written,
|
||||
> approved form; re-asking would violate the operator's recorded
|
||||
> plan-over-menu preference. `framing: preserve` is documented, not guessed:
|
||||
> I2 continues the frozen plan arc unchanged (STATE.md: "I2 [NESTE]").
|
||||
|
||||
## TL;DR
|
||||
|
||||
Implement the ingest layer's first source type on the MAF side, exactly as frozen in
|
||||
`shared/ingest-spec.md`: fail-fast pydantic manifest contract, CSV/file-catalogue
|
||||
connector, deterministic materialization into an OKF bundle with provenance
|
||||
frontmatter + index generation — offline, zero model calls, TDD. Framing: **preserve**
|
||||
— the "I2 [NESTE]" step of the operator-approved, frozen ingest session plan.
|
||||
|
||||
## Intent
|
||||
|
||||
The 8-step optimiser loop consumes hand-curated OKF bundles today; the frozen ingest
|
||||
målbilde closes the gap to real data sources while preserving the architecture-defining
|
||||
rule that **data reaches the model ONLY via OKF bundles** (no RAG, no query-time
|
||||
retrieval — method spec §3). I2 is the first implementation session of that program:
|
||||
it proves the frozen contract (`shared/ingest-spec.md`, authored and guarded in I1) is
|
||||
actually implementable from the spec alone, on the reference (MAF) side, using the
|
||||
offline `file` source type so the whole session stays at zero model calls and zero
|
||||
network (cost discipline D6). It also closes the plan's key assumption 2 — that an
|
||||
ingest manifest can be validated fail-fast without network — which is currently
|
||||
marked RISK with only pattern-level evidence (`contracts.py` is in-process). Finally,
|
||||
I2 turns the planning session's ephemeral ad-hoc proof (a synthetic generated bundle
|
||||
passing `navigate_bundle`/`bundle_context`) into a durable load-bearing test, so the
|
||||
seam can never silently detach.
|
||||
|
||||
## Goal
|
||||
|
||||
`uv run pytest` green (without network or credentials) on a repo where: a pydantic
|
||||
manifest contract validates ingest manifests fail-fast per spec §4 (polymorphic on
|
||||
`source.type`, verdict-layer reservation enforced before any source call); a
|
||||
file-catalogue/CSV connector executes extractions per §4 (boundary-checked path
|
||||
resolution, `max_rows` fail-fast); materialization per §5–§7 writes
|
||||
`ingest-{id}.md` concept files with the exact 7-key provenance frontmatter and
|
||||
generates/updates `index.md` per §6 (idempotent by target, curated/promoted links
|
||||
preserved); and a golden extraction case at `examples/ingest-golden-file/` (spec §11
|
||||
layout) reproduces byte for byte from fixture + fixed `ingested_at`. The existing
|
||||
loop consumes the generated bundle through UNCHANGED `okf.py` navigation code. Every
|
||||
seam in the I2 load-bearing set has a test proven RED at detach.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **SQL connector** — I4 (the pydantic manifest schema MAY validate the `sql`/`http`
|
||||
variants per §4, but no `sql` connector execution in I2; see Open Questions).
|
||||
- **HTTP/MCP connector and the network opt-in gate test** — I6 (gated: network
|
||||
capability). The `file` source type requires no network flag (§5, §8).
|
||||
- **Sibling repo (D7) work** — I3/I5 (gated: other repo). No files outside this repo.
|
||||
- **Commons/spec changes** — any needed spec change means STOP and a separate gated
|
||||
commons round (session-plan frame rule); never folded into I2. The golden case is
|
||||
authored LOCALLY in this repo in I2; sharing it via commons is a later gated round.
|
||||
- **Wiring ingest into `run_project`** — ingest is an explicit operator-run step
|
||||
BEFORE the loop (spec §2, §9); the run path is untouched.
|
||||
- **Extension points explicitly deferred by the spec:** per-row file splitting,
|
||||
multiple manifests per bundle, incremental re-ingest, approval registry /
|
||||
signed `approved_by` (spec §5, §9, §10).
|
||||
- **S12 release work** — independently gated (LICENSE is an operator-owned choice).
|
||||
|
||||
Note on spec conformance: full §1 conformance requires both `file` and `sql`
|
||||
connectors — I2 is deliberately a partial-conformance milestone; §1 conformance
|
||||
completes at I4. Reviewers must not read §1 as an I2 failure.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **The spec is normative and frozen.** Implement from `shared/ingest-spec.md` alone;
|
||||
MUST/MUST NOT clauses are binding. On any discovered spec defect or needed change:
|
||||
stop, report — separate gated commons round.
|
||||
- **Offline, deterministic, zero model calls** (D6 + spec §1, §11): tests run without
|
||||
credentials and without network; no live sources anywhere.
|
||||
- **TDD Iron Law:** no production code without a failing test first. Load-bearing
|
||||
tests must be proven RED at detach (detach → observe red → reattach), the method
|
||||
spec §11 regime.
|
||||
- **`okf.py` navigation code stays UNCHANGED** — the navigability criterion is
|
||||
explicitly "consumable via UNCHANGED `okf.py`". Reusing its existing primitives
|
||||
(e.g. the index-linking primitive, whose idempotent-by-target semantics §6
|
||||
references) is expected; modifying them is not in scope.
|
||||
- **Context-layer neutrality invariant:** `okf.py` and `shared/` remain pure stdlib,
|
||||
MAF-free (`test_okf_is_maf_free`). The new ingest module makes zero model calls and
|
||||
MUST NOT import `agent_framework`/`mcp` (it is a deterministic pre-loop step);
|
||||
pydantic is fine (already a core dependency, `contracts.py` pattern).
|
||||
- **Commons is PULL-ONLY**; nothing in I2 writes to commons or `shared/`.
|
||||
- **One session = one closed delivery:** green suite, STATE.md overwritten,
|
||||
commit + push to Forgejo; never a broken state across the session boundary.
|
||||
- **Surgical scope:** new ingest module + tests + golden example; no refactoring of
|
||||
adjacent code.
|
||||
|
||||
## Preferences
|
||||
|
||||
- Follow the existing `contracts.py` pydantic style for the manifest contract
|
||||
(fail-fast, explicit models, type hints everywhere).
|
||||
- Mirror the existing load-bearing test idiom (e.g.
|
||||
`tests/test_step8_promotion_loadbearing.py`): each test names its seam and fails
|
||||
when the seam is detached.
|
||||
- Reuse `okf.py`'s existing index-linking primitive for §6 index updates rather than
|
||||
reimplementing link semantics.
|
||||
- Golden case directory per spec §11 convention: `examples/ingest-golden-file/`
|
||||
with `manifest.json`, `fixture/`, `ingested-at.txt`, `expected-bundle/`.
|
||||
- Small synthetic fixture data (cost discipline — no large test payloads).
|
||||
- Code/comments/docs in English; Conventional Commits.
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
- **Bit-determinism:** same source content + manifest + `ingested_at` → byte-identical
|
||||
bundle; repeated runs idempotent (spec §10, §11).
|
||||
- **Zero new runtime dependencies** (pydantic and stdlib suffice; CSV via stdlib `csv`).
|
||||
- **Fail-fast, never silent:** malformed manifest, verdict-typed mapping, path escape,
|
||||
`max_rows` overflow, non-CSV-representable cell value, curated-file collision — all
|
||||
ERRORS, never warnings/truncation/coercion (spec §3, §4, §5, §8).
|
||||
- **File hygiene per spec §5:** LF-only, exactly one trailing newline, single-line
|
||||
frontmatter values, whitespace-collapsed `source_query`.
|
||||
- **Source calls are logged** (which source, when, row count — spec §8); test-visible
|
||||
via stdlib `logging`, no new dependency.
|
||||
- **Contract completeness:** the plan traces spec §12's cross-check table
|
||||
field-by-field (`manifest_version` … `generated`, golden-case entries) so no
|
||||
contract field is silently dropped — including `bundle_summary` (§4/§6) and the
|
||||
`ingest_manifest` stamp format `{stem}@{hash16}` (§5).
|
||||
|
||||
## Success Criteria
|
||||
|
||||
All commands run from the repo root; "without network/credentials" is the standing
|
||||
test condition (spec §11).
|
||||
|
||||
1. **Golden extraction, bit-deterministic:** a test materializes
|
||||
`examples/ingest-golden-file/` from `manifest.json` + `fixture/` +
|
||||
`ingested-at.txt` and compares against `expected-bundle/` file by file, byte for
|
||||
byte — `uv run pytest tests/ -k ingest` exits 0. Running materialization twice
|
||||
over the same target produces identical bytes (idempotence, §10).
|
||||
2. **Load-bearing: provenance detach** — with the §7 stamping detached (e.g.
|
||||
provenance keys not written), the suite goes RED. Proof recorded (detach → red →
|
||||
reattach) in the session log.
|
||||
3. **Load-bearing: navigability via UNCHANGED `okf.py`** — the generated bundle is
|
||||
consumed by the existing navigation code (`navigate_bundle`/`bundle_context`
|
||||
seam): generated files reachable via index cross-links, unknown provenance fields
|
||||
preserved. Test goes RED if index generation/linking is detached. `git diff` for
|
||||
the delivery shows no modification to `okf.py`.
|
||||
4. **Load-bearing: verdict reservation** — a manifest with `okf_type: verdict`
|
||||
(case-insensitive) or a generated-filename collision with `index.md`/the
|
||||
`promoted-verdict-*` namespace is REJECTED at validation, before any source call;
|
||||
test goes RED if the reservation is removed.
|
||||
5. **Load-bearing: re-ingest layer safety** — re-materialization over a bundle
|
||||
containing a promoted verdict (created via the real `promote_verdict`) preserves
|
||||
the verdict file AND its index link; test goes RED if replacement semantics stop
|
||||
honouring the ingest-stamp ownership rule (§3, §6).
|
||||
6. **Fail-fast manifest validation without network (closes key assumption 2):**
|
||||
malformed manifests (missing fields, bad `id` grammar, empty `extractions`,
|
||||
non-positive `max_rows`, unknown `source.type`) raise before any source access —
|
||||
asserted by tests that run with no network available.
|
||||
7. **Full suite + gates green:** `uv run pytest` exits 0 — the pre-existing suite
|
||||
passes unchanged (baseline captured at session start before any I2 change) plus
|
||||
the new ingest tests; `uv run ruff check .` exits 0, `uv run ruff format --check .`
|
||||
clean, `uv run mypy src` exits 0.
|
||||
8. **Honesty of artefacts:** no artefact delivered in I2 claims SQL/HTTP support,
|
||||
live-source integration, or commons sharing of the golden case. Concretely:
|
||||
case-insensitive grep for `sql`, `http`, `live`, `commons` over the new module's
|
||||
docstrings, new test files' docstrings, and any touched README/docs sections
|
||||
yields no support-claiming hits (mentions that explicitly DEFER — "I4", "I6",
|
||||
"extension point" — are fine); the plan fixes the exact grep invocation.
|
||||
|
||||
## Research Plan
|
||||
|
||||
No external research needed — the codebase and this brief contain sufficient context
|
||||
for planning. Rationale (recorded in lieu of an interview): the task is fully
|
||||
specified by the frozen `shared/ingest-spec.md`; the implementation surface is
|
||||
stdlib + pydantic patterns already established in this repo (`contracts.py`,
|
||||
`okf.py`, `verdicts.py`); the session is offline by rule, so there is no external
|
||||
API/library/version decision to research. Key assumption 4 (D7 SDK hooks) belongs to
|
||||
I3, not I2.
|
||||
|
||||
## Open Questions / Assumptions
|
||||
|
||||
- **[ASSUMPTION] Manifest schema breadth:** the pydantic contract is built
|
||||
polymorphic per spec §4 (validating `file`, `sql`, `http` source variants), while
|
||||
I2 implements ONLY the `file` connector execution path. This is the cheapest
|
||||
spec-complete reading; the plan may narrow it to `file`-only validation with
|
||||
justification (spec conformance for `sql` arrives in I4 either way).
|
||||
- **[ASSUMPTION] Invocation surface:** spec §9 requires ingest to be an explicit
|
||||
operator command. For I2 a public programmatic API (module function) satisfies the
|
||||
procedural gate for tests; whether to also add a `python -m` CLI entry in I2 is a
|
||||
plan decision (cheap, but not demanded by the I2 verification set).
|
||||
- **[ASSUMPTION] Module placement:** new module `src/portfolio_optimiser/ingest.py`
|
||||
(name/path is a plan decision; the constraint is only that `okf.py` stays unchanged
|
||||
and the module is MAF-free).
|
||||
- **[ASSUMPTION] Expected-bundle authoring:** `expected-bundle/` in the golden case is
|
||||
generated ONCE by the implementation after review of its content (then frozen in
|
||||
git), rather than hand-typed — acceptable because the golden test's job is
|
||||
regression pinning (spec §11), while correctness is carried by the unit/load-bearing
|
||||
tests. The plan should sequence this so the golden freeze happens LAST, after all
|
||||
behaviour tests are green.
|
||||
|
||||
## Prior Attempts
|
||||
|
||||
None as implementation — this is the first ingest implementation session. Two prior
|
||||
results are load-bearing context, not failures: (1) key assumption 1 was proven ad hoc
|
||||
on 2026-07-03 (a synthetic bundle with provenance frontmatter passed
|
||||
`navigate_bundle`/`bundle_context`; unknown fields preserved, context rendered) — I2
|
||||
makes that proof durable; (2) I1 delivered the frozen spec into `shared/` with the
|
||||
framework guard + structure test extended
|
||||
(`tests/test_method_spec_loadbearing.py`, commits `4df2140` + `7ba0fae`), so the
|
||||
contract I2 implements is already guarded in this repo.
|
||||
|
||||
## Metadata
|
||||
|
||||
- **Created:** 2026-07-03
|
||||
- **Interview turns:** 0 (brief derived from frozen, operator-approved plan + spec;
|
||||
see provenance note at top)
|
||||
- **Auto-research opted in:** no (0 topics)
|
||||
- **Source:** manual
|
||||
|
||||
---
|
||||
|
||||
## How to continue
|
||||
|
||||
Manual (default):
|
||||
|
||||
```bash
|
||||
# No research topics — go straight to plan:
|
||||
/trekplan --project .claude/projects/2026-07-03-i2-ingest-csv-maf/
|
||||
|
||||
# Then execute:
|
||||
/trekexecute --project .claude/projects/2026-07-03-i2-ingest-csv-maf/
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue