732 lines
54 KiB
Markdown
732 lines
54 KiB
Markdown
# I2 — MAF reference implementation: ingest file catalogue/CSV (offline)
|
||
|
||
> **Plan quality: B+** (88/100) — APPROVE_WITH_NOTES
|
||
>
|
||
> Generated by trekplan v5.9 on 2026-07-03 — `plan_version: 1.7`
|
||
|
||
## Context
|
||
|
||
The 8-step optimiser loop consumes hand-curated OKF bundles today; the frozen ingest
|
||
målbilde closes the gap to real data sources while preserving the architecture-defining
|
||
rule that data reaches the model ONLY via OKF bundles (no RAG, no query-time retrieval —
|
||
method spec §3). I2 is the first implementation session of that program: it proves the
|
||
frozen contract (`shared/ingest-spec.md`, authored and guarded in I1) is implementable
|
||
from the spec alone, on the reference (MAF) side, using the offline `file` source type so
|
||
the session stays at zero model calls and zero network (cost discipline D6). It closes the
|
||
session plan's key assumption 2 (fail-fast manifest validation without network, currently
|
||
RISK-marked) and turns the planning session's ephemeral ad-hoc navigability proof into a
|
||
durable load-bearing test. Source: brief Intent, verbatim intent trace.
|
||
|
||
Scope note (brief, Non-Goals): I2 is deliberately a **partial-conformance milestone** —
|
||
spec §1 full conformance (`file` + `sql`) completes at I4. The manifest contract validates
|
||
all three source variants; only the `file` connector executes in I2.
|
||
|
||
## Architecture Diagram
|
||
|
||
```mermaid
|
||
graph TD
|
||
subgraph "New in this plan"
|
||
M[manifest.json] -->|"load_manifest() — pydantic fail-fast (§4), verdict reservation (§3)"| C[ingest.py contract models]
|
||
C -->|"file connector: safe_resolve boundary + csv.reader + max_rows cap"| X[extraction rows in memory]
|
||
X -->|"render: escaped markdown table + §5/§7 provenance frontmatter"| S[staged files in memory]
|
||
S -->|"replacement semantics: delete stamped only, collision fail (§3)"| B[OKF bundle dir]
|
||
S -->|"index create/update: bundle_summary body, managed-line removal, link_in_index append (§6)"| I[index.md]
|
||
end
|
||
subgraph "Existing, UNCHANGED"
|
||
B --> N["okf.navigate_bundle / bundle_context"]
|
||
I --> N
|
||
V["verdicts.promote_verdict"] -->|"promoted-verdict-*.md + index link — must survive re-ingest"| B
|
||
R["retrieval.safe_resolve / PathSecurityError"] -.reused by.-> C
|
||
L["okf.link_in_index / render_frontmatter / parse_frontmatter"] -.reused by.-> S
|
||
end
|
||
```
|
||
|
||
## Codebase Analysis
|
||
|
||
- **Tech stack:** Python ≥3.10, pydantic ≥2.11, `uv`, pytest ≥8 (`asyncio_mode=auto`; ingest
|
||
tests are plain sync), ruff (line-length 100), mypy (py3.10 target). No new runtime
|
||
dependencies needed: stdlib `csv`, `hashlib`, `logging`, `pathlib` + pydantic suffice.
|
||
- **Key patterns:** fail-fast pydantic startup contracts (`contracts.py:30-96`); frozen
|
||
dataclasses for value objects, `BaseModel` for validated external input; PEP 604 unions,
|
||
keyword-only options via bare `*`; condition-noun exceptions (`PromotionRefused`
|
||
`verdicts.py:447`, `PathSecurityError` `retrieval.py:24`, dominant base `RuntimeError`);
|
||
narrative module docstrings citing målbilde/spec sections; load-bearing tests with
|
||
detach-RED docstrings (`tests/test_step8_promotion_loadbearing.py`).
|
||
- **Relevant files:** `src/portfolio_optimiser/okf.py` (UNCHANGED — reuse
|
||
`parse_frontmatter:31`, `render_frontmatter:146`, `link_in_index:168`,
|
||
`navigate_bundle:113`, `bundle_context:133`), `src/portfolio_optimiser/retrieval.py`
|
||
(`safe_resolve:63`, `is_within_dir:48`, `PathSecurityError:24`),
|
||
`src/portfolio_optimiser/verdicts.py` (`promote_verdict:465` — structural analogue and
|
||
the SC5 test's real promotion; SHA-256 `[:16]` idiom at `verdicts.py:95,461`),
|
||
`tests/test_method_spec_loadbearing.py:156-174` (recorded I2 obligation: field-level
|
||
cross-check), `tests/conftest.py` (no ingest-relevant fixtures — none needed),
|
||
`shared/examples/bygg-energi-mikro/` (copytree fixture for the re-ingest safety test).
|
||
- **Reusable code:** `safe_resolve` for the §4 root boundary (fail-closed);
|
||
`render_frontmatter` for §5 key order + single-lining (collapses whitespace in ALL
|
||
values — acceptable: §5 requires single-line values everywhere; golden pins it);
|
||
`link_in_index` for §6 appends (idempotent by `](target)` — exactly the primitive §6
|
||
references); `parse_frontmatter` for stamp detection (`generated == "true"` string
|
||
compare — it returns strings, never booleans); `promote_verdict` as the shape template
|
||
(explicit required `timestamp`, no wall-clock default).
|
||
- **Deliberately NOT reused:** `okf.write_concept_file` for the final write — it delegates
|
||
to `Path.write_text` (platform newline translation risk) and does not guarantee exactly
|
||
one trailing newline (`okf.py:164`). The ingest module builds each file's full content as
|
||
a string and writes bytes (`encode("utf-8")`) for the §5 LF-only guarantee on CONCEPT
|
||
files. Honest limit: index APPENDS reuse the frozen `link_in_index` (`Path.write_text`,
|
||
`okf.py:180`), so `index.md` LF-ness is platform-scoped — nil on the LF-only
|
||
development/target platform; recorded as Assumption 6.
|
||
`contracts.py` is NOT imported: it imports `backends` → `agent_framework`
|
||
(`contracts.py:25`), which would break the ingest module's MAF-free guard. Manifest
|
||
models live in `ingest.py` itself.
|
||
- **External tech (researched):** none — offline task, frozen spec, established repo
|
||
patterns (brief Research Plan: 0 topics).
|
||
- **Recent git activity:** I1 landed the spec + guards 1h before planning (`4df2140`,
|
||
`7ba0fae`); `tests/test_method_spec_loadbearing.py` is freshly touched — extend, don't
|
||
conflict. All work is sequential on `main`, clean tree. Commons subtree is PULL-ONLY
|
||
(violation observed + cleaned 2026-07-03); **nothing in this plan writes under `shared/`**.
|
||
- **Codebase-analysis correction (honesty):** the brief's Research Plan rationale mentions
|
||
JSON-Schema validation; exploration verified `datasource.py` contains no schema
|
||
validation — the fail-fast validation pattern lives in `contracts.py` as pydantic. The
|
||
plan follows the pydantic pattern (which the brief's Preferences already mandate).
|
||
|
||
## Implementation Plan
|
||
|
||
Every step is TDD (Iron Law): the step's tests are written and observed RED before the
|
||
production code that turns them green. `okf.py` and everything under `shared/` are
|
||
untouchable in every step (see per-step `forbidden_paths`). All commands run offline.
|
||
|
||
### Step 1: Fail-fast manifest contract (pydantic, polymorphic, verdict reservation)
|
||
|
||
- **Files:** `src/portfolio_optimiser/ingest.py` (new), `tests/test_ingest_manifest.py` (new)
|
||
- **Changes:** Create the ingest module with its narrative docstring (purpose, spec §
|
||
references, MAF-free invariant, D7-portability) and the manifest contract per spec §4:
|
||
`FileSource` (`type: Literal["file"]`, `root: str`), `SqlSource` (`type: Literal["sql"]`,
|
||
`connection_ref: str`), `HttpSource` (`type: Literal["http"]`, `base_url: str` — reject
|
||
embedded credentials (userinfo `@`) via validator, optional `credential_ref: str`), all
|
||
with common `id: str` constrained to `^[a-z0-9][a-z0-9-]*$`; `source` as a discriminated
|
||
union on `type` (`Field(discriminator="type")`). `Extraction`: `id` (same grammar),
|
||
`title` (non-empty, single-line — reject `\n`/`\r`), `query: str`, `okf_type` (non-empty,
|
||
single-line, `@field_validator` rejecting `verdict` case-insensitively — spec §3, before
|
||
any source call), `max_rows: int = Field(gt=0)`. `ManifestV1`: `manifest_version:
|
||
Literal[1]`, `source`, `bundle_summary: str` (non-empty), `extractions` (`min_length=1`,
|
||
`@model_validator` rejecting duplicate extraction ids). Public loader
|
||
`load_manifest(path: str | Path) -> tuple[ManifestV1, str]` that reads raw bytes ONCE,
|
||
computes the `{stem}@{sha256(raw)[:16]}` stamp (spec §5), `json.loads`, validates —
|
||
raising `pydantic.ValidationError` / `json.JSONDecodeError` before ANY source access.
|
||
Define `IngestError(RuntimeError)` (condition-noun family; used by later steps for
|
||
materialization-time refusals). `title` is whitespace-normalized at validation
|
||
(`" ".join(title.split())` after the single-line check) so the frontmatter rendering
|
||
(which collapses runs, `okf.py:154`) and the index label are guaranteed identical.
|
||
The `HttpSource` credential validator is defined precisely: reject when
|
||
`urllib.parse.urlsplit(base_url)` yields a non-`None` `username` or `password`
|
||
(userinfo is THE credential-embedding URL mechanism §4 targets; query-param token
|
||
heuristics are out of scope — documented in the validator docstring).
|
||
**Baseline capture (SC7):** BEFORE writing anything in this step, run `uv run pytest -q`
|
||
on the clean tree and record the summary line in the session log — this is the
|
||
pre-existing-suite baseline SC7's "unchanged-green" is checked against. (new file)
|
||
- **Reuses:** pydantic contract idiom from `contracts.py:30-96` (`Field` constraints,
|
||
`Literal` enums, `@model_validator(mode="after")` returning `self`); SHA-256 `[:16]`
|
||
idiom from `verdicts.py:95`; docstring style from `okf.py:1-12`.
|
||
- **Test first:**
|
||
- File: `tests/test_ingest_manifest.py` (new)
|
||
- Verifies: valid `file` manifest loads with correct stamp (stem + 16-hex of raw bytes);
|
||
each missing top-level field raises `ValidationError`; `manifest_version != 1` rejected;
|
||
empty `extractions` rejected; bad `id` grammar (uppercase, leading `-`, empty) rejected
|
||
for both source and extraction ids; duplicate extraction ids rejected; `max_rows <= 0`
|
||
rejected; unknown `source.type` rejected; `file` source missing `root` rejected;
|
||
`okf_type` of `verdict`/`Verdict`/`VERDICT` rejected; multi-line `title` rejected;
|
||
`base_url` with embedded credentials rejected; **fail-fast ordering proof**: a manifest
|
||
that is malformed AND whose `source.root` does not exist raises `ValidationError`
|
||
without touching the missing root (the ingest analogue of
|
||
`test_no_chat_client_call_on_malformed_contract` in `tests/test_contracts.py`);
|
||
`sql`/`http` variants VALIDATE (schema breadth per brief assumption) though no
|
||
connector executes them in I2.
|
||
- Pattern: `tests/test_contracts.py` (inline dict constants, one `pytest.raises` per
|
||
malformation)
|
||
- **Verify:** `uv run pytest tests/test_ingest_manifest.py -q` → expected: all pass (after
|
||
observed RED on first run without implementation)
|
||
- **On failure:** revert — `rm -f src/portfolio_optimiser/ingest.py tests/test_ingest_manifest.py`
|
||
(both files are NEW and untracked until this step's checkpoint — `git checkout --` cannot
|
||
remove untracked files)
|
||
- **Checkpoint:** `git commit -m "feat(ingest): fail-fast manifest contract with verdict reservation (I2)"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/ingest.py
|
||
- tests/test_ingest_manifest.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(ingest\\): fail-fast manifest contract with verdict reservation \\(I2\\)$"
|
||
bash_syntax_check: []
|
||
forbidden_paths:
|
||
- src/portfolio_optimiser/okf.py
|
||
- shared/ingest-spec.md
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/ingest.py
|
||
pattern: "load_manifest"
|
||
- path: src/portfolio_optimiser/ingest.py
|
||
pattern: "discriminator"
|
||
```
|
||
|
||
### Step 2: CSV connector — boundary check, streaming row cap, cell escaping, table body
|
||
|
||
- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_materialize.py` (new)
|
||
- **Changes:** Add the `file` connector + renderer. Connector: resolve the manifest's
|
||
`root` against the **manifest file's parent directory** when relative (pinned decision —
|
||
extraction must not depend on cwd), then `safe_resolve(root, extraction.query)` for the
|
||
fail-closed §4 boundary check; open with `encoding="utf-8-sig"` (BOM never leaks into
|
||
the first header cell) and `newline=""`; parse with `csv.reader`. First row = header;
|
||
fail (`IngestError`) on empty file (no header). Stream data rows counting against
|
||
`max_rows` — raise `IngestError` the moment the cap is exceeded (§8: error, never silent
|
||
truncation; no fetch-all-then-count). Fail on ragged rows (data row width ≠ header
|
||
width) — silent padding/truncation is coercion. Renderer: cell text **verbatim** with
|
||
escaping in this exact order: `\` → `\\` FIRST, then `|` → `\|`, then any newline (CR,
|
||
LF, CRLF) → single space (pinned decision: on the `file`/CSV path every cell is a
|
||
string, so §5's integer/float/NULL clauses bite typed `sql` values in I4 — a spec-text
|
||
reading recorded as Assumption 1). Newline replacement treats CRLF as ONE unit: replace
|
||
`\r\n` first, then lone `\r`, then lone `\n` — each with a single space (a per-character
|
||
replace would turn CRLF into two spaces). A well-formed manifest whose `root` directory
|
||
or resolved `query` CSV does not exist raises `IngestError` with a clear message (never a
|
||
bare `FileNotFoundError` leaking from `open()`). Body = markdown table: `| h1 | h2 |`
|
||
header row, `| --- | --- |` separator, data rows in source order, LF line endings, ending
|
||
in exactly one `\n`. **Header cells are escaped identically to data cells** (spec §5
|
||
separates "column names" from "cell values" without giving headers their own rule —
|
||
pinned decision: same escaping/newline-collapse, or a header containing `|` or an
|
||
embedded newline breaks the table). The connector itself is PURE (returns header +
|
||
rows); the §8 source-call logging lives in `materialize` (Step 3), which owns
|
||
`ingested_at`.
|
||
- **Reuses:** `retrieval.safe_resolve`/`PathSecurityError` (`retrieval.py:63,24`) —
|
||
identical fail-closed semantics `okf.py` itself relies on.
|
||
- **Test first:**
|
||
- File: `tests/test_ingest_materialize.py` (new)
|
||
- Verifies: table rendering (header/separator/rows, source order); escaping of `\`,
|
||
`|`, embedded quoted newline (one assertion each, plus one cell containing BOTH `\`
|
||
and `|` proving escape order, plus an explicit embedded-CRLF cell → exactly ONE
|
||
space); a header cell containing `|` and one containing an embedded newline →
|
||
escaped/collapsed identically to data cells; header-only CSV → header + separator,
|
||
zero data rows; empty CSV file → `IngestError`; ragged row → `IngestError`; `max_rows` fixture with cap+1 rows →
|
||
`IngestError`; `../escape` query → `PathSecurityError`; missing `root` dir and
|
||
missing `query` file → `IngestError` (not bare `FileNotFoundError`); BOM fixture →
|
||
first header cell clean.
|
||
- Pattern: local `tmp_path` CSV-builder helper (conftest's LLM fixtures are irrelevant
|
||
to ingest — no conftest changes)
|
||
- **Verify:** `uv run pytest tests/test_ingest_materialize.py -q` → expected: all pass
|
||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py && rm -f tests/test_ingest_materialize.py`
|
||
(`ingest.py` is tracked after Step 1's checkpoint; the test file is new in this step)
|
||
- **Checkpoint:** `git commit -m "feat(ingest): CSV connector with boundary check, row cap and escaped table body (I2)"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/ingest.py
|
||
- tests/test_ingest_materialize.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(ingest\\): CSV connector with boundary check, row cap and escaped table body \\(I2\\)$"
|
||
bash_syntax_check: []
|
||
forbidden_paths:
|
||
- src/portfolio_optimiser/okf.py
|
||
- tests/conftest.py
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/ingest.py
|
||
pattern: "safe_resolve"
|
||
- path: src/portfolio_optimiser/ingest.py
|
||
pattern: "utf-8-sig"
|
||
```
|
||
|
||
### Step 3: Materialization — provenance frontmatter, LF byte-writer, in-memory staging
|
||
|
||
- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_materialize.py`
|
||
- **Changes:** Add the per-extraction concept-file builder and the public entry point
|
||
`materialize(manifest_path: str | Path, bundle_dir: str | Path, *, ingested_at: str) ->
|
||
list[Path]` (explicit required `ingested_at` keyword, NO wall-clock default — mirrors
|
||
`promote_verdict`; validate against the regex `^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$` —
|
||
NOT `datetime.fromisoformat`, which rejects the `Z` suffix on Python 3.10, the repo's
|
||
version floor — then stamp the string verbatim). This programmatic function IS the I2
|
||
invocation surface (recorded decision, resolving the brief's open question: a `python -m`
|
||
CLI is not demanded by the I2 verification set and is deferred). `materialize` performs
|
||
the §8 source-call log per extraction on logger `portfolio_optimiser.ingest`: source id,
|
||
the `ingested_at` argument (deterministic "when" — never wall-clock), row count; never
|
||
log cell contents or (in later source types) resolved secrets. It creates the bundle
|
||
directory first: `Path(bundle_dir).mkdir(parents=True, exist_ok=True)` — `safe_resolve`
|
||
never creates directories, and unlike `okf.write_concept_file` the byte-writer has no
|
||
implicit mkdir. Frontmatter: ordered dict with EXACTLY the §5 keys in §5 order — `type`
|
||
(= `okf_type`), `title`, `source_system` (= `source.id`), `source_query` (the query,
|
||
whitespace-collapsed), `ingested_at`, `ingest_manifest` (the Step-1 stamp), `generated`
|
||
(`true`) — rendered via `okf.render_frontmatter` (insertion-ordered, single-lining).
|
||
Filenames: `ingest-{extraction.id}.md` (§5; grammar keeps the namespace disjoint from
|
||
`index.md`/`promoted-verdict-*` by construction). File content assembled as ONE string
|
||
`---\n{fm}\n---\n\n{body}` (body already ends in exactly one `\n`) and written as bytes
|
||
(`content.encode("utf-8")`) to a `safe_resolve`d path — LF-only and exactly one trailing
|
||
newline guaranteed on every platform (§5). **All extractions execute and render in
|
||
memory BEFORE the first disk mutation** (crash-window mitigation for §5's non-atomic
|
||
replace sequence; recovery = idempotent re-run, §10).
|
||
- **Reuses:** `okf.render_frontmatter` (`okf.py:146`), `retrieval.safe_resolve`;
|
||
`promote_verdict`'s explicit-timestamp shape (`verdicts.py:465-472`).
|
||
- **Test first:**
|
||
- File: `tests/test_ingest_materialize.py` (extend)
|
||
- Verifies: generated file's RAW TEXT has the 7 frontmatter keys in exact §5 order (read
|
||
the file text, not just the parsed dict); `okf.parse_frontmatter` round-trips the
|
||
values (`generated == "true"` as string; `ingest_manifest == {stem}@{hash16}` computed
|
||
independently in the test from the manifest's raw bytes; `ingested_at` verbatim;
|
||
`source_query` whitespace-collapsed); file bytes are LF-only with exactly one trailing
|
||
`\n` (`read_bytes()` assertions: no `\r`, endswith exactly one `b"\n"`);
|
||
missing/invalid `ingested_at` (no argument has no default — non-`Z`/non-ISO string
|
||
raises `ValueError`; the golden timestamp `2026-07-03T12:00:00Z` passes the regex);
|
||
materializing into a NON-EXISTENT nested `bundle_dir` succeeds (mkdir path); §8 log
|
||
record (via `caplog`) carries source id, the explicit `ingested_at` string, and row
|
||
count; two `materialize` runs with identical inputs produce byte-identical files
|
||
(§10 idempotence at file level).
|
||
- Pattern: `tests/test_okf.py` frontmatter round-trip assertions
|
||
- **Verify:** `uv run pytest tests/test_ingest_materialize.py -q` → expected: all pass
|
||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_materialize.py`
|
||
- **Checkpoint:** `git commit -m "feat(ingest): deterministic materialization with §5/§7 provenance stamp (I2)"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/ingest.py
|
||
- tests/test_ingest_materialize.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(ingest\\): deterministic materialization with .5/.7 provenance stamp \\(I2\\)$"
|
||
bash_syntax_check: []
|
||
forbidden_paths:
|
||
- src/portfolio_optimiser/okf.py
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/ingest.py
|
||
pattern: "ingested_at"
|
||
- path: src/portfolio_optimiser/ingest.py
|
||
pattern: "ingest_manifest"
|
||
```
|
||
|
||
### Step 4: Index generation + stamped-replacement semantics
|
||
|
||
- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_materialize.py`
|
||
- **Changes:** Complete `materialize` with §3/§5/§6 bundle semantics. (a) **Ownership
|
||
scan:** classify a bundle file as ingest-owned iff `okf.parse_frontmatter` yields
|
||
`generated == "true"` AND an `ingest_manifest` key (string comparisons; sorted directory
|
||
scan for determinism). (b) **Collision gate:** if a target filename `ingest-{id}.md`
|
||
exists WITHOUT the stamp → `IngestError`, nothing written (§3: never overwrite curated
|
||
content). (c) **Replace:** delete every stamped file, then write the new staged set.
|
||
(d) **Index:** if `index.md` missing → create it with `bundle_summary` as the body (no
|
||
frontmatter — spec-minimal, pinned decision; content = `{bundle_summary}\n` written as
|
||
bytes). If present → preserve every unmanaged line byte-for-byte; remove ONLY
|
||
whole lines matching the managed-link form `- [<label>](<target>)` whose `<target>` is a
|
||
previously-stamped file deleted in this run and NOT re-created (exact full-line
|
||
target match, never bare-substring — a promoted verdict's or curated link must be
|
||
unreachable by this filter). (e) **Link:** `okf.link_in_index(bundle_dir,
|
||
"ingest-{id}.md", title)` per extraction in manifest order (idempotent by target: links
|
||
for re-created targets keep their existing position — byte-stability across re-ingest).
|
||
(f) **Label refresh:** for a re-created target whose existing managed line's label no
|
||
longer equals the (normalized) extraction `title`, rewrite THAT line in place —
|
||
`- [{new title}](ingest-{id}.md)` — preserving its position; §6 defines the label AS
|
||
the title, so a title change must never leave a stale label behind. The refresh uses
|
||
the SAME anchoring as the removal filter: only an exact full-line managed form
|
||
`- [<label>](ingest-{id}.md)` targeting THIS run's ingest files is rewritten;
|
||
non-conforming lines (curated prose mentioning the target inline, links to non-ingest
|
||
files) are untouched. Index update is the LAST disk mutation.
|
||
- **Reuses:** `okf.link_in_index` (`okf.py:168` — the exact primitive spec §6 cites),
|
||
`okf.parse_frontmatter` (`okf.py:31`).
|
||
- **Test first:**
|
||
- File: `tests/test_ingest_materialize.py` (extend)
|
||
- Verifies: fresh dir → `index.md` created with `bundle_summary` body + one link per
|
||
extraction in manifest order; existing `index.md` with a curated line → line preserved
|
||
byte-for-byte after materialization; re-materialization with an extraction REMOVED
|
||
from the manifest → its file gone AND its index link gone, other links intact;
|
||
re-materialization with a CHANGED extraction `title` (same id) → the managed link's
|
||
label updated in place, position preserved (no stale label); unstamped
|
||
`ingest-foo.md` present → `IngestError` and bundle unmodified (assert directory bytes
|
||
unchanged); re-materialization twice with identical inputs → byte-identical bundle
|
||
including `index.md` (§10).
|
||
- Pattern: `tests/test_okf.py::test_link_in_index_makes_concept_file_navigable`
|
||
- **Verify:** `uv run pytest tests/test_ingest_materialize.py -q` → expected: all pass
|
||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_materialize.py`
|
||
- **Checkpoint:** `git commit -m "feat(ingest): index generation and stamped-replacement semantics (I2)"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/ingest.py
|
||
- tests/test_ingest_materialize.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(ingest\\): index generation and stamped-replacement semantics \\(I2\\)$"
|
||
bash_syntax_check: []
|
||
forbidden_paths:
|
||
- src/portfolio_optimiser/okf.py
|
||
- src/portfolio_optimiser/verdicts.py
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/ingest.py
|
||
pattern: "link_in_index"
|
||
- path: src/portfolio_optimiser/ingest.py
|
||
pattern: "bundle_summary"
|
||
```
|
||
|
||
### Step 5: Load-bearing seam quartet + MAF-free guard (detach-RED proofs)
|
||
|
||
- **Files:** `tests/test_ingest_loadbearing.py` (new)
|
||
- **Changes:** The durable seam tests per spec §11's table, in the repo's load-bearing
|
||
idiom (module docstring naming each seam + its detach-RED condition; scenarios built so
|
||
the seam is the ONLY thing preventing the observable artifact). (1) **Verdict
|
||
reservation:** an otherwise fully-valid manifest with `okf_type: "Verdict"` →
|
||
`ValidationError` at load, zero files/links written (RED if the case-insensitive
|
||
reservation validator is removed). Plus the defensive namespace invariant: the id
|
||
grammar + `ingest-` prefix make `promoted-verdict-*`/`index.md` collisions structurally
|
||
unreachable — asserted as invariant, with the honest note that its load-bearing bite is
|
||
the `okf_type` check. (2) **Provenance stamping:** materialize, then assert the §7 layer
|
||
on every generated file via UNCHANGED `okf.parse_frontmatter` (RED if the stamp layer is
|
||
dropped). (3) **Navigability via UNCHANGED okf.py** (the durable version of the planning
|
||
session's ad-hoc proof): `okf.navigate_bundle` reaches every `ingest-{id}.md` through
|
||
index cross-links; each is classified by its `okf_type`; provenance fields ride through
|
||
`frontmatter` (unknown-field preservation); `okf.bundle_context` renders the generated
|
||
content (RED if index linking detaches — file unreachable). (4) **Re-ingest layer
|
||
safety — with the removal filter ACTIVE:** on a `shutil.copytree` copy of
|
||
`shared/examples/bygg-energi-mikro/`, run `materialize` (manifest with ≥2 extractions),
|
||
then the REAL `verdicts.promote_verdict` (approved verdict, explicit timestamp), then
|
||
`materialize` AGAIN with a REDUCED manifest (one extraction dropped) — the §6
|
||
managed-line removal filter must actually FIRE while the promoted link and the bundle's
|
||
curated managed-form links are present (an identical-manifest second run never removes
|
||
anything, so it cannot catch a loose filter — the green-but-dead trap): the dropped
|
||
extraction's file AND index link are gone; the promoted verdict file and its index link
|
||
survive verbatim; every curated link survives verbatim; only stamped files were
|
||
replaced (RED if the filter over-matches — `promote_verdict` writes exactly the
|
||
managed-line shape a bare-substring filter would wrongly delete — or if replacement
|
||
stops honouring the stamp ownership rule). (5) **MAF-free guard:** AST-parse
|
||
`src/portfolio_optimiser/ingest.py` (pattern: `tests/test_okf.py::test_okf_is_maf_free`)
|
||
asserting no import roots in `{agent_framework, mcp}` AND no `portfolio_optimiser.*`
|
||
imports outside `{okf, retrieval}` (transitively stdlib-pure — `contracts`/`backends`
|
||
would smuggle MAF in). **Execution rule:** each of (1)–(4) is proven RED by temporarily
|
||
detaching its seam (comment out the validator / stamp keys / `link_in_index` call / the
|
||
stamp filter), observing the failure, reattaching; the detach-proof log (seam → detached
|
||
how → observed failure) goes into the session verification log for STATE/commit message.
|
||
- **Reuses:** `_copy_bundle` copytree idiom (`tests/test_step8_promotion_loadbearing.py:51`),
|
||
AST guard (`tests/test_okf.py:165-181`), real `promote_verdict` (`verdicts.py:465`).
|
||
- **Test first:**
|
||
- File: `tests/test_ingest_loadbearing.py` (new — this whole step IS tests; the seams it
|
||
guards were built in Steps 1–4, so these must pass immediately and the detach-proofs
|
||
demonstrate they can fail)
|
||
- Verifies: the four §11 seams + module purity, as above
|
||
- Pattern: `tests/test_step8_promotion_loadbearing.py` (docstring naming detach-RED per test)
|
||
- **Verify:** `uv run pytest tests/test_ingest_loadbearing.py -q` → expected: all pass;
|
||
then 4 documented detach→RED→reattach cycles each showing ≥1 failure
|
||
- **On failure:** escalate — a quartet test that cannot go RED at detach means the seam is
|
||
not actually load-bearing; stop and redesign the test before proceeding (do not weaken
|
||
the assertion). Cleanup on abandon: `rm -f tests/test_ingest_loadbearing.py` (new file —
|
||
untracked until this step's checkpoint)
|
||
- **Checkpoint:** `git commit -m "test(ingest): load-bearing seam quartet + MAF-free guard, detach-proven (I2)"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- tests/test_ingest_loadbearing.py
|
||
min_file_count: 1
|
||
commit_message_pattern: "^test\\(ingest\\): load-bearing seam quartet \\+ MAF-free guard, detach-proven \\(I2\\)$"
|
||
bash_syntax_check: []
|
||
forbidden_paths:
|
||
- src/portfolio_optimiser/okf.py
|
||
- shared/examples/bygg-energi-mikro/index.md
|
||
must_contain:
|
||
- path: tests/test_ingest_loadbearing.py
|
||
pattern: "promote_verdict"
|
||
- path: tests/test_ingest_loadbearing.py
|
||
pattern: "navigate_bundle"
|
||
```
|
||
|
||
### Step 6: Spec §12 field cross-check (discharge the I1-recorded obligation)
|
||
|
||
- **Files:** `tests/test_method_spec_loadbearing.py`
|
||
- **Changes:** Add `test_ingest_spec_documents_every_contract_field`, mirroring the
|
||
existing `test_spec_documents_every_contract_field` (line 106): iterate the REAL pydantic
|
||
models' `model_fields` (ManifestV1, recursively into the three source variants and
|
||
Extraction) plus the seven §5 frontmatter keys plus the four golden-case entries
|
||
(`manifest.json`, `fixture/`, `ingested-at.txt`, `expected-bundle/`), asserting each
|
||
token appears in `shared/ingest-spec.md` (the §12 cross-check table's completeness,
|
||
test-enforced at field level). This discharges the deferred note recorded at
|
||
`tests/test_method_spec_loadbearing.py:156-162` ("arrives with the I2 contracts").
|
||
RED when contract code and spec drift apart. Surgical: only ADD the test — the I1
|
||
structure/neutrality guards in the file are untouched.
|
||
- **Reuses:** the file's own test-3 pattern (`model_fields`-driven spec assertion).
|
||
- **Test first:**
|
||
- File: `tests/test_method_spec_loadbearing.py` (existing)
|
||
- Verifies: every machine-readable contract field is documented in the frozen spec; a
|
||
hypothetical new model field without a spec entry turns it RED (detach-proof: add a
|
||
dummy field to a throwaway copy check — verified by temporarily adding a fake field
|
||
name to the iteration list, observing RED, removing)
|
||
- Pattern: `tests/test_method_spec_loadbearing.py::test_spec_documents_every_contract_field`
|
||
- **Verify:** `uv run pytest tests/test_method_spec_loadbearing.py -q` → expected: all pass
|
||
- **On failure:** escalate — a genuine field↔spec mismatch means either the contract
|
||
deviates from the frozen spec (fix the contract) or the spec is defective (STOP: spec
|
||
changes are a separate GATED commons round, never folded into I2)
|
||
- **Checkpoint:** `git commit -m "test(spec-guard): ingest contract field cross-check vs spec §12 (I2)"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- tests/test_method_spec_loadbearing.py
|
||
min_file_count: 1
|
||
commit_message_pattern: "^test\\(spec-guard\\): ingest contract field cross-check vs spec .12 \\(I2\\)$"
|
||
bash_syntax_check: []
|
||
forbidden_paths:
|
||
- shared/ingest-spec.md
|
||
- shared/method-spec.md
|
||
must_contain:
|
||
- path: tests/test_method_spec_loadbearing.py
|
||
pattern: "test_ingest_spec_documents_every_contract_field"
|
||
```
|
||
|
||
### Step 7: Golden extraction case + byte-for-byte regression test
|
||
|
||
- **Files:** `examples/ingest-golden-file/` (new directory), `tests/test_ingest_golden.py` (new)
|
||
- **Changes:** Author the golden case per spec §11 layout, LOCALLY in this repo (brief
|
||
Non-Goal: commons sharing is a later gated round; `shared/` is untouchable). New
|
||
top-level `examples/` directory (does not exist yet; outside ruff's `src` config —
|
||
fixture data, not lint targets): `manifest.json` (file source, `root: "fixture"`
|
||
resolved against the manifest's directory, 2 extractions: a plain cost table AND an
|
||
edge-case table whose quoted CSV cells exercise `\`, `|`, an embedded newline, a cell
|
||
with both `\` and `|`, AND number-shaped cells that DISCRIMINATE the text-verbatim
|
||
reading — `007` and `1.50`, strings whose numeric normalization differs byte-wise
|
||
(`7`, `1.5`); Assumption 1 is only pinned if a number-normalizing implementation CANNOT
|
||
reproduce the golden — a cell like `30000.0` discriminates nothing since
|
||
`repr(30000.0)` round-trips identically), `fixture/*.csv`, `ingested-at.txt`
|
||
(`2026-07-03T12:00:00Z`, one line). `expected-bundle/` is generated ONCE by the (now fully behaviour-tested)
|
||
implementation, its content REVIEWED against the spec §5/§6/§7 rules by hand, then
|
||
frozen in git (brief Assumption 4: golden = regression pin; correctness is carried by
|
||
Steps 1–6's tests; the freeze happens LAST). Golden test: read `ingested-at.txt`, run
|
||
`materialize` into `tmp_path`, then assert (a) the produced file SET equals the
|
||
`expected-bundle/` file set (catches extra AND missing files) and (b) every file's
|
||
`read_bytes()` matches exactly. A second `materialize` run over the same output must
|
||
leave every byte unchanged (golden-level idempotence, §10).
|
||
- **Reuses:** `BUNDLE_DIR`-style module-level fixture-path idiom (`tests/test_okf.py`,
|
||
`tests/test_step8_promotion_loadbearing.py`).
|
||
- **Test first:**
|
||
- File: `tests/test_ingest_golden.py` (new — written BEFORE `expected-bundle/` exists;
|
||
observed RED against the empty expectation, then the reviewed freeze turns it green)
|
||
- Verifies: byte-for-byte golden regression + file-set equality + repeat-run idempotence
|
||
- Pattern: `tests/test_okf.py` fixture-path idiom
|
||
- **Verify:** `uv run pytest tests/test_ingest_golden.py -q` → expected: all pass; then the
|
||
full gate battery: `uv run pytest` (pre-existing suite unchanged-green vs the Step-1
|
||
baseline + all new tests), `uv run ruff check .` → 0 issues,
|
||
`uv run ruff format --check .` → clean, `uv run mypy src` → 0 errors; honesty grep
|
||
(brief SC8 — scoped to PROSE per the brief: docstrings/comments of the new module and
|
||
new test files, plus any touched docs):
|
||
`grep -rinE "sql|http|live|commons" src/portfolio_optimiser/ingest.py tests/test_ingest_*.py` →
|
||
every PROSE hit is a defer-mention (I4/I6/extension point/gated commons round) or a
|
||
spec citation; hits that are polymorphic-contract SCHEMA IDENTIFIERS (`SqlSource`,
|
||
`HttpSource`, `Literal["sql"]`, `Literal["http"]`, `connection_ref`, `base_url`,
|
||
`credential_ref`) are expected and allowed — they validate variants per §4 and claim
|
||
no delivered connector support
|
||
- **On failure:** escalate — a golden mismatch after Steps 1–6 are green means a
|
||
determinism defect (platform newline, ordering, escaping); diagnose root cause before
|
||
re-freezing — NEVER re-freeze to make a mismatch disappear. Cleanup on abandon:
|
||
`rm -rf examples/ tests/test_ingest_golden.py` (new, untracked until checkpoint)
|
||
- **Checkpoint:** `git commit -m "test(ingest): golden extraction case examples/ingest-golden-file, byte-frozen (I2)"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- examples/ingest-golden-file/manifest.json
|
||
- examples/ingest-golden-file/ingested-at.txt
|
||
- examples/ingest-golden-file/fixture/costs.csv
|
||
- examples/ingest-golden-file/expected-bundle/index.md
|
||
- tests/test_ingest_golden.py
|
||
min_file_count: 5
|
||
commit_message_pattern: "^test\\(ingest\\): golden extraction case examples/ingest-golden-file, byte-frozen \\(I2\\)$"
|
||
bash_syntax_check: []
|
||
forbidden_paths:
|
||
- src/portfolio_optimiser/okf.py
|
||
- shared/ingest-spec.md
|
||
must_contain:
|
||
- path: tests/test_ingest_golden.py
|
||
pattern: "read_bytes"
|
||
- path: examples/ingest-golden-file/ingested-at.txt
|
||
pattern: "2026-07-03T12:00:00Z"
|
||
```
|
||
|
||
## Execution Strategy
|
||
|
||
Seven steps, strictly sequential — every step builds on the previous step's code and the
|
||
session's operating model is single-session TDD (one session = one closed delivery). No
|
||
parallel waves.
|
||
|
||
### Session 1: I2 complete (this session)
|
||
- **Steps:** 1, 2, 3, 4, 5, 6, 7
|
||
- **Wave:** 1
|
||
- **Depends on:** none
|
||
- **Scope fence:**
|
||
- Touch: `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_*.py`,
|
||
`tests/test_ingest_golden.py`, `tests/test_method_spec_loadbearing.py` (additive only),
|
||
`examples/ingest-golden-file/**`
|
||
- Never touch: `src/portfolio_optimiser/okf.py`, `src/portfolio_optimiser/run.py`,
|
||
`src/portfolio_optimiser/verdicts.py`, `src/portfolio_optimiser/contracts.py`,
|
||
`shared/**`, `tests/conftest.py`, `pyproject.toml` (no new deps needed)
|
||
|
||
### Execution Order
|
||
- **Wave 1:** Session 1 (sequential steps 1→7)
|
||
|
||
### Grouping rules applied
|
||
- All steps share `src/portfolio_optimiser/ingest.py` → one session by necessity.
|
||
- TDD dependency chain (contract → connector → materializer → semantics → seams → guard →
|
||
golden freeze) forbids reordering.
|
||
|
||
## Alternatives Considered
|
||
|
||
| Approach | Pros | Cons | Why rejected |
|
||
|----------|------|------|--------------|
|
||
| Add a link-removal primitive to `okf.py` | Symmetric API next to `link_in_index` | Violates the brief's hard constraint (navigability criterion is "consumable via UNCHANGED okf.py"); widens D7-shared surface mid-program | Brief Constraints; removal lives in `ingest.py` |
|
||
| Define manifest models in `contracts.py` | One contracts home | `contracts.py:25` imports `backends` → `agent_framework`; ingest module would fail its MAF-free guard; also violates surgical-scope | Brief Constraints (MAF-free ingest seam) |
|
||
| Reuse `okf.write_concept_file` for file writes | Maximum reuse, path-safety built in | `Path.write_text` translates newlines per platform and doesn't guarantee exactly one trailing `\n` — §5 LF-only byte-determinism at risk | Risk assessment (High); ingest writes bytes itself, still via `safe_resolve` + `render_frontmatter` |
|
||
| Validate only the `file` source variant in I2 | Smaller Step 1 | Spec §4's schema is polymorphic; I4 would retrofit the union; brief assumption already commits to schema breadth with connector-narrowness | Brief Assumption 1 (schema polymorphic, connector `file`-only) |
|
||
| Hand-author `expected-bundle/` before implementing | "Purer" golden TDD | Byte-level hand-typing of hash stamps and escaping is error-prone; correctness is already carried by Steps 1–6 behaviour tests; golden is a regression PIN (spec §11) | Brief Assumption 4 (generate once, review, freeze LAST) |
|
||
| Remove + re-append index links for re-created targets | Simpler §6 filter | Moves ingest links to index end whenever anything sits between; unnecessary byte-churn breaks repeat-run stability | Idempotence (§10): keep links in place for re-created targets — WITH the Step-4(f) in-place label refresh, so a changed title never leaves a stale label (plan-critic major finding) |
|
||
|
||
## Test Strategy
|
||
|
||
- **Framework:** pytest ≥8, plain sync tests (ingest is deterministic — no asyncio), no new
|
||
markers, no conftest changes, no mocking library (repo convention: hand-built doubles;
|
||
ingest needs only `tmp_path` + small CSV builders).
|
||
- **Existing patterns:** fail-fast contract tests (`tests/test_contracts.py`), detach-RED
|
||
load-bearing modules (`tests/test_step8_promotion_loadbearing.py`), navigability
|
||
round-trips (`tests/test_okf.py`), spec cross-check (`tests/test_method_spec_loadbearing.py`).
|
||
- **New tests in this plan:** ~30 tests across 4 new files + 1 extended file.
|
||
|
||
### Tests to write
|
||
|
||
| Type | File | Verifies | Model test |
|
||
|------|------|----------|------------|
|
||
| Unit | `tests/test_ingest_manifest.py` | §4 schema fail-fast, verdict reservation, grammar, fail-before-source-access | `tests/test_contracts.py` |
|
||
| Unit | `tests/test_ingest_materialize.py` | connector boundary/cap/escaping, §5 frontmatter order, LF bytes, §6 index semantics, collision gate, idempotence | `tests/test_okf.py` |
|
||
| Load-bearing | `tests/test_ingest_loadbearing.py` | the four §11 seams + MAF-free module guard, each detach-RED-proven | `tests/test_step8_promotion_loadbearing.py` |
|
||
| Spec guard | `tests/test_method_spec_loadbearing.py` | §12 field-level contract↔spec cross-check | its own test 3 |
|
||
| Golden | `tests/test_ingest_golden.py` | byte-for-byte regression + file-set equality + repeat-run idempotence | `tests/test_okf.py` fixture idiom |
|
||
|
||
## Risks and Mitigations
|
||
|
||
| Priority | Risk | Location | Impact | Mitigation |
|
||
|----------|------|----------|--------|------------|
|
||
| Critical | §6 link removal implemented as substring match deletes a promoted verdict's or curated link | new removal filter in `ingest.py`; `okf.py:168-181` is append-only | Corrupted `index.md`; §11 re-ingest seam broken | Exact whole-line managed-link match, only for targets deleted this run and not re-created (Step 4); re-ingest safety test uses the REAL `promote_verdict` (Step 5) |
|
||
| Critical | Crash mid-replace leaves half-materialized bundle (§5 sequence is non-atomic) | `materialize` orchestration | Loop navigates a broken bundle until re-run | Full in-memory staging before first disk mutation; index update last; recovery = idempotent re-run (§10) — documented, not hidden |
|
||
| High | Reformatting numeric-looking CSV text (e.g. `"30000.0"`, `"007"`) breaks golden bytes | renderer, spec §5 reading | Golden divergence; sibling (I3) mismatch | Pinned decision: CSV cells text-verbatim; §5 number rules are typed-`sql` territory (I4). Recorded as Assumption 1; golden fixture pins it |
|
||
| High | Escape-order bug (`|` before `\`) double-escapes | renderer | Silent cell corruption | Fixed order `\`→`\\` then `|`→`\|`; both-chars cell in unit test AND golden fixture |
|
||
| High | Platform newline translation / trailing-newline drift breaks byte-determinism | file writes | Golden mismatch cross-platform | Bytes writer with explicit `\n` for concept files (Step 3); `read_bytes()` assertions; `okf.write_concept_file` not used for writes. Index appends stay on the frozen `link_in_index` → LF-ness there is platform-scoped (Assumption 6), nil on macOS/Linux |
|
||
| Medium | BOM/CRLF/embedded-newline CSV mishandling | connector | First header cell polluted; golden flaky | `utf-8-sig` + `newline=""` + `csv.reader` (pinned); BOM + CRLF + embedded-newline unit fixtures |
|
||
| Medium | Stamp misclassification (string `"true"` vs bool) deletes curated file or strands stale ingest files | ownership scan; `okf.py:46` returns strings | §3 violation | Classify on `generated == "true"` AND `ingest_manifest` present; collision gate test (unstamped `ingest-foo.md` → error, bundle unmodified) |
|
||
| Medium | `link_in_index` raises when `index.md` missing | `okf.py:176` | Mid-run crash on fresh bundle | Create `index.md` (bundle_summary body) before any linking (Step 4 order) |
|
||
| Medium | Title containing `](x.md)` injects a parseable index link | index label = raw title; `okf.py:28` | Spurious navigation targets (skipped if missing — fail-closed) | Spec-exact validation only (single-line); limitation documented in module docstring — spec change would be a gated commons round |
|
||
| Low | Golden re-frozen to hide a real determinism defect | Step 7 | Regression pin becomes a lie | Step 7 On-failure rule: diagnose root cause; NEVER re-freeze to green a mismatch |
|
||
|
||
## Assumptions
|
||
|
||
| # | Assumption | Why unverifiable | Impact if wrong |
|
||
|---|-----------|-----------------|-----------------|
|
||
| 1 | §5's integer/float/NULL rendering rules apply to typed (`sql`) values; `file`/CSV cells are text-verbatim (escape-only) | Spec prose lists both under one bullet; the frozen spec cannot be amended inside I2 | Golden bytes + sibling (I3) diverge; if I3 reads it differently, a GATED commons clarification round is triggered — the golden fixture makes the chosen reading explicit and shareable |
|
||
| 2 | Fresh `index.md` is created WITHOUT frontmatter (body = `bundle_summary` only) — spec §6 is silent on index frontmatter | Spec silence; curated example (`bygg-energi-mikro/index.md`) has `type: index` but is hand-authored, not generated | Cosmetic divergence pinned by the golden; `navigate_bundle` handles both (frontmatter-less parse returns `{}`) — verified `okf.py:38-39` |
|
||
| 3 | Relative `root` resolves against the manifest file's directory | Spec §4 does not name the resolution base | Golden becomes cwd-dependent if wrong; pinned decision documented in docstring + spec cross-check exercise for I3 |
|
||
| 4 | `utf-8-sig` reading (BOM tolerated and stripped) is the correct connector-side choice | Spec silent on encoding | A BOM-carrying golden fixture in the sibling could differ; our golden contains no BOM in `fixture/`, so the shared surface is unaffected |
|
||
| 5 | Empty CSV (no header), ragged rows, and missing `root`/`query` paths are `IngestError` fail-fast (never silent coercion/leaked `FileNotFoundError`) | Spec §5's MUST-fail clause addresses typed values; these input-shape failures are unaddressed spec silence | I3 sibling could choose differently; the decisions are documented in the module docstring and exercised by unit tests so the I3 cross-check surfaces any divergence — if the sibling disagrees, a GATED commons clarification round resolves it |
|
||
| 6 | LF-only bytes are platform-scoped for `index.md`: concept files go through the ingest byte-writer (LF everywhere), but index appends REUSE the frozen `okf.link_in_index`, whose `Path.write_text` translates newlines per platform (and `read_text` universal-newline-decodes a CRLF curated index on round-trip) | The brief mandates reusing `link_in_index` and forbids changing `okf.py` — the platform behaviour is frozen with it | On the LF-only development/target platform (macOS/Linux) impact is nil and the golden holds; a Windows deployment would need its own golden verification — documented limitation, not a universal cross-platform claim |
|
||
|
||
*6 items — above the template's 3-item investigation threshold, accepted deliberately:
|
||
all are spec-SILENCE or platform-scope points (not unknowns about this codebase) pinned
|
||
by the golden fixture and/or documented decisions — exactly the mechanism the program
|
||
uses to surface such choices to I3; none blocks execution.*
|
||
|
||
## Verification
|
||
|
||
*Per-step manifests verify automatically during execution. These are the end-to-end
|
||
checks crossing step boundaries — they restate the brief's Success Criteria (SC1–SC8).*
|
||
|
||
- [ ] SC1: `uv run pytest tests/test_ingest_golden.py -q` → all pass (byte-for-byte +
|
||
repeat-run idempotence)
|
||
- [ ] SC2–SC5: `uv run pytest tests/test_ingest_loadbearing.py -q` → all pass, AND the
|
||
session log records four detach→RED→reattach proofs (provenance, navigability,
|
||
verdict reservation, re-ingest safety)
|
||
- [ ] SC3 (unchanged okf.py): `git diff main@{session-start} -- src/portfolio_optimiser/okf.py`
|
||
→ empty output
|
||
- [ ] SC6: `uv run pytest tests/test_ingest_manifest.py -q` → all pass with no network
|
||
available (key assumption 2 closed)
|
||
- [ ] SC7: `uv run pytest` → exit 0, pre-existing suite unchanged-green vs the baseline
|
||
recorded in Step 1 (pre-change `uv run pytest -q` summary in the session log) + new
|
||
tests; `uv run ruff check .` → exit 0; `uv run ruff format --check .` → clean;
|
||
`uv run mypy src` → exit 0
|
||
- [ ] SC8: `grep -rinE "sql|http|live|commons" src/portfolio_optimiser/ingest.py tests/test_ingest*.py`
|
||
→ every PROSE hit (docstrings/comments — the brief's scope) is a defer-mention
|
||
(I4/I6/extension point/gated) or spec citation; polymorphic-contract schema
|
||
identifiers (`SqlSource`/`HttpSource`/`Literal["sql"|"http"]`/`connection_ref`/
|
||
`base_url`/`credential_ref`) are allowed hits claiming no delivered support
|
||
- [ ] Layer hygiene: `uv run pytest tests/test_okf.py tests/test_method_spec_loadbearing.py -q`
|
||
→ all pass (MAF-free + spec guards intact)
|
||
|
||
## Estimated Scope
|
||
|
||
- **Files to modify:** 1 (`tests/test_method_spec_loadbearing.py`, additive)
|
||
- **Files to create:** 5 code/test files (`src/portfolio_optimiser/ingest.py`,
|
||
`tests/test_ingest_manifest.py`, `tests/test_ingest_materialize.py`,
|
||
`tests/test_ingest_loadbearing.py`, `tests/test_ingest_golden.py`) + the
|
||
`examples/ingest-golden-file/` fixture tree (~6 small files)
|
||
- **Complexity:** medium
|
||
|
||
## Plan Quality Score
|
||
|
||
| Dimension | Weight | Score | Notes |
|
||
|-----------|--------|-------|-------|
|
||
| Structural integrity | 0.15 | 90 | strict TDD dependency chain; Step-2/3 logging ownership clarified post-review |
|
||
| Step quality | 0.20 | 88 | 1 module + 1 test file per step; all 3 critic majors fixed in revision |
|
||
| Coverage completeness | 0.20 | 88 | all 8 SCs + §11 seams mapped; title-refresh, fresh-dir, missing-source cases added post-review |
|
||
| Specification quality | 0.15 | 88 | 5 spec-silence points pinned as explicit assumptions |
|
||
| Risk & pre-mortem | 0.15 | 88 | risk table incl. 2 critical collisions; review added revert/mkdir/staleness fixes |
|
||
| Headless readiness | 0.10 | 85 | On-failure clauses corrected for untracked files; baseline capture now explicit in Step 1 |
|
||
| Manifest quality | 0.05 | 88 | Step-7 manifest now pins the byte-frozen golden artifacts |
|
||
| **Weighted total** | **1.00** | **88** | **Grade: B+** |
|
||
|
||
**Adversarial review:**
|
||
- **Plan critic:** APPROVE_WITH_NOTES (81/100, grade B) — 0 blockers, 3 major, 10 minor;
|
||
all 3 majors and 9 of 10 minors resolved in the revision below (the remaining minor —
|
||
SC4's filename-namespace clause being structurally covered — is accepted as honest
|
||
by-construction coverage, matching scope-guardian's reading).
|
||
- **Scope guardian:** ALIGNED — 0 creep, 0 dependency issues, 3 gaps (1 major: SC8 grep
|
||
scope; 2 minor: baseline step, SC4 structural note); the major and the baseline gap are
|
||
resolved below.
|
||
|
||
## Revisions
|
||
|
||
*Added by adversarial review (plan-critic + scope-guardian, deduped 16 findings).*
|
||
|
||
| # | Finding | Severity | Resolution |
|
||
|---|---------|----------|------------|
|
||
| 1 | Step 1 On-failure `git checkout --` cannot remove untracked new files | major | Step 1 revert → `rm -f`; Step 2 revert split (tracked checkout + `rm` new test file); Steps 5/7 got explicit cleanup-on-abandon commands |
|
||
| 2 | Changed extraction `title` on re-ingest left a stale index label (kept-in-place links) | major | Step 4 gained (f) in-place label refresh preserving line position + a changed-title test; Alternatives row updated |
|
||
| 3 | `materialize` wrote bytes without creating `bundle_dir` (byte-writer has no mkdir) | major | Step 3 now mkdirs `bundle_dir` first + fresh-nested-dir test added |
|
||
| 4 | SC8 whole-file grep unachievable — schema identifiers (`SqlSource` etc.) would false-positive | major | SC8 (Step 7 + Verification) rescoped to PROSE hits per the brief, with schema identifiers as an explicitly allowed category |
|
||
| 5 | `ingested_at` validation via `fromisoformat` breaks on py3.10 (`Z` unsupported) | minor | Pinned to an explicit regex; py3.10 trap documented in Step 3 |
|
||
| 6 | Missing `root`/`query` leaked bare `FileNotFoundError` | minor | Wrapped in `IngestError` + tests added (Step 2) |
|
||
| 7 | `render_frontmatter` collapses title whitespace → frontmatter/label divergence | minor | `title` whitespace-normalized at validation so both representations agree (Step 1) |
|
||
| 8 | CRLF could become two spaces | minor | CRLF replaced as one unit (`\r\n` first) + explicit CRLF test (Step 2) |
|
||
| 9 | Step 2 logged `ingested_at` it does not own | minor | Connector is pure; §8 logging moved to `materialize` (Step 3) with the caplog test |
|
||
| 10 | `HttpSource` credential validator under-defined | minor | Defined as `urlsplit` userinfo check; query-param heuristics explicitly out of scope (Step 1) |
|
||
| 11 | Ragged/empty-file decisions unpinned vs I3 sibling | minor | Added as Assumption 5 (documented + unit-tested divergence surface) |
|
||
| 12 | No baseline-capture step for SC7 | minor | Step 1 now opens with a pre-change `uv run pytest -q` baseline recording; SC7 references it |
|
||
| 13 | Step 7 manifest omitted the byte-frozen golden artifacts | minor | `expected_paths` now includes `fixture/costs.csv` + `expected-bundle/index.md`, `min_file_count: 5` |
|
||
| 14 | Invocation surface (API vs CLI) unstated | minor | Recorded in Step 3: programmatic `materialize` IS the I2 surface; CLI deferred |
|
||
| 15 | SC4 filename-namespace clause covered by construction, not detach-testable | minor | Accepted as-is: honestly documented invariant (both reviewers concur); load-bearing bite is the `okf_type` check |
|
||
|
||
## Adversarial Pass 2 (independent second pass, v5.1.1 high-effort)
|
||
|
||
*Mandated by the brief's `effort: high` plan signal. The prescribed `gemini-bridge` pass
|
||
was SUBSTITUTED with an independent fresh-context contrarian reviewer: Gemini deep-research
|
||
result retrieval is verified broken as of 2026-07-03 (research completes, result
|
||
unretrievable — operator memory), so a bridge run would burn time with no retrievable
|
||
output. The substitute preserves the intent — an independent perspective that does not
|
||
re-tread Pass 1's dedup space (it was given Pass 1's findings as excluded ground).*
|
||
|
||
Verdict: **PASS2_FINDINGS(5)** — 0 blocker, 3 major, 2 minor. All five incorporated:
|
||
|
||
| # | Finding | Severity | Resolution |
|
||
|---|---------|----------|------------|
|
||
| P2-1 | Re-ingest safety test ran `materialize` twice with an IDENTICAL manifest — the §6 removal filter never fired, so the plan's own Critical risk (loose filter deleting a promoted/curated link) was untested (green-but-dead) | major | Step 5(4) rewritten: second run uses a REDUCED manifest so removal fires with the promoted verdict + curated managed-form links present; over-match → RED |
|
||
| P2-2 | Golden fixture had no cell discriminating text-verbatim vs number-normalization — Assumption 1 was not actually pinned (`30000.0` round-trips identically) | major | Step 7 fixture now mandates `007` and `1.50` cells whose normalized forms differ byte-wise |
|
||
| P2-3 | Index writes route through the frozen `link_in_index` (`Path.write_text`) — the "LF-only on every platform" claim contradicted the plan's own reason for avoiding `write_concept_file` | major | Determinism claim rescoped: concept files LF-guaranteed via byte-writer; `index.md` LF-ness platform-scoped (nil on macOS/Linux) — Assumption 6 + risk-row update |
|
||
| P2-4 | Header-cell escaping unspecified (spec §5 separates "column names" from "cell values") — a `|`/newline header would break the table | minor | Step 2: headers escaped identically to data cells (pinned) + header edge-case tests |
|
||
| P2-5 | Step 4(f) label refresh did not restate the removal filter's full-line/ingest-target anchoring | minor | Step 4(f) now requires the same exact managed-line anchoring; non-conforming lines untouched |
|
||
|
||
Probed and clean (per the reviewer): §12 cross-check test CAN go red; file-deletion
|
||
ownership seam holds against real `promote_verdict`; no invariant violations (okf.py
|
||
unchanged, `shared/` untouched, MAF-free, no new deps); escape order correct; golden
|
||
`tmp_path` second run does exercise re-ingest-over-existing.
|