llm-ingestion-okf/docs/plan/phase-1-door-a.md

# Phase 1 plan — Door A: ingest-spec implementation + §11 golden fixtures

Status: approved roadmap phase (see `CLAUDE.md`); this plan details it before any code.
Normative basis: `ingest-spec.md` v1 in `portfolio-optimiser-commons` (RFC-2119).
This plan was written against spec revision `sha256:959484e1952d28fa…` — if the spec has
changed when implementation starts, re-read it first and update this plan.

## Goal

A stdlib-only Python implementation of the full ingest-spec contract:
manifest → connector (`file`/`sql`/`http`) → deterministic materialization of
`ingest-{id}.md` files → index generation — plus the §11 golden fixture suite
(byte-exact) that no repo ships today. Zero model calls; zero runtime dependencies.

## Deliverables

1. **Manifest layer** — parse + fail-fast schema validation BEFORE any source call
   (spec §4): `manifest_version == 1`, required top-level fields, polymorphic
   `source.type`, extraction list validation (id grammar `[a-z0-9][a-z0-9-]*`,
   unique ids, single-line titles, positive `max_rows`), and the verdict
   reservation (spec §3): `okf_type` must not be `verdict` (case-insensitive) and
   generated names must not fall in `promoted-verdict-*` or be `index.md`.
   Hand-rolled validation (no pydantic — zero-dep rule); typed error hierarchy
   rooted in `IngestError`.
2. **Connectors** (spec §4):
   - `file`: CSV under `root`, boundary-checked fail-closed path resolution,
     `utf-8-sig` decoding, streaming row cap, ragged-row rejection.
   - `sql`: sqlite via `connection_ref` env-var resolution, single read-only
     SELECT, bool-value rejection, sqlite errors wrapped in typed errors,
     streaming row cap.
   - `http`: GET against `base_url` + path, `credential_ref` env-var resolution,
     hard network opt-in gate (spec §8): refuse fail-fast unless the per-run flag
     is set; the flag is a run argument, never a manifest field. Transport behind
     a small seam so tests inject a mock (spec §11: no live sources in tests).
3. **Materialization** (spec §5): explicit required `ingested_at`
   (ISO-8601 UTC `Z`, regex-validated), frontmatter with exactly the seven keys in
   spec order, body rendering per source type (cell escaping `\` → `\\`, `|` →
   `\|`, newline → space; integers plain decimal; floats shortest round-trip;
   SQL NULL → empty string; any other type is an error), `ingest_manifest` stamp
   `{stem}@{sha256(manifest bytes)[:16]}`, LF-only output with exactly one
   trailing newline. Replacement semantics: stage everything in memory, run the
   collision gate (a generated name colliding with an unstamped existing file is
   an error) BEFORE any mutation, then remove stamped files, write the new set,
   update the index.
4. **Index generation** (spec §6): create with `bundle_summary` when missing;
   otherwise preserve every unmanaged line byte for byte; managed links
   `- [{title}](ingest-{id}.md)` appended in extraction order, idempotent by
   target; on re-materialization remove only links whose target was an
   ingest-owned file removed this run.
5. **Public API** — one entry point, roughly
   `materialize_bundle(manifest_path, bundle_dir, ingested_at, *, allow_network=False) -> IngestResult`,
   plus the typed error hierarchy. Exact naming settled during TDD; the shape
   (three explicit inputs + opt-in flag) is fixed by spec §5/§8.
6. **Golden fixtures** (spec §11): `examples/ingest-golden-file/` and
   `examples/ingest-golden-sql/` (conformance MUSTs), plus
   `examples/ingest-golden-http/` against mock payloads (documented as the
   optional extension exercised). Each case: `manifest.json`, `fixture/`,
   `ingested-at.txt`, `expected-bundle/`. Runnable offline without credentials.
7. **Load-bearing test table** (spec §11): one named test per seam — provenance
   stamping, navigability, verdict reservation, re-ingest layer safety over a
   promoted verdict, golden byte regression, network gate. (The spec-integrity
   seam belongs to commons, which owns the spec.)

## Implementation baseline

The stricter behaviors named in `CLAUDE.md` are the floor, not options: streaming
row caps, `utf-8-sig`, in-memory staging with pre-mutation collision gate,
validated `ingested_at`, typed `IngestError`.

## TDD order

Each step starts with a failing test:

1. Manifest validation (schema, id grammar, verdict reservation, fail-fast order —
   validation error must occur with zero source calls).
2. Body renderers (escaping, number forms, NULL, type rejection) as pure functions.
3. `file` connector (fixture catalogue; boundary check; BOM; caps).
4. `sql` connector (fixture sqlite file; env-var resolution; read-only; caps).
5. Materialization + collision gate + replacement semantics.
6. Index generation (fresh, preserve, idempotent, removal rules).
7. `http` connector + network gate (mock transport).
8. Golden fixtures authored, then the byte-for-byte conformance test.
9. Load-bearing table completed; each seam test verified red when its seam is
   detached (mutation check by hand, noted in the test docstring).

## Key assumptions (each with its test)

| # | Assumption | Test |
|---|---|---|
| A1 | Spec v1 unchanged since this plan | Compare `sha256(ingest-spec.md)` against the hash above at implementation start |
| A2 | stdlib suffices (`csv`, `sqlite3`, `json`, `hashlib`, `urllib`, `re`) | Import smoke test + CI assert that `pyproject.toml` declares no runtime dependencies |
| A3 | Byte determinism holds across runs/platforms | Run materialization twice → `diff -r` empty; golden suite in CI |
| A4 | Float shortest round-trip (`repr`) is stable in Python ≥ 3.10 | Guaranteed since Python 3.1; covered by golden bytes anyway |

## Non-goals

- Consumer adoption (`portfolio-optimiser-claude`, then `portfolio-optimiser`)
  is separate coordinated work in those repos — this phase only makes the library
  adoption-ready. Spec changes, if any prove necessary, go via commons.
- Doors B/C, configurable contracts, Node — later phases.
- Per-row file splitting, multi-manifest bundles, incremental re-ingest,
  approval registries — spec-declared extension points, not v1.

## Verification

All must pass before the phase is declared done:

1. `pytest` green, including the golden byte-for-byte comparison for all three
   fixture cases.
2. `mypy --strict src/` clean; `ruff check .` and `ruff format --check .` clean.
3. Determinism: run the materializer twice on the same golden inputs into two
   fresh directories; `diff -r` between them and against `expected-bundle/` is
   empty.
4. Idempotence: re-run over the just-materialized bundle; `diff -r` still empty.
5. Layer safety: fixture bundle containing a `promoted-verdict-*` file and its
   index link → after re-ingest both survive byte-identical (named test).
6. Verdict reservation: manifest with `okf_type: Verdict` rejected at validation,
   before any source call (named test asserts no source I/O happened).
7. Network gate: `http` manifest without the opt-in flag fails fast with a typed
   error; no socket use (mock transport asserts zero calls).
8. No wall-clock: `grep -rn "datetime.now\|time.time\|utcnow" src/` returns
   nothing.
9. Zero deps: `pyproject.toml` has an empty runtime dependency list.
10. Offline: full test suite passes with network access disabled.