llm-ingestion-okf/docs/plan/phase-1-door-a.md

Phase 1 plan — Door A: ingest-spec implementation + §11 golden fixtures

Status: approved roadmap phase (see CLAUDE.md); this plan details it before any code. Normative basis: ingest-spec.md v1 in portfolio-optimiser-commons (RFC-2119). This plan was written against spec revision sha256:959484e1952d28fa… — if the spec has changed when implementation starts, re-read it first and update this plan.

Goal

A stdlib-only Python implementation of the full ingest-spec contract: manifest → connector (file/sql/http) → deterministic materialization of ingest-{id}.md files → index generation — plus the §11 golden fixture suite (byte-exact) that no repo ships today. Zero model calls; zero runtime dependencies.

Deliverables

1. Manifest layer — parse + fail-fast schema validation BEFORE any source call (spec §4): manifest_version == 1, required top-level fields, polymorphic source.type, extraction list validation (id grammar [a-z0-9][a-z0-9-]*, unique ids, single-line titles, positive max_rows), and the verdict reservation (spec §3): okf_type must not be verdict (case-insensitive) and generated names must not fall in promoted-verdict-* or be index.md. Hand-rolled validation (no pydantic — zero-dep rule); typed error hierarchy rooted in IngestError.
2. Connectors (spec §4):
   • file: CSV under root, boundary-checked fail-closed path resolution, utf-8-sig decoding, streaming row cap, ragged-row rejection.
   • sql: sqlite via connection_ref env-var resolution, single read-only SELECT, bool-value rejection, sqlite errors wrapped in typed errors, streaming row cap.
   • http: GET against base_url + path, credential_ref env-var resolution, hard network opt-in gate (spec §8): refuse fail-fast unless the per-run flag is set; the flag is a run argument, never a manifest field. Transport behind a small seam so tests inject a mock (spec §11: no live sources in tests).
3. Materialization (spec §5): explicit required ingested_at (ISO-8601 UTC Z, regex-validated), frontmatter with exactly the seven keys in spec order, body rendering per source type (cell escaping \ → \\, | → \|, newline → space; integers plain decimal; floats shortest round-trip; SQL NULL → empty string; any other type is an error), ingest_manifest stamp {stem}@{sha256(manifest bytes)[:16]}, LF-only output with exactly one trailing newline. Replacement semantics: stage everything in memory, run the collision gate (a generated name colliding with an unstamped existing file is an error) BEFORE any mutation, then remove stamped files, write the new set, update the index.
4. Index generation (spec §6): create with bundle_summary when missing; otherwise preserve every unmanaged line byte for byte; managed links - [{title}](ingest-{id}.md) appended in extraction order, idempotent by target; on re-materialization remove only links whose target was an ingest-owned file removed this run.
5. Public API — one entry point, roughly materialize_bundle(manifest_path, bundle_dir, ingested_at, *, allow_network=False) -> IngestResult, plus the typed error hierarchy. Exact naming settled during TDD; the shape (three explicit inputs + opt-in flag) is fixed by spec §5/§8.
6. Golden fixtures (spec §11): examples/ingest-golden-file/ and examples/ingest-golden-sql/ (conformance MUSTs), plus examples/ingest-golden-http/ against mock payloads (documented as the optional extension exercised). Each case: manifest.json, fixture/, ingested-at.txt, expected-bundle/. Runnable offline without credentials.
7. Load-bearing test table (spec §11): one named test per seam — provenance stamping, navigability, verdict reservation, re-ingest layer safety over a promoted verdict, golden byte regression, network gate. (The spec-integrity seam belongs to commons, which owns the spec.)

Implementation baseline

The stricter behaviors named in CLAUDE.md are the floor, not options: streaming row caps, utf-8-sig, in-memory staging with pre-mutation collision gate, validated ingested_at, typed IngestError.

TDD order

Each step starts with a failing test:

1. Manifest validation (schema, id grammar, verdict reservation, fail-fast order — validation error must occur with zero source calls).
2. Body renderers (escaping, number forms, NULL, type rejection) as pure functions.
3. file connector (fixture catalogue; boundary check; BOM; caps).
4. sql connector (fixture sqlite file; env-var resolution; read-only; caps).
5. Materialization + collision gate + replacement semantics.
6. Index generation (fresh, preserve, idempotent, removal rules).
7. http connector + network gate (mock transport).
8. Golden fixtures authored, then the byte-for-byte conformance test.
9. Load-bearing table completed; each seam test verified red when its seam is detached (mutation check by hand, noted in the test docstring).

Key assumptions (each with its test)

#	Assumption	Test
A1	Spec v1 unchanged since this plan	Compare sha256(ingest-spec.md) against the hash above at implementation start
A2	stdlib suffices (csv, sqlite3, json, hashlib, urllib, re)	Import smoke test + CI assert that pyproject.toml declares no runtime dependencies
A3	Byte determinism holds across runs/platforms	Run materialization twice → diff -r empty; golden suite in CI
A4	Float shortest round-trip (repr) is stable in Python ≥ 3.10	Guaranteed since Python 3.1; covered by golden bytes anyway

Non-goals

• Consumer adoption (portfolio-optimiser-claude, then portfolio-optimiser) is separate coordinated work in those repos — this phase only makes the library adoption-ready. Spec changes, if any prove necessary, go via commons.
• Doors B/C, configurable contracts, Node — later phases.
• Per-row file splitting, multi-manifest bundles, incremental re-ingest, approval registries — spec-declared extension points, not v1.

Verification

All must pass before the phase is declared done:

1. pytest green, including the golden byte-for-byte comparison for all three fixture cases.
2. mypy --strict src/ clean; ruff check . and ruff format --check . clean.
3. Determinism: run the materializer twice on the same golden inputs into two fresh directories; diff -r between them and against expected-bundle/ is empty.
4. Idempotence: re-run over the just-materialized bundle; diff -r still empty.
5. Layer safety: fixture bundle containing a promoted-verdict-* file and its index link → after re-ingest both survive byte-identical (named test).
6. Verdict reservation: manifest with okf_type: Verdict rejected at validation, before any source call (named test asserts no source I/O happened).
7. Network gate: http manifest without the opt-in flag fails fast with a typed error; no socket use (mock transport asserts zero calls).
8. No wall-clock: grep -rn "datetime.now\|time.time\|utcnow" src/ returns nothing.
9. Zero deps: pyproject.toml has an empty runtime dependency list.
10. Offline: full test suite passes with network access disabled.