From bc52c12ddb332e2c74a6dba789ee5a98e35c431c Mon Sep 17 00:00:00 2001 From: Kjell Tore Guttormsen Date: Sat, 4 Jul 2026 08:50:16 +0200 Subject: [PATCH] docs(i6): brief for http source extension point (I6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit /trekbrief for the http source family (MAF-only). framing=refine, brief-reviewer 5/5 PROCEED, validator PASS, 0 research topics. Contract: activate http behind a run-argument opt-in network flag, local-mock-only (no live source), body verbatim in fenced code block (§5), golden examples/ingest-golden-http/. Two deferred [ASSUMPTION]s for /trekplan to pin: max_rows unit + injectable-GET mock seam. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_015U5UK6gVFJNvTEdDcyUb8W --- .../2026-07-04-i6-ingest-http-maf/brief.md | 220 ++++++++++++++++++ 1 file changed, 220 insertions(+) create mode 100644 .claude/projects/2026-07-04-i6-ingest-http-maf/brief.md diff --git a/.claude/projects/2026-07-04-i6-ingest-http-maf/brief.md b/.claude/projects/2026-07-04-i6-ingest-http-maf/brief.md new file mode 100644 index 0000000..6612100 --- /dev/null +++ b/.claude/projects/2026-07-04-i6-ingest-http-maf/brief.md @@ -0,0 +1,220 @@ +--- +type: trekbrief +brief_version: "2.2" +created: 2026-07-04 +task: "I6 — wire the HTTP source connector behind an opt-in network flag (MAF-only), tested against a local mock, no live source" +slug: i6-ingest-http-maf +project_dir: .claude/projects/2026-07-04-i6-ingest-http-maf/ +research_topics: 0 +research_status: skipped +auto_research: false +interview_turns: 1 +source: interview +framing: refine +phase_signals: + - phase: research + effort: high + model: opus + - phase: plan + effort: high + model: opus + - phase: execute + effort: high + model: opus + - phase: review + effort: high + model: opus +--- + +# Task: I6 — HTTP source connector as a demonstrated extension point (MAF-only) + +> Generated by `/trekbrief` on 2026-07-04. +> This brief is the contract between requirements and planning. `/trekplan` +> reads it to produce the implementation plan. Every decision in the plan must +> trace back to content in this brief. + +## TL;DR + +Refine of the frozen ingest program (I1–I5). Activate the `http` source family — +already declared an OPTIONAL extension point in the spec — by wiring the connector +behind an explicit per-run network opt-in flag, tested ONLY against a local mock +(no live source; that gate stays unconsumed). Proves the extension-point pattern +and the "no silent egress" frame. MAF-only; `okf.py`/`shared/` and the spec untouched. + +## Intent + +The ingest layer now implements the `file`/CSV (I2/I3) and `sql` (I4/I5) source +families on both stacks, built from the shared frozen spec. That spec (§1, §4) +declares `type: "http"` an OPTIONAL extension point: the schema validates +`HttpSource` today, but `materialize` refuses it fail-fast +(`src/portfolio_optimiser/ingest.py:394`, "gated extension point (I6) with no +connector"). I6 activates that stub to demonstrate — honestly and end-to-end — +HOW a third, network-transport source family plugs into the same connector / +materialization / gate contracts, WITHOUT consuming the platform's "first live +source" milestone. This realizes two locked framework principles at once: the +90%-principle (build the generic core plus clear extension points, do not chase +the last 10%) and the security frame's "local-only default, no silent egress" +(§8) — showing a network source can be added strictly behind an explicit gate +that the manifest itself cannot grant. + +## Goal + +`materialize` (and the surrounding run path) accept an explicit opt-in network +flag that is a RUN argument, never a manifest field (§8). When a manifest's +source is `HttpSource`: without the flag the connector refuses fail-fast with +`IngestError`; with the flag it fetches each extraction (the extraction `query` +is a path joined to `base_url`, retrieved with GET — §4/§5), resolves an optional +`credential_ref` from the environment by name (mirroring the I4 `connection_ref` +seam), and materializes one `ingest-{id}.md` per extraction whose body is the +**response body verbatim inside a fenced code block** (§5, distinct from the +markdown table used for `file`/`sql`), enforcing `max_rows` fail-fast. A golden +case `examples/ingest-golden-http/` reproduces byte-deterministic output against +committed mock payloads — with no network and no credentials (§9/§11). +`docs/extending.md` documents the extension-point pattern and points at the D7 +sibling hook. `okf.py` and `shared/` are untouched; the frozen spec is unchanged. + +## Non-Goals + +- **No live HTTP source.** Tested only against a local/in-process (or injected) + mock. The "first live source" gate in the ingest brief remains unconsumed. +- **No MCP client wired into the run path.** MCP stays the documented + "extension of this family" framing (§4) plus the existing server-side + demonstration in `datasource.py` (`build_mcp_server`); I6 does not add a new + MCP-over-HTTP client connector. +- **No spec change.** The ingest spec is FROZEN; any change is a separate gated + commons round. I6 implements, it does not amend. +- **No change to `okf.py` or `shared/`.** The MAF-free, D7-portable context seam + and the pull-only commons subtree stay untouched. +- **No change to the D7 sibling repo this session.** MAF-only; the D7 http mirror + is a later, separate slice. +- **No manifest-level network grant.** The flag is a run argument only. +- **Not building "typed index everywhere."** That OKF-standard remains a deferred, + separately-coordinated commons RFC. + +## Constraints + +- The frozen ingest spec (`shared/ingest-spec.md`) is authoritative. §4/§5 define + the http query (path + GET), body rendering (verbatim in a fenced code block), + frontmatter keys, and credential-by-name; §8/§9 define the network gate and the + HITL layers; §11 defines the golden format. I6 conforms; it does not extend the + spec. +- Test suite MUST run **without network access and without credentials** (§9, + §11) — full green offline. +- Prefer **zero new runtime dependencies** — the connector uses the stdlib for + GET, and tests use a stdlib/injected mock. +- **Cost discipline (D6):** null API-spend; no model calls in the ingest program. +- **Commons is pull-only.** Do not touch `shared/`; never `git subtree push`. + +## Preferences + +- Mirror the I4 SQL implementation shape: a per-source-type value/body render + dispatch, credential/connection resolution by env-name, a committed golden case, + and load-bearing tests that go RED when a seam is detached. +- **Transport seam:** an injectable GET callable (defaulting to a stdlib + `urllib` GET) so golden tests supply canned mock payloads with no socket — + the hermetic reading of §11 "without network access". +- TDD (Iron Law: failing test first). Surgical changes; match existing + `ingest.py` style and docstring-pinning convention. + +## Non-Functional Requirements + +- **Hermetic tests:** the full `pytest` suite is green with no network and no + credentials. +- **Determinism:** the golden http extraction is byte-identical on repeat runs + (fixed `ingested_at`, committed mock payloads). +- **Fail-fast security:** no silent egress; flag absent → refuse before any + source call. + +## Success Criteria + +- **Network gate (load-bearing):** without the opt-in flag, an `http` manifest is + refused fail-fast — a test asserts `IngestError`, and reverting the gate seam + turns that test RED. (`uv run pytest -k ` passes; detach → fail.) +- **Golden determinism:** with the flag, the golden case materializes + byte-identically — `uv run pytest -k ` passes and output + matches committed `examples/ingest-golden-http/` bytes. +- **Offline suite:** `uv run pytest` exits 0 with no network and no credentials + set (mock in-process / injected transport). +- **max_rows fail-fast:** an http extraction whose response exceeds `max_rows` is + an `IngestError` (never silent truncation) — asserted by a test. +- **No live claim:** `grep -rin "live" docs/ src/ examples/` surfaces only + "local mock" / "no live source" framing; the "first live source" gate is still + documented as unconsumed. +- **Context seam untouched:** `git diff --stat` shows no change to `okf.py` or + `shared/`. +- **Lint/format/type green:** `uv run ruff check .`, `uv run ruff format --check .`, + and `uv run mypy src` each exit 0. +- **Null API-spend:** the run makes no model or network calls (the golden run uses + the injected mock). +- **Docs:** `docs/extending.md` exists and documents the extension-point pattern + with the D7 `create_sdk_mcp_server` hook pointer. + +Do NOT accept vague criteria ("http works", "the connector is implemented") — +each criterion above is command-checkable or detach-provable. + +## Research Plan + +No external research needed — the codebase and this brief contain sufficient +context for planning. Verified basis: the frozen spec (`shared/ingest-spec.md` +§4/§5/§8/§9/§11) fully defines the http source semantics; the implementation +mirrors the already-landed I4 SQL patterns (env-resolved credential-by-name, +per-type render dispatch, golden-case structure, detach-proven load-bearing +tests); the local mock uses the Python stdlib (no obscure or platform-specific +behavior — the search-first rule is not triggered). The single external-adjacent +item, the current `create_sdk_mcp_server` API for the `docs/extending.md` D7 +pointer, is a low-stakes documentation pointer for the sibling repo, recorded as +an assumption to verify at doc-writing time — it does not gate the plan. + +## Open Questions / Assumptions + +- **[ASSUMPTION] `max_rows` for http.** The response body renders verbatim (no + rows). The plan must pin the enforced interpretation — most consistent with §8 + ("source calls are logged … row count"; caps "enforced fail-fast"): the + response body's line count is the enforced "size", fail-fast on exceed. To be + locked in the plan like the I4 SQL type/number rules, and pinned in the + `ingest.py` docstring + golden. +- **[ASSUMPTION] Local-mock strategy.** An injectable GET callable (defaulting to + a stdlib `urllib` GET) so golden tests supply canned payloads with no socket — + the hermetic reading of §11. A localhost `http.server` thread is a documented + fallback; the plan decides. +- **[ASSUMPTION] `source_system` frontmatter value for http.** Mirror the + existing `file`/`sql` renderer convention (plan to confirm from the current + materializer — likely the source `type` or a `base_url`-derived value; must + match the golden bytes). +- **[ASSUMPTION] Opt-in flag threading + name.** The flag threads through + `materialize` and `run_project` as a run argument (e.g. `allow_network: bool = + False`); the exact name/signature is pinned in the plan. +- **[ASSUMPTION] `create_sdk_mcp_server` D7 pointer.** Assumed current Claude + Agent SDK primitive for the docs pointer; verify at doc-writing time (cheap, + non-gating). + +## Prior Attempts + +None for the http family specifically. I2 (CSV, `e4ee8bd`→`994f5be`) and I4 (SQL, +`d7e5f2f`→`1b7612b`) are the direct precedents on this stack — identical +connector / materialization / golden / load-bearing pattern — and I6 mirrors them +for the network-transport family. The `HttpSource` schema and its fail-fast +rejection already exist (`ingest.py:84` and `:394-396`) as the gated stub that I6 +activates behind the new flag. + +## Metadata + +- **Created:** 2026-07-04 +- **Interview turns:** 1 (framing); required-section content pre-populated from + the detailed task directive + the frozen spec, then reviewed. +- **Auto-research opted in:** no +- **Source:** trekbrief interview + +--- + +## How to continue + +Manual (default): + +```bash +# No research topics — go straight to planning: +/trekplan --project .claude/projects/2026-07-04-i6-ingest-http-maf/ + +# Then execute: +/trekexecute --project .claude/projects/2026-07-04-i6-ingest-http-maf/ +```