portfolio-optimiser/.claude/projects/2026-07-04-i6-ingest-http-maf/brief.md

---
type: trekbrief
brief_version: "2.2"
created: 2026-07-04
task: "I6 — wire the HTTP source connector behind an opt-in network flag (MAF-only), tested against a local mock, no live source"
slug: i6-ingest-http-maf
project_dir: .claude/projects/2026-07-04-i6-ingest-http-maf/
research_topics: 0
research_status: skipped
auto_research: false
interview_turns: 1
source: interview
framing: refine
phase_signals:
  - phase: research
    effort: high
    model: opus
  - phase: plan
    effort: high
    model: opus
  - phase: execute
    effort: high
    model: opus
  - phase: review
    effort: high
    model: opus
---

# Task: I6 — HTTP source connector as a demonstrated extension point (MAF-only)

> Generated by `/trekbrief` on 2026-07-04.
> This brief is the contract between requirements and planning. `/trekplan`
> reads it to produce the implementation plan. Every decision in the plan must
> trace back to content in this brief.

## TL;DR

Refine of the frozen ingest program (I1–I5). Activate the `http` source family —
already declared an OPTIONAL extension point in the spec — by wiring the connector
behind an explicit per-run network opt-in flag, tested ONLY against a local mock
(no live source; that gate stays unconsumed). Proves the extension-point pattern
and the "no silent egress" frame. MAF-only; `okf.py`/`shared/` and the spec untouched.

## Intent

The ingest layer now implements the `file`/CSV (I2/I3) and `sql` (I4/I5) source
families on both stacks, built from the shared frozen spec. That spec (§1, §4)
declares `type: "http"` an OPTIONAL extension point: the schema validates
`HttpSource` today, but `materialize` refuses it fail-fast
(`src/portfolio_optimiser/ingest.py:394`, "gated extension point (I6) with no
connector"). I6 activates that stub to demonstrate — honestly and end-to-end —
HOW a third, network-transport source family plugs into the same connector /
materialization / gate contracts, WITHOUT consuming the platform's "first live
source" milestone. This realizes two locked framework principles at once: the
90%-principle (build the generic core plus clear extension points, do not chase
the last 10%) and the security frame's "local-only default, no silent egress"
(§8) — showing a network source can be added strictly behind an explicit gate
that the manifest itself cannot grant.

## Goal

`materialize` (and the surrounding run path) accept an explicit opt-in network
flag that is a RUN argument, never a manifest field (§8). When a manifest's
source is `HttpSource`: without the flag the connector refuses fail-fast with
`IngestError`; with the flag it fetches each extraction (the extraction `query`
is a path joined to `base_url`, retrieved with GET — §4/§5), resolves an optional
`credential_ref` from the environment by name (mirroring the I4 `connection_ref`
seam), and materializes one `ingest-{id}.md` per extraction whose body is the
**response body verbatim inside a fenced code block** (§5, distinct from the
markdown table used for `file`/`sql`), enforcing `max_rows` fail-fast. A golden
case `examples/ingest-golden-http/` reproduces byte-deterministic output against
committed mock payloads — with no network and no credentials (§9/§11).
`docs/extending.md` documents the extension-point pattern and points at the D7
sibling hook. `okf.py` and `shared/` are untouched; the frozen spec is unchanged.

## Non-Goals

- **No live HTTP source.** Tested only against a local/in-process (or injected)
  mock. The "first live source" gate in the ingest brief remains unconsumed.
- **No MCP client wired into the run path.** MCP stays the documented
  "extension of this family" framing (§4) plus the existing server-side
  demonstration in `datasource.py` (`build_mcp_server`); I6 does not add a new
  MCP-over-HTTP client connector.
- **No spec change.** The ingest spec is FROZEN; any change is a separate gated
  commons round. I6 implements, it does not amend.
- **No change to `okf.py` or `shared/`.** The MAF-free, D7-portable context seam
  and the pull-only commons subtree stay untouched.
- **No change to the D7 sibling repo this session.** MAF-only; the D7 http mirror
  is a later, separate slice.
- **No manifest-level network grant.** The flag is a run argument only.
- **Not building "typed index everywhere."** That OKF-standard remains a deferred,
  separately-coordinated commons RFC.

## Constraints

- The frozen ingest spec (`shared/ingest-spec.md`) is authoritative. §4/§5 define
  the http query (path + GET), body rendering (verbatim in a fenced code block),
  frontmatter keys, and credential-by-name; §8/§9 define the network gate and the
  HITL layers; §11 defines the golden format. I6 conforms; it does not extend the
  spec.
- Test suite MUST run **without network access and without credentials** (§9,
  §11) — full green offline.
- Prefer **zero new runtime dependencies** — the connector uses the stdlib for
  GET, and tests use a stdlib/injected mock.
- **Cost discipline (D6):** null API-spend; no model calls in the ingest program.
- **Commons is pull-only.** Do not touch `shared/`; never `git subtree push`.

## Preferences

- Mirror the I4 SQL implementation shape: a per-source-type value/body render
  dispatch, credential/connection resolution by env-name, a committed golden case,
  and load-bearing tests that go RED when a seam is detached.
- **Transport seam:** an injectable GET callable (defaulting to a stdlib
  `urllib` GET) so golden tests supply canned mock payloads with no socket —
  the hermetic reading of §11 "without network access".
- TDD (Iron Law: failing test first). Surgical changes; match existing
  `ingest.py` style and docstring-pinning convention.

## Non-Functional Requirements

- **Hermetic tests:** the full `pytest` suite is green with no network and no
  credentials.
- **Determinism:** the golden http extraction is byte-identical on repeat runs
  (fixed `ingested_at`, committed mock payloads).
- **Fail-fast security:** no silent egress; flag absent → refuse before any
  source call.

## Success Criteria

- **Network gate (load-bearing):** without the opt-in flag, an `http` manifest is
  refused fail-fast — a test asserts `IngestError`, and reverting the gate seam
  turns that test RED. (`uv run pytest -k <gate test>` passes; detach → fail.)
- **Golden determinism:** with the flag, the golden case materializes
  byte-identically — `uv run pytest -k <golden http test>` passes and output
  matches committed `examples/ingest-golden-http/` bytes.
- **Offline suite:** `uv run pytest` exits 0 with no network and no credentials
  set (mock in-process / injected transport).
- **max_rows fail-fast:** an http extraction whose response exceeds `max_rows` is
  an `IngestError` (never silent truncation) — asserted by a test.
- **No live claim:** `grep -rin "live" docs/ src/ examples/` surfaces only
  "local mock" / "no live source" framing; the "first live source" gate is still
  documented as unconsumed.
- **Context seam untouched:** `git diff --stat` shows no change to `okf.py` or
  `shared/`.
- **Lint/format/type green:** `uv run ruff check .`, `uv run ruff format --check .`,
  and `uv run mypy src` each exit 0.
- **Null API-spend:** the run makes no model or network calls (the golden run uses
  the injected mock).
- **Docs:** `docs/extending.md` exists and documents the extension-point pattern
  with the D7 `create_sdk_mcp_server` hook pointer.

Do NOT accept vague criteria ("http works", "the connector is implemented") —
each criterion above is command-checkable or detach-provable.

## Research Plan

No external research needed — the codebase and this brief contain sufficient
context for planning. Verified basis: the frozen spec (`shared/ingest-spec.md`
§4/§5/§8/§9/§11) fully defines the http source semantics; the implementation
mirrors the already-landed I4 SQL patterns (env-resolved credential-by-name,
per-type render dispatch, golden-case structure, detach-proven load-bearing
tests); the local mock uses the Python stdlib (no obscure or platform-specific
behavior — the search-first rule is not triggered). The single external-adjacent
item, the current `create_sdk_mcp_server` API for the `docs/extending.md` D7
pointer, is a low-stakes documentation pointer for the sibling repo, recorded as
an assumption to verify at doc-writing time — it does not gate the plan.

## Open Questions / Assumptions

- **[ASSUMPTION] `max_rows` for http.** The response body renders verbatim (no
  rows). The plan must pin the enforced interpretation — most consistent with §8
  ("source calls are logged … row count"; caps "enforced fail-fast"): the
  response body's line count is the enforced "size", fail-fast on exceed. To be
  locked in the plan like the I4 SQL type/number rules, and pinned in the
  `ingest.py` docstring + golden.
- **[ASSUMPTION] Local-mock strategy.** An injectable GET callable (defaulting to
  a stdlib `urllib` GET) so golden tests supply canned payloads with no socket —
  the hermetic reading of §11. A localhost `http.server` thread is a documented
  fallback; the plan decides.
- **[ASSUMPTION] `source_system` frontmatter value for http.** Mirror the
  existing `file`/`sql` renderer convention (plan to confirm from the current
  materializer — likely the source `type` or a `base_url`-derived value; must
  match the golden bytes).
- **[ASSUMPTION] Opt-in flag threading + name.** The flag threads through
  `materialize` and `run_project` as a run argument (e.g. `allow_network: bool =
  False`); the exact name/signature is pinned in the plan.
- **[ASSUMPTION] `create_sdk_mcp_server` D7 pointer.** Assumed current Claude
  Agent SDK primitive for the docs pointer; verify at doc-writing time (cheap,
  non-gating).

## Prior Attempts

None for the http family specifically. I2 (CSV, `e4ee8bd`→`994f5be`) and I4 (SQL,
`d7e5f2f`→`1b7612b`) are the direct precedents on this stack — identical
connector / materialization / golden / load-bearing pattern — and I6 mirrors them
for the network-transport family. The `HttpSource` schema and its fail-fast
rejection already exist (`ingest.py:84` and `:394-396`) as the gated stub that I6
activates behind the new flag.

## Metadata

- **Created:** 2026-07-04
- **Interview turns:** 1 (framing); required-section content pre-populated from
  the detailed task directive + the frozen spec, then reviewed.
- **Auto-research opted in:** no
- **Source:** trekbrief interview

---

## How to continue

Manual (default):

```bash
# No research topics — go straight to planning:
/trekplan --project .claude/projects/2026-07-04-i6-ingest-http-maf/

# Then execute:
/trekexecute --project .claude/projects/2026-07-04-i6-ingest-http-maf/
```