portfolio-optimiser/.claude/projects/2026-07-04-i6-ingest-http-maf/brief.md

type	brief_version	created	task	slug	project_dir	research_topics	research_status	auto_research	interview_turns	source	framing	phase_signals
trekbrief	2.2	2026-07-04	I6 — wire the HTTP source connector behind an opt-in network flag (MAF-only), tested against a local mock, no live source	i6-ingest-http-maf	.claude/projects/2026-07-04-i6-ingest-http-maf/	0	skipped	false	1	interview	refine	phase effort model research high opus phase effort model plan high opus phase effort model execute high opus phase effort model review high opus

Task: I6 — HTTP source connector as a demonstrated extension point (MAF-only)

Generated by /trekbrief on 2026-07-04. This brief is the contract between requirements and planning. /trekplan reads it to produce the implementation plan. Every decision in the plan must trace back to content in this brief.

TL;DR

Refine of the frozen ingest program (I1–I5). Activate the http source family — already declared an OPTIONAL extension point in the spec — by wiring the connector behind an explicit per-run network opt-in flag, tested ONLY against a local mock (no live source; that gate stays unconsumed). Proves the extension-point pattern and the "no silent egress" frame. MAF-only; okf.py/shared/ and the spec untouched.

Intent

The ingest layer now implements the file/CSV (I2/I3) and sql (I4/I5) source families on both stacks, built from the shared frozen spec. That spec (§1, §4) declares type: "http" an OPTIONAL extension point: the schema validates HttpSource today, but materialize refuses it fail-fast (src/portfolio_optimiser/ingest.py:394, "gated extension point (I6) with no connector"). I6 activates that stub to demonstrate — honestly and end-to-end — HOW a third, network-transport source family plugs into the same connector / materialization / gate contracts, WITHOUT consuming the platform's "first live source" milestone. This realizes two locked framework principles at once: the 90%-principle (build the generic core plus clear extension points, do not chase the last 10%) and the security frame's "local-only default, no silent egress" (§8) — showing a network source can be added strictly behind an explicit gate that the manifest itself cannot grant.

Goal

materialize (and the surrounding run path) accept an explicit opt-in network flag that is a RUN argument, never a manifest field (§8). When a manifest's source is HttpSource: without the flag the connector refuses fail-fast with IngestError; with the flag it fetches each extraction (the extraction query is a path joined to base_url, retrieved with GET — §4/§5), resolves an optional credential_ref from the environment by name (mirroring the I4 connection_ref seam), and materializes one ingest-{id}.md per extraction whose body is the response body verbatim inside a fenced code block (§5, distinct from the markdown table used for file/sql), enforcing max_rows fail-fast. A golden case examples/ingest-golden-http/ reproduces byte-deterministic output against committed mock payloads — with no network and no credentials (§9/§11). docs/extending.md documents the extension-point pattern and points at the D7 sibling hook. okf.py and shared/ are untouched; the frozen spec is unchanged.

Non-Goals

• No live HTTP source. Tested only against a local/in-process (or injected) mock. The "first live source" gate in the ingest brief remains unconsumed.
• No MCP client wired into the run path. MCP stays the documented "extension of this family" framing (§4) plus the existing server-side demonstration in datasource.py (build_mcp_server); I6 does not add a new MCP-over-HTTP client connector.
• No spec change. The ingest spec is FROZEN; any change is a separate gated commons round. I6 implements, it does not amend.
• No change to okf.py or shared/. The MAF-free, D7-portable context seam and the pull-only commons subtree stay untouched.
• No change to the D7 sibling repo this session. MAF-only; the D7 http mirror is a later, separate slice.
• No manifest-level network grant. The flag is a run argument only.
• Not building "typed index everywhere." That OKF-standard remains a deferred, separately-coordinated commons RFC.

Constraints

• The frozen ingest spec (shared/ingest-spec.md) is authoritative. §4/§5 define the http query (path + GET), body rendering (verbatim in a fenced code block), frontmatter keys, and credential-by-name; §8/§9 define the network gate and the HITL layers; §11 defines the golden format. I6 conforms; it does not extend the spec.
• Test suite MUST run without network access and without credentials (§9, §11) — full green offline.
• Prefer zero new runtime dependencies — the connector uses the stdlib for GET, and tests use a stdlib/injected mock.
• Cost discipline (D6): null API-spend; no model calls in the ingest program.
• Commons is pull-only. Do not touch shared/; never git subtree push.

Preferences

• Mirror the I4 SQL implementation shape: a per-source-type value/body render dispatch, credential/connection resolution by env-name, a committed golden case, and load-bearing tests that go RED when a seam is detached.
• Transport seam: an injectable GET callable (defaulting to a stdlib urllib GET) so golden tests supply canned mock payloads with no socket — the hermetic reading of §11 "without network access".
• TDD (Iron Law: failing test first). Surgical changes; match existing ingest.py style and docstring-pinning convention.

Non-Functional Requirements

• Hermetic tests: the full pytest suite is green with no network and no credentials.
• Determinism: the golden http extraction is byte-identical on repeat runs (fixed ingested_at, committed mock payloads).
• Fail-fast security: no silent egress; flag absent → refuse before any source call.

Success Criteria

• Network gate (load-bearing): without the opt-in flag, an http manifest is refused fail-fast — a test asserts IngestError, and reverting the gate seam turns that test RED. (uv run pytest -k <gate test> passes; detach → fail.)
• Golden determinism: with the flag, the golden case materializes byte-identically — uv run pytest -k <golden http test> passes and output matches committed examples/ingest-golden-http/ bytes.
• Offline suite: uv run pytest exits 0 with no network and no credentials set (mock in-process / injected transport).
• max_rows fail-fast: an http extraction whose response exceeds max_rows is an IngestError (never silent truncation) — asserted by a test.
• No live claim: grep -rin "live" docs/ src/ examples/ surfaces only "local mock" / "no live source" framing; the "first live source" gate is still documented as unconsumed.
• Context seam untouched: git diff --stat shows no change to okf.py or shared/.
• Lint/format/type green: uv run ruff check ., uv run ruff format --check ., and uv run mypy src each exit 0.
• Null API-spend: the run makes no model or network calls (the golden run uses the injected mock).
• Docs: docs/extending.md exists and documents the extension-point pattern with the D7 create_sdk_mcp_server hook pointer.

Do NOT accept vague criteria ("http works", "the connector is implemented") — each criterion above is command-checkable or detach-provable.

Research Plan

No external research needed — the codebase and this brief contain sufficient context for planning. Verified basis: the frozen spec (shared/ingest-spec.md §4/§5/§8/§9/§11) fully defines the http source semantics; the implementation mirrors the already-landed I4 SQL patterns (env-resolved credential-by-name, per-type render dispatch, golden-case structure, detach-proven load-bearing tests); the local mock uses the Python stdlib (no obscure or platform-specific behavior — the search-first rule is not triggered). The single external-adjacent item, the current create_sdk_mcp_server API for the docs/extending.md D7 pointer, is a low-stakes documentation pointer for the sibling repo, recorded as an assumption to verify at doc-writing time — it does not gate the plan.

Open Questions / Assumptions

• [ASSUMPTION] max_rows for http. The response body renders verbatim (no rows). The plan must pin the enforced interpretation — most consistent with §8 ("source calls are logged … row count"; caps "enforced fail-fast"): the response body's line count is the enforced "size", fail-fast on exceed. To be locked in the plan like the I4 SQL type/number rules, and pinned in the ingest.py docstring + golden.
• [ASSUMPTION] Local-mock strategy. An injectable GET callable (defaulting to a stdlib urllib GET) so golden tests supply canned payloads with no socket — the hermetic reading of §11. A localhost http.server thread is a documented fallback; the plan decides.
• [ASSUMPTION] source_system frontmatter value for http. Mirror the existing file/sql renderer convention (plan to confirm from the current materializer — likely the source type or a base_url-derived value; must match the golden bytes).
• [ASSUMPTION] Opt-in flag threading + name. The flag threads through materialize and run_project as a run argument (e.g. allow_network: bool = False); the exact name/signature is pinned in the plan.
• [ASSUMPTION] create_sdk_mcp_server D7 pointer. Assumed current Claude Agent SDK primitive for the docs pointer; verify at doc-writing time (cheap, non-gating).

Prior Attempts

None for the http family specifically. I2 (CSV, e4ee8bd→994f5be) and I4 (SQL, d7e5f2f→1b7612b) are the direct precedents on this stack — identical connector / materialization / golden / load-bearing pattern — and I6 mirrors them for the network-transport family. The HttpSource schema and its fail-fast rejection already exist (ingest.py:84 and :394-396) as the gated stub that I6 activates behind the new flag.

Metadata

• Created: 2026-07-04
• Interview turns: 1 (framing); required-section content pre-populated from the detailed task directive + the frozen spec, then reviewed.
• Auto-research opted in: no
• Source: trekbrief interview

---

How to continue

Manual (default):

# No research topics — go straight to planning:
/trekplan --project .claude/projects/2026-07-04-i6-ingest-http-maf/

# Then execute:
/trekexecute --project .claude/projects/2026-07-04-i6-ingest-http-maf/