docs(i6): brief for http source extension point (I6)
/trekbrief for the http source family (MAF-only). framing=refine, brief-reviewer 5/5 PROCEED, validator PASS, 0 research topics. Contract: activate http behind a run-argument opt-in network flag, local-mock-only (no live source), body verbatim in fenced code block (§5), golden examples/ingest-golden-http/. Two deferred [ASSUMPTION]s for /trekplan to pin: max_rows unit + injectable-GET mock seam. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015U5UK6gVFJNvTEdDcyUb8W
This commit is contained in:
parent
1b7612b22d
commit
bc52c12ddb
1 changed files with 220 additions and 0 deletions
220
.claude/projects/2026-07-04-i6-ingest-http-maf/brief.md
Normal file
220
.claude/projects/2026-07-04-i6-ingest-http-maf/brief.md
Normal file
|
|
@ -0,0 +1,220 @@
|
|||
---
|
||||
type: trekbrief
|
||||
brief_version: "2.2"
|
||||
created: 2026-07-04
|
||||
task: "I6 — wire the HTTP source connector behind an opt-in network flag (MAF-only), tested against a local mock, no live source"
|
||||
slug: i6-ingest-http-maf
|
||||
project_dir: .claude/projects/2026-07-04-i6-ingest-http-maf/
|
||||
research_topics: 0
|
||||
research_status: skipped
|
||||
auto_research: false
|
||||
interview_turns: 1
|
||||
source: interview
|
||||
framing: refine
|
||||
phase_signals:
|
||||
- phase: research
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: plan
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: execute
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: review
|
||||
effort: high
|
||||
model: opus
|
||||
---
|
||||
|
||||
# Task: I6 — HTTP source connector as a demonstrated extension point (MAF-only)
|
||||
|
||||
> Generated by `/trekbrief` on 2026-07-04.
|
||||
> This brief is the contract between requirements and planning. `/trekplan`
|
||||
> reads it to produce the implementation plan. Every decision in the plan must
|
||||
> trace back to content in this brief.
|
||||
|
||||
## TL;DR
|
||||
|
||||
Refine of the frozen ingest program (I1–I5). Activate the `http` source family —
|
||||
already declared an OPTIONAL extension point in the spec — by wiring the connector
|
||||
behind an explicit per-run network opt-in flag, tested ONLY against a local mock
|
||||
(no live source; that gate stays unconsumed). Proves the extension-point pattern
|
||||
and the "no silent egress" frame. MAF-only; `okf.py`/`shared/` and the spec untouched.
|
||||
|
||||
## Intent
|
||||
|
||||
The ingest layer now implements the `file`/CSV (I2/I3) and `sql` (I4/I5) source
|
||||
families on both stacks, built from the shared frozen spec. That spec (§1, §4)
|
||||
declares `type: "http"` an OPTIONAL extension point: the schema validates
|
||||
`HttpSource` today, but `materialize` refuses it fail-fast
|
||||
(`src/portfolio_optimiser/ingest.py:394`, "gated extension point (I6) with no
|
||||
connector"). I6 activates that stub to demonstrate — honestly and end-to-end —
|
||||
HOW a third, network-transport source family plugs into the same connector /
|
||||
materialization / gate contracts, WITHOUT consuming the platform's "first live
|
||||
source" milestone. This realizes two locked framework principles at once: the
|
||||
90%-principle (build the generic core plus clear extension points, do not chase
|
||||
the last 10%) and the security frame's "local-only default, no silent egress"
|
||||
(§8) — showing a network source can be added strictly behind an explicit gate
|
||||
that the manifest itself cannot grant.
|
||||
|
||||
## Goal
|
||||
|
||||
`materialize` (and the surrounding run path) accept an explicit opt-in network
|
||||
flag that is a RUN argument, never a manifest field (§8). When a manifest's
|
||||
source is `HttpSource`: without the flag the connector refuses fail-fast with
|
||||
`IngestError`; with the flag it fetches each extraction (the extraction `query`
|
||||
is a path joined to `base_url`, retrieved with GET — §4/§5), resolves an optional
|
||||
`credential_ref` from the environment by name (mirroring the I4 `connection_ref`
|
||||
seam), and materializes one `ingest-{id}.md` per extraction whose body is the
|
||||
**response body verbatim inside a fenced code block** (§5, distinct from the
|
||||
markdown table used for `file`/`sql`), enforcing `max_rows` fail-fast. A golden
|
||||
case `examples/ingest-golden-http/` reproduces byte-deterministic output against
|
||||
committed mock payloads — with no network and no credentials (§9/§11).
|
||||
`docs/extending.md` documents the extension-point pattern and points at the D7
|
||||
sibling hook. `okf.py` and `shared/` are untouched; the frozen spec is unchanged.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **No live HTTP source.** Tested only against a local/in-process (or injected)
|
||||
mock. The "first live source" gate in the ingest brief remains unconsumed.
|
||||
- **No MCP client wired into the run path.** MCP stays the documented
|
||||
"extension of this family" framing (§4) plus the existing server-side
|
||||
demonstration in `datasource.py` (`build_mcp_server`); I6 does not add a new
|
||||
MCP-over-HTTP client connector.
|
||||
- **No spec change.** The ingest spec is FROZEN; any change is a separate gated
|
||||
commons round. I6 implements, it does not amend.
|
||||
- **No change to `okf.py` or `shared/`.** The MAF-free, D7-portable context seam
|
||||
and the pull-only commons subtree stay untouched.
|
||||
- **No change to the D7 sibling repo this session.** MAF-only; the D7 http mirror
|
||||
is a later, separate slice.
|
||||
- **No manifest-level network grant.** The flag is a run argument only.
|
||||
- **Not building "typed index everywhere."** That OKF-standard remains a deferred,
|
||||
separately-coordinated commons RFC.
|
||||
|
||||
## Constraints
|
||||
|
||||
- The frozen ingest spec (`shared/ingest-spec.md`) is authoritative. §4/§5 define
|
||||
the http query (path + GET), body rendering (verbatim in a fenced code block),
|
||||
frontmatter keys, and credential-by-name; §8/§9 define the network gate and the
|
||||
HITL layers; §11 defines the golden format. I6 conforms; it does not extend the
|
||||
spec.
|
||||
- Test suite MUST run **without network access and without credentials** (§9,
|
||||
§11) — full green offline.
|
||||
- Prefer **zero new runtime dependencies** — the connector uses the stdlib for
|
||||
GET, and tests use a stdlib/injected mock.
|
||||
- **Cost discipline (D6):** null API-spend; no model calls in the ingest program.
|
||||
- **Commons is pull-only.** Do not touch `shared/`; never `git subtree push`.
|
||||
|
||||
## Preferences
|
||||
|
||||
- Mirror the I4 SQL implementation shape: a per-source-type value/body render
|
||||
dispatch, credential/connection resolution by env-name, a committed golden case,
|
||||
and load-bearing tests that go RED when a seam is detached.
|
||||
- **Transport seam:** an injectable GET callable (defaulting to a stdlib
|
||||
`urllib` GET) so golden tests supply canned mock payloads with no socket —
|
||||
the hermetic reading of §11 "without network access".
|
||||
- TDD (Iron Law: failing test first). Surgical changes; match existing
|
||||
`ingest.py` style and docstring-pinning convention.
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
- **Hermetic tests:** the full `pytest` suite is green with no network and no
|
||||
credentials.
|
||||
- **Determinism:** the golden http extraction is byte-identical on repeat runs
|
||||
(fixed `ingested_at`, committed mock payloads).
|
||||
- **Fail-fast security:** no silent egress; flag absent → refuse before any
|
||||
source call.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- **Network gate (load-bearing):** without the opt-in flag, an `http` manifest is
|
||||
refused fail-fast — a test asserts `IngestError`, and reverting the gate seam
|
||||
turns that test RED. (`uv run pytest -k <gate test>` passes; detach → fail.)
|
||||
- **Golden determinism:** with the flag, the golden case materializes
|
||||
byte-identically — `uv run pytest -k <golden http test>` passes and output
|
||||
matches committed `examples/ingest-golden-http/` bytes.
|
||||
- **Offline suite:** `uv run pytest` exits 0 with no network and no credentials
|
||||
set (mock in-process / injected transport).
|
||||
- **max_rows fail-fast:** an http extraction whose response exceeds `max_rows` is
|
||||
an `IngestError` (never silent truncation) — asserted by a test.
|
||||
- **No live claim:** `grep -rin "live" docs/ src/ examples/` surfaces only
|
||||
"local mock" / "no live source" framing; the "first live source" gate is still
|
||||
documented as unconsumed.
|
||||
- **Context seam untouched:** `git diff --stat` shows no change to `okf.py` or
|
||||
`shared/`.
|
||||
- **Lint/format/type green:** `uv run ruff check .`, `uv run ruff format --check .`,
|
||||
and `uv run mypy src` each exit 0.
|
||||
- **Null API-spend:** the run makes no model or network calls (the golden run uses
|
||||
the injected mock).
|
||||
- **Docs:** `docs/extending.md` exists and documents the extension-point pattern
|
||||
with the D7 `create_sdk_mcp_server` hook pointer.
|
||||
|
||||
Do NOT accept vague criteria ("http works", "the connector is implemented") —
|
||||
each criterion above is command-checkable or detach-provable.
|
||||
|
||||
## Research Plan
|
||||
|
||||
No external research needed — the codebase and this brief contain sufficient
|
||||
context for planning. Verified basis: the frozen spec (`shared/ingest-spec.md`
|
||||
§4/§5/§8/§9/§11) fully defines the http source semantics; the implementation
|
||||
mirrors the already-landed I4 SQL patterns (env-resolved credential-by-name,
|
||||
per-type render dispatch, golden-case structure, detach-proven load-bearing
|
||||
tests); the local mock uses the Python stdlib (no obscure or platform-specific
|
||||
behavior — the search-first rule is not triggered). The single external-adjacent
|
||||
item, the current `create_sdk_mcp_server` API for the `docs/extending.md` D7
|
||||
pointer, is a low-stakes documentation pointer for the sibling repo, recorded as
|
||||
an assumption to verify at doc-writing time — it does not gate the plan.
|
||||
|
||||
## Open Questions / Assumptions
|
||||
|
||||
- **[ASSUMPTION] `max_rows` for http.** The response body renders verbatim (no
|
||||
rows). The plan must pin the enforced interpretation — most consistent with §8
|
||||
("source calls are logged … row count"; caps "enforced fail-fast"): the
|
||||
response body's line count is the enforced "size", fail-fast on exceed. To be
|
||||
locked in the plan like the I4 SQL type/number rules, and pinned in the
|
||||
`ingest.py` docstring + golden.
|
||||
- **[ASSUMPTION] Local-mock strategy.** An injectable GET callable (defaulting to
|
||||
a stdlib `urllib` GET) so golden tests supply canned payloads with no socket —
|
||||
the hermetic reading of §11. A localhost `http.server` thread is a documented
|
||||
fallback; the plan decides.
|
||||
- **[ASSUMPTION] `source_system` frontmatter value for http.** Mirror the
|
||||
existing `file`/`sql` renderer convention (plan to confirm from the current
|
||||
materializer — likely the source `type` or a `base_url`-derived value; must
|
||||
match the golden bytes).
|
||||
- **[ASSUMPTION] Opt-in flag threading + name.** The flag threads through
|
||||
`materialize` and `run_project` as a run argument (e.g. `allow_network: bool =
|
||||
False`); the exact name/signature is pinned in the plan.
|
||||
- **[ASSUMPTION] `create_sdk_mcp_server` D7 pointer.** Assumed current Claude
|
||||
Agent SDK primitive for the docs pointer; verify at doc-writing time (cheap,
|
||||
non-gating).
|
||||
|
||||
## Prior Attempts
|
||||
|
||||
None for the http family specifically. I2 (CSV, `e4ee8bd`→`994f5be`) and I4 (SQL,
|
||||
`d7e5f2f`→`1b7612b`) are the direct precedents on this stack — identical
|
||||
connector / materialization / golden / load-bearing pattern — and I6 mirrors them
|
||||
for the network-transport family. The `HttpSource` schema and its fail-fast
|
||||
rejection already exist (`ingest.py:84` and `:394-396`) as the gated stub that I6
|
||||
activates behind the new flag.
|
||||
|
||||
## Metadata
|
||||
|
||||
- **Created:** 2026-07-04
|
||||
- **Interview turns:** 1 (framing); required-section content pre-populated from
|
||||
the detailed task directive + the frozen spec, then reviewed.
|
||||
- **Auto-research opted in:** no
|
||||
- **Source:** trekbrief interview
|
||||
|
||||
---
|
||||
|
||||
## How to continue
|
||||
|
||||
Manual (default):
|
||||
|
||||
```bash
|
||||
# No research topics — go straight to planning:
|
||||
/trekplan --project .claude/projects/2026-07-04-i6-ingest-http-maf/
|
||||
|
||||
# Then execute:
|
||||
/trekexecute --project .claude/projects/2026-07-04-i6-ingest-http-maf/
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue