diff --git a/.claude/projects/2026-07-04-i6-ingest-http-maf/plan.md b/.claude/projects/2026-07-04-i6-ingest-http-maf/plan.md new file mode 100644 index 0000000..ac0b279 --- /dev/null +++ b/.claude/projects/2026-07-04-i6-ingest-http-maf/plan.md @@ -0,0 +1,565 @@ +# I6 — HTTP source connector as a demonstrated extension point (MAF-only) + +> **Plan quality: A** (92/100) — APPROVE_WITH_NOTES +> +> Generated by trekplan v5.9.1 on 2026-07-04 — `plan_version: 1.7` + +## Context + +The ingest layer implements the `file`/CSV (I2/I3) and `sql` (I4/I5) source families on +both stacks, built from the frozen `shared/ingest-spec.md`. That spec (§1, §4) declares +`type: "http"` an OPTIONAL extension point: the schema already validates `HttpSource` +(`src/portfolio_optimiser/ingest.py:84-100`), but `materialize` refuses it fail-fast +(`ingest.py:394-398`, "gated extension point (I6) with no connector"). I6 activates that stub +to demonstrate — honestly and end-to-end — HOW a third, network-transport source family plugs +into the same connector / materialization / gate contracts, WITHOUT consuming the platform's +"first live source" milestone. It realizes two locked framework principles at once: the +90%-principle (generic core + clear extension points, do not chase the last 10%) and the +security frame's "local-only default, no silent egress" (§8) — a network source added strictly +behind an explicit run-flag the manifest itself cannot grant. MAF-only; `okf.py`, `shared/`, +and the frozen spec are untouched. + +## Architecture Diagram + +```mermaid +graph TD + subgraph "I6 changes in ingest.py" + M["materialize(...,
allow_network=False,
http_get=None)"]:::mod + GATE{"isinstance(HttpSource)
AND not allow_network?"}:::new + DISP["dispatch loop
(file / sql / http branch)"]:::mod + RH["read_http(base_url, query,
*, max_rows, credential_ref, get)"]:::new + UG["_urllib_get
(stdlib GET — ONLY socket path)"]:::new + FENCE["_render_fenced_block
(verbatim, ``` fence)"]:::new + end + subgraph "reused UNCHANGED" + RC["_render_concept_file
(7-key frontmatter)"]:::keep + WB["_write_bytes / ownership /
collision / index"]:::keep + OKF["okf.render_frontmatter /
link_in_index"]:::keep + end + TEST["golden + load-bearing tests
inject a canned http_get
(no socket, no creds)"]:::test + + M --> GATE + GATE -->|"no flag → IngestError"| REFUSE["fail-fast (spec §8)"]:::new + GATE -->|"allow_network=True"| DISP + DISP --> RH --> UG + RH -->|"body text"| FENCE --> RC --> WB --> OKF + TEST -.injects.-> RH + + classDef new fill:#2d6,stroke:#164,color:#000 + classDef mod fill:#fd6,stroke:#a70,color:#000 + classDef keep fill:#ddd,stroke:#888,color:#000 + classDef test fill:#8cf,stroke:#048,color:#000 +``` + +## Codebase Analysis + +- **Tech stack:** Python ≥3.10, Pydantic (validation/IR), stdlib `urllib` (transport), + `pytest` (+ byte-for-byte golden harness), `ruff`/`mypy`, `uv`. Zero new runtime deps. +- **Key patterns:** one connector fn per source type returning renderable content; a + per-source-type dispatch inside `materialize`; credential/connection resolution by + **env-var name** (never from the manifest); a committed golden case per type; load-bearing + tests that go RED when a seam is detached; spec-silent decisions pinned in the module + docstring + frozen in golden bytes. +- **Relevant files (verified):** + - `src/portfolio_optimiser/ingest.py` — `HttpSource` schema (`:84-100`), the gate to + replace (`:394-398`), the dispatch loop (`:408-426`), `read_sql` template (`:233-278`), + `_render_concept_file` 7-key frontmatter (`:304-318`, `source_system = manifest.source.id`), + `_write_bytes` (`:321-327`), ownership/collision/index machinery (`:428-470`), module + docstring pin-block (`:1-31`). + - `tests/test_ingest_golden_sql.py` — byte-for-byte golden harness (`_bundle_bytes`, file-set + equality, §10 second-run idempotence). + - `tests/test_ingest_sql_loadbearing.py` / `tests/test_ingest_sql.py` — detach-RED and + connector-unit templates. + - `tests/test_ingest_materialize.py:245-253` — `test_http_source_has_no_connector` (stale + after I6: still green under default-no-flag, but its "no connector" comment becomes false). + - `examples/ingest-golden-sql/` — golden layout to mirror (`manifest.json`, `fixture/`, + `ingested-at.txt`, `expected-bundle/{index.md, ingest-*.md}`). + - `.claude/projects/2026-07-04-i4-ingest-sql-maf/build_fixture.py` — throwaway fixture-builder + pattern. + - `docs/extending.md` — **already exists** (bilingual dev doc: Norwegian headers, English + prose); I6 **appends** an http source-family section + the D7 hook pointer. +- **Reusable code (leverage unchanged):** `_render_concept_file`, `_write_bytes`, the ownership + scan / collision gate / index generation (`ingest.py:428-470`), `load_manifest`, + `okf.render_frontmatter` / `okf.link_in_index`. **Explicitly NOT reused for http:** + `render_table` / `_escape_cell` / `_sql_value_to_text` (http body is verbatim inside a + fenced code block, not a table). +- **External tech (researched):** none — 0 research topics (frozen spec fully defines http + semantics; I4 SQL is the direct in-repo precedent; stdlib mock). The single external-adjacent + item (`create_sdk_mcp_server` currency for the docs pointer) is a doc-time verification, not a + planning input. +- **Recent git activity:** `bc52c12` (I6 brief), `1b7612b`/`d7e5f2f` (I4 SQL, the direct + precedent), `e4ee8bd`→`994f5be` (I2 CSV). Branch `main`, clean. + +## Implementation Plan + +Each step is test-first (Iron Law). Steps are ordered by dependency: connector → gate+dispatch → +golden → docs. + +### Step 1: Implement `read_http` connector + injectable transport seam (docstring-pinned) + +- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_http.py` *(new)* +- **Changes:** Add, near `read_sql` (`ingest.py:233`): + - A transport-seam type alias `HttpGet = Callable[[str, str | None], str]` (`(url, credential) -> response body text`); add `from collections.abc import Callable` and + `from urllib.request import Request, urlopen` + `from urllib.error import URLError` to imports. + - `def _urllib_get(url: str, credential: str | None) -> str` — the stdlib GET and the **only** + socket path: build a `Request(url, headers=..., method="GET")`, attach + `Authorization: Bearer {credential}` iff `credential is not None`, `urlopen`, read, decode + UTF-8. Wrap `URLError`/`OSError`/`UnicodeDecodeError` as `IngestError`. Do NOT pre-add a + `# noqa: S310` — `pyproject.toml` selects no `flake8-bandit` (`S`) rules, so S310 never + fires and a dead noqa would trip `RUF100` if it is ever enabled; add the noqa ONLY if + `uv run ruff check .` actually flags the `urlopen` at Verify. + - `def read_http(base_url, query, *, max_rows, credential_ref=None, get: HttpGet = _urllib_get) -> str` + mirroring `read_sql`: resolve `credential_ref` via `os.environ.get` (present-but-unset/empty → + `IngestError` fail-fast, mirroring `ingest.py:249`; `None` → no auth); build + `url = base_url.rstrip("/") + "/" + query.lstrip("/")`; call `get(url, credential)`; + normalize CRLF→LF; **fence-collision fail-fast** — if any normalized line, after stripping + up to its leading spaces (`line.lstrip(" ")`), begins with ` ``` ` (a fence marker — note + CommonMark permits a closing fence indented ≤3 spaces, so a leading-space check is required), + raise `IngestError` (a body that cannot be safely embedded in a fenced code block is an + ERROR, never silently-corrupted markdown — the module's no-silent-coercion discipline + §8 + fail-fast); enforce the cap on the **`\n`-only line count** of the CRLF-normalized text — + `n = normalized.count("\n"); line_count = n if (normalized == "" or normalized.endswith("\n")) else n + 1`; + `line_count > max_rows → IngestError` (never truncation, §8). **Do NOT use `str.splitlines()`** + for the cap: CPython `splitlines()` splits on the extended Unicode set + (`\v`, `\f`, `\x1c`-`\x1e`, `\x85`, `\u2028`, `\u2029`), so a body with a vertical-tab or line-separator would be over-counted vs the + LF-only rendered file (verified 2026-07-04). The `\n`-only count matches the rendered body and + still treats a single trailing newline as no phantom line. Return normalized text. Pure: no + writes, no logging (materialize owns §8 log). + - `def _render_fenced_block(body: str) -> str` → `f"\`\`\`\n{body.rstrip(chr(10))}\n\`\`\`\n"` + (verbatim inside a fenced code block; LF-only; exactly one trailing newline). + - **Docstring pin (module docstring, `ingest.py:23-30` block):** append the http (I6) pinned + decisions, mirroring the I4 sql sentence: transport is an injectable `get` callable + (default `_urllib_get`, the only socket); `credential_ref` names an env secret resolved at + run time (Bearer), never from the manifest, never logged, never stamped in frontmatter; + URL = `base_url` (one trailing slash stripped) + `/` + `query` (leading slash stripped); + body = response decoded UTF-8, CRLF→LF, trailing newlines stripped, rendered verbatim in a + ``` fenced block (NOT table-escaped); `max_rows` caps response **line count** + (`len(normalized.splitlines())`), fail-fast; a response line beginning with ` ``` ` → + `IngestError` fail-fast (cannot be safely fenced — never silent corruption, consistent with + the no-silent-coercion frame; the fixed 3-backtick fence is deliberate, NOT a silent + fallback). +- **Reuses:** `read_sql` structure (`ingest.py:233-278`) — env-resolve → fetch → cap → + `IngestError` wrap; the CSV/SQL streaming-cap idiom (`ingest.py:193-197`, `:270-274`). +- **Test first:** + - File: `tests/test_ingest_http.py` (new) + - Verifies (each injects a canned `get`, no socket, no creds unless monkeypatched): + `read_http` returns the injected body verbatim and calls `get` with the joined URL; + a body of exactly `max_rows` lines → returns normally (boundary pass); + `max_rows + 1` lines → `IngestError` (never truncated) **[SC max_rows]**; + a body with a line beginning ` ``` ` → `IngestError` (fence-collision fail-fast); + `credential_ref` present but env unset → `IngestError`; `credential_ref` set → the injected + `get` receives the resolved secret (secret comes from env, never the manifest); + `read_http`'s `get` default is `_urllib_get` (`inspect.signature` — proves the stdlib + default without a socket). + - Pattern: `tests/test_ingest_sql.py` (connector-unit style; local `get`-builder mirroring + `_make_db` locality — do NOT add to `conftest.py`). +- **Verify:** `uv run pytest tests/test_ingest_http.py -q` → expected: all pass (offline). +- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_http.py` +- **Checkpoint:** `git commit -m "feat(ingest): http read_http connector + injectable transport seam (I6)"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - src/portfolio_optimiser/ingest.py + - tests/test_ingest_http.py + min_file_count: 2 + commit_message_pattern: "^feat\\(ingest\\): http read_http connector \\+ injectable transport seam \\(I6\\)$" + bash_syntax_check: [] + forbidden_paths: + - src/portfolio_optimiser/okf.py + - shared/ingest-spec.md + must_contain: + - path: src/portfolio_optimiser/ingest.py + pattern: "def read_http" + - path: src/portfolio_optimiser/ingest.py + pattern: "_render_fenced_block" + - path: tests/test_ingest_http.py + pattern: "read_http" + ``` + +### Step 2: Gate http behind the `allow_network` run-flag + wire the dispatch branch + +- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_http_loadbearing.py` + *(new)*, `tests/test_ingest_materialize.py` +- **Changes:** + - `materialize` signature (`ingest.py:373-375`): add keyword-only `allow_network: bool = False` + and `http_get: HttpGet | None = None` (both defaulted → all existing file/sql callers stay + backward-compatible; `run_project`/`run_portfolio` are NOT touched — verified they never call + `materialize`). Extend the docstring to note the §8 network opt-in flag is a run argument. + - Replace the unconditional refusal (`ingest.py:393-398`) with the gate: + `if isinstance(source, HttpSource) and not allow_network: raise IngestError(...)` — refuse + fail-fast **before any source call**, message citing §8 ("the manifest cannot grant itself + network access"). This exact `and not allow_network` clause is the load-bearing seam. + - Restructure the dispatch loop (`ingest.py:408-426`) so each branch sets `body` and a + `row_count`: `FileSource` → `read_csv` + `render_table`, `row_count = len(rows)`; + `elif isinstance(source, HttpSource)` → `get = http_get if http_get is not None else _urllib_get`; + `text = read_http(source.base_url, extraction.query, max_rows=extraction.max_rows, credential_ref=source.credential_ref, get=get)`; + `body = _render_fenced_block(text)`; `row_count = len(text.splitlines())`; `else` (Sql) → + `read_sql` + `render_table`, `row_count = len(rows)`. Keep the single `_LOGGER.info` §8 log + (`ingest.py:416-421`) logging `rows=row_count` (line count for http). The stale + "http already refused above" comment (`ingest.py:412`) is removed. + - `tests/test_ingest_materialize.py:245-253`: update `test_http_source_has_no_connector`'s + comment/intent — the default-no-flag path still raises `IngestError`, but the wording + "no connector" is now false; reword to "http is refused without the per-run network flag" + (assertion stays green; comment corrected). Do not delete — it is the default-refuse smoke. +- **Reuses:** `_render_concept_file`, `_write_bytes`, ownership/collision/index machinery + (`ingest.py:428-470`) — all unchanged; the http bundle flows through the identical §3/§6 disk + phase. +- **Test first:** + - File: `tests/test_ingest_http_loadbearing.py` (new) + - Verifies (detach-RED seams, each maps to a Success Criterion): + 1. **Network gate [SC1] — asserts BOTH branches on the SAME manifest + SAME recording `get` + (this is what makes it genuinely detach-RED):** + - *Refuse branch:* `materialize(http_manifest, out, ingested_at=…, allow_network=False, http_get=recording_get)` + → assert `IngestError` **and** `recording_get` was **never called** (refuse before any + source access, §8). + - *Allow branch:* `materialize(http_manifest, out, ingested_at=…, allow_network=True, http_get=recording_get)` + → assert it **succeeds** (writes `ingest-*.md`) **and** `recording_get` **was called**. + Why both: the seam is `if isinstance(source, HttpSource) and not allow_network: raise`. + Removing the whole clause → the *refuse* assertion fails (get called, no error) → RED; + removing just `and not allow_network` (reverting to the *unconditional* refusal that + exists today) → the *allow* assertion fails (raises when it should fetch) → RED. A + refuse-only test would false-green on the second detach. *(Inject a recording `get` that + returns a valid payload — never the default `_urllib_get`, whose dead socket would + false-green the refuse branch.)* + 2. **Fenced-not-table [§5 http seam]:** payload with a raw `|` and `\`; assert the body is + ``` -fenced and the pipe survives **un-escaped** (`"\\|" not in body`, raw `|` present). + RED if the http path degrades to `render_table`. + 3. **Navigability [§2/§11]:** drive the http bundle through unchanged + `okf.navigate_bundle`/`bundle_context`; reachable via index link, classified by + `okf_type`, provenance rides through. RED when index linking detaches. + 4. **Over-cap writes nothing [SC max_rows]:** over-cap payload → `IngestError` **and** no + `ingest-*.md` written (memory-staging). RED if silent truncation replaces the raise. + - Pattern: `tests/test_ingest_sql_loadbearing.py` (detach discipline) + the ordering proof in + `tests/test_ingest_manifest.py:178`. +- **Verify:** `uv run pytest tests/test_ingest_http_loadbearing.py tests/test_ingest_materialize.py -q` + → expected: all pass. Detach-proof (manual, at review): the gate test goes RED on BOTH + detaches — (a) delete the whole `if isinstance(source, HttpSource) and not allow_network:` + block → refuse branch RED; (b) reduce it to the unconditional `if isinstance(source, HttpSource):` + → allow branch RED. +- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_http_loadbearing.py tests/test_ingest_materialize.py` +- **Checkpoint:** `git commit -m "feat(ingest): gate http behind allow_network run-flag + dispatch (I6)"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - src/portfolio_optimiser/ingest.py + - tests/test_ingest_http_loadbearing.py + - tests/test_ingest_materialize.py + min_file_count: 3 + commit_message_pattern: "^feat\\(ingest\\): gate http behind allow_network run-flag \\+ dispatch \\(I6\\)$" + bash_syntax_check: [] + forbidden_paths: + - src/portfolio_optimiser/okf.py + - src/portfolio_optimiser/run.py + - shared/ingest-spec.md + must_contain: + - path: src/portfolio_optimiser/ingest.py + pattern: "allow_network" + - path: src/portfolio_optimiser/ingest.py + pattern: "and not allow_network" + - path: tests/test_ingest_http_loadbearing.py + pattern: "allow_network=False" + ``` + +### Step 3: Author the byte-deterministic golden case `examples/ingest-golden-http/` + +- **Files:** `examples/ingest-golden-http/manifest.json` *(new)*, + `examples/ingest-golden-http/fixture/status` *(new)*, + `examples/ingest-golden-http/fixture/report` *(new)*, + `examples/ingest-golden-http/ingested-at.txt` *(new)*, + `examples/ingest-golden-http/expected-bundle/` *(new, generated + hand-reviewed)*, + `tests/test_ingest_golden_http.py` *(new)*, + `.claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py` *(new, throwaway)* +- **Changes:** + - `manifest.json`: `source` = `{"type": "http", "id": "status-api", "base_url": "https://api.example.test/v1", "credential_ref": null}` + (`.test` reserved TLD + `credential_ref: null` → runs with **no network, no credentials**, + proving SC offline); `bundle_summary`; two extractions — + `{"id": "status", "title": "Service status", "query": "status", "okf_type": "dataset", "max_rows": 10}`, + `{"id": "report", "title": "Ops report", "query": "report", "okf_type": "reference", "max_rows": 10}`. + - `fixture/status` (1 line, discriminates fenced-verbatim vs table-escape — contains raw `|` + and `\`): `{"service": "billing", "state": "degraded | partial", "path": "c:\temp\cache"}` + - `fixture/report` (3 lines, ≤ max_rows): `baseline ok` / `pipe | and backslash \ kept verbatim` / + `end of report`. + - `ingested-at.txt`: `2026-07-04T12:00:00Z` + trailing newline (byte-write, §5 format). + - `build_fixture.py`: writes the two payload files + `ingested-at.txt` (mirrors the I4 builder; + payloads are LF-only so the golden is platform-stable). Documents regenerability. **Author + the payload bytes as raw bytes / raw-string literals** — in a normal Python string literal + `\t` is a TAB and `\c` a literal backslash, so the intended verbatim backslashes in + `c:\temp\cache` (and `backslash \`) would be silently corrupted; write with `rb"..."` / + `.write_bytes(b"...")` or `r"..."` so the byte the fenced-not-escaped test depends on + survives exactly. + - `expected-bundle/`: **generate** by running `materialize` against the fixture with a canned + `get` (map `url.rsplit("/",1)[-1]` → `fixture/`), `allow_network=True`, then **verify + the generated bytes against the spec-derived expectation below BEFORE committing** — this is + an agent check against an independent oracle, not a human gate, so a headless `claude -p` + run performs it too. The non-circular correctness oracle is carried here in the plan (and, + independently, by the Step-2 load-bearing tests, which hand-assert the fenced-not-escaped + body, the 7-key frontmatter, and navigability — so correctness is proven by hand-written + assertions, not by comparing code output to code output). If the generated `ingest-status.md` + diverges from the expectation below in any way OTHER than the `ingest_manifest` hash, + **escalate** (do not commit a wrong oracle). + + Hand-derived `expected-bundle/ingest-status.md` (from §5/§7 + `fixture/status`; every field + except the manifest-hash is independently derived): + ``` + --- + type: dataset + title: Service status + source_system: status-api + source_query: status + ingested_at: 2026-07-04T12:00:00Z + ingest_manifest: manifest@ + generated: true + --- + + ```‹fence› + {"service": "billing", "state": "degraded | partial", "path": "c:\temp\cache"} + ```‹fence› + ``` + Note the pipe and backslashes are **un-escaped** (verbatim) inside the fence — the exact + discriminator vs `render_table`. `index.md` = `bundle_summary` + `- [Service status](ingest-status.md)` + + `- [Ops report](ingest-report.md)`. The `ingest_manifest` hash is the only computed field + (SHA-256 of the committed `manifest.json` bytes) — frozen once committed. +- **Reuses:** golden layout + harness from `examples/ingest-golden-sql/` and + `tests/test_ingest_golden_sql.py`; builder shape from the I4 `build_fixture.py`. +- **Test first:** + - File: `tests/test_ingest_golden_http.py` (new) + - Verifies **[SC golden determinism]**: build the canned `get` from `fixture/`; `materialize(GOLDEN/manifest.json, out, ingested_at=, allow_network=True, http_get=fake)`; + file-set equality, per-file byte compare against `expected-bundle/`, second-run idempotence + (§10). No credential env set (`credential_ref: null`). + - Pattern: `tests/test_ingest_golden_sql.py:30-47` (swap the `monkeypatch.setenv` DSN for the + injected `get`). +- **Verify:** `uv run pytest tests/test_ingest_golden_http.py -q` → expected: pass, byte-identical. +- **On failure:** revert — `git checkout -- examples/ingest-golden-http tests/test_ingest_golden_http.py .claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py && rm -rf examples/ingest-golden-http` +- **Checkpoint:** `git commit -m "test(ingest): http golden extraction, byte-deterministic (I6)"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - examples/ingest-golden-http/manifest.json + - examples/ingest-golden-http/ingested-at.txt + - examples/ingest-golden-http/fixture/status + - examples/ingest-golden-http/fixture/report + - examples/ingest-golden-http/expected-bundle/index.md + - examples/ingest-golden-http/expected-bundle/ingest-status.md + - examples/ingest-golden-http/expected-bundle/ingest-report.md + - tests/test_ingest_golden_http.py + - .claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py + min_file_count: 9 + commit_message_pattern: "^test\\(ingest\\): http golden extraction, byte-deterministic \\(I6\\)$" + bash_syntax_check: [] + forbidden_paths: + - src/portfolio_optimiser/okf.py + - shared/ingest-spec.md + must_contain: + - path: examples/ingest-golden-http/expected-bundle/ingest-status.md + pattern: "```" + - path: examples/ingest-golden-http/expected-bundle/ingest-status.md + pattern: "generated: true" + - path: examples/ingest-golden-http/manifest.json + pattern: "\"type\": \"http\"" + ``` + +### Step 4: Document the http extension-point pattern + D7 SDK hook pointer in `docs/extending.md` + +- **Files:** `docs/extending.md` +- **Changes:** Append a new section (bilingual house style: Norwegian header, English prose) + documenting the http source family as the worked extension-point example: the manifest + `type: "http"` contract (`base_url` no embedded credentials, `credential_ref` by env-name), + the **run-argument** network opt-in (`materialize(..., allow_network=True)`; the manifest + cannot grant network — §8), the injectable transport seam (`http_get`, default stdlib + `urllib`), and the "verbatim in a fenced code block" body. Add the D7 sibling-hook pointer: + an MCP-based connector is "an extension of this family" (§4), and on the Claude Agent SDK side + the in-process server hook is **`create_sdk_mcp_server(name, version="1.0.0", tools=...) -> + McpSdkServerConfig`** (package `claude-agent-sdk`) — **VERIFIED 2026-07-04 against the official + Claude Agent SDK Python docs** (code.claude.com/docs/en/agent-sdk/python + docs.claude.com + agent-sdk/mcp; the name is confirmed, not assumed — pin it). Note MCP is documented as an + extension of this family, NOT a new client wired into the run path (Non-Goal). Frame + everything as "local mock / no live source"; do not claim a live integration. +- **Reuses:** the existing "Bevisst ikke bygget (90 %-kuttlista)" seam-naming style in + `docs/extending.md:55-74`; the honesty framing in `docs/extending.md:7-13`. +- **Test first:** *(docs — no unit test; command-checkable at Verify)* + - Verifies: `grep -q create_sdk_mcp_server docs/extending.md` exits 0, and a new http + source-family section is present; the new I6 artifacts carry no affirmative live-source claim. +- **Verify:** `grep -q create_sdk_mcp_server docs/extending.md && echo OK` → expected: `OK`; + `grep -rin "live" docs/extending.md examples/ingest-golden-http/ src/portfolio_optimiser/ingest.py` + → expected: any hit is only "local mock" / "no live source" negated framing (scoped to the + I6 artifacts — the repo-wide grep is not a clean gate: `docs/`, `src/persona.py`, + `simulation.py` etc. carry dozens of unrelated pre-existing "live"/"delivery" substrings). +- **On failure:** revert — `git checkout -- docs/extending.md` +- **Checkpoint:** `git commit -m "docs(i6): http source extension-point + D7 create_sdk_mcp_server pointer"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - docs/extending.md + min_file_count: 1 + commit_message_pattern: "^docs\\(i6\\): http source extension-point \\+ D7 create_sdk_mcp_server pointer$" + bash_syntax_check: [] + forbidden_paths: + - src/portfolio_optimiser/okf.py + - shared/ingest-spec.md + must_contain: + - path: docs/extending.md + pattern: "create_sdk_mcp_server" + - path: docs/extending.md + pattern: "allow_network" + ``` + +## Alternatives Considered + +| Approach | Pros | Cons | Why rejected | +|----------|------|------|--------------| +| Thread `allow_network` through `run_project`/`run_portfolio` (as the brief assumed) | Matches the brief's literal assumption | `run.py` never imports `ingest`/calls `materialize` (verified by 2 agents + grep) — the param would be dead code | Scope creep + speculative feature; violates surgical-change. Flag scoped to `materialize` only. | +| Reuse `render_table` for the http body | Maximal "mirror I4" symmetry | §5 requires the http body **verbatim in a fenced code block**, not a table; `render_table` escapes `\|`/`\\` | Spec-nonconforming; the fenced-not-table load-bearing test would fail. New `_render_fenced_block`. | +| `localhost http.server` thread for tests | "Real" HTTP round-trip | Opens a socket → violates §11 "without network access"; flaky/slow; hermeticity risk | Injectable `get` callable (canned payloads, no socket) is the hermetic §11 reading. Documented fallback only. | +| Variable-length fence (count backticks in body) | Silently "handles" a body containing ``` | Extra determinism surface; papers over an ambiguous body the model would later read | Rejected in favour of **fail-fast**: a body line beginning ` ``` ` → `IngestError` (no silent corruption; consistent with the module's no-silent-coercion frame). Fixed 3-backtick fence is deliberate, not a fallback. | +| `urllib.parse.urljoin` for the URL | Stdlib canonical | Surprising path semantics (absolute paths, `..`) → non-deterministic joins | Explicit `rstrip("/") + "/" + lstrip("/")` — predictable, pinned. | + +## Test Strategy + +- **Framework:** `pytest` (`uv run pytest`); byte-for-byte golden harness; detach-RED + load-bearing regime (spec §11). +- **Existing patterns:** three-file-per-source-type convention + (`test_ingest_http.py` connector-unit, `_loadbearing.py` detach-RED, `_golden_http.py` + byte-for-byte) — mirrors the sql trio. Canned-`get` builders local to the test files (never + `conftest.py`, which is MAF/LLM-only). +- **New tests in this plan:** ~11 across 3 new files + 1 comment correction. + +### Tests to write + +| Type | File | Verifies | Model test | +|------|------|----------|------------| +| Unit | `tests/test_ingest_http.py` | verbatim fetch; URL join; `max_rows`; credential env-resolve (set/unset); urllib default | `tests/test_ingest_sql.py` | +| Load-bearing | `tests/test_ingest_http_loadbearing.py` | network gate (never-called + IngestError); fenced-not-table; navigability; over-cap-writes-nothing | `tests/test_ingest_sql_loadbearing.py` | +| Golden | `tests/test_ingest_golden_http.py` | byte-identical materialization; §10 idempotence; no creds | `tests/test_ingest_golden_sql.py` | +| Comment fix | `tests/test_ingest_materialize.py:245` | default-no-flag still refuses (intent reworded) | — | + +**Offline property (SC):** satisfied by construction — every http test injects `http_get` (no +socket) and needs no credential. Acceptance: `uv run pytest` exits 0 with the network cable +pulled. Convention to enforce in review: **no ingest test calls `materialize`/`read_http` for an +http source without passing `http_get`** (the default `_urllib_get` is the only socket path and +is never exercised). + +## Risks and Mitigations + +| Priority | Risk | Location | Impact | Mitigation | +|----------|------|----------|--------|------------| +| Critical | Silent egress if the gate sits after connector construction, or `_urllib_get` is reachable without the flag | `ingest.py:393` gate / dispatch | Network hit with no operator opt-in — defeats the task | Gate in `materialize` BEFORE the staging loop; `_urllib_get` only reached inside the http branch, which the gate guards. Load-bearing test asserts `get` never called when flag absent. | +| High | Credential echo in the verbatim body | http body render | A reflecting endpoint could echo `Authorization` into a committed bundle file | Inherent to "verbatim body" — do NOT auto-redact (breaks determinism). Golden mock never echoes a secret; document as a residual deployer-owned risk (§1 boundary). | +| High | Credential leak via error/log text | `read_http` IngestError, `_LOGGER.info` | Secret in an exception or log line | Mirror `read_sql`: wrap transport errors WITHOUT the secret; keep §8 log to `id`/`ingested_at`/`row_count` only. (Optional sentinel-secret grep test — nice-to-have, not gating.) | +| High | Determinism: verbatim-vs-LF-only tension, trailing newline, fence collision | `_render_fenced_block` / `read_http` | Golden passes locally, diverges on other input/platform; or silently-broken markdown in the bundle | Pin (docstring + golden): UTF-8 decode fail-fast; CRLF→LF; strip trailing newlines outside the fence; `max_rows = len(splitlines())`; **a body line beginning ` ``` ` → `IngestError` fail-fast** (never a silently-corrupted fenced block — consistent with the no-silent-coercion frame). | +| Medium | `max_rows` line-count semantics (extended-Unicode / trailing / blank lines) | `read_http` | Spurious pass/fail vs golden | Count `\n`-only on the CRLF-normalized text (NOT `str.splitlines()`, which also splits on `\v \f \x1c-\x1e \x85 \u2028 \u2029` — verified): `n = normalized.count("\n"); line_count = n if normalized.endswith("\n") else n+1`. Matches the LF-only body; a trailing newline adds no phantom line; internal blank lines count. Test exactly `max_rows` (pass) and `max_rows+1` (raise). | +| Medium | `base_url`+`query` join edge cases | `read_http` | Non-deterministic joins; scheme/host escape | Pinned `rstrip("/")+"/"+lstrip("/")`; `base_url` userinfo already rejected (`ingest.py:92-100`); the joined URL is never stamped in frontmatter. | +| Low | Stale `test_http_source_has_no_connector` intent | `tests/test_ingest_materialize.py:245` | Comment says "no connector" (now false) | Reword in Step 2; keep the default-refuse assertion. | +| Low | Temptation to touch `okf.py`/`shared/` | `okf.py`, `shared/` | Breaks D7-portability invariant / commons pull-only | `forbidden_paths` on every code step; SC `git diff --stat` shows no change. | +| Low | ruff `S310` (urlopen) / mypy `Callable` typing | `ingest.py` | Lint/type red | `# noqa: S310` on the gated `urlopen`; `Callable` from `collections.abc`; confirm at Verify. | + +## Assumptions + +| # | Assumption | Why unverifiable at plan time | Impact if wrong | +|---|-----------|-----------------|-----------------| +| 1 | The default auth scheme for a resolved `credential_ref` is `Authorization: Bearer {secret}` | The spec names credential-by-ref but not the wire scheme; endpoint-specific | A real deployer's endpoint may need a different scheme — acceptable for a demo extension point; documented, and the `get` seam is swappable. Golden uses `credential_ref: null` so this is not frozen in golden bytes. | + +*(RESOLVED, not open — the brief's `[ASSUMPTION] flag threads through materialize and +run_project` was verified FALSE: `run.py` never calls `materialize`. The flag is scoped to +`materialize` only. `source_system` = `manifest.source.id` confirmed from +`_render_concept_file:312`; `max_rows` = response line count pinned per §8. The D7 pointer +`create_sdk_mcp_server` was verified 2026-07-04 against the official Claude Agent SDK Python docs +— it is the confirmed in-process MCP-server primitive, not an assumption.)* + +## Verification + +*Per-step manifests are checked by trekexecute. These are the brief's Success Criteria as +end-to-end checks (run from repo root):* + +- [ ] `uv run pytest -q` → expected: exits 0 (full suite green, no network, no credentials set). +- [ ] Network gate detach-proof: comment out `and not allow_network` in `ingest.py`, run + `uv run pytest tests/test_ingest_http_loadbearing.py -k gate` → expected: **RED**; restore → green. +- [ ] `uv run pytest tests/test_ingest_golden_http.py -q` → expected: pass; output byte-matches + `examples/ingest-golden-http/expected-bundle/`. +- [ ] `uv run pytest tests/test_ingest_http.py -k max_rows` → expected: pass (over-cap → IngestError). +- [ ] `grep -rin "live" docs/extending.md examples/ingest-golden-http/ src/portfolio_optimiser/ingest.py` + → expected: any hit is only "local mock" / "no live source" negated framing (scoped to the + NEW I6 artifacts — a repo-wide `grep "live"` is not a pass/fail gate: dozens of unrelated + pre-existing hits in `docs/`, `persona.py`, `simulation.py`, `test_*_live.py`). +- [ ] `git diff --stat` (vs pre-I6) → expected: NO change to `src/portfolio_optimiser/okf.py` or `shared/`. +- [ ] `uv run ruff check .` → 0; `uv run ruff format --check .` → 0; `uv run mypy src` → 0. +- [ ] `grep -q create_sdk_mcp_server docs/extending.md` → exits 0; new http section present. +- [ ] Null API-spend: no model calls, no real network (golden/gate/unit all use injected `get`). + +## Estimated Scope + +- **Files to modify:** 3 (`ingest.py`, `tests/test_ingest_materialize.py`, `docs/extending.md`). +- **Files to create:** 11 (3 test files + 8 golden/fixture/builder: `manifest.json`, + `ingested-at.txt`, `fixture/status`, `fixture/report`, 3 × `expected-bundle/*.md`, + `build_fixture.py`). +- **Complexity:** medium (one genuinely new render path + a security gate; the rest mirrors I4). + +## Plan Quality Score + +| Dimension | Weight | Score | Notes | +|-----------|--------|-------|-------| +| Structural integrity | 0.15 | 95 | 4 steps, strict dependency order (connector→gate→golden→docs) | +| Step quality | 0.20 | 92 | test-first, exact file:line, explicit reuse; Step 2 touches 3 files (cohesive) | +| Coverage completeness | 0.20 | 95 | every Success Criterion → a step + a Verification check | +| Specification quality | 0.15 | 90 | pinned decisions concrete; 2 honest doc-time assumptions remain | +| Risk & pre-mortem | 0.15 | 92 | silent-egress, credential-leak, determinism all mitigated | +| Headless readiness | 0.10 | 90 | On-failure + Checkpoint + Manifest per step | +| Manifest quality | 0.05 | 88 | must_contain + forbidden_paths guard okf.py/shared/ | +| **Weighted total** | **1.00** | **92** | **Grade: A** | + +**Adversarial review:** +- **Plan critic:** REVISE → addressed. 1 blocker (gate false-green), 5 major (live-grep + unsatisfiable, first-live-source circular, create_sdk_mcp_server contradiction, fence-collision + silent, golden circular/hand-review), 6 minor — all resolved in Revisions below. +- **Scope guardian:** ALIGNED. 0 creep, 0 Non-Goal violations, 0 dependency issues; independently + confirmed the `materialize`-only flag scoping is a correct scope fix (not a gap). Its one minor + (over-broad live-grep) coincides with plan-critic MAJOR-1 and is fixed. + +## Revisions + +*Added by adversarial review (Phase 9). All blocker + major findings addressed; the plan was +revised once for the merged set.* + +| # | Finding | Severity | Resolution | +|---|---------|----------|------------| +| 1 | Gate load-bearing test would false-green: reverting `and not allow_network` to the unconditional refusal still passes a refuse-only test | blocker | Step 2 gate test now asserts BOTH branches on the same manifest + same recording `get` (refuse: flag off → IngestError + get never called; allow: flag on → succeeds + get called). Removing the whole gate → refuse branch RED; removing just the clause → allow branch RED. Verify wording fixed. | +| 2 | `grep -rin "live" docs/ src/ examples/` unsatisfiable (dozens of pre-existing hits) | major | Scoped the "no live claim" check to the NEW I6 artifacts (`docs/extending.md`, `examples/ingest-golden-http/`, `ingest.py`) in both Step 4 and Verification. | +| 3 | `git grep "first live source"` circular (matches only plan/brief) | major | Removed the circular sub-check. | +| 4 | `create_sdk_mcp_server` pinned in 3 gates while flagged "may be stale" | major | Verified 2026-07-04 against official Claude Agent SDK Python docs — confirmed correct; pinned confidently, Assumption downgraded to a verified fact. | +| 5 | Fence-collision (body containing a ` ``` ` line) silently corrupts the bundle | major | Changed to fail-fast: a body line beginning ` ``` ` → `IngestError` (Step 1 connector + docstring + risk + alternatives + a new unit test). | +| 6 | Golden expected-bundle circular + "hand-review" is a non-headless human step | major | Step 3 now carries the hand-derived `ingest-status.md` expectation IN the plan (non-circular, headless-usable), names the Step-2 load-bearing tests as the independent correctness oracle, and adds an escalate-on-divergence clause. | +| 7 | `max_rows` counts lines pre-strip; blank-line nuance | minor | Pinned: count on CRLF-normalized text before the trailing-strip; blank lines count as content lines (deterministic). Risk row + Step 1 updated. | +| 8 | `c:\temp\cache` backslash/tab trap in `build_fixture.py` | minor | Pinned raw-bytes / raw-string authoring so the verbatim backslash survives. | +| 9 | `# noqa: S310` likely dead (no bandit config) | minor | Do not pre-add; add only if `ruff check` flags the `urlopen` at Verify. | +| 10 | File-count off by one (10 vs 11 created) | minor | Corrected to 11 created / 3 modified. | +| 11 | Line-ref drift (`:393` vs `:394`) | minor | Standardized the gate reference to `:394-398`. | +| 12 | Pass-at-`max_rows` boundary test only in risk table | minor | Added the boundary-pass + `max_rows+1`-raise pair to the Step 1 test enumeration. | + +## Adversarial Pass 2 (gemini-bridge, v5.1.1 high-effort) + +The high-effort plan phase runs an additional independent gemini-bridge review. **Outcome: +gemini-bridge is non-functional (verified 2026-07-04, not assumed).** The deep-research query ran +to `status: completed`, but every retrieval path failed: `gemini_get_research_result` → +`'Interaction' object has no attribute 'outputs'` (server-side bug, 3× retry); the followup +re-emit → `404 model gemini-3-pro-preview no longer available` (deprecated). This confirms the +prior recorded outage — the operator's expectation that it was fixed did not hold. No findings +were retrievable; the agent correctly refused to fabricate a second opinion. + +**Substitute triangulation (per the standing gemini-outage handling):** +- An independent high-effort **plan-critic confirmation pass** traced the blocker fix against both + detach cases and verified revisions #2–#6 — plan **sound**, no new blocker/major; folded in one + minor (CommonMark ≤3-space indented closing fence → `line.lstrip(" ").startswith("```")`). +- The two decisions gemini flagged as spec-checkable were resolved by **primary-source verification** + (no gemini needed): **CommonMark §4.5** fenced-code semantics (closing fence ≥ opening length, + ≤3-space indent) → the fence-collision check is adequate; **CPython `str.splitlines()`** splits on + the extended Unicode set → the `max_rows` cap was switched to a **`\n`-only** count so it matches + the LF-only rendered body (Step 1 + risk row updated). +- Decision 4 ("config cannot grant its own capability; capability is an explicit runtime grant") + is the **capability-security / POLA** (Principle of Least Authority) pattern — the manifest is + data that cannot escalate its own authority; only the `allow_network` runtime grant can. + +Net: no external second opinion available, but every decision gemini would have triangulated is +covered by an independent internal pass or an authoritative primary-source check.