# I6 — HTTP source connector as a demonstrated extension point (MAF-only) > **Plan quality: A** (92/100) — APPROVE_WITH_NOTES > > Generated by trekplan v5.9.1 on 2026-07-04 — `plan_version: 1.7` ## Context The ingest layer implements the `file`/CSV (I2/I3) and `sql` (I4/I5) source families on both stacks, built from the frozen `shared/ingest-spec.md`. That spec (§1, §4) declares `type: "http"` an OPTIONAL extension point: the schema already validates `HttpSource` (`src/portfolio_optimiser/ingest.py:84-100`), but `materialize` refuses it fail-fast (`ingest.py:394-398`, "gated extension point (I6) with no connector"). I6 activates that stub to demonstrate — honestly and end-to-end — HOW a third, network-transport source family plugs into the same connector / materialization / gate contracts, WITHOUT consuming the platform's "first live source" milestone. It realizes two locked framework principles at once: the 90%-principle (generic core + clear extension points, do not chase the last 10%) and the security frame's "local-only default, no silent egress" (§8) — a network source added strictly behind an explicit run-flag the manifest itself cannot grant. MAF-only; `okf.py`, `shared/`, and the frozen spec are untouched. ## Architecture Diagram ```mermaid graph TD subgraph "I6 changes in ingest.py" M["materialize(...,
allow_network=False,
http_get=None)"]:::mod GATE{"isinstance(HttpSource)
AND not allow_network?"}:::new DISP["dispatch loop
(file / sql / http branch)"]:::mod RH["read_http(base_url, query,
*, max_rows, credential_ref, get)"]:::new UG["_urllib_get
(stdlib GET — ONLY socket path)"]:::new FENCE["_render_fenced_block
(verbatim, ``` fence)"]:::new end subgraph "reused UNCHANGED" RC["_render_concept_file
(7-key frontmatter)"]:::keep WB["_write_bytes / ownership /
collision / index"]:::keep OKF["okf.render_frontmatter /
link_in_index"]:::keep end TEST["golden + load-bearing tests
inject a canned http_get
(no socket, no creds)"]:::test M --> GATE GATE -->|"no flag → IngestError"| REFUSE["fail-fast (spec §8)"]:::new GATE -->|"allow_network=True"| DISP DISP --> RH --> UG RH -->|"body text"| FENCE --> RC --> WB --> OKF TEST -.injects.-> RH classDef new fill:#2d6,stroke:#164,color:#000 classDef mod fill:#fd6,stroke:#a70,color:#000 classDef keep fill:#ddd,stroke:#888,color:#000 classDef test fill:#8cf,stroke:#048,color:#000 ``` ## Codebase Analysis - **Tech stack:** Python ≥3.10, Pydantic (validation/IR), stdlib `urllib` (transport), `pytest` (+ byte-for-byte golden harness), `ruff`/`mypy`, `uv`. Zero new runtime deps. - **Key patterns:** one connector fn per source type returning renderable content; a per-source-type dispatch inside `materialize`; credential/connection resolution by **env-var name** (never from the manifest); a committed golden case per type; load-bearing tests that go RED when a seam is detached; spec-silent decisions pinned in the module docstring + frozen in golden bytes. - **Relevant files (verified):** - `src/portfolio_optimiser/ingest.py` — `HttpSource` schema (`:84-100`), the gate to replace (`:394-398`), the dispatch loop (`:408-426`), `read_sql` template (`:233-278`), `_render_concept_file` 7-key frontmatter (`:304-318`, `source_system = manifest.source.id`), `_write_bytes` (`:321-327`), ownership/collision/index machinery (`:428-470`), module docstring pin-block (`:1-31`). - `tests/test_ingest_golden_sql.py` — byte-for-byte golden harness (`_bundle_bytes`, file-set equality, §10 second-run idempotence). - `tests/test_ingest_sql_loadbearing.py` / `tests/test_ingest_sql.py` — detach-RED and connector-unit templates. - `tests/test_ingest_materialize.py:245-253` — `test_http_source_has_no_connector` (stale after I6: still green under default-no-flag, but its "no connector" comment becomes false). - `examples/ingest-golden-sql/` — golden layout to mirror (`manifest.json`, `fixture/`, `ingested-at.txt`, `expected-bundle/{index.md, ingest-*.md}`). - `.claude/projects/2026-07-04-i4-ingest-sql-maf/build_fixture.py` — throwaway fixture-builder pattern. - `docs/extending.md` — **already exists** (bilingual dev doc: Norwegian headers, English prose); I6 **appends** an http source-family section + the D7 hook pointer. - **Reusable code (leverage unchanged):** `_render_concept_file`, `_write_bytes`, the ownership scan / collision gate / index generation (`ingest.py:428-470`), `load_manifest`, `okf.render_frontmatter` / `okf.link_in_index`. **Explicitly NOT reused for http:** `render_table` / `_escape_cell` / `_sql_value_to_text` (http body is verbatim inside a fenced code block, not a table). - **External tech (researched):** none — 0 research topics (frozen spec fully defines http semantics; I4 SQL is the direct in-repo precedent; stdlib mock). The single external-adjacent item (`create_sdk_mcp_server` currency for the docs pointer) is a doc-time verification, not a planning input. - **Recent git activity:** `bc52c12` (I6 brief), `1b7612b`/`d7e5f2f` (I4 SQL, the direct precedent), `e4ee8bd`→`994f5be` (I2 CSV). Branch `main`, clean. ## Implementation Plan Each step is test-first (Iron Law). Steps are ordered by dependency: connector → gate+dispatch → golden → docs. ### Step 1: Implement `read_http` connector + injectable transport seam (docstring-pinned) - **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_http.py` *(new)* - **Changes:** Add, near `read_sql` (`ingest.py:233`): - A transport-seam type alias `HttpGet = Callable[[str, str | None], str]` (`(url, credential) -> response body text`); add `from collections.abc import Callable` and `from urllib.request import Request, urlopen` + `from urllib.error import URLError` to imports. - `def _urllib_get(url: str, credential: str | None) -> str` — the stdlib GET and the **only** socket path: build a `Request(url, headers=..., method="GET")`, attach `Authorization: Bearer {credential}` iff `credential is not None`, `urlopen`, read, decode UTF-8. Wrap `URLError`/`OSError`/`UnicodeDecodeError` as `IngestError`. Do NOT pre-add a `# noqa: S310` — `pyproject.toml` selects no `flake8-bandit` (`S`) rules, so S310 never fires and a dead noqa would trip `RUF100` if it is ever enabled; add the noqa ONLY if `uv run ruff check .` actually flags the `urlopen` at Verify. - `def read_http(base_url, query, *, max_rows, credential_ref=None, get: HttpGet = _urllib_get) -> str` mirroring `read_sql`: resolve `credential_ref` via `os.environ.get` (present-but-unset/empty → `IngestError` fail-fast, mirroring `ingest.py:249`; `None` → no auth); build `url = base_url.rstrip("/") + "/" + query.lstrip("/")`; call `get(url, credential)`; normalize CRLF→LF; **fence-collision fail-fast** — if any normalized line, after stripping up to its leading spaces (`line.lstrip(" ")`), begins with ` ``` ` (a fence marker — note CommonMark permits a closing fence indented ≤3 spaces, so a leading-space check is required), raise `IngestError` (a body that cannot be safely embedded in a fenced code block is an ERROR, never silently-corrupted markdown — the module's no-silent-coercion discipline + §8 fail-fast); enforce the cap on the **`\n`-only line count** of the CRLF-normalized text — `n = normalized.count("\n"); line_count = n if (normalized == "" or normalized.endswith("\n")) else n + 1`; `line_count > max_rows → IngestError` (never truncation, §8). **Do NOT use `str.splitlines()`** for the cap: CPython `splitlines()` splits on the extended Unicode set (`\v`, `\f`, `\x1c`-`\x1e`, `\x85`, `\u2028`, `\u2029`), so a body with a vertical-tab or line-separator would be over-counted vs the LF-only rendered file (verified 2026-07-04). The `\n`-only count matches the rendered body and still treats a single trailing newline as no phantom line. Return normalized text. Pure: no writes, no logging (materialize owns §8 log). - `def _render_fenced_block(body: str) -> str` → `f"\`\`\`\n{body.rstrip(chr(10))}\n\`\`\`\n"` (verbatim inside a fenced code block; LF-only; exactly one trailing newline). - **Docstring pin (module docstring, `ingest.py:23-30` block):** append the http (I6) pinned decisions, mirroring the I4 sql sentence: transport is an injectable `get` callable (default `_urllib_get`, the only socket); `credential_ref` names an env secret resolved at run time (Bearer), never from the manifest, never logged, never stamped in frontmatter; URL = `base_url` (one trailing slash stripped) + `/` + `query` (leading slash stripped); body = response decoded UTF-8, CRLF→LF, trailing newlines stripped, rendered verbatim in a ``` fenced block (NOT table-escaped); `max_rows` caps response **line count** (`len(normalized.splitlines())`), fail-fast; a response line beginning with ` ``` ` → `IngestError` fail-fast (cannot be safely fenced — never silent corruption, consistent with the no-silent-coercion frame; the fixed 3-backtick fence is deliberate, NOT a silent fallback). - **Reuses:** `read_sql` structure (`ingest.py:233-278`) — env-resolve → fetch → cap → `IngestError` wrap; the CSV/SQL streaming-cap idiom (`ingest.py:193-197`, `:270-274`). - **Test first:** - File: `tests/test_ingest_http.py` (new) - Verifies (each injects a canned `get`, no socket, no creds unless monkeypatched): `read_http` returns the injected body verbatim and calls `get` with the joined URL; a body of exactly `max_rows` lines → returns normally (boundary pass); `max_rows + 1` lines → `IngestError` (never truncated) **[SC max_rows]**; a body with a line beginning ` ``` ` → `IngestError` (fence-collision fail-fast); `credential_ref` present but env unset → `IngestError`; `credential_ref` set → the injected `get` receives the resolved secret (secret comes from env, never the manifest); `read_http`'s `get` default is `_urllib_get` (`inspect.signature` — proves the stdlib default without a socket). - Pattern: `tests/test_ingest_sql.py` (connector-unit style; local `get`-builder mirroring `_make_db` locality — do NOT add to `conftest.py`). - **Verify:** `uv run pytest tests/test_ingest_http.py -q` → expected: all pass (offline). - **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_http.py` - **Checkpoint:** `git commit -m "feat(ingest): http read_http connector + injectable transport seam (I6)"` - **Manifest:** ```yaml manifest: expected_paths: - src/portfolio_optimiser/ingest.py - tests/test_ingest_http.py min_file_count: 2 commit_message_pattern: "^feat\\(ingest\\): http read_http connector \\+ injectable transport seam \\(I6\\)$" bash_syntax_check: [] forbidden_paths: - src/portfolio_optimiser/okf.py - shared/ingest-spec.md must_contain: - path: src/portfolio_optimiser/ingest.py pattern: "def read_http" - path: src/portfolio_optimiser/ingest.py pattern: "_render_fenced_block" - path: tests/test_ingest_http.py pattern: "read_http" ``` ### Step 2: Gate http behind the `allow_network` run-flag + wire the dispatch branch - **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_http_loadbearing.py` *(new)*, `tests/test_ingest_materialize.py` - **Changes:** - `materialize` signature (`ingest.py:373-375`): add keyword-only `allow_network: bool = False` and `http_get: HttpGet | None = None` (both defaulted → all existing file/sql callers stay backward-compatible; `run_project`/`run_portfolio` are NOT touched — verified they never call `materialize`). Extend the docstring to note the §8 network opt-in flag is a run argument. - Replace the unconditional refusal (`ingest.py:393-398`) with the gate: `if isinstance(source, HttpSource) and not allow_network: raise IngestError(...)` — refuse fail-fast **before any source call**, message citing §8 ("the manifest cannot grant itself network access"). This exact `and not allow_network` clause is the load-bearing seam. - Restructure the dispatch loop (`ingest.py:408-426`) so each branch sets `body` and a `row_count`: `FileSource` → `read_csv` + `render_table`, `row_count = len(rows)`; `elif isinstance(source, HttpSource)` → `get = http_get if http_get is not None else _urllib_get`; `text = read_http(source.base_url, extraction.query, max_rows=extraction.max_rows, credential_ref=source.credential_ref, get=get)`; `body = _render_fenced_block(text)`; `row_count = len(text.splitlines())`; `else` (Sql) → `read_sql` + `render_table`, `row_count = len(rows)`. Keep the single `_LOGGER.info` §8 log (`ingest.py:416-421`) logging `rows=row_count` (line count for http). The stale "http already refused above" comment (`ingest.py:412`) is removed. - `tests/test_ingest_materialize.py:245-253`: update `test_http_source_has_no_connector`'s comment/intent — the default-no-flag path still raises `IngestError`, but the wording "no connector" is now false; reword to "http is refused without the per-run network flag" (assertion stays green; comment corrected). Do not delete — it is the default-refuse smoke. - **Reuses:** `_render_concept_file`, `_write_bytes`, ownership/collision/index machinery (`ingest.py:428-470`) — all unchanged; the http bundle flows through the identical §3/§6 disk phase. - **Test first:** - File: `tests/test_ingest_http_loadbearing.py` (new) - Verifies (detach-RED seams, each maps to a Success Criterion): 1. **Network gate [SC1] — asserts BOTH branches on the SAME manifest + SAME recording `get` (this is what makes it genuinely detach-RED):** - *Refuse branch:* `materialize(http_manifest, out, ingested_at=…, allow_network=False, http_get=recording_get)` → assert `IngestError` **and** `recording_get` was **never called** (refuse before any source access, §8). - *Allow branch:* `materialize(http_manifest, out, ingested_at=…, allow_network=True, http_get=recording_get)` → assert it **succeeds** (writes `ingest-*.md`) **and** `recording_get` **was called**. Why both: the seam is `if isinstance(source, HttpSource) and not allow_network: raise`. Removing the whole clause → the *refuse* assertion fails (get called, no error) → RED; removing just `and not allow_network` (reverting to the *unconditional* refusal that exists today) → the *allow* assertion fails (raises when it should fetch) → RED. A refuse-only test would false-green on the second detach. *(Inject a recording `get` that returns a valid payload — never the default `_urllib_get`, whose dead socket would false-green the refuse branch.)* 2. **Fenced-not-table [§5 http seam]:** payload with a raw `|` and `\`; assert the body is ``` -fenced and the pipe survives **un-escaped** (`"\\|" not in body`, raw `|` present). RED if the http path degrades to `render_table`. 3. **Navigability [§2/§11]:** drive the http bundle through unchanged `okf.navigate_bundle`/`bundle_context`; reachable via index link, classified by `okf_type`, provenance rides through. RED when index linking detaches. 4. **Over-cap writes nothing [SC max_rows]:** over-cap payload → `IngestError` **and** no `ingest-*.md` written (memory-staging). RED if silent truncation replaces the raise. - Pattern: `tests/test_ingest_sql_loadbearing.py` (detach discipline) + the ordering proof in `tests/test_ingest_manifest.py:178`. - **Verify:** `uv run pytest tests/test_ingest_http_loadbearing.py tests/test_ingest_materialize.py -q` → expected: all pass. Detach-proof (manual, at review): the gate test goes RED on BOTH detaches — (a) delete the whole `if isinstance(source, HttpSource) and not allow_network:` block → refuse branch RED; (b) reduce it to the unconditional `if isinstance(source, HttpSource):` → allow branch RED. - **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_http_loadbearing.py tests/test_ingest_materialize.py` - **Checkpoint:** `git commit -m "feat(ingest): gate http behind allow_network run-flag + dispatch (I6)"` - **Manifest:** ```yaml manifest: expected_paths: - src/portfolio_optimiser/ingest.py - tests/test_ingest_http_loadbearing.py - tests/test_ingest_materialize.py min_file_count: 3 commit_message_pattern: "^feat\\(ingest\\): gate http behind allow_network run-flag \\+ dispatch \\(I6\\)$" bash_syntax_check: [] forbidden_paths: - src/portfolio_optimiser/okf.py - src/portfolio_optimiser/run.py - shared/ingest-spec.md must_contain: - path: src/portfolio_optimiser/ingest.py pattern: "allow_network" - path: src/portfolio_optimiser/ingest.py pattern: "and not allow_network" - path: tests/test_ingest_http_loadbearing.py pattern: "allow_network=False" ``` ### Step 3: Author the byte-deterministic golden case `examples/ingest-golden-http/` - **Files:** `examples/ingest-golden-http/manifest.json` *(new)*, `examples/ingest-golden-http/fixture/status` *(new)*, `examples/ingest-golden-http/fixture/report` *(new)*, `examples/ingest-golden-http/ingested-at.txt` *(new)*, `examples/ingest-golden-http/expected-bundle/` *(new, generated + hand-reviewed)*, `tests/test_ingest_golden_http.py` *(new)*, `.claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py` *(new, throwaway)* - **Changes:** - `manifest.json`: `source` = `{"type": "http", "id": "status-api", "base_url": "https://api.example.test/v1", "credential_ref": null}` (`.test` reserved TLD + `credential_ref: null` → runs with **no network, no credentials**, proving SC offline); `bundle_summary`; two extractions — `{"id": "status", "title": "Service status", "query": "status", "okf_type": "dataset", "max_rows": 10}`, `{"id": "report", "title": "Ops report", "query": "report", "okf_type": "reference", "max_rows": 10}`. - `fixture/status` (1 line, discriminates fenced-verbatim vs table-escape — contains raw `|` and `\`): `{"service": "billing", "state": "degraded | partial", "path": "c:\temp\cache"}` - `fixture/report` (3 lines, ≤ max_rows): `baseline ok` / `pipe | and backslash \ kept verbatim` / `end of report`. - `ingested-at.txt`: `2026-07-04T12:00:00Z` + trailing newline (byte-write, §5 format). - `build_fixture.py`: writes the two payload files + `ingested-at.txt` (mirrors the I4 builder; payloads are LF-only so the golden is platform-stable). Documents regenerability. **Author the payload bytes as raw bytes / raw-string literals** — in a normal Python string literal `\t` is a TAB and `\c` a literal backslash, so the intended verbatim backslashes in `c:\temp\cache` (and `backslash \`) would be silently corrupted; write with `rb"..."` / `.write_bytes(b"...")` or `r"..."` so the byte the fenced-not-escaped test depends on survives exactly. - `expected-bundle/`: **generate** by running `materialize` against the fixture with a canned `get` (map `url.rsplit("/",1)[-1]` → `fixture/`), `allow_network=True`, then **verify the generated bytes against the spec-derived expectation below BEFORE committing** — this is an agent check against an independent oracle, not a human gate, so a headless `claude -p` run performs it too. The non-circular correctness oracle is carried here in the plan (and, independently, by the Step-2 load-bearing tests, which hand-assert the fenced-not-escaped body, the 7-key frontmatter, and navigability — so correctness is proven by hand-written assertions, not by comparing code output to code output). If the generated `ingest-status.md` diverges from the expectation below in any way OTHER than the `ingest_manifest` hash, **escalate** (do not commit a wrong oracle). Hand-derived `expected-bundle/ingest-status.md` (from §5/§7 + `fixture/status`; every field except the manifest-hash is independently derived): ``` --- type: dataset title: Service status source_system: status-api source_query: status ingested_at: 2026-07-04T12:00:00Z ingest_manifest: manifest@ generated: true --- ```‹fence› {"service": "billing", "state": "degraded | partial", "path": "c:\temp\cache"} ```‹fence› ``` Note the pipe and backslashes are **un-escaped** (verbatim) inside the fence — the exact discriminator vs `render_table`. `index.md` = `bundle_summary` + `- [Service status](ingest-status.md)` + `- [Ops report](ingest-report.md)`. The `ingest_manifest` hash is the only computed field (SHA-256 of the committed `manifest.json` bytes) — frozen once committed. - **Reuses:** golden layout + harness from `examples/ingest-golden-sql/` and `tests/test_ingest_golden_sql.py`; builder shape from the I4 `build_fixture.py`. - **Test first:** - File: `tests/test_ingest_golden_http.py` (new) - Verifies **[SC golden determinism]**: build the canned `get` from `fixture/`; `materialize(GOLDEN/manifest.json, out, ingested_at=, allow_network=True, http_get=fake)`; file-set equality, per-file byte compare against `expected-bundle/`, second-run idempotence (§10). No credential env set (`credential_ref: null`). - Pattern: `tests/test_ingest_golden_sql.py:30-47` (swap the `monkeypatch.setenv` DSN for the injected `get`). - **Verify:** `uv run pytest tests/test_ingest_golden_http.py -q` → expected: pass, byte-identical. - **On failure:** revert — `git checkout -- examples/ingest-golden-http tests/test_ingest_golden_http.py .claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py && rm -rf examples/ingest-golden-http` - **Checkpoint:** `git commit -m "test(ingest): http golden extraction, byte-deterministic (I6)"` - **Manifest:** ```yaml manifest: expected_paths: - examples/ingest-golden-http/manifest.json - examples/ingest-golden-http/ingested-at.txt - examples/ingest-golden-http/fixture/status - examples/ingest-golden-http/fixture/report - examples/ingest-golden-http/expected-bundle/index.md - examples/ingest-golden-http/expected-bundle/ingest-status.md - examples/ingest-golden-http/expected-bundle/ingest-report.md - tests/test_ingest_golden_http.py - .claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py min_file_count: 9 commit_message_pattern: "^test\\(ingest\\): http golden extraction, byte-deterministic \\(I6\\)$" bash_syntax_check: [] forbidden_paths: - src/portfolio_optimiser/okf.py - shared/ingest-spec.md must_contain: - path: examples/ingest-golden-http/expected-bundle/ingest-status.md pattern: "```" - path: examples/ingest-golden-http/expected-bundle/ingest-status.md pattern: "generated: true" - path: examples/ingest-golden-http/manifest.json pattern: "\"type\": \"http\"" ``` ### Step 4: Document the http extension-point pattern + D7 SDK hook pointer in `docs/extending.md` - **Files:** `docs/extending.md` - **Changes:** Append a new section (bilingual house style: Norwegian header, English prose) documenting the http source family as the worked extension-point example: the manifest `type: "http"` contract (`base_url` no embedded credentials, `credential_ref` by env-name), the **run-argument** network opt-in (`materialize(..., allow_network=True)`; the manifest cannot grant network — §8), the injectable transport seam (`http_get`, default stdlib `urllib`), and the "verbatim in a fenced code block" body. Add the D7 sibling-hook pointer: an MCP-based connector is "an extension of this family" (§4), and on the Claude Agent SDK side the in-process server hook is **`create_sdk_mcp_server(name, version="1.0.0", tools=...) -> McpSdkServerConfig`** (package `claude-agent-sdk`) — **VERIFIED 2026-07-04 against the official Claude Agent SDK Python docs** (code.claude.com/docs/en/agent-sdk/python + docs.claude.com agent-sdk/mcp; the name is confirmed, not assumed — pin it). Note MCP is documented as an extension of this family, NOT a new client wired into the run path (Non-Goal). Frame everything as "local mock / no live source"; do not claim a live integration. - **Reuses:** the existing "Bevisst ikke bygget (90 %-kuttlista)" seam-naming style in `docs/extending.md:55-74`; the honesty framing in `docs/extending.md:7-13`. - **Test first:** *(docs — no unit test; command-checkable at Verify)* - Verifies: `grep -q create_sdk_mcp_server docs/extending.md` exits 0, and a new http source-family section is present; the new I6 artifacts carry no affirmative live-source claim. - **Verify:** `grep -q create_sdk_mcp_server docs/extending.md && echo OK` → expected: `OK`; `grep -rin "live" docs/extending.md examples/ingest-golden-http/ src/portfolio_optimiser/ingest.py` → expected: any hit is only "local mock" / "no live source" negated framing (scoped to the I6 artifacts — the repo-wide grep is not a clean gate: `docs/`, `src/persona.py`, `simulation.py` etc. carry dozens of unrelated pre-existing "live"/"delivery" substrings). - **On failure:** revert — `git checkout -- docs/extending.md` - **Checkpoint:** `git commit -m "docs(i6): http source extension-point + D7 create_sdk_mcp_server pointer"` - **Manifest:** ```yaml manifest: expected_paths: - docs/extending.md min_file_count: 1 commit_message_pattern: "^docs\\(i6\\): http source extension-point \\+ D7 create_sdk_mcp_server pointer$" bash_syntax_check: [] forbidden_paths: - src/portfolio_optimiser/okf.py - shared/ingest-spec.md must_contain: - path: docs/extending.md pattern: "create_sdk_mcp_server" - path: docs/extending.md pattern: "allow_network" ``` ## Alternatives Considered | Approach | Pros | Cons | Why rejected | |----------|------|------|--------------| | Thread `allow_network` through `run_project`/`run_portfolio` (as the brief assumed) | Matches the brief's literal assumption | `run.py` never imports `ingest`/calls `materialize` (verified by 2 agents + grep) — the param would be dead code | Scope creep + speculative feature; violates surgical-change. Flag scoped to `materialize` only. | | Reuse `render_table` for the http body | Maximal "mirror I4" symmetry | §5 requires the http body **verbatim in a fenced code block**, not a table; `render_table` escapes `\|`/`\\` | Spec-nonconforming; the fenced-not-table load-bearing test would fail. New `_render_fenced_block`. | | `localhost http.server` thread for tests | "Real" HTTP round-trip | Opens a socket → violates §11 "without network access"; flaky/slow; hermeticity risk | Injectable `get` callable (canned payloads, no socket) is the hermetic §11 reading. Documented fallback only. | | Variable-length fence (count backticks in body) | Silently "handles" a body containing ``` | Extra determinism surface; papers over an ambiguous body the model would later read | Rejected in favour of **fail-fast**: a body line beginning ` ``` ` → `IngestError` (no silent corruption; consistent with the module's no-silent-coercion frame). Fixed 3-backtick fence is deliberate, not a fallback. | | `urllib.parse.urljoin` for the URL | Stdlib canonical | Surprising path semantics (absolute paths, `..`) → non-deterministic joins | Explicit `rstrip("/") + "/" + lstrip("/")` — predictable, pinned. | ## Test Strategy - **Framework:** `pytest` (`uv run pytest`); byte-for-byte golden harness; detach-RED load-bearing regime (spec §11). - **Existing patterns:** three-file-per-source-type convention (`test_ingest_http.py` connector-unit, `_loadbearing.py` detach-RED, `_golden_http.py` byte-for-byte) — mirrors the sql trio. Canned-`get` builders local to the test files (never `conftest.py`, which is MAF/LLM-only). - **New tests in this plan:** ~11 across 3 new files + 1 comment correction. ### Tests to write | Type | File | Verifies | Model test | |------|------|----------|------------| | Unit | `tests/test_ingest_http.py` | verbatim fetch; URL join; `max_rows`; credential env-resolve (set/unset); urllib default | `tests/test_ingest_sql.py` | | Load-bearing | `tests/test_ingest_http_loadbearing.py` | network gate (never-called + IngestError); fenced-not-table; navigability; over-cap-writes-nothing | `tests/test_ingest_sql_loadbearing.py` | | Golden | `tests/test_ingest_golden_http.py` | byte-identical materialization; §10 idempotence; no creds | `tests/test_ingest_golden_sql.py` | | Comment fix | `tests/test_ingest_materialize.py:245` | default-no-flag still refuses (intent reworded) | — | **Offline property (SC):** satisfied by construction — every http test injects `http_get` (no socket) and needs no credential. Acceptance: `uv run pytest` exits 0 with the network cable pulled. Convention to enforce in review: **no ingest test calls `materialize`/`read_http` for an http source without passing `http_get`** (the default `_urllib_get` is the only socket path and is never exercised). ## Risks and Mitigations | Priority | Risk | Location | Impact | Mitigation | |----------|------|----------|--------|------------| | Critical | Silent egress if the gate sits after connector construction, or `_urllib_get` is reachable without the flag | `ingest.py:393` gate / dispatch | Network hit with no operator opt-in — defeats the task | Gate in `materialize` BEFORE the staging loop; `_urllib_get` only reached inside the http branch, which the gate guards. Load-bearing test asserts `get` never called when flag absent. | | High | Credential echo in the verbatim body | http body render | A reflecting endpoint could echo `Authorization` into a committed bundle file | Inherent to "verbatim body" — do NOT auto-redact (breaks determinism). Golden mock never echoes a secret; document as a residual deployer-owned risk (§1 boundary). | | High | Credential leak via error/log text | `read_http` IngestError, `_LOGGER.info` | Secret in an exception or log line | Mirror `read_sql`: wrap transport errors WITHOUT the secret; keep §8 log to `id`/`ingested_at`/`row_count` only. (Optional sentinel-secret grep test — nice-to-have, not gating.) | | High | Determinism: verbatim-vs-LF-only tension, trailing newline, fence collision | `_render_fenced_block` / `read_http` | Golden passes locally, diverges on other input/platform; or silently-broken markdown in the bundle | Pin (docstring + golden): UTF-8 decode fail-fast; CRLF→LF; strip trailing newlines outside the fence; `max_rows = len(splitlines())`; **a body line beginning ` ``` ` → `IngestError` fail-fast** (never a silently-corrupted fenced block — consistent with the no-silent-coercion frame). | | Medium | `max_rows` line-count semantics (extended-Unicode / trailing / blank lines) | `read_http` | Spurious pass/fail vs golden | Count `\n`-only on the CRLF-normalized text (NOT `str.splitlines()`, which also splits on `\v \f \x1c-\x1e \x85 \u2028 \u2029` — verified): `n = normalized.count("\n"); line_count = n if normalized.endswith("\n") else n+1`. Matches the LF-only body; a trailing newline adds no phantom line; internal blank lines count. Test exactly `max_rows` (pass) and `max_rows+1` (raise). | | Medium | `base_url`+`query` join edge cases | `read_http` | Non-deterministic joins; scheme/host escape | Pinned `rstrip("/")+"/"+lstrip("/")`; `base_url` userinfo already rejected (`ingest.py:92-100`); the joined URL is never stamped in frontmatter. | | Low | Stale `test_http_source_has_no_connector` intent | `tests/test_ingest_materialize.py:245` | Comment says "no connector" (now false) | Reword in Step 2; keep the default-refuse assertion. | | Low | Temptation to touch `okf.py`/`shared/` | `okf.py`, `shared/` | Breaks D7-portability invariant / commons pull-only | `forbidden_paths` on every code step; SC `git diff --stat` shows no change. | | Low | ruff `S310` (urlopen) / mypy `Callable` typing | `ingest.py` | Lint/type red | `# noqa: S310` on the gated `urlopen`; `Callable` from `collections.abc`; confirm at Verify. | ## Assumptions | # | Assumption | Why unverifiable at plan time | Impact if wrong | |---|-----------|-----------------|-----------------| | 1 | The default auth scheme for a resolved `credential_ref` is `Authorization: Bearer {secret}` | The spec names credential-by-ref but not the wire scheme; endpoint-specific | A real deployer's endpoint may need a different scheme — acceptable for a demo extension point; documented, and the `get` seam is swappable. Golden uses `credential_ref: null` so this is not frozen in golden bytes. | *(RESOLVED, not open — the brief's `[ASSUMPTION] flag threads through materialize and run_project` was verified FALSE: `run.py` never calls `materialize`. The flag is scoped to `materialize` only. `source_system` = `manifest.source.id` confirmed from `_render_concept_file:312`; `max_rows` = response line count pinned per §8. The D7 pointer `create_sdk_mcp_server` was verified 2026-07-04 against the official Claude Agent SDK Python docs — it is the confirmed in-process MCP-server primitive, not an assumption.)* ## Verification *Per-step manifests are checked by trekexecute. These are the brief's Success Criteria as end-to-end checks (run from repo root):* - [ ] `uv run pytest -q` → expected: exits 0 (full suite green, no network, no credentials set). - [ ] Network gate detach-proof: comment out `and not allow_network` in `ingest.py`, run `uv run pytest tests/test_ingest_http_loadbearing.py -k gate` → expected: **RED**; restore → green. - [ ] `uv run pytest tests/test_ingest_golden_http.py -q` → expected: pass; output byte-matches `examples/ingest-golden-http/expected-bundle/`. - [ ] `uv run pytest tests/test_ingest_http.py -k max_rows` → expected: pass (over-cap → IngestError). - [ ] `grep -rin "live" docs/extending.md examples/ingest-golden-http/ src/portfolio_optimiser/ingest.py` → expected: any hit is only "local mock" / "no live source" negated framing (scoped to the NEW I6 artifacts — a repo-wide `grep "live"` is not a pass/fail gate: dozens of unrelated pre-existing hits in `docs/`, `persona.py`, `simulation.py`, `test_*_live.py`). - [ ] `git diff --stat` (vs pre-I6) → expected: NO change to `src/portfolio_optimiser/okf.py` or `shared/`. - [ ] `uv run ruff check .` → 0; `uv run ruff format --check .` → 0; `uv run mypy src` → 0. - [ ] `grep -q create_sdk_mcp_server docs/extending.md` → exits 0; new http section present. - [ ] Null API-spend: no model calls, no real network (golden/gate/unit all use injected `get`). ## Estimated Scope - **Files to modify:** 3 (`ingest.py`, `tests/test_ingest_materialize.py`, `docs/extending.md`). - **Files to create:** 11 (3 test files + 8 golden/fixture/builder: `manifest.json`, `ingested-at.txt`, `fixture/status`, `fixture/report`, 3 × `expected-bundle/*.md`, `build_fixture.py`). - **Complexity:** medium (one genuinely new render path + a security gate; the rest mirrors I4). ## Plan Quality Score | Dimension | Weight | Score | Notes | |-----------|--------|-------|-------| | Structural integrity | 0.15 | 95 | 4 steps, strict dependency order (connector→gate→golden→docs) | | Step quality | 0.20 | 92 | test-first, exact file:line, explicit reuse; Step 2 touches 3 files (cohesive) | | Coverage completeness | 0.20 | 95 | every Success Criterion → a step + a Verification check | | Specification quality | 0.15 | 90 | pinned decisions concrete; 2 honest doc-time assumptions remain | | Risk & pre-mortem | 0.15 | 92 | silent-egress, credential-leak, determinism all mitigated | | Headless readiness | 0.10 | 90 | On-failure + Checkpoint + Manifest per step | | Manifest quality | 0.05 | 88 | must_contain + forbidden_paths guard okf.py/shared/ | | **Weighted total** | **1.00** | **92** | **Grade: A** | **Adversarial review:** - **Plan critic:** REVISE → addressed. 1 blocker (gate false-green), 5 major (live-grep unsatisfiable, first-live-source circular, create_sdk_mcp_server contradiction, fence-collision silent, golden circular/hand-review), 6 minor — all resolved in Revisions below. - **Scope guardian:** ALIGNED. 0 creep, 0 Non-Goal violations, 0 dependency issues; independently confirmed the `materialize`-only flag scoping is a correct scope fix (not a gap). Its one minor (over-broad live-grep) coincides with plan-critic MAJOR-1 and is fixed. ## Revisions *Added by adversarial review (Phase 9). All blocker + major findings addressed; the plan was revised once for the merged set.* | # | Finding | Severity | Resolution | |---|---------|----------|------------| | 1 | Gate load-bearing test would false-green: reverting `and not allow_network` to the unconditional refusal still passes a refuse-only test | blocker | Step 2 gate test now asserts BOTH branches on the same manifest + same recording `get` (refuse: flag off → IngestError + get never called; allow: flag on → succeeds + get called). Removing the whole gate → refuse branch RED; removing just the clause → allow branch RED. Verify wording fixed. | | 2 | `grep -rin "live" docs/ src/ examples/` unsatisfiable (dozens of pre-existing hits) | major | Scoped the "no live claim" check to the NEW I6 artifacts (`docs/extending.md`, `examples/ingest-golden-http/`, `ingest.py`) in both Step 4 and Verification. | | 3 | `git grep "first live source"` circular (matches only plan/brief) | major | Removed the circular sub-check. | | 4 | `create_sdk_mcp_server` pinned in 3 gates while flagged "may be stale" | major | Verified 2026-07-04 against official Claude Agent SDK Python docs — confirmed correct; pinned confidently, Assumption downgraded to a verified fact. | | 5 | Fence-collision (body containing a ` ``` ` line) silently corrupts the bundle | major | Changed to fail-fast: a body line beginning ` ``` ` → `IngestError` (Step 1 connector + docstring + risk + alternatives + a new unit test). | | 6 | Golden expected-bundle circular + "hand-review" is a non-headless human step | major | Step 3 now carries the hand-derived `ingest-status.md` expectation IN the plan (non-circular, headless-usable), names the Step-2 load-bearing tests as the independent correctness oracle, and adds an escalate-on-divergence clause. | | 7 | `max_rows` counts lines pre-strip; blank-line nuance | minor | Pinned: count on CRLF-normalized text before the trailing-strip; blank lines count as content lines (deterministic). Risk row + Step 1 updated. | | 8 | `c:\temp\cache` backslash/tab trap in `build_fixture.py` | minor | Pinned raw-bytes / raw-string authoring so the verbatim backslash survives. | | 9 | `# noqa: S310` likely dead (no bandit config) | minor | Do not pre-add; add only if `ruff check` flags the `urlopen` at Verify. | | 10 | File-count off by one (10 vs 11 created) | minor | Corrected to 11 created / 3 modified. | | 11 | Line-ref drift (`:393` vs `:394`) | minor | Standardized the gate reference to `:394-398`. | | 12 | Pass-at-`max_rows` boundary test only in risk table | minor | Added the boundary-pass + `max_rows+1`-raise pair to the Step 1 test enumeration. | ## Adversarial Pass 2 (gemini-bridge, v5.1.1 high-effort) The high-effort plan phase runs an additional independent gemini-bridge review. **Outcome: gemini-bridge is non-functional (verified 2026-07-04, not assumed).** The deep-research query ran to `status: completed`, but every retrieval path failed: `gemini_get_research_result` → `'Interaction' object has no attribute 'outputs'` (server-side bug, 3× retry); the followup re-emit → `404 model gemini-3-pro-preview no longer available` (deprecated). This confirms the prior recorded outage — the operator's expectation that it was fixed did not hold. No findings were retrievable; the agent correctly refused to fabricate a second opinion. **Substitute triangulation (per the standing gemini-outage handling):** - An independent high-effort **plan-critic confirmation pass** traced the blocker fix against both detach cases and verified revisions #2–#6 — plan **sound**, no new blocker/major; folded in one minor (CommonMark ≤3-space indented closing fence → `line.lstrip(" ").startswith("```")`). - The two decisions gemini flagged as spec-checkable were resolved by **primary-source verification** (no gemini needed): **CommonMark §4.5** fenced-code semantics (closing fence ≥ opening length, ≤3-space indent) → the fence-collision check is adequate; **CPython `str.splitlines()`** splits on the extended Unicode set → the `max_rows` cap was switched to a **`\n`-only** count so it matches the LF-only rendered body (Step 1 + risk row updated). - Decision 4 ("config cannot grant its own capability; capability is an explicit runtime grant") is the **capability-security / POLA** (Principle of Least Authority) pattern — the manifest is data that cannot escalate its own authority; only the `allow_network` runtime grant can. Net: no external second opinion available, but every decision gemini would have triangulated is covered by an independent internal pass or an authoritative primary-source check.