portfolio-optimiser/.claude/projects/2026-07-04-i6-ingest-http-maf/plan.md

I6 — HTTP source connector as a demonstrated extension point (MAF-only)

Plan quality: A (92/100) — APPROVE_WITH_NOTES

Generated by trekplan v5.9.1 on 2026-07-04 — plan_version: 1.7

Context

The ingest layer implements the file/CSV (I2/I3) and sql (I4/I5) source families on both stacks, built from the frozen shared/ingest-spec.md. That spec (§1, §4) declares type: "http" an OPTIONAL extension point: the schema already validates HttpSource (src/portfolio_optimiser/ingest.py:84-100), but materialize refuses it fail-fast (ingest.py:394-398, "gated extension point (I6) with no connector"). I6 activates that stub to demonstrate — honestly and end-to-end — HOW a third, network-transport source family plugs into the same connector / materialization / gate contracts, WITHOUT consuming the platform's "first live source" milestone. It realizes two locked framework principles at once: the 90%-principle (generic core + clear extension points, do not chase the last 10%) and the security frame's "local-only default, no silent egress" (§8) — a network source added strictly behind an explicit run-flag the manifest itself cannot grant. MAF-only; okf.py, shared/, and the frozen spec are untouched.

Architecture Diagram

graph TD
    subgraph "I6 changes in ingest.py"
        M["materialize(...,<br/>allow_network=False,<br/>http_get=None)"]:::mod
        GATE{"isinstance(HttpSource)<br/>AND not allow_network?"}:::new
        DISP["dispatch loop<br/>(file / sql / http branch)"]:::mod
        RH["read_http(base_url, query,<br/>*, max_rows, credential_ref, get)"]:::new
        UG["_urllib_get<br/>(stdlib GET — ONLY socket path)"]:::new
        FENCE["_render_fenced_block<br/>(verbatim, ``` fence)"]:::new
    end
    subgraph "reused UNCHANGED"
        RC["_render_concept_file<br/>(7-key frontmatter)"]:::keep
        WB["_write_bytes / ownership /<br/>collision / index"]:::keep
        OKF["okf.render_frontmatter /<br/>link_in_index"]:::keep
    end
    TEST["golden + load-bearing tests<br/>inject a canned http_get<br/>(no socket, no creds)"]:::test

    M --> GATE
    GATE -->|"no flag → IngestError"| REFUSE["fail-fast (spec §8)"]:::new
    GATE -->|"allow_network=True"| DISP
    DISP --> RH --> UG
    RH -->|"body text"| FENCE --> RC --> WB --> OKF
    TEST -.injects.-> RH

    classDef new fill:#2d6,stroke:#164,color:#000
    classDef mod fill:#fd6,stroke:#a70,color:#000
    classDef keep fill:#ddd,stroke:#888,color:#000
    classDef test fill:#8cf,stroke:#048,color:#000

Codebase Analysis

• Tech stack: Python ≥3.10, Pydantic (validation/IR), stdlib urllib (transport), pytest (+ byte-for-byte golden harness), ruff/mypy, uv. Zero new runtime deps.
• Key patterns: one connector fn per source type returning renderable content; a per-source-type dispatch inside materialize; credential/connection resolution by env-var name (never from the manifest); a committed golden case per type; load-bearing tests that go RED when a seam is detached; spec-silent decisions pinned in the module docstring + frozen in golden bytes.
• Relevant files (verified):
  • src/portfolio_optimiser/ingest.py — HttpSource schema (:84-100), the gate to replace (:394-398), the dispatch loop (:408-426), read_sql template (:233-278), _render_concept_file 7-key frontmatter (:304-318, source_system = manifest.source.id), _write_bytes (:321-327), ownership/collision/index machinery (:428-470), module docstring pin-block (:1-31).
  • tests/test_ingest_golden_sql.py — byte-for-byte golden harness (_bundle_bytes, file-set equality, §10 second-run idempotence).
  • tests/test_ingest_sql_loadbearing.py / tests/test_ingest_sql.py — detach-RED and connector-unit templates.
  • tests/test_ingest_materialize.py:245-253 — test_http_source_has_no_connector (stale after I6: still green under default-no-flag, but its "no connector" comment becomes false).
  • examples/ingest-golden-sql/ — golden layout to mirror (manifest.json, fixture/, ingested-at.txt, expected-bundle/{index.md, ingest-*.md}).
  • .claude/projects/2026-07-04-i4-ingest-sql-maf/build_fixture.py — throwaway fixture-builder pattern.
  • docs/extending.md — already exists (bilingual dev doc: Norwegian headers, English prose); I6 appends an http source-family section + the D7 hook pointer.
• Reusable code (leverage unchanged): _render_concept_file, _write_bytes, the ownership scan / collision gate / index generation (ingest.py:428-470), load_manifest, okf.render_frontmatter / okf.link_in_index. Explicitly NOT reused for http: render_table / _escape_cell / _sql_value_to_text (http body is verbatim inside a fenced code block, not a table).
• External tech (researched): none — 0 research topics (frozen spec fully defines http semantics; I4 SQL is the direct in-repo precedent; stdlib mock). The single external-adjacent item (create_sdk_mcp_server currency for the docs pointer) is a doc-time verification, not a planning input.
• Recent git activity: bc52c12 (I6 brief), 1b7612b/d7e5f2f (I4 SQL, the direct precedent), e4ee8bd→994f5be (I2 CSV). Branch main, clean.

Implementation Plan

Each step is test-first (Iron Law). Steps are ordered by dependency: connector → gate+dispatch → golden → docs.

Step 1: Implement read_http connector + injectable transport seam (docstring-pinned)

• Files: src/portfolio_optimiser/ingest.py, tests/test_ingest_http.py (new)
• Changes: Add, near read_sql (ingest.py:233):
  • A transport-seam type alias HttpGet = Callable[[str, str | None], str] ((url, credential) -> response body text); add from collections.abc import Callable and from urllib.request import Request, urlopen + from urllib.error import URLError to imports.
  • def _urllib_get(url: str, credential: str | None) -> str — the stdlib GET and the only socket path: build a Request(url, headers=..., method="GET"), attach Authorization: Bearer {credential} iff credential is not None, urlopen, read, decode UTF-8. Wrap URLError/OSError/UnicodeDecodeError as IngestError. Do NOT pre-add a # noqa: S310 — pyproject.toml selects no flake8-bandit (S) rules, so S310 never fires and a dead noqa would trip RUF100 if it is ever enabled; add the noqa ONLY if uv run ruff check . actually flags the urlopen at Verify.
  • def read_http(base_url, query, *, max_rows, credential_ref=None, get: HttpGet = _urllib_get) -> str mirroring read_sql: resolve credential_ref via os.environ.get (present-but-unset/empty → IngestError fail-fast, mirroring ingest.py:249; None → no auth); build url = base_url.rstrip("/") + "/" + query.lstrip("/"); call get(url, credential); normalize CRLF→LF; fence-collision fail-fast — if any normalized line, after stripping up to its leading spaces (line.lstrip(" ")), begins with ``` (a fence marker — note CommonMark permits a closing fence indented ≤3 spaces, so a leading-space check is required), raise IngestError (a body that cannot be safely embedded in a fenced code block is an ERROR, never silently-corrupted markdown — the module's no-silent-coercion discipline + §8 fail-fast); enforce the cap on the \n-only line count of the CRLF-normalized text — n = normalized.count("\n"); line_count = n if (normalized == "" or normalized.endswith("\n")) else n + 1; line_count > max_rows → IngestError (never truncation, §8). Do NOT use str.splitlines() for the cap: CPython splitlines() splits on the extended Unicode set (\v, \f, \x1c-\x1e, \x85, \u2028, \u2029), so a body with a vertical-tab or line-separator would be over-counted vs the LF-only rendered file (verified 2026-07-04). The \n-only count matches the rendered body and still treats a single trailing newline as no phantom line. Return normalized text. Pure: no writes, no logging (materialize owns §8 log).
  • def _render_fenced_block(body: str) -> str → f"\``\n{body.rstrip(chr(10))}\n```\n"` (verbatim inside a fenced code block; LF-only; exactly one trailing newline).
  • Docstring pin (module docstring, ingest.py:23-30 block): append the http (I6) pinned decisions, mirroring the I4 sql sentence: transport is an injectable get callable (default _urllib_get, the only socket); credential_ref names an env secret resolved at run time (Bearer), never from the manifest, never logged, never stamped in frontmatter; URL = base_url (one trailing slash stripped) + / + query (leading slash stripped); body = response decoded UTF-8, CRLF→LF, trailing newlines stripped, rendered verbatim in a fenced block (NOT table-escaped); `max_rows` caps response **line count** (`len(normalized.splitlines())`), fail-fast; a response line beginning with ` →IngestError` fail-fast (cannot be safely fenced — never silent corruption, consistent with the no-silent-coercion frame; the fixed 3-backtick fence is deliberate, NOT a silent fallback).
• Reuses: read_sql structure (ingest.py:233-278) — env-resolve → fetch → cap → IngestError wrap; the CSV/SQL streaming-cap idiom (ingest.py:193-197, :270-274).
• Test first:
  • File: tests/test_ingest_http.py (new)
  • Verifies (each injects a canned get, no socket, no creds unless monkeypatched): read_http returns the injected body verbatim and calls get with the joined URL; a body of exactly max_rows lines → returns normally (boundary pass); max_rows + 1 lines → IngestError (never truncated) [SC max_rows]; a body with a line beginning ``` → IngestError (fence-collision fail-fast); credential_ref present but env unset → IngestError; credential_ref set → the injected get receives the resolved secret (secret comes from env, never the manifest); read_http's get default is _urllib_get (inspect.signature — proves the stdlib default without a socket).
  • Pattern: tests/test_ingest_sql.py (connector-unit style; local get-builder mirroring _make_db locality — do NOT add to conftest.py).
• Verify: uv run pytest tests/test_ingest_http.py -q → expected: all pass (offline).
• On failure: revert — git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_http.py
• Checkpoint: git commit -m "feat(ingest): http read_http connector + injectable transport seam (I6)"
• Manifest:

  manifest:
    expected_paths:
      - src/portfolio_optimiser/ingest.py
      - tests/test_ingest_http.py
    min_file_count: 2
    commit_message_pattern: "^feat\\(ingest\\): http read_http connector \\+ injectable transport seam \\(I6\\)$"
    bash_syntax_check: []
    forbidden_paths:
      - src/portfolio_optimiser/okf.py
      - shared/ingest-spec.md
    must_contain:
      - path: src/portfolio_optimiser/ingest.py
        pattern: "def read_http"
      - path: src/portfolio_optimiser/ingest.py
        pattern: "_render_fenced_block"
      - path: tests/test_ingest_http.py
        pattern: "read_http"
  

Step 2: Gate http behind the allow_network run-flag + wire the dispatch branch

• Files: src/portfolio_optimiser/ingest.py, tests/test_ingest_http_loadbearing.py (new), tests/test_ingest_materialize.py
• Changes:
  • materialize signature (ingest.py:373-375): add keyword-only allow_network: bool = False and http_get: HttpGet | None = None (both defaulted → all existing file/sql callers stay backward-compatible; run_project/run_portfolio are NOT touched — verified they never call materialize). Extend the docstring to note the §8 network opt-in flag is a run argument.
  • Replace the unconditional refusal (ingest.py:393-398) with the gate: if isinstance(source, HttpSource) and not allow_network: raise IngestError(...) — refuse fail-fast before any source call, message citing §8 ("the manifest cannot grant itself network access"). This exact and not allow_network clause is the load-bearing seam.
  • Restructure the dispatch loop (ingest.py:408-426) so each branch sets body and a row_count: FileSource → read_csv + render_table, row_count = len(rows); elif isinstance(source, HttpSource) → get = http_get if http_get is not None else _urllib_get; text = read_http(source.base_url, extraction.query, max_rows=extraction.max_rows, credential_ref=source.credential_ref, get=get); body = _render_fenced_block(text); row_count = len(text.splitlines()); else (Sql) → read_sql + render_table, row_count = len(rows). Keep the single _LOGGER.info §8 log (ingest.py:416-421) logging rows=row_count (line count for http). The stale "http already refused above" comment (ingest.py:412) is removed.
  • tests/test_ingest_materialize.py:245-253: update test_http_source_has_no_connector's comment/intent — the default-no-flag path still raises IngestError, but the wording "no connector" is now false; reword to "http is refused without the per-run network flag" (assertion stays green; comment corrected). Do not delete — it is the default-refuse smoke.
• Reuses: _render_concept_file, _write_bytes, ownership/collision/index machinery (ingest.py:428-470) — all unchanged; the http bundle flows through the identical §3/§6 disk phase.
• Test first:
  • File: tests/test_ingest_http_loadbearing.py (new)
  • Verifies (detach-RED seams, each maps to a Success Criterion):
    1. Network gate [SC1] — asserts BOTH branches on the SAME manifest + SAME recording get (this is what makes it genuinely detach-RED):
       • Refuse branch: materialize(http_manifest, out, ingested_at=…, allow_network=False, http_get=recording_get) → assert IngestError and recording_get was never called (refuse before any source access, §8).
       • Allow branch: materialize(http_manifest, out, ingested_at=…, allow_network=True, http_get=recording_get) → assert it succeeds (writes ingest-*.md) and recording_get was called. Why both: the seam is if isinstance(source, HttpSource) and not allow_network: raise. Removing the whole clause → the refuse assertion fails (get called, no error) → RED; removing just and not allow_network (reverting to the unconditional refusal that exists today) → the allow assertion fails (raises when it should fetch) → RED. A refuse-only test would false-green on the second detach. (Inject a recording get that returns a valid payload — never the default _urllib_get, whose dead socket would false-green the refuse branch.)
    2. Fenced-not-table [§5 http seam]: payload with a raw | and \; assert the body is ``` -fenced and the pipe survives un-escaped ("\\|" not in body, raw | present). RED if the http path degrades to render_table.
    3. Navigability [§2/§11]: drive the http bundle through unchanged okf.navigate_bundle/bundle_context; reachable via index link, classified by okf_type, provenance rides through. RED when index linking detaches.
    4. Over-cap writes nothing [SC max_rows]: over-cap payload → IngestError and no ingest-*.md written (memory-staging). RED if silent truncation replaces the raise.
  • Pattern: tests/test_ingest_sql_loadbearing.py (detach discipline) + the ordering proof in tests/test_ingest_manifest.py:178.
• Verify: uv run pytest tests/test_ingest_http_loadbearing.py tests/test_ingest_materialize.py -q → expected: all pass. Detach-proof (manual, at review): the gate test goes RED on BOTH detaches — (a) delete the whole if isinstance(source, HttpSource) and not allow_network: block → refuse branch RED; (b) reduce it to the unconditional if isinstance(source, HttpSource): → allow branch RED.
• On failure: revert — git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_http_loadbearing.py tests/test_ingest_materialize.py
• Checkpoint: git commit -m "feat(ingest): gate http behind allow_network run-flag + dispatch (I6)"
• Manifest:

  manifest:
    expected_paths:
      - src/portfolio_optimiser/ingest.py
      - tests/test_ingest_http_loadbearing.py
      - tests/test_ingest_materialize.py
    min_file_count: 3
    commit_message_pattern: "^feat\\(ingest\\): gate http behind allow_network run-flag \\+ dispatch \\(I6\\)$"
    bash_syntax_check: []
    forbidden_paths:
      - src/portfolio_optimiser/okf.py
      - src/portfolio_optimiser/run.py
      - shared/ingest-spec.md
    must_contain:
      - path: src/portfolio_optimiser/ingest.py
        pattern: "allow_network"
      - path: src/portfolio_optimiser/ingest.py
        pattern: "and not allow_network"
      - path: tests/test_ingest_http_loadbearing.py
        pattern: "allow_network=False"
  

Step 3: Author the byte-deterministic golden case examples/ingest-golden-http/

• Files: examples/ingest-golden-http/manifest.json (new), examples/ingest-golden-http/fixture/status (new), examples/ingest-golden-http/fixture/report (new), examples/ingest-golden-http/ingested-at.txt (new), examples/ingest-golden-http/expected-bundle/ (new, generated + hand-reviewed), tests/test_ingest_golden_http.py (new), .claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py (new, throwaway)
• Changes:

  • manifest.json: source = {"type": "http", "id": "status-api", "base_url": "https://api.example.test/v1", "credential_ref": null} (.test reserved TLD + credential_ref: null → runs with no network, no credentials, proving SC offline); bundle_summary; two extractions — {"id": "status", "title": "Service status", "query": "status", "okf_type": "dataset", "max_rows": 10}, {"id": "report", "title": "Ops report", "query": "report", "okf_type": "reference", "max_rows": 10}.

  • fixture/status (1 line, discriminates fenced-verbatim vs table-escape — contains raw | and \): {"service": "billing", "state": "degraded | partial", "path": "c:\temp\cache"}

  • fixture/report (3 lines, ≤ max_rows): baseline ok / pipe | and backslash \ kept verbatim / end of report.

  • ingested-at.txt: 2026-07-04T12:00:00Z + trailing newline (byte-write, §5 format).

  • build_fixture.py: writes the two payload files + ingested-at.txt (mirrors the I4 builder; payloads are LF-only so the golden is platform-stable). Documents regenerability. Author the payload bytes as raw bytes / raw-string literals — in a normal Python string literal \t is a TAB and \c a literal backslash, so the intended verbatim backslashes in c:\temp\cache (and backslash \) would be silently corrupted; write with rb"..." / .write_bytes(b"...") or r"..." so the byte the fenced-not-escaped test depends on survives exactly.

  • expected-bundle/: generate by running materialize against the fixture with a canned get (map url.rsplit("/",1)[-1] → fixture/<name>), allow_network=True, then verify the generated bytes against the spec-derived expectation below BEFORE committing — this is an agent check against an independent oracle, not a human gate, so a headless claude -p run performs it too. The non-circular correctness oracle is carried here in the plan (and, independently, by the Step-2 load-bearing tests, which hand-assert the fenced-not-escaped body, the 7-key frontmatter, and navigability — so correctness is proven by hand-written assertions, not by comparing code output to code output). If the generated ingest-status.md diverges from the expectation below in any way OTHER than the ingest_manifest hash, escalate (do not commit a wrong oracle).

    Hand-derived expected-bundle/ingest-status.md (from §5/§7 + fixture/status; every field except the manifest-hash is independently derived):

    ---
    type: dataset
    title: Service status
    source_system: status-api
    source_query: status
    ingested_at: 2026-07-04T12:00:00Z
    ingest_manifest: manifest@<sha256(manifest.json)[:16]>
    generated: true
    ---
    
    ```‹fence›
    {"service": "billing", "state": "degraded | partial", "path": "c:\temp\cache"}
    ```‹fence›
    

    Note the pipe and backslashes are un-escaped (verbatim) inside the fence — the exact discriminator vs render_table. index.md = bundle_summary + - [Service status](ingest-status.md)

    • - [Ops report](ingest-report.md). The ingest_manifest hash is the only computed field (SHA-256 of the committed manifest.json bytes) — frozen once committed.
• Reuses: golden layout + harness from examples/ingest-golden-sql/ and tests/test_ingest_golden_sql.py; builder shape from the I4 build_fixture.py.
• Test first:
  • File: tests/test_ingest_golden_http.py (new)
  • Verifies [SC golden determinism]: build the canned get from fixture/; materialize(GOLDEN/manifest.json, out, ingested_at=<ingested-at.txt>, allow_network=True, http_get=fake); file-set equality, per-file byte compare against expected-bundle/, second-run idempotence (§10). No credential env set (credential_ref: null).
  • Pattern: tests/test_ingest_golden_sql.py:30-47 (swap the monkeypatch.setenv DSN for the injected get).
• Verify: uv run pytest tests/test_ingest_golden_http.py -q → expected: pass, byte-identical.
• On failure: revert — git checkout -- examples/ingest-golden-http tests/test_ingest_golden_http.py .claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py && rm -rf examples/ingest-golden-http
• Checkpoint: git commit -m "test(ingest): http golden extraction, byte-deterministic (I6)"
• Manifest:

  manifest:
    expected_paths:
      - examples/ingest-golden-http/manifest.json
      - examples/ingest-golden-http/ingested-at.txt
      - examples/ingest-golden-http/fixture/status
      - examples/ingest-golden-http/fixture/report
      - examples/ingest-golden-http/expected-bundle/index.md
      - examples/ingest-golden-http/expected-bundle/ingest-status.md
      - examples/ingest-golden-http/expected-bundle/ingest-report.md
      - tests/test_ingest_golden_http.py
      - .claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py
    min_file_count: 9
    commit_message_pattern: "^test\\(ingest\\): http golden extraction, byte-deterministic \\(I6\\)$"
    bash_syntax_check: []
    forbidden_paths:
      - src/portfolio_optimiser/okf.py
      - shared/ingest-spec.md
    must_contain:
      - path: examples/ingest-golden-http/expected-bundle/ingest-status.md
        pattern: "```"
      - path: examples/ingest-golden-http/expected-bundle/ingest-status.md
        pattern: "generated: true"
      - path: examples/ingest-golden-http/manifest.json
        pattern: "\"type\": \"http\""
  

Step 4: Document the http extension-point pattern + D7 SDK hook pointer in docs/extending.md

• Files: docs/extending.md
• Changes: Append a new section (bilingual house style: Norwegian header, English prose) documenting the http source family as the worked extension-point example: the manifest type: "http" contract (base_url no embedded credentials, credential_ref by env-name), the run-argument network opt-in (materialize(..., allow_network=True); the manifest cannot grant network — §8), the injectable transport seam (http_get, default stdlib urllib), and the "verbatim in a fenced code block" body. Add the D7 sibling-hook pointer: an MCP-based connector is "an extension of this family" (§4), and on the Claude Agent SDK side the in-process server hook is create_sdk_mcp_server(name, version="1.0.0", tools=...) -> McpSdkServerConfig (package claude-agent-sdk) — VERIFIED 2026-07-04 against the official Claude Agent SDK Python docs (code.claude.com/docs/en/agent-sdk/python + docs.claude.com agent-sdk/mcp; the name is confirmed, not assumed — pin it). Note MCP is documented as an extension of this family, NOT a new client wired into the run path (Non-Goal). Frame everything as "local mock / no live source"; do not claim a live integration.
• Reuses: the existing "Bevisst ikke bygget (90 %-kuttlista)" seam-naming style in docs/extending.md:55-74; the honesty framing in docs/extending.md:7-13.
• Test first: (docs — no unit test; command-checkable at Verify)
  • Verifies: grep -q create_sdk_mcp_server docs/extending.md exits 0, and a new http source-family section is present; the new I6 artifacts carry no affirmative live-source claim.
• Verify: grep -q create_sdk_mcp_server docs/extending.md && echo OK → expected: OK; grep -rin "live" docs/extending.md examples/ingest-golden-http/ src/portfolio_optimiser/ingest.py → expected: any hit is only "local mock" / "no live source" negated framing (scoped to the I6 artifacts — the repo-wide grep is not a clean gate: docs/, src/persona.py, simulation.py etc. carry dozens of unrelated pre-existing "live"/"delivery" substrings).
• On failure: revert — git checkout -- docs/extending.md
• Checkpoint: git commit -m "docs(i6): http source extension-point + D7 create_sdk_mcp_server pointer"
• Manifest:

  manifest:
    expected_paths:
      - docs/extending.md
    min_file_count: 1
    commit_message_pattern: "^docs\\(i6\\): http source extension-point \\+ D7 create_sdk_mcp_server pointer$"
    bash_syntax_check: []
    forbidden_paths:
      - src/portfolio_optimiser/okf.py
      - shared/ingest-spec.md
    must_contain:
      - path: docs/extending.md
        pattern: "create_sdk_mcp_server"
      - path: docs/extending.md
        pattern: "allow_network"
  

Alternatives Considered

Approach	Pros	Cons	Why rejected
Thread allow_network through run_project/run_portfolio (as the brief assumed)	Matches the brief's literal assumption	run.py never imports ingest/calls materialize (verified by 2 agents + grep) — the param would be dead code	Scope creep + speculative feature; violates surgical-change. Flag scoped to materialize only.
Reuse render_table for the http body	Maximal "mirror I4" symmetry	§5 requires the http body verbatim in a fenced code block, not a table; render_table escapes |/\\	Spec-nonconforming; the fenced-not-table load-bearing test would fail. New _render_fenced_block.
localhost http.server thread for tests	"Real" HTTP round-trip	Opens a socket → violates §11 "without network access"; flaky/slow; hermeticity risk	Injectable get callable (canned payloads, no socket) is the hermetic §11 reading. Documented fallback only.
Variable-length fence (count backticks in body)	Silently "handles" a body containing ```	Extra determinism surface; papers over an ambiguous body the model would later read	Rejected in favour of fail-fast: a body line beginning ``` → IngestError (no silent corruption; consistent with the module's no-silent-coercion frame). Fixed 3-backtick fence is deliberate, not a fallback.
urllib.parse.urljoin for the URL	Stdlib canonical	Surprising path semantics (absolute paths, ..) → non-deterministic joins	Explicit rstrip("/") + "/" + lstrip("/") — predictable, pinned.

Test Strategy

• Framework: pytest (uv run pytest); byte-for-byte golden harness; detach-RED load-bearing regime (spec §11).
• Existing patterns: three-file-per-source-type convention (test_ingest_http.py connector-unit, _loadbearing.py detach-RED, _golden_http.py byte-for-byte) — mirrors the sql trio. Canned-get builders local to the test files (never conftest.py, which is MAF/LLM-only).
• New tests in this plan: ~11 across 3 new files + 1 comment correction.

Tests to write

Type	File	Verifies	Model test
Unit	tests/test_ingest_http.py	verbatim fetch; URL join; max_rows; credential env-resolve (set/unset); urllib default	tests/test_ingest_sql.py
Load-bearing	tests/test_ingest_http_loadbearing.py	network gate (never-called + IngestError); fenced-not-table; navigability; over-cap-writes-nothing	tests/test_ingest_sql_loadbearing.py
Golden	tests/test_ingest_golden_http.py	byte-identical materialization; §10 idempotence; no creds	tests/test_ingest_golden_sql.py
Comment fix	tests/test_ingest_materialize.py:245	default-no-flag still refuses (intent reworded)	—

Offline property (SC): satisfied by construction — every http test injects http_get (no socket) and needs no credential. Acceptance: uv run pytest exits 0 with the network cable pulled. Convention to enforce in review: no ingest test calls materialize/read_http for an http source without passing http_get (the default _urllib_get is the only socket path and is never exercised).

Risks and Mitigations

Priority	Risk	Location	Impact	Mitigation
Critical	Silent egress if the gate sits after connector construction, or _urllib_get is reachable without the flag	ingest.py:393 gate / dispatch	Network hit with no operator opt-in — defeats the task	Gate in materialize BEFORE the staging loop; _urllib_get only reached inside the http branch, which the gate guards. Load-bearing test asserts get never called when flag absent.
High	Credential echo in the verbatim body	http body render	A reflecting endpoint could echo Authorization into a committed bundle file	Inherent to "verbatim body" — do NOT auto-redact (breaks determinism). Golden mock never echoes a secret; document as a residual deployer-owned risk (§1 boundary).
High	Credential leak via error/log text	read_http IngestError, _LOGGER.info	Secret in an exception or log line	Mirror read_sql: wrap transport errors WITHOUT the secret; keep §8 log to id/ingested_at/row_count only. (Optional sentinel-secret grep test — nice-to-have, not gating.)
High	Determinism: verbatim-vs-LF-only tension, trailing newline, fence collision	_render_fenced_block / read_http	Golden passes locally, diverges on other input/platform; or silently-broken markdown in the bundle	Pin (docstring + golden): UTF-8 decode fail-fast; CRLF→LF; strip trailing newlines outside the fence; max_rows = len(splitlines()); a body line beginning ``` → IngestError fail-fast (never a silently-corrupted fenced block — consistent with the no-silent-coercion frame).
Medium	max_rows line-count semantics (extended-Unicode / trailing / blank lines)	read_http	Spurious pass/fail vs golden	Count \n-only on the CRLF-normalized text (NOT str.splitlines(), which also splits on \v \f \x1c-\x1e \x85 \u2028 \u2029 — verified): n = normalized.count("\n"); line_count = n if normalized.endswith("\n") else n+1. Matches the LF-only body; a trailing newline adds no phantom line; internal blank lines count. Test exactly max_rows (pass) and max_rows+1 (raise).
Medium	base_url+query join edge cases	read_http	Non-deterministic joins; scheme/host escape	Pinned rstrip("/")+"/"+lstrip("/"); base_url userinfo already rejected (ingest.py:92-100); the joined URL is never stamped in frontmatter.
Low	Stale test_http_source_has_no_connector intent	tests/test_ingest_materialize.py:245	Comment says "no connector" (now false)	Reword in Step 2; keep the default-refuse assertion.
Low	Temptation to touch okf.py/shared/	okf.py, shared/	Breaks D7-portability invariant / commons pull-only	forbidden_paths on every code step; SC git diff --stat shows no change.
Low	ruff S310 (urlopen) / mypy Callable typing	ingest.py	Lint/type red	# noqa: S310 on the gated urlopen; Callable from collections.abc; confirm at Verify.

Assumptions

#	Assumption	Why unverifiable at plan time	Impact if wrong
1	The default auth scheme for a resolved credential_ref is Authorization: Bearer {secret}	The spec names credential-by-ref but not the wire scheme; endpoint-specific	A real deployer's endpoint may need a different scheme — acceptable for a demo extension point; documented, and the get seam is swappable. Golden uses credential_ref: null so this is not frozen in golden bytes.

(RESOLVED, not open — the brief's [ASSUMPTION] flag threads through materialize and run_project was verified FALSE: run.py never calls materialize. The flag is scoped to materialize only. source_system = manifest.source.id confirmed from _render_concept_file:312; max_rows = response line count pinned per §8. The D7 pointer create_sdk_mcp_server was verified 2026-07-04 against the official Claude Agent SDK Python docs — it is the confirmed in-process MCP-server primitive, not an assumption.)

Verification

Per-step manifests are checked by trekexecute. These are the brief's Success Criteria as end-to-end checks (run from repo root):

• uv run pytest -q → expected: exits 0 (full suite green, no network, no credentials set).
• Network gate detach-proof: comment out and not allow_network in ingest.py, run uv run pytest tests/test_ingest_http_loadbearing.py -k gate → expected: RED; restore → green.
• uv run pytest tests/test_ingest_golden_http.py -q → expected: pass; output byte-matches examples/ingest-golden-http/expected-bundle/.
• uv run pytest tests/test_ingest_http.py -k max_rows → expected: pass (over-cap → IngestError).
• grep -rin "live" docs/extending.md examples/ingest-golden-http/ src/portfolio_optimiser/ingest.py → expected: any hit is only "local mock" / "no live source" negated framing (scoped to the NEW I6 artifacts — a repo-wide grep "live" is not a pass/fail gate: dozens of unrelated pre-existing hits in docs/, persona.py, simulation.py, test_*_live.py).
• git diff --stat (vs pre-I6) → expected: NO change to src/portfolio_optimiser/okf.py or shared/.
• uv run ruff check . → 0; uv run ruff format --check . → 0; uv run mypy src → 0.
• grep -q create_sdk_mcp_server docs/extending.md → exits 0; new http section present.
• Null API-spend: no model calls, no real network (golden/gate/unit all use injected get).

Estimated Scope

• Files to modify: 3 (ingest.py, tests/test_ingest_materialize.py, docs/extending.md).
• Files to create: 11 (3 test files + 8 golden/fixture/builder: manifest.json, ingested-at.txt, fixture/status, fixture/report, 3 × expected-bundle/*.md, build_fixture.py).
• Complexity: medium (one genuinely new render path + a security gate; the rest mirrors I4).

Plan Quality Score

Dimension	Weight	Score	Notes
Structural integrity	0.15	95	4 steps, strict dependency order (connector→gate→golden→docs)
Step quality	0.20	92	test-first, exact file:line, explicit reuse; Step 2 touches 3 files (cohesive)
Coverage completeness	0.20	95	every Success Criterion → a step + a Verification check
Specification quality	0.15	90	pinned decisions concrete; 2 honest doc-time assumptions remain
Risk & pre-mortem	0.15	92	silent-egress, credential-leak, determinism all mitigated
Headless readiness	0.10	90	On-failure + Checkpoint + Manifest per step
Manifest quality	0.05	88	must_contain + forbidden_paths guard okf.py/shared/
Weighted total	1.00	92	Grade: A

Adversarial review:

• Plan critic: REVISE → addressed. 1 blocker (gate false-green), 5 major (live-grep unsatisfiable, first-live-source circular, create_sdk_mcp_server contradiction, fence-collision silent, golden circular/hand-review), 6 minor — all resolved in Revisions below.
• Scope guardian: ALIGNED. 0 creep, 0 Non-Goal violations, 0 dependency issues; independently confirmed the materialize-only flag scoping is a correct scope fix (not a gap). Its one minor (over-broad live-grep) coincides with plan-critic MAJOR-1 and is fixed.

Revisions

Added by adversarial review (Phase 9). All blocker + major findings addressed; the plan was revised once for the merged set.

#	Finding	Severity	Resolution
1	Gate load-bearing test would false-green: reverting and not allow_network to the unconditional refusal still passes a refuse-only test	blocker	Step 2 gate test now asserts BOTH branches on the same manifest + same recording get (refuse: flag off → IngestError + get never called; allow: flag on → succeeds + get called). Removing the whole gate → refuse branch RED; removing just the clause → allow branch RED. Verify wording fixed.
2	grep -rin "live" docs/ src/ examples/ unsatisfiable (dozens of pre-existing hits)	major	Scoped the "no live claim" check to the NEW I6 artifacts (docs/extending.md, examples/ingest-golden-http/, ingest.py) in both Step 4 and Verification.
3	git grep "first live source" circular (matches only plan/brief)	major	Removed the circular sub-check.
4	create_sdk_mcp_server pinned in 3 gates while flagged "may be stale"	major	Verified 2026-07-04 against official Claude Agent SDK Python docs — confirmed correct; pinned confidently, Assumption downgraded to a verified fact.
5	Fence-collision (body containing a ``` line) silently corrupts the bundle	major	Changed to fail-fast: a body line beginning ``` → IngestError (Step 1 connector + docstring + risk + alternatives + a new unit test).
6	Golden expected-bundle circular + "hand-review" is a non-headless human step	major	Step 3 now carries the hand-derived ingest-status.md expectation IN the plan (non-circular, headless-usable), names the Step-2 load-bearing tests as the independent correctness oracle, and adds an escalate-on-divergence clause.
7	max_rows counts lines pre-strip; blank-line nuance	minor	Pinned: count on CRLF-normalized text before the trailing-strip; blank lines count as content lines (deterministic). Risk row + Step 1 updated.
8	c:\temp\cache backslash/tab trap in build_fixture.py	minor	Pinned raw-bytes / raw-string authoring so the verbatim backslash survives.
9	# noqa: S310 likely dead (no bandit config)	minor	Do not pre-add; add only if ruff check flags the urlopen at Verify.
10	File-count off by one (10 vs 11 created)	minor	Corrected to 11 created / 3 modified.
11	Line-ref drift (:393 vs :394)	minor	Standardized the gate reference to :394-398.
12	Pass-at-max_rows boundary test only in risk table	minor	Added the boundary-pass + max_rows+1-raise pair to the Step 1 test enumeration.

Adversarial Pass 2 (gemini-bridge, v5.1.1 high-effort)

The high-effort plan phase runs an additional independent gemini-bridge review. Outcome: gemini-bridge is non-functional (verified 2026-07-04, not assumed). The deep-research query ran to status: completed, but every retrieval path failed: gemini_get_research_result → 'Interaction' object has no attribute 'outputs' (server-side bug, 3× retry); the followup re-emit → 404 model gemini-3-pro-preview no longer available (deprecated). This confirms the prior recorded outage — the operator's expectation that it was fixed did not hold. No findings were retrievable; the agent correctly refused to fabricate a second opinion.

Substitute triangulation (per the standing gemini-outage handling):

• An independent high-effort plan-critic confirmation pass traced the blocker fix against both detach cases and verified revisions #2–#6 — plan sound, no new blocker/major; folded in one minor (CommonMark ≤3-space indented closing fence → line.lstrip(" ").startswith("```")).
• The two decisions gemini flagged as spec-checkable were resolved by primary-source verification (no gemini needed): CommonMark §4.5 fenced-code semantics (closing fence ≥ opening length, ≤3-space indent) → the fence-collision check is adequate; CPython str.splitlines() splits on the extended Unicode set → the max_rows cap was switched to a \n-only count so it matches the LF-only rendered body (Step 1 + risk row updated).
• Decision 4 ("config cannot grant its own capability; capability is an explicit runtime grant") is the capability-security / POLA (Principle of Least Authority) pattern — the manifest is data that cannot escalate its own authority; only the allow_network runtime grant can.

Net: no external second opinion available, but every decision gemini would have triangulated is covered by an independent internal pass or an authoritative primary-source check.