Kjell Tore Guttormsen
2208cdf3e9
/trekplan for the http source family (MAF-only). Validator green (4 steps, plan_version 1.7). 5 exploration agents; scope-guardian ALIGNED; plan-critic REVISE -> all resolved (1 blocker: gate false-green, now asserts both branches so it goes RED on either detach; 5 major inc. materialize-only flag scoping verified vs run.py, fence-collision -> fail-fast, create_sdk_mcp_server verified vs official Agent SDK docs, live-grep scoped to I6 artifacts, golden oracle hand-derived in-plan). High-effort gemini-bridge pass: re-confirmed BROKEN (retrieval bug + deprecated model) -> substituted plan-critic + primary-source checks (CommonMark 4.5 fence, CPython str.splitlines -> \n-only max_rows count). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015U5UK6gVFJNvTEdDcyUb8W
565 lines
41 KiB
Markdown
565 lines
41 KiB
Markdown
# I6 — HTTP source connector as a demonstrated extension point (MAF-only)
|
||
|
||
> **Plan quality: A** (92/100) — APPROVE_WITH_NOTES
|
||
>
|
||
> Generated by trekplan v5.9.1 on 2026-07-04 — `plan_version: 1.7`
|
||
|
||
## Context
|
||
|
||
The ingest layer implements the `file`/CSV (I2/I3) and `sql` (I4/I5) source families on
|
||
both stacks, built from the frozen `shared/ingest-spec.md`. That spec (§1, §4) declares
|
||
`type: "http"` an OPTIONAL extension point: the schema already validates `HttpSource`
|
||
(`src/portfolio_optimiser/ingest.py:84-100`), but `materialize` refuses it fail-fast
|
||
(`ingest.py:394-398`, "gated extension point (I6) with no connector"). I6 activates that stub
|
||
to demonstrate — honestly and end-to-end — HOW a third, network-transport source family plugs
|
||
into the same connector / materialization / gate contracts, WITHOUT consuming the platform's
|
||
"first live source" milestone. It realizes two locked framework principles at once: the
|
||
90%-principle (generic core + clear extension points, do not chase the last 10%) and the
|
||
security frame's "local-only default, no silent egress" (§8) — a network source added strictly
|
||
behind an explicit run-flag the manifest itself cannot grant. MAF-only; `okf.py`, `shared/`,
|
||
and the frozen spec are untouched.
|
||
|
||
## Architecture Diagram
|
||
|
||
```mermaid
|
||
graph TD
|
||
subgraph "I6 changes in ingest.py"
|
||
M["materialize(...,<br/>allow_network=False,<br/>http_get=None)"]:::mod
|
||
GATE{"isinstance(HttpSource)<br/>AND not allow_network?"}:::new
|
||
DISP["dispatch loop<br/>(file / sql / http branch)"]:::mod
|
||
RH["read_http(base_url, query,<br/>*, max_rows, credential_ref, get)"]:::new
|
||
UG["_urllib_get<br/>(stdlib GET — ONLY socket path)"]:::new
|
||
FENCE["_render_fenced_block<br/>(verbatim, ``` fence)"]:::new
|
||
end
|
||
subgraph "reused UNCHANGED"
|
||
RC["_render_concept_file<br/>(7-key frontmatter)"]:::keep
|
||
WB["_write_bytes / ownership /<br/>collision / index"]:::keep
|
||
OKF["okf.render_frontmatter /<br/>link_in_index"]:::keep
|
||
end
|
||
TEST["golden + load-bearing tests<br/>inject a canned http_get<br/>(no socket, no creds)"]:::test
|
||
|
||
M --> GATE
|
||
GATE -->|"no flag → IngestError"| REFUSE["fail-fast (spec §8)"]:::new
|
||
GATE -->|"allow_network=True"| DISP
|
||
DISP --> RH --> UG
|
||
RH -->|"body text"| FENCE --> RC --> WB --> OKF
|
||
TEST -.injects.-> RH
|
||
|
||
classDef new fill:#2d6,stroke:#164,color:#000
|
||
classDef mod fill:#fd6,stroke:#a70,color:#000
|
||
classDef keep fill:#ddd,stroke:#888,color:#000
|
||
classDef test fill:#8cf,stroke:#048,color:#000
|
||
```
|
||
|
||
## Codebase Analysis
|
||
|
||
- **Tech stack:** Python ≥3.10, Pydantic (validation/IR), stdlib `urllib` (transport),
|
||
`pytest` (+ byte-for-byte golden harness), `ruff`/`mypy`, `uv`. Zero new runtime deps.
|
||
- **Key patterns:** one connector fn per source type returning renderable content; a
|
||
per-source-type dispatch inside `materialize`; credential/connection resolution by
|
||
**env-var name** (never from the manifest); a committed golden case per type; load-bearing
|
||
tests that go RED when a seam is detached; spec-silent decisions pinned in the module
|
||
docstring + frozen in golden bytes.
|
||
- **Relevant files (verified):**
|
||
- `src/portfolio_optimiser/ingest.py` — `HttpSource` schema (`:84-100`), the gate to
|
||
replace (`:394-398`), the dispatch loop (`:408-426`), `read_sql` template (`:233-278`),
|
||
`_render_concept_file` 7-key frontmatter (`:304-318`, `source_system = manifest.source.id`),
|
||
`_write_bytes` (`:321-327`), ownership/collision/index machinery (`:428-470`), module
|
||
docstring pin-block (`:1-31`).
|
||
- `tests/test_ingest_golden_sql.py` — byte-for-byte golden harness (`_bundle_bytes`, file-set
|
||
equality, §10 second-run idempotence).
|
||
- `tests/test_ingest_sql_loadbearing.py` / `tests/test_ingest_sql.py` — detach-RED and
|
||
connector-unit templates.
|
||
- `tests/test_ingest_materialize.py:245-253` — `test_http_source_has_no_connector` (stale
|
||
after I6: still green under default-no-flag, but its "no connector" comment becomes false).
|
||
- `examples/ingest-golden-sql/` — golden layout to mirror (`manifest.json`, `fixture/`,
|
||
`ingested-at.txt`, `expected-bundle/{index.md, ingest-*.md}`).
|
||
- `.claude/projects/2026-07-04-i4-ingest-sql-maf/build_fixture.py` — throwaway fixture-builder
|
||
pattern.
|
||
- `docs/extending.md` — **already exists** (bilingual dev doc: Norwegian headers, English
|
||
prose); I6 **appends** an http source-family section + the D7 hook pointer.
|
||
- **Reusable code (leverage unchanged):** `_render_concept_file`, `_write_bytes`, the ownership
|
||
scan / collision gate / index generation (`ingest.py:428-470`), `load_manifest`,
|
||
`okf.render_frontmatter` / `okf.link_in_index`. **Explicitly NOT reused for http:**
|
||
`render_table` / `_escape_cell` / `_sql_value_to_text` (http body is verbatim inside a
|
||
fenced code block, not a table).
|
||
- **External tech (researched):** none — 0 research topics (frozen spec fully defines http
|
||
semantics; I4 SQL is the direct in-repo precedent; stdlib mock). The single external-adjacent
|
||
item (`create_sdk_mcp_server` currency for the docs pointer) is a doc-time verification, not a
|
||
planning input.
|
||
- **Recent git activity:** `bc52c12` (I6 brief), `1b7612b`/`d7e5f2f` (I4 SQL, the direct
|
||
precedent), `e4ee8bd`→`994f5be` (I2 CSV). Branch `main`, clean.
|
||
|
||
## Implementation Plan
|
||
|
||
Each step is test-first (Iron Law). Steps are ordered by dependency: connector → gate+dispatch →
|
||
golden → docs.
|
||
|
||
### Step 1: Implement `read_http` connector + injectable transport seam (docstring-pinned)
|
||
|
||
- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_http.py` *(new)*
|
||
- **Changes:** Add, near `read_sql` (`ingest.py:233`):
|
||
- A transport-seam type alias `HttpGet = Callable[[str, str | None], str]` (`(url, credential) -> response body text`); add `from collections.abc import Callable` and
|
||
`from urllib.request import Request, urlopen` + `from urllib.error import URLError` to imports.
|
||
- `def _urllib_get(url: str, credential: str | None) -> str` — the stdlib GET and the **only**
|
||
socket path: build a `Request(url, headers=..., method="GET")`, attach
|
||
`Authorization: Bearer {credential}` iff `credential is not None`, `urlopen`, read, decode
|
||
UTF-8. Wrap `URLError`/`OSError`/`UnicodeDecodeError` as `IngestError`. Do NOT pre-add a
|
||
`# noqa: S310` — `pyproject.toml` selects no `flake8-bandit` (`S`) rules, so S310 never
|
||
fires and a dead noqa would trip `RUF100` if it is ever enabled; add the noqa ONLY if
|
||
`uv run ruff check .` actually flags the `urlopen` at Verify.
|
||
- `def read_http(base_url, query, *, max_rows, credential_ref=None, get: HttpGet = _urllib_get) -> str`
|
||
mirroring `read_sql`: resolve `credential_ref` via `os.environ.get` (present-but-unset/empty →
|
||
`IngestError` fail-fast, mirroring `ingest.py:249`; `None` → no auth); build
|
||
`url = base_url.rstrip("/") + "/" + query.lstrip("/")`; call `get(url, credential)`;
|
||
normalize CRLF→LF; **fence-collision fail-fast** — if any normalized line, after stripping
|
||
up to its leading spaces (`line.lstrip(" ")`), begins with ` ``` ` (a fence marker — note
|
||
CommonMark permits a closing fence indented ≤3 spaces, so a leading-space check is required),
|
||
raise `IngestError` (a body that cannot be safely embedded in a fenced code block is an
|
||
ERROR, never silently-corrupted markdown — the module's no-silent-coercion discipline + §8
|
||
fail-fast); enforce the cap on the **`\n`-only line count** of the CRLF-normalized text —
|
||
`n = normalized.count("\n"); line_count = n if (normalized == "" or normalized.endswith("\n")) else n + 1`;
|
||
`line_count > max_rows → IngestError` (never truncation, §8). **Do NOT use `str.splitlines()`**
|
||
for the cap: CPython `splitlines()` splits on the extended Unicode set
|
||
(`\v`, `\f`, `\x1c`-`\x1e`, `\x85`, `\u2028`, `\u2029`), so a body with a vertical-tab or line-separator would be over-counted vs the
|
||
LF-only rendered file (verified 2026-07-04). The `\n`-only count matches the rendered body and
|
||
still treats a single trailing newline as no phantom line. Return normalized text. Pure: no
|
||
writes, no logging (materialize owns §8 log).
|
||
- `def _render_fenced_block(body: str) -> str` → `f"\`\`\`\n{body.rstrip(chr(10))}\n\`\`\`\n"`
|
||
(verbatim inside a fenced code block; LF-only; exactly one trailing newline).
|
||
- **Docstring pin (module docstring, `ingest.py:23-30` block):** append the http (I6) pinned
|
||
decisions, mirroring the I4 sql sentence: transport is an injectable `get` callable
|
||
(default `_urllib_get`, the only socket); `credential_ref` names an env secret resolved at
|
||
run time (Bearer), never from the manifest, never logged, never stamped in frontmatter;
|
||
URL = `base_url` (one trailing slash stripped) + `/` + `query` (leading slash stripped);
|
||
body = response decoded UTF-8, CRLF→LF, trailing newlines stripped, rendered verbatim in a
|
||
``` fenced block (NOT table-escaped); `max_rows` caps response **line count**
|
||
(`len(normalized.splitlines())`), fail-fast; a response line beginning with ` ``` ` →
|
||
`IngestError` fail-fast (cannot be safely fenced — never silent corruption, consistent with
|
||
the no-silent-coercion frame; the fixed 3-backtick fence is deliberate, NOT a silent
|
||
fallback).
|
||
- **Reuses:** `read_sql` structure (`ingest.py:233-278`) — env-resolve → fetch → cap →
|
||
`IngestError` wrap; the CSV/SQL streaming-cap idiom (`ingest.py:193-197`, `:270-274`).
|
||
- **Test first:**
|
||
- File: `tests/test_ingest_http.py` (new)
|
||
- Verifies (each injects a canned `get`, no socket, no creds unless monkeypatched):
|
||
`read_http` returns the injected body verbatim and calls `get` with the joined URL;
|
||
a body of exactly `max_rows` lines → returns normally (boundary pass);
|
||
`max_rows + 1` lines → `IngestError` (never truncated) **[SC max_rows]**;
|
||
a body with a line beginning ` ``` ` → `IngestError` (fence-collision fail-fast);
|
||
`credential_ref` present but env unset → `IngestError`; `credential_ref` set → the injected
|
||
`get` receives the resolved secret (secret comes from env, never the manifest);
|
||
`read_http`'s `get` default is `_urllib_get` (`inspect.signature` — proves the stdlib
|
||
default without a socket).
|
||
- Pattern: `tests/test_ingest_sql.py` (connector-unit style; local `get`-builder mirroring
|
||
`_make_db` locality — do NOT add to `conftest.py`).
|
||
- **Verify:** `uv run pytest tests/test_ingest_http.py -q` → expected: all pass (offline).
|
||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_http.py`
|
||
- **Checkpoint:** `git commit -m "feat(ingest): http read_http connector + injectable transport seam (I6)"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/ingest.py
|
||
- tests/test_ingest_http.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(ingest\\): http read_http connector \\+ injectable transport seam \\(I6\\)$"
|
||
bash_syntax_check: []
|
||
forbidden_paths:
|
||
- src/portfolio_optimiser/okf.py
|
||
- shared/ingest-spec.md
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/ingest.py
|
||
pattern: "def read_http"
|
||
- path: src/portfolio_optimiser/ingest.py
|
||
pattern: "_render_fenced_block"
|
||
- path: tests/test_ingest_http.py
|
||
pattern: "read_http"
|
||
```
|
||
|
||
### Step 2: Gate http behind the `allow_network` run-flag + wire the dispatch branch
|
||
|
||
- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_http_loadbearing.py`
|
||
*(new)*, `tests/test_ingest_materialize.py`
|
||
- **Changes:**
|
||
- `materialize` signature (`ingest.py:373-375`): add keyword-only `allow_network: bool = False`
|
||
and `http_get: HttpGet | None = None` (both defaulted → all existing file/sql callers stay
|
||
backward-compatible; `run_project`/`run_portfolio` are NOT touched — verified they never call
|
||
`materialize`). Extend the docstring to note the §8 network opt-in flag is a run argument.
|
||
- Replace the unconditional refusal (`ingest.py:393-398`) with the gate:
|
||
`if isinstance(source, HttpSource) and not allow_network: raise IngestError(...)` — refuse
|
||
fail-fast **before any source call**, message citing §8 ("the manifest cannot grant itself
|
||
network access"). This exact `and not allow_network` clause is the load-bearing seam.
|
||
- Restructure the dispatch loop (`ingest.py:408-426`) so each branch sets `body` and a
|
||
`row_count`: `FileSource` → `read_csv` + `render_table`, `row_count = len(rows)`;
|
||
`elif isinstance(source, HttpSource)` → `get = http_get if http_get is not None else _urllib_get`;
|
||
`text = read_http(source.base_url, extraction.query, max_rows=extraction.max_rows, credential_ref=source.credential_ref, get=get)`;
|
||
`body = _render_fenced_block(text)`; `row_count = len(text.splitlines())`; `else` (Sql) →
|
||
`read_sql` + `render_table`, `row_count = len(rows)`. Keep the single `_LOGGER.info` §8 log
|
||
(`ingest.py:416-421`) logging `rows=row_count` (line count for http). The stale
|
||
"http already refused above" comment (`ingest.py:412`) is removed.
|
||
- `tests/test_ingest_materialize.py:245-253`: update `test_http_source_has_no_connector`'s
|
||
comment/intent — the default-no-flag path still raises `IngestError`, but the wording
|
||
"no connector" is now false; reword to "http is refused without the per-run network flag"
|
||
(assertion stays green; comment corrected). Do not delete — it is the default-refuse smoke.
|
||
- **Reuses:** `_render_concept_file`, `_write_bytes`, ownership/collision/index machinery
|
||
(`ingest.py:428-470`) — all unchanged; the http bundle flows through the identical §3/§6 disk
|
||
phase.
|
||
- **Test first:**
|
||
- File: `tests/test_ingest_http_loadbearing.py` (new)
|
||
- Verifies (detach-RED seams, each maps to a Success Criterion):
|
||
1. **Network gate [SC1] — asserts BOTH branches on the SAME manifest + SAME recording `get`
|
||
(this is what makes it genuinely detach-RED):**
|
||
- *Refuse branch:* `materialize(http_manifest, out, ingested_at=…, allow_network=False, http_get=recording_get)`
|
||
→ assert `IngestError` **and** `recording_get` was **never called** (refuse before any
|
||
source access, §8).
|
||
- *Allow branch:* `materialize(http_manifest, out, ingested_at=…, allow_network=True, http_get=recording_get)`
|
||
→ assert it **succeeds** (writes `ingest-*.md`) **and** `recording_get` **was called**.
|
||
Why both: the seam is `if isinstance(source, HttpSource) and not allow_network: raise`.
|
||
Removing the whole clause → the *refuse* assertion fails (get called, no error) → RED;
|
||
removing just `and not allow_network` (reverting to the *unconditional* refusal that
|
||
exists today) → the *allow* assertion fails (raises when it should fetch) → RED. A
|
||
refuse-only test would false-green on the second detach. *(Inject a recording `get` that
|
||
returns a valid payload — never the default `_urllib_get`, whose dead socket would
|
||
false-green the refuse branch.)*
|
||
2. **Fenced-not-table [§5 http seam]:** payload with a raw `|` and `\`; assert the body is
|
||
``` -fenced and the pipe survives **un-escaped** (`"\\|" not in body`, raw `|` present).
|
||
RED if the http path degrades to `render_table`.
|
||
3. **Navigability [§2/§11]:** drive the http bundle through unchanged
|
||
`okf.navigate_bundle`/`bundle_context`; reachable via index link, classified by
|
||
`okf_type`, provenance rides through. RED when index linking detaches.
|
||
4. **Over-cap writes nothing [SC max_rows]:** over-cap payload → `IngestError` **and** no
|
||
`ingest-*.md` written (memory-staging). RED if silent truncation replaces the raise.
|
||
- Pattern: `tests/test_ingest_sql_loadbearing.py` (detach discipline) + the ordering proof in
|
||
`tests/test_ingest_manifest.py:178`.
|
||
- **Verify:** `uv run pytest tests/test_ingest_http_loadbearing.py tests/test_ingest_materialize.py -q`
|
||
→ expected: all pass. Detach-proof (manual, at review): the gate test goes RED on BOTH
|
||
detaches — (a) delete the whole `if isinstance(source, HttpSource) and not allow_network:`
|
||
block → refuse branch RED; (b) reduce it to the unconditional `if isinstance(source, HttpSource):`
|
||
→ allow branch RED.
|
||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_http_loadbearing.py tests/test_ingest_materialize.py`
|
||
- **Checkpoint:** `git commit -m "feat(ingest): gate http behind allow_network run-flag + dispatch (I6)"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/ingest.py
|
||
- tests/test_ingest_http_loadbearing.py
|
||
- tests/test_ingest_materialize.py
|
||
min_file_count: 3
|
||
commit_message_pattern: "^feat\\(ingest\\): gate http behind allow_network run-flag \\+ dispatch \\(I6\\)$"
|
||
bash_syntax_check: []
|
||
forbidden_paths:
|
||
- src/portfolio_optimiser/okf.py
|
||
- src/portfolio_optimiser/run.py
|
||
- shared/ingest-spec.md
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/ingest.py
|
||
pattern: "allow_network"
|
||
- path: src/portfolio_optimiser/ingest.py
|
||
pattern: "and not allow_network"
|
||
- path: tests/test_ingest_http_loadbearing.py
|
||
pattern: "allow_network=False"
|
||
```
|
||
|
||
### Step 3: Author the byte-deterministic golden case `examples/ingest-golden-http/`
|
||
|
||
- **Files:** `examples/ingest-golden-http/manifest.json` *(new)*,
|
||
`examples/ingest-golden-http/fixture/status` *(new)*,
|
||
`examples/ingest-golden-http/fixture/report` *(new)*,
|
||
`examples/ingest-golden-http/ingested-at.txt` *(new)*,
|
||
`examples/ingest-golden-http/expected-bundle/` *(new, generated + hand-reviewed)*,
|
||
`tests/test_ingest_golden_http.py` *(new)*,
|
||
`.claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py` *(new, throwaway)*
|
||
- **Changes:**
|
||
- `manifest.json`: `source` = `{"type": "http", "id": "status-api", "base_url": "https://api.example.test/v1", "credential_ref": null}`
|
||
(`.test` reserved TLD + `credential_ref: null` → runs with **no network, no credentials**,
|
||
proving SC offline); `bundle_summary`; two extractions —
|
||
`{"id": "status", "title": "Service status", "query": "status", "okf_type": "dataset", "max_rows": 10}`,
|
||
`{"id": "report", "title": "Ops report", "query": "report", "okf_type": "reference", "max_rows": 10}`.
|
||
- `fixture/status` (1 line, discriminates fenced-verbatim vs table-escape — contains raw `|`
|
||
and `\`): `{"service": "billing", "state": "degraded | partial", "path": "c:\temp\cache"}`
|
||
- `fixture/report` (3 lines, ≤ max_rows): `baseline ok` / `pipe | and backslash \ kept verbatim` /
|
||
`end of report`.
|
||
- `ingested-at.txt`: `2026-07-04T12:00:00Z` + trailing newline (byte-write, §5 format).
|
||
- `build_fixture.py`: writes the two payload files + `ingested-at.txt` (mirrors the I4 builder;
|
||
payloads are LF-only so the golden is platform-stable). Documents regenerability. **Author
|
||
the payload bytes as raw bytes / raw-string literals** — in a normal Python string literal
|
||
`\t` is a TAB and `\c` a literal backslash, so the intended verbatim backslashes in
|
||
`c:\temp\cache` (and `backslash \`) would be silently corrupted; write with `rb"..."` /
|
||
`.write_bytes(b"...")` or `r"..."` so the byte the fenced-not-escaped test depends on
|
||
survives exactly.
|
||
- `expected-bundle/`: **generate** by running `materialize` against the fixture with a canned
|
||
`get` (map `url.rsplit("/",1)[-1]` → `fixture/<name>`), `allow_network=True`, then **verify
|
||
the generated bytes against the spec-derived expectation below BEFORE committing** — this is
|
||
an agent check against an independent oracle, not a human gate, so a headless `claude -p`
|
||
run performs it too. The non-circular correctness oracle is carried here in the plan (and,
|
||
independently, by the Step-2 load-bearing tests, which hand-assert the fenced-not-escaped
|
||
body, the 7-key frontmatter, and navigability — so correctness is proven by hand-written
|
||
assertions, not by comparing code output to code output). If the generated `ingest-status.md`
|
||
diverges from the expectation below in any way OTHER than the `ingest_manifest` hash,
|
||
**escalate** (do not commit a wrong oracle).
|
||
|
||
Hand-derived `expected-bundle/ingest-status.md` (from §5/§7 + `fixture/status`; every field
|
||
except the manifest-hash is independently derived):
|
||
```
|
||
---
|
||
type: dataset
|
||
title: Service status
|
||
source_system: status-api
|
||
source_query: status
|
||
ingested_at: 2026-07-04T12:00:00Z
|
||
ingest_manifest: manifest@<sha256(manifest.json)[:16]>
|
||
generated: true
|
||
---
|
||
|
||
```‹fence›
|
||
{"service": "billing", "state": "degraded | partial", "path": "c:\temp\cache"}
|
||
```‹fence›
|
||
```
|
||
Note the pipe and backslashes are **un-escaped** (verbatim) inside the fence — the exact
|
||
discriminator vs `render_table`. `index.md` = `bundle_summary` + `- [Service status](ingest-status.md)`
|
||
+ `- [Ops report](ingest-report.md)`. The `ingest_manifest` hash is the only computed field
|
||
(SHA-256 of the committed `manifest.json` bytes) — frozen once committed.
|
||
- **Reuses:** golden layout + harness from `examples/ingest-golden-sql/` and
|
||
`tests/test_ingest_golden_sql.py`; builder shape from the I4 `build_fixture.py`.
|
||
- **Test first:**
|
||
- File: `tests/test_ingest_golden_http.py` (new)
|
||
- Verifies **[SC golden determinism]**: build the canned `get` from `fixture/`; `materialize(GOLDEN/manifest.json, out, ingested_at=<ingested-at.txt>, allow_network=True, http_get=fake)`;
|
||
file-set equality, per-file byte compare against `expected-bundle/`, second-run idempotence
|
||
(§10). No credential env set (`credential_ref: null`).
|
||
- Pattern: `tests/test_ingest_golden_sql.py:30-47` (swap the `monkeypatch.setenv` DSN for the
|
||
injected `get`).
|
||
- **Verify:** `uv run pytest tests/test_ingest_golden_http.py -q` → expected: pass, byte-identical.
|
||
- **On failure:** revert — `git checkout -- examples/ingest-golden-http tests/test_ingest_golden_http.py .claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py && rm -rf examples/ingest-golden-http`
|
||
- **Checkpoint:** `git commit -m "test(ingest): http golden extraction, byte-deterministic (I6)"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- examples/ingest-golden-http/manifest.json
|
||
- examples/ingest-golden-http/ingested-at.txt
|
||
- examples/ingest-golden-http/fixture/status
|
||
- examples/ingest-golden-http/fixture/report
|
||
- examples/ingest-golden-http/expected-bundle/index.md
|
||
- examples/ingest-golden-http/expected-bundle/ingest-status.md
|
||
- examples/ingest-golden-http/expected-bundle/ingest-report.md
|
||
- tests/test_ingest_golden_http.py
|
||
- .claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py
|
||
min_file_count: 9
|
||
commit_message_pattern: "^test\\(ingest\\): http golden extraction, byte-deterministic \\(I6\\)$"
|
||
bash_syntax_check: []
|
||
forbidden_paths:
|
||
- src/portfolio_optimiser/okf.py
|
||
- shared/ingest-spec.md
|
||
must_contain:
|
||
- path: examples/ingest-golden-http/expected-bundle/ingest-status.md
|
||
pattern: "```"
|
||
- path: examples/ingest-golden-http/expected-bundle/ingest-status.md
|
||
pattern: "generated: true"
|
||
- path: examples/ingest-golden-http/manifest.json
|
||
pattern: "\"type\": \"http\""
|
||
```
|
||
|
||
### Step 4: Document the http extension-point pattern + D7 SDK hook pointer in `docs/extending.md`
|
||
|
||
- **Files:** `docs/extending.md`
|
||
- **Changes:** Append a new section (bilingual house style: Norwegian header, English prose)
|
||
documenting the http source family as the worked extension-point example: the manifest
|
||
`type: "http"` contract (`base_url` no embedded credentials, `credential_ref` by env-name),
|
||
the **run-argument** network opt-in (`materialize(..., allow_network=True)`; the manifest
|
||
cannot grant network — §8), the injectable transport seam (`http_get`, default stdlib
|
||
`urllib`), and the "verbatim in a fenced code block" body. Add the D7 sibling-hook pointer:
|
||
an MCP-based connector is "an extension of this family" (§4), and on the Claude Agent SDK side
|
||
the in-process server hook is **`create_sdk_mcp_server(name, version="1.0.0", tools=...) ->
|
||
McpSdkServerConfig`** (package `claude-agent-sdk`) — **VERIFIED 2026-07-04 against the official
|
||
Claude Agent SDK Python docs** (code.claude.com/docs/en/agent-sdk/python + docs.claude.com
|
||
agent-sdk/mcp; the name is confirmed, not assumed — pin it). Note MCP is documented as an
|
||
extension of this family, NOT a new client wired into the run path (Non-Goal). Frame
|
||
everything as "local mock / no live source"; do not claim a live integration.
|
||
- **Reuses:** the existing "Bevisst ikke bygget (90 %-kuttlista)" seam-naming style in
|
||
`docs/extending.md:55-74`; the honesty framing in `docs/extending.md:7-13`.
|
||
- **Test first:** *(docs — no unit test; command-checkable at Verify)*
|
||
- Verifies: `grep -q create_sdk_mcp_server docs/extending.md` exits 0, and a new http
|
||
source-family section is present; the new I6 artifacts carry no affirmative live-source claim.
|
||
- **Verify:** `grep -q create_sdk_mcp_server docs/extending.md && echo OK` → expected: `OK`;
|
||
`grep -rin "live" docs/extending.md examples/ingest-golden-http/ src/portfolio_optimiser/ingest.py`
|
||
→ expected: any hit is only "local mock" / "no live source" negated framing (scoped to the
|
||
I6 artifacts — the repo-wide grep is not a clean gate: `docs/`, `src/persona.py`,
|
||
`simulation.py` etc. carry dozens of unrelated pre-existing "live"/"delivery" substrings).
|
||
- **On failure:** revert — `git checkout -- docs/extending.md`
|
||
- **Checkpoint:** `git commit -m "docs(i6): http source extension-point + D7 create_sdk_mcp_server pointer"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- docs/extending.md
|
||
min_file_count: 1
|
||
commit_message_pattern: "^docs\\(i6\\): http source extension-point \\+ D7 create_sdk_mcp_server pointer$"
|
||
bash_syntax_check: []
|
||
forbidden_paths:
|
||
- src/portfolio_optimiser/okf.py
|
||
- shared/ingest-spec.md
|
||
must_contain:
|
||
- path: docs/extending.md
|
||
pattern: "create_sdk_mcp_server"
|
||
- path: docs/extending.md
|
||
pattern: "allow_network"
|
||
```
|
||
|
||
## Alternatives Considered
|
||
|
||
| Approach | Pros | Cons | Why rejected |
|
||
|----------|------|------|--------------|
|
||
| Thread `allow_network` through `run_project`/`run_portfolio` (as the brief assumed) | Matches the brief's literal assumption | `run.py` never imports `ingest`/calls `materialize` (verified by 2 agents + grep) — the param would be dead code | Scope creep + speculative feature; violates surgical-change. Flag scoped to `materialize` only. |
|
||
| Reuse `render_table` for the http body | Maximal "mirror I4" symmetry | §5 requires the http body **verbatim in a fenced code block**, not a table; `render_table` escapes `\|`/`\\` | Spec-nonconforming; the fenced-not-table load-bearing test would fail. New `_render_fenced_block`. |
|
||
| `localhost http.server` thread for tests | "Real" HTTP round-trip | Opens a socket → violates §11 "without network access"; flaky/slow; hermeticity risk | Injectable `get` callable (canned payloads, no socket) is the hermetic §11 reading. Documented fallback only. |
|
||
| Variable-length fence (count backticks in body) | Silently "handles" a body containing ``` | Extra determinism surface; papers over an ambiguous body the model would later read | Rejected in favour of **fail-fast**: a body line beginning ` ``` ` → `IngestError` (no silent corruption; consistent with the module's no-silent-coercion frame). Fixed 3-backtick fence is deliberate, not a fallback. |
|
||
| `urllib.parse.urljoin` for the URL | Stdlib canonical | Surprising path semantics (absolute paths, `..`) → non-deterministic joins | Explicit `rstrip("/") + "/" + lstrip("/")` — predictable, pinned. |
|
||
|
||
## Test Strategy
|
||
|
||
- **Framework:** `pytest` (`uv run pytest`); byte-for-byte golden harness; detach-RED
|
||
load-bearing regime (spec §11).
|
||
- **Existing patterns:** three-file-per-source-type convention
|
||
(`test_ingest_http.py` connector-unit, `_loadbearing.py` detach-RED, `_golden_http.py`
|
||
byte-for-byte) — mirrors the sql trio. Canned-`get` builders local to the test files (never
|
||
`conftest.py`, which is MAF/LLM-only).
|
||
- **New tests in this plan:** ~11 across 3 new files + 1 comment correction.
|
||
|
||
### Tests to write
|
||
|
||
| Type | File | Verifies | Model test |
|
||
|------|------|----------|------------|
|
||
| Unit | `tests/test_ingest_http.py` | verbatim fetch; URL join; `max_rows`; credential env-resolve (set/unset); urllib default | `tests/test_ingest_sql.py` |
|
||
| Load-bearing | `tests/test_ingest_http_loadbearing.py` | network gate (never-called + IngestError); fenced-not-table; navigability; over-cap-writes-nothing | `tests/test_ingest_sql_loadbearing.py` |
|
||
| Golden | `tests/test_ingest_golden_http.py` | byte-identical materialization; §10 idempotence; no creds | `tests/test_ingest_golden_sql.py` |
|
||
| Comment fix | `tests/test_ingest_materialize.py:245` | default-no-flag still refuses (intent reworded) | — |
|
||
|
||
**Offline property (SC):** satisfied by construction — every http test injects `http_get` (no
|
||
socket) and needs no credential. Acceptance: `uv run pytest` exits 0 with the network cable
|
||
pulled. Convention to enforce in review: **no ingest test calls `materialize`/`read_http` for an
|
||
http source without passing `http_get`** (the default `_urllib_get` is the only socket path and
|
||
is never exercised).
|
||
|
||
## Risks and Mitigations
|
||
|
||
| Priority | Risk | Location | Impact | Mitigation |
|
||
|----------|------|----------|--------|------------|
|
||
| Critical | Silent egress if the gate sits after connector construction, or `_urllib_get` is reachable without the flag | `ingest.py:393` gate / dispatch | Network hit with no operator opt-in — defeats the task | Gate in `materialize` BEFORE the staging loop; `_urllib_get` only reached inside the http branch, which the gate guards. Load-bearing test asserts `get` never called when flag absent. |
|
||
| High | Credential echo in the verbatim body | http body render | A reflecting endpoint could echo `Authorization` into a committed bundle file | Inherent to "verbatim body" — do NOT auto-redact (breaks determinism). Golden mock never echoes a secret; document as a residual deployer-owned risk (§1 boundary). |
|
||
| High | Credential leak via error/log text | `read_http` IngestError, `_LOGGER.info` | Secret in an exception or log line | Mirror `read_sql`: wrap transport errors WITHOUT the secret; keep §8 log to `id`/`ingested_at`/`row_count` only. (Optional sentinel-secret grep test — nice-to-have, not gating.) |
|
||
| High | Determinism: verbatim-vs-LF-only tension, trailing newline, fence collision | `_render_fenced_block` / `read_http` | Golden passes locally, diverges on other input/platform; or silently-broken markdown in the bundle | Pin (docstring + golden): UTF-8 decode fail-fast; CRLF→LF; strip trailing newlines outside the fence; `max_rows = len(splitlines())`; **a body line beginning ` ``` ` → `IngestError` fail-fast** (never a silently-corrupted fenced block — consistent with the no-silent-coercion frame). |
|
||
| Medium | `max_rows` line-count semantics (extended-Unicode / trailing / blank lines) | `read_http` | Spurious pass/fail vs golden | Count `\n`-only on the CRLF-normalized text (NOT `str.splitlines()`, which also splits on `\v \f \x1c-\x1e \x85 \u2028 \u2029` — verified): `n = normalized.count("\n"); line_count = n if normalized.endswith("\n") else n+1`. Matches the LF-only body; a trailing newline adds no phantom line; internal blank lines count. Test exactly `max_rows` (pass) and `max_rows+1` (raise). |
|
||
| Medium | `base_url`+`query` join edge cases | `read_http` | Non-deterministic joins; scheme/host escape | Pinned `rstrip("/")+"/"+lstrip("/")`; `base_url` userinfo already rejected (`ingest.py:92-100`); the joined URL is never stamped in frontmatter. |
|
||
| Low | Stale `test_http_source_has_no_connector` intent | `tests/test_ingest_materialize.py:245` | Comment says "no connector" (now false) | Reword in Step 2; keep the default-refuse assertion. |
|
||
| Low | Temptation to touch `okf.py`/`shared/` | `okf.py`, `shared/` | Breaks D7-portability invariant / commons pull-only | `forbidden_paths` on every code step; SC `git diff --stat` shows no change. |
|
||
| Low | ruff `S310` (urlopen) / mypy `Callable` typing | `ingest.py` | Lint/type red | `# noqa: S310` on the gated `urlopen`; `Callable` from `collections.abc`; confirm at Verify. |
|
||
|
||
## Assumptions
|
||
|
||
| # | Assumption | Why unverifiable at plan time | Impact if wrong |
|
||
|---|-----------|-----------------|-----------------|
|
||
| 1 | The default auth scheme for a resolved `credential_ref` is `Authorization: Bearer {secret}` | The spec names credential-by-ref but not the wire scheme; endpoint-specific | A real deployer's endpoint may need a different scheme — acceptable for a demo extension point; documented, and the `get` seam is swappable. Golden uses `credential_ref: null` so this is not frozen in golden bytes. |
|
||
|
||
*(RESOLVED, not open — the brief's `[ASSUMPTION] flag threads through materialize and
|
||
run_project` was verified FALSE: `run.py` never calls `materialize`. The flag is scoped to
|
||
`materialize` only. `source_system` = `manifest.source.id` confirmed from
|
||
`_render_concept_file:312`; `max_rows` = response line count pinned per §8. The D7 pointer
|
||
`create_sdk_mcp_server` was verified 2026-07-04 against the official Claude Agent SDK Python docs
|
||
— it is the confirmed in-process MCP-server primitive, not an assumption.)*
|
||
|
||
## Verification
|
||
|
||
*Per-step manifests are checked by trekexecute. These are the brief's Success Criteria as
|
||
end-to-end checks (run from repo root):*
|
||
|
||
- [ ] `uv run pytest -q` → expected: exits 0 (full suite green, no network, no credentials set).
|
||
- [ ] Network gate detach-proof: comment out `and not allow_network` in `ingest.py`, run
|
||
`uv run pytest tests/test_ingest_http_loadbearing.py -k gate` → expected: **RED**; restore → green.
|
||
- [ ] `uv run pytest tests/test_ingest_golden_http.py -q` → expected: pass; output byte-matches
|
||
`examples/ingest-golden-http/expected-bundle/`.
|
||
- [ ] `uv run pytest tests/test_ingest_http.py -k max_rows` → expected: pass (over-cap → IngestError).
|
||
- [ ] `grep -rin "live" docs/extending.md examples/ingest-golden-http/ src/portfolio_optimiser/ingest.py`
|
||
→ expected: any hit is only "local mock" / "no live source" negated framing (scoped to the
|
||
NEW I6 artifacts — a repo-wide `grep "live"` is not a pass/fail gate: dozens of unrelated
|
||
pre-existing hits in `docs/`, `persona.py`, `simulation.py`, `test_*_live.py`).
|
||
- [ ] `git diff --stat` (vs pre-I6) → expected: NO change to `src/portfolio_optimiser/okf.py` or `shared/`.
|
||
- [ ] `uv run ruff check .` → 0; `uv run ruff format --check .` → 0; `uv run mypy src` → 0.
|
||
- [ ] `grep -q create_sdk_mcp_server docs/extending.md` → exits 0; new http section present.
|
||
- [ ] Null API-spend: no model calls, no real network (golden/gate/unit all use injected `get`).
|
||
|
||
## Estimated Scope
|
||
|
||
- **Files to modify:** 3 (`ingest.py`, `tests/test_ingest_materialize.py`, `docs/extending.md`).
|
||
- **Files to create:** 11 (3 test files + 8 golden/fixture/builder: `manifest.json`,
|
||
`ingested-at.txt`, `fixture/status`, `fixture/report`, 3 × `expected-bundle/*.md`,
|
||
`build_fixture.py`).
|
||
- **Complexity:** medium (one genuinely new render path + a security gate; the rest mirrors I4).
|
||
|
||
## Plan Quality Score
|
||
|
||
| Dimension | Weight | Score | Notes |
|
||
|-----------|--------|-------|-------|
|
||
| Structural integrity | 0.15 | 95 | 4 steps, strict dependency order (connector→gate→golden→docs) |
|
||
| Step quality | 0.20 | 92 | test-first, exact file:line, explicit reuse; Step 2 touches 3 files (cohesive) |
|
||
| Coverage completeness | 0.20 | 95 | every Success Criterion → a step + a Verification check |
|
||
| Specification quality | 0.15 | 90 | pinned decisions concrete; 2 honest doc-time assumptions remain |
|
||
| Risk & pre-mortem | 0.15 | 92 | silent-egress, credential-leak, determinism all mitigated |
|
||
| Headless readiness | 0.10 | 90 | On-failure + Checkpoint + Manifest per step |
|
||
| Manifest quality | 0.05 | 88 | must_contain + forbidden_paths guard okf.py/shared/ |
|
||
| **Weighted total** | **1.00** | **92** | **Grade: A** |
|
||
|
||
**Adversarial review:**
|
||
- **Plan critic:** REVISE → addressed. 1 blocker (gate false-green), 5 major (live-grep
|
||
unsatisfiable, first-live-source circular, create_sdk_mcp_server contradiction, fence-collision
|
||
silent, golden circular/hand-review), 6 minor — all resolved in Revisions below.
|
||
- **Scope guardian:** ALIGNED. 0 creep, 0 Non-Goal violations, 0 dependency issues; independently
|
||
confirmed the `materialize`-only flag scoping is a correct scope fix (not a gap). Its one minor
|
||
(over-broad live-grep) coincides with plan-critic MAJOR-1 and is fixed.
|
||
|
||
## Revisions
|
||
|
||
*Added by adversarial review (Phase 9). All blocker + major findings addressed; the plan was
|
||
revised once for the merged set.*
|
||
|
||
| # | Finding | Severity | Resolution |
|
||
|---|---------|----------|------------|
|
||
| 1 | Gate load-bearing test would false-green: reverting `and not allow_network` to the unconditional refusal still passes a refuse-only test | blocker | Step 2 gate test now asserts BOTH branches on the same manifest + same recording `get` (refuse: flag off → IngestError + get never called; allow: flag on → succeeds + get called). Removing the whole gate → refuse branch RED; removing just the clause → allow branch RED. Verify wording fixed. |
|
||
| 2 | `grep -rin "live" docs/ src/ examples/` unsatisfiable (dozens of pre-existing hits) | major | Scoped the "no live claim" check to the NEW I6 artifacts (`docs/extending.md`, `examples/ingest-golden-http/`, `ingest.py`) in both Step 4 and Verification. |
|
||
| 3 | `git grep "first live source"` circular (matches only plan/brief) | major | Removed the circular sub-check. |
|
||
| 4 | `create_sdk_mcp_server` pinned in 3 gates while flagged "may be stale" | major | Verified 2026-07-04 against official Claude Agent SDK Python docs — confirmed correct; pinned confidently, Assumption downgraded to a verified fact. |
|
||
| 5 | Fence-collision (body containing a ` ``` ` line) silently corrupts the bundle | major | Changed to fail-fast: a body line beginning ` ``` ` → `IngestError` (Step 1 connector + docstring + risk + alternatives + a new unit test). |
|
||
| 6 | Golden expected-bundle circular + "hand-review" is a non-headless human step | major | Step 3 now carries the hand-derived `ingest-status.md` expectation IN the plan (non-circular, headless-usable), names the Step-2 load-bearing tests as the independent correctness oracle, and adds an escalate-on-divergence clause. |
|
||
| 7 | `max_rows` counts lines pre-strip; blank-line nuance | minor | Pinned: count on CRLF-normalized text before the trailing-strip; blank lines count as content lines (deterministic). Risk row + Step 1 updated. |
|
||
| 8 | `c:\temp\cache` backslash/tab trap in `build_fixture.py` | minor | Pinned raw-bytes / raw-string authoring so the verbatim backslash survives. |
|
||
| 9 | `# noqa: S310` likely dead (no bandit config) | minor | Do not pre-add; add only if `ruff check` flags the `urlopen` at Verify. |
|
||
| 10 | File-count off by one (10 vs 11 created) | minor | Corrected to 11 created / 3 modified. |
|
||
| 11 | Line-ref drift (`:393` vs `:394`) | minor | Standardized the gate reference to `:394-398`. |
|
||
| 12 | Pass-at-`max_rows` boundary test only in risk table | minor | Added the boundary-pass + `max_rows+1`-raise pair to the Step 1 test enumeration. |
|
||
|
||
## Adversarial Pass 2 (gemini-bridge, v5.1.1 high-effort)
|
||
|
||
The high-effort plan phase runs an additional independent gemini-bridge review. **Outcome:
|
||
gemini-bridge is non-functional (verified 2026-07-04, not assumed).** The deep-research query ran
|
||
to `status: completed`, but every retrieval path failed: `gemini_get_research_result` →
|
||
`'Interaction' object has no attribute 'outputs'` (server-side bug, 3× retry); the followup
|
||
re-emit → `404 model gemini-3-pro-preview no longer available` (deprecated). This confirms the
|
||
prior recorded outage — the operator's expectation that it was fixed did not hold. No findings
|
||
were retrievable; the agent correctly refused to fabricate a second opinion.
|
||
|
||
**Substitute triangulation (per the standing gemini-outage handling):**
|
||
- An independent high-effort **plan-critic confirmation pass** traced the blocker fix against both
|
||
detach cases and verified revisions #2–#6 — plan **sound**, no new blocker/major; folded in one
|
||
minor (CommonMark ≤3-space indented closing fence → `line.lstrip(" ").startswith("```")`).
|
||
- The two decisions gemini flagged as spec-checkable were resolved by **primary-source verification**
|
||
(no gemini needed): **CommonMark §4.5** fenced-code semantics (closing fence ≥ opening length,
|
||
≤3-space indent) → the fence-collision check is adequate; **CPython `str.splitlines()`** splits on
|
||
the extended Unicode set → the `max_rows` cap was switched to a **`\n`-only** count so it matches
|
||
the LF-only rendered body (Step 1 + risk row updated).
|
||
- Decision 4 ("config cannot grant its own capability; capability is an explicit runtime grant")
|
||
is the **capability-security / POLA** (Principle of Least Authority) pattern — the manifest is
|
||
data that cannot escalate its own authority; only the `allow_network` runtime grant can.
|
||
|
||
Net: no external second opinion available, but every decision gemini would have triangulated is
|
||
covered by an independent internal pass or an authoritative primary-source check.
|