open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 1 Activity

feat(ingest): add the MCP connector as a transport inside the http family (S2.2)

Browse source 
Commons settled on 2026-08-01 that MCP is an extension of the `http` source
family, not a fourth family (`shared/ingest-spec.md` §4). This implements it
with ZERO schema change and zero spec amendment.

The transport discriminator lives in `base_url`, not in new manifest fields:
the shared library rejects unknown manifest keys fail-fast, and we consume it
pull-only at a pinned v0.3.1, so `server_ref`/`tool` as fields would have meant
a spec amendment plus a library release. It buys nothing — the library already
joins `base_url` + `/` + `query`, so `mcp+stdio://<server_ref>` + `<tool>`
reproduces exactly the two-part structure the (now stale) reference plan wanted.

Staying inside the family INHERITS what a fourth family would have had to write
and could have forgotten: the §8 network grant (measured to fire before any tool
call), the `max_rows` cap, §5 verbatim fenced rendering, and the §7 provenance
stamp. The discriminator gates rather than labels — `mcp_get` refuses a URL it
does not own, so an MCP transport can never quietly serve an `https://` manifest
and leave the bundle's provenance claiming a transport that was never used.

Parsing is string-based, not `urlsplit`-based: `urlsplit().hostname` lowercases
the host, which would silently break the case-sensitive env lookup `server_ref`
depends on.

`ingest.py` is untouched — it is AST-guarded mcp-free, so the transport lives in
its own module and is opt-in at the call site. `ingest_mcp.py` imports the open
`mcp` protocol client but never `agent_framework`, keeping the seam D7-portable.

Load-bearing, six mutations all measured RED: detach the scheme guard · make the
refusal unconditional · swap parsing to `urlsplit().hostname` · skip non-text
content instead of raising · force `allow_network=True` · smuggle in a MAF
import. Both source files restored byte-identical (`shasum -c`) after each.

Honesty: `stdio_call_tool` (the real stdio path) is written but never executed
end to end — every test injects a canned tool call, so the suite spawns no
subprocess and opens no socket. No golden fixture, and MCP stays unwired in the
optimiser run path. Stated in docs/extending.md rather than implied away.

555 -> 578 tests; ruff + mypy green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0112FPR5TX6pDLiNicBPzE8i
This commit is contained in:
Kjell Tore Guttormsen 2026-08-02 21:14:57 +02:00
commit ddd6338f02
4 changed files with 683 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

55
docs/extending.md
View file

@ -105,15 +105,52 @@ client wired into the optimiser run path — that stays a **Non-Goal** here: the
`FunctionTool` seam is the default in the run path, and MCP is demonstrated (via `build_mcp_server`
in `datasource.py`), not wired in.
**Where the D7 sibling stands (målbilde §11 boundary).** The Claude Agent SDK sibling (D7) built
the **file/CSV and SQL** connectors — mirroring I3/I5 — with bit-identical golden extractions.
**HTTP and MCP are demonstrated on the MAF side only** (MAF-only), against a local mock; the
sibling ships no network connector and no live-source integration. On D7 the in-process server
hook is `create_sdk_mcp_server(name, version="1.0.0", tools=...) -> McpSdkServerConfig` (package
`claude-agent-sdk`) — verified 2026-07-04 against the official Claude Agent SDK Python docs — but
that is a **documented hook a deployer would reach for, not a shipped D7 connector**; no D7 HTTP
or MCP session is planned. A deployer who wants a network- or MCP-mediated source extends this
family behind the same explicit, per-run network grant; nothing here contacts a live endpoint.
**The connector (S2.2, 2026-08-02).** `ingest_mcp.py` implements it. There is **no fourth source
family and no schema change**: an MCP source is an ordinary `type: "http"` source whose transport is
declared by the `base_url` scheme, and whose two parts fall straight out of the join the library
already performs (`base_url` + `/` + `query`):
```json
{ "type": "http", "id": "docs", "base_url": "mcp+stdio://PORTFOLIO_DOCS_MCP" }
```
```python
from portfolio_optimiser.ingest import materialize
from portfolio_optimiser.ingest_mcp import mcp_get, stdio_call_tool
materialize(manifest, bundle_dir, ingested_at="2026-08-02T00:00:00Z",
allow_network=True, http_get=mcp_get(stdio_call_tool()))
```
`server_ref` is the **name of an environment variable** holding the server command — mirroring the
`sql` family's `connection_ref`, so the manifest carries a reference and never an executable path —
and the extraction's `query` is the tool name. Parsing is string-based, not `urlsplit`-based:
`urlsplit().hostname` lowercases the host, which would silently break a case-sensitive env lookup.
Two properties are worth stating because they are **inherited, not written**: the §8 network grant
covers MCP for free (an MCP source *is* `http`, so it is refused before any tool call unless
`allow_network=True`), as do the `max_rows` cap, verbatim fenced rendering, and the §7 provenance
stamp. Had MCP become a fourth family, each of those would have been ours to write — and ours to
forget. The transport discriminator **gates rather than labels**: `mcp_get` refuses a URL it does
not own, so an MCP transport can never quietly serve a plain `https://` manifest and leave the
bundle's provenance claiming a transport that was never used.
**Not verified against a live MCP server.** Every test injects a canned tool call, so the suite
spawns no subprocess and opens no socket (cost discipline). `stdio_call_tool` — the real stdio path
— is therefore **written but never executed end to end**; a deployer using it should expect to shake
it out. The seam it plugs into (`mcp_get`, the discriminator, the gate) *is* measured, including six
detach mutations. There is no `examples/ingest-golden-mcp/` fixture, and MCP remains **unwired in
the optimiser run path** — the in-process `FunctionTool` seam stays the default there.
**Where the D7 sibling stands (målbilde §11 boundary).** The Claude Agent SDK sibling built the
**file/CSV and SQL** connectors — mirroring I3/I5 — with bit-identical golden extractions. **HTTP
and MCP are implemented on the MAF side only**; the sibling ships no network connector and no
live-source integration. `ingest_mcp.py` is deliberately MAF-free (it imports the open `mcp`
protocol client, never `agent_framework`), so the seam is portable to D7 unchanged. On D7 the
in-process server hook is `create_sdk_mcp_server(name, version="1.0.0", tools=...) ->
McpSdkServerConfig` (package `claude-agent-sdk`) — verified 2026-07-04 against the official Claude
Agent SDK Python docs — but that is a **documented hook a deployer would reach for, not a shipped D7
connector**; no D7 MCP session is planned. Nothing here contacts a live endpoint.
## Bytt ut henteren (Embedder / Retriever, S3.1)