The hole: `read_http`'s default transport is the library's `urllib_get`, which invokes the
stdlib opener with no `timeout=`. urllib's documented fallback is then the process-wide default
socket timeout — `None` out of the box — so an http source that accepts a connection and never
answers hangs a run indefinitely. That contradicts the invariant that nothing runs unbounded.
The spec text for S2.4 ("a timeout parameter on `_urllib_get`") could NOT be followed literally:
that function is UPSTREAM library code (`llm_ingestion_okf.connectors`, pinned v0.3.1, pull-only),
signature `(url, credential) -> str` — measured, not assumed. Same failure class as S2.2's
"implement it in `ingest.py`": spec text that says "change X" has to be checked against whether
X is ours at all.
So the fix goes in FRONT of the library: `timeout_get` scopes `socket.setdefaulttimeout` around
a delegate call to the library's own `urllib_get`, and `materialize` now hands the library that
wrapped transport instead of letting it resolve its own untimed default. This meets S2.4's own
verification criterion — a bound WITHOUT a second socket path — and avoids duplicating the
credential-header logic. An explicitly injected `http_get` is passed through UNWRAPPED: a
caller-owned transport (MCP fronts a subprocess with its own `timeout_seconds`) keeps its own
policy, and a process-global side effect is not ours to impose on it.
Honest limit, carried in the code comment, the test docstring and `docs/extending.md`, not just
in the commit: the default socket timeout is PROCESS-global. Under `concurrency=k` the runner is
asyncio on one thread, so the scoping holds; driving `read_http` from a thread-pool executor
would make it unsafe.
Half of S2.4's scope was already delivered upstream — transport failures are categorised as
`SourceError(code="http_transport")`. Coarser than the plan envisaged, but not ours to rewrite.
Two pre-existing guards went red on the first pass, both on PROSE only: `ingest.py` must not
contain "urlopen" (no forked connector) or "ingest_mcp" (AST-guarded mcp-free). No code violated
either — my docstrings merely named them. The guards were left exactly as strict as they were and
the prose was reworded; weakening a real guard to save a comment is the trade this repo refuses.
578 -> 583 tests. Five mutations MEASURED red (restored from scratchpad + `shasum -c` each time,
never `git checkout`):
1. remove the timeout scoping entirely -> RED
2. apply the bound AFTER the delegate call -> RED
3. set the bound but never restore it (no finally)-> RED (the unconditional control)
4. hand the library a bare None again (pre-S2.4) -> RED (the wiring)
5. make the wrapping unconditional -> RED (the conditional control)
Mutations 1 and 2 take ~10s to fail rather than failing instantly: that is the loopback test's
join deadline expiring. It is the measurement that the bound actually BITES — a black-hole
listener on 127.0.0.1 that completes the handshake and never answers, run on a daemon thread so
a detached seam fails an assertion instead of hanging the suite forever. Every other assertion
here only proves we set a global; that one proves the global does something.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdLGwd33vqhkToh98Ym34P
Commons settled on 2026-08-01 that MCP is an extension of the `http` source
family, not a fourth family (`shared/ingest-spec.md` §4). This implements it
with ZERO schema change and zero spec amendment.
The transport discriminator lives in `base_url`, not in new manifest fields:
the shared library rejects unknown manifest keys fail-fast, and we consume it
pull-only at a pinned v0.3.1, so `server_ref`/`tool` as fields would have meant
a spec amendment plus a library release. It buys nothing — the library already
joins `base_url` + `/` + `query`, so `mcp+stdio://<server_ref>` + `<tool>`
reproduces exactly the two-part structure the (now stale) reference plan wanted.
Staying inside the family INHERITS what a fourth family would have had to write
and could have forgotten: the §8 network grant (measured to fire before any tool
call), the `max_rows` cap, §5 verbatim fenced rendering, and the §7 provenance
stamp. The discriminator gates rather than labels — `mcp_get` refuses a URL it
does not own, so an MCP transport can never quietly serve an `https://` manifest
and leave the bundle's provenance claiming a transport that was never used.
Parsing is string-based, not `urlsplit`-based: `urlsplit().hostname` lowercases
the host, which would silently break the case-sensitive env lookup `server_ref`
depends on.
`ingest.py` is untouched — it is AST-guarded mcp-free, so the transport lives in
its own module and is opt-in at the call site. `ingest_mcp.py` imports the open
`mcp` protocol client but never `agent_framework`, keeping the seam D7-portable.
Load-bearing, six mutations all measured RED: detach the scheme guard · make the
refusal unconditional · swap parsing to `urlsplit().hostname` · skip non-text
content instead of raising · force `allow_network=True` · smuggle in a MAF
import. Both source files restored byte-identical (`shasum -c`) after each.
Honesty: `stdio_call_tool` (the real stdio path) is written but never executed
end to end — every test injects a canned tool call, so the suite spawns no
subprocess and opens no socket. No golden fixture, and MCP stays unwired in the
optimiser run path. Stated in docs/extending.md rather than implied away.
555 -> 578 tests; ruff + mypy green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0112FPR5TX6pDLiNicBPzE8i
Door A (manifest -> connector -> deterministic materialization -> index) is no
longer implemented here. src/portfolio_optimiser/ingest.py becomes a thin
consumer seam over the shared library, git-pinned to v0.3.1 on the same Forgejo
channel portfolio-optimiser-claude uses. Net -626/+385; ingest.py 599 -> 145 lines.
shared/ingest-spec.md remains the normative spec: the library implements it, it
does not replace it. Spec changes continue to go via commons.
Acceptance criterion met and proven: all three golden bundles (file/sql/http)
are byte-exact before and after, including the idempotence re-run. examples/ and
shared/ carry ZERO modifications -- the fasit was not adjusted to fit.
The rejection set was verified equivalent, not assumed: all 22 malformations the
repo's pydantic models refused are refused by the library, with typed codes
(okf_type_reserved, credential_embedded, extraction_id_duplicate, ...).
Test rebinding (invariants preserved, vehicle changed): the library has zero
runtime dependencies by design, so pydantic is unavailable to it.
ManifestV1.model_validate(dict) -> load_manifest_bytes(bytes); ValidationError ->
ManifestError; model_fields -> dataclasses.fields; PathSecurityError ->
SourceError(path_escape); ValueError -> MaterializationError(ingested_at_invalid).
Tests now also pin the refusal `code`, the library's documented stability
contract -- a sharper assertion than "some validation error was raised".
Two accepted behavioural deltas, recorded rather than silently dropped:
- Title whitespace is stored verbatim instead of collapsed at validation, so the
frontmatter title and the index label are no longer guaranteed identical for
irregular whitespace. Both behaviours are spec-conformant (the spec is SILENT;
the old one was a repo-local pinned decision). Queued as a commons-amendment
candidate so both stacks pin the same answer. Goldens unaffected.
- The section 8 audit log moves to logger llm_ingestion_okf.materialize. Nothing
in the repo consumed the old channel.
Also: the `type` discriminator is no longer a dataclass field, so the spec
cross-check asserts it explicitly -- without that line the swap would have
silently narrowed the test.
New tests/test_ingest_library_seam.py pins the seam itself: the restated section 5
stamp formula against the stamp the library actually writes (the one place the
adapter does not purely delegate, since v0.3.1 exposes no stamp helper), the
local-only allow_network default, the list[Path] unwrapping, and a guard that the
adapter never regrows local Door A machinery. All four verified RED when detached,
as were both golden regressions under a byte-level render mutation.
Door A is UNGATED: it calls no guard before writing to disk. Gating untrusted
content remains the caller's responsibility (guard wiring still planned).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B4jNN186eVqfe1x5DnTU6r
Findings 4-7 from the 2026-07-02 status analysis, per the session plan (S1):
- CHANGELOG rewritten truthfully (was: 'Plan phase - no framework code yet')
- README stack line names the split GA packages, not the agent-framework meta-package
- CLAUDE.md: MCP downgraded to extension point (in-process FunctionTool is the default seam)
- Verdict conflict semantics documented as chosen (store first-write-wins per id,
disk/wiki last-write-wins per file; full B10 taxonomy deliberately deferred)
- docs/extending.md: explicit 90%-principle cut-list (B10, B11, U12, U14, concurrent fan-out)
- .gitignore covers .trekexecute-progress-* (docs/.DS_Store was already untracked/ignored -
the plan's git rm --cached assumption was stale; no-op)
No code behavior changed (docstring only in verdicts.py). Suite 152/4 green, mypy clean,
ruff format --check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AaQCFnfsh3tfq1VfzdJpoi
