feat(ingest): bound the default http transport in time, in front of the pinned library (S2.4)

The hole: `read_http`'s default transport is the library's `urllib_get`, which invokes the
stdlib opener with no `timeout=`. urllib's documented fallback is then the process-wide default
socket timeout — `None` out of the box — so an http source that accepts a connection and never
answers hangs a run indefinitely. That contradicts the invariant that nothing runs unbounded.

The spec text for S2.4 ("a timeout parameter on `_urllib_get`") could NOT be followed literally:
that function is UPSTREAM library code (`llm_ingestion_okf.connectors`, pinned v0.3.1, pull-only),
signature `(url, credential) -> str` — measured, not assumed. Same failure class as S2.2's
"implement it in `ingest.py`": spec text that says "change X" has to be checked against whether
X is ours at all.

So the fix goes in FRONT of the library: `timeout_get` scopes `socket.setdefaulttimeout` around
a delegate call to the library's own `urllib_get`, and `materialize` now hands the library that
wrapped transport instead of letting it resolve its own untimed default. This meets S2.4's own
verification criterion — a bound WITHOUT a second socket path — and avoids duplicating the
credential-header logic. An explicitly injected `http_get` is passed through UNWRAPPED: a
caller-owned transport (MCP fronts a subprocess with its own `timeout_seconds`) keeps its own
policy, and a process-global side effect is not ours to impose on it.

Honest limit, carried in the code comment, the test docstring and `docs/extending.md`, not just
in the commit: the default socket timeout is PROCESS-global. Under `concurrency=k` the runner is
asyncio on one thread, so the scoping holds; driving `read_http` from a thread-pool executor
would make it unsafe.

Half of S2.4's scope was already delivered upstream — transport failures are categorised as
`SourceError(code="http_transport")`. Coarser than the plan envisaged, but not ours to rewrite.

Two pre-existing guards went red on the first pass, both on PROSE only: `ingest.py` must not
contain "urlopen" (no forked connector) or "ingest_mcp" (AST-guarded mcp-free). No code violated
either — my docstrings merely named them. The guards were left exactly as strict as they were and
the prose was reworded; weakening a real guard to save a comment is the trade this repo refuses.

578 -> 583 tests. Five mutations MEASURED red (restored from scratchpad + `shasum -c` each time,
never `git checkout`):
  1. remove the timeout scoping entirely            -> RED
  2. apply the bound AFTER the delegate call        -> RED
  3. set the bound but never restore it (no finally)-> RED  (the unconditional control)
  4. hand the library a bare None again (pre-S2.4)  -> RED  (the wiring)
  5. make the wrapping unconditional                -> RED  (the conditional control)

Mutations 1 and 2 take ~10s to fail rather than failing instantly: that is the loopback test's
join deadline expiring. It is the measurement that the bound actually BITES — a black-hole
listener on 127.0.0.1 that completes the handshake and never answers, run on a daemon thread so
a detached seam fails an assertion instead of hanging the suite forever. Every other assertion
here only proves we set a global; that one proves the global does something.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdLGwd33vqhkToh98Ym34P

docs/extending.md

@@ -98,6 +98,21 @@ implementation (the default `_urllib_get` is the only socket path). The golden c
 (`examples/ingest-golden-http/`) and every test inject a canned `get` over committed fixture
 payloads, so the suite runs offline against a **local mock** — no live source, no credentials.
 
+**The default transport is time-bounded (S2.4).** The library's `_urllib_get` calls the stdlib
+opener without a timeout, which falls back to the process-wide default socket timeout — `None`
+out of the box — so a source that accepts a connection and then never answers would hang a run,
+contradicting the invariant that nothing runs unbounded. `materialize` therefore hands the library
+that same socket path wrapped in `ingest.timeout_get`, which scopes `socket.setdefaulttimeout`
+(`HTTP_TIMEOUT_SECONDS`, 30s) around the delegate call: the bound applies **without** a second
+socket path and **without** duplicating the credential header, so the pinned library stays
+untouched. An explicitly injected `http_get` is passed through **unwrapped** — a caller-supplied
+transport owns its own timeout policy.
+
+**Honest limit:** the default socket timeout is *process-global*. Under `concurrency=k` the runner
+is asyncio on a single thread, so the scoping holds. Driving `read_http` from a thread-pool
+executor would make it unsafe, and the bound would have to move to a per-call timeout argument —
+i.e. to owning a socket path locally.
+
 ### D7 sibling hook — MCP as an extension of this family
 
 The spec (§4) documents an **MCP-based connector as an extension of the `http` family**, not a new

src/portfolio_optimiser/ingest.py

@@ -33,6 +33,7 @@ dependencies and imports no ``agent_framework`` / ``mcp``. Guarded by
 from __future__ import annotations
 
 import hashlib
+import socket
 from pathlib import Path
 
 from llm_ingestion_okf import (
@@ -64,10 +65,10 @@ from llm_ingestion_okf.render import render_fenced_block, render_table
 
 # Two historical PRIVATE names the repo's existing tests bind to, re-exported so those bindings
 # survive the adoption unchanged. `_urllib_get` backs an identity assertion that the default http
-# transport is the real socket path rather than a stub (tests/test_ingest_http.py);
+# transport is the real socket path rather than a stub (tests/test_ingest_http.py), and is the
-# `_sql_value_to_text` is exercised directly for the §5 bool/BLOB refusals
+# delegate `timeout_get` wraps below; `_sql_value_to_text` is exercised directly for the §5
-# (tests/test_ingest_sql.py). Not in `__all__` — they are not part of this module's contract.
+# bool/BLOB refusals (tests/test_ingest_sql.py). Not in `__all__` — not part of this contract.
-from llm_ingestion_okf.connectors import urllib_get as _urllib_get  # noqa: F401
+from llm_ingestion_okf.connectors import urllib_get as _urllib_get
 from llm_ingestion_okf.render import sql_value_to_text as _sql_value_to_text  # noqa: F401
 
 #: The repo's historical name for the §4 top-level model. The library calls it ``Manifest`` and
@@ -75,7 +76,15 @@ from llm_ingestion_okf.render import sql_value_to_text as _sql_value_to_text  #
 #: repo-local name stable for callers that bound to it before the library adoption.
 ManifestV1 = Manifest
 
+#: The wall-clock bound applied to the default http transport (S2.4). A source that accepts a
+#: connection and then never answers must not be able to hang a run: the repo requires stop
+#: criteria and budget caps at startup precisely so nothing runs unbounded, and an untimed socket
+#: is the same failure wearing a different hat. Chosen to match the MCP transport's own
+#: `timeout_seconds` default so both transports in the `http` family fail on the same clock.
+HTTP_TIMEOUT_SECONDS = 30.0
+
 __all__ = [
+    "HTTP_TIMEOUT_SECONDS",
     "Extraction",
     "FileSource",
     "HttpGet",
@@ -99,9 +108,41 @@ __all__ = [
     "read_sql",
     "render_fenced_block",
     "render_table",
+    "timeout_get",
 ]
 
 
+def timeout_get(get: HttpGet, *, timeout: float = HTTP_TIMEOUT_SECONDS) -> HttpGet:
+    """Wrap an http transport so its socket operations are time-bounded (S2.4).
+
+    The library's ``urllib_get`` is the only socket path, and it invokes the stdlib opener with no
+    ``timeout=`` argument. urllib's documented fallback in that case is the process-wide default
+    socket timeout — ``None`` unless set — so an unanswered connection blocks forever. Scoping
+    ``socket.setdefaulttimeout`` around the delegate call therefore bounds the real transport
+    WITHOUT opening a second socket path and WITHOUT re-implementing the credential header, which
+    is what S2.4 asks for: the fix belongs in front of the pinned library, not inside it.
+
+    The previous default is restored in a ``finally``, so a transport failure cannot leak the
+    bound onto unrelated sockets.
+
+    KNOWN CAVEAT — the default socket timeout is PROCESS-global, not per-call. Under
+    ``concurrency=k`` the runner is asyncio on a single thread, so this scoping holds. If
+    ``read_http`` is ever driven from a thread-pool executor, this is NOT thread-safe and the
+    bound has to move to a per-call timeout argument on the opener — i.e. to owning a socket
+    path in this module, which is exactly what this wrapper exists to avoid.
+    """
+
+    def bounded(url: str, credential: str | None) -> str:
+        previous = socket.getdefaulttimeout()
+        socket.setdefaulttimeout(timeout)
+        try:
+            return get(url, credential)
+        finally:
+            socket.setdefaulttimeout(previous)
+
+    return bounded
+
+
 def load_manifest(path: str | Path) -> tuple[Manifest, str]:
     """Validate a manifest fail-fast, BEFORE any source access (§4, §9).
 
@@ -141,13 +182,21 @@ def materialize(
     ``http`` source is refused fail-fast unless it is set, so the manifest itself can never
     grant network access. ``http_get`` optionally injects the transport seam; both are ignored
     for ``file``/``sql`` sources. Call ``materialize_bundle`` directly when the stamp is
-    wanted alongside the paths."""
+    wanted alongside the paths.
+
+    The DEFAULT transport is time-bounded (S2.4): rather than letting the library resolve its own
+    untimed ``urllib_get``, this passes that same socket path wrapped in :func:`timeout_get`, so
+    an unanswered http source fails typed instead of hanging the run. An explicitly injected
+    ``http_get`` is passed through UNWRAPPED — a caller-supplied transport (the MCP one fronts a
+    subprocess with its own ``timeout_seconds``; test stubs touch nothing) owns its own timeout
+    policy, and a process-global socket bound is not ours to impose on it."""
+    transport = http_get if http_get is not None else timeout_get(_urllib_get)
     return list(
         materialize_bundle(
             Path(manifest_path),
             Path(bundle_dir),
             ingested_at,
             allow_network=allow_network,
-            http_get=http_get,
+            http_get=transport,
         ).written
     )

tests/test_ingest_http_timeout_loadbearing.py

@@ -0,0 +1,222 @@
+"""S2.4 load-bearing seam — the ``http`` transport is TIME-BOUNDED (ingest spec §11's detach-RED
+regime).
+
+The hole this closes: ``read_http``'s default transport is the library's ``urllib_get``, which
+calls ``urlopen(request)`` with no ``timeout=``. With no timeout argument urllib falls back to the
+process-wide default socket timeout, which is ``None`` out of the box — so a source that accepts a
+connection and then never answers blocks a run forever. That contradicts the repo invariant that
+nothing runs unbounded (stop criteria + budget caps are required at startup, never an open loop).
+
+The fix is a wrapper, NOT a fork: ``ingest.timeout_get`` scopes ``socket.setdefaulttimeout`` around
+a delegate call to the library's own ``urllib_get``. This is the documented urllib fallback path,
+so the bound applies WITHOUT introducing a second socket path and WITHOUT duplicating the
+credential-header logic — S2.4's own verification criterion.
+
+KNOWN CAVEAT (also carried in the implementation's comment): the default socket timeout is
+PROCESS-global. Under ``concurrency=k`` the runner is asyncio on a single thread, so the scoping
+holds. If ``read_http`` is ever driven from a thread-pool executor, this scoping is NOT
+thread-safe and the bound must move to a per-call ``urlopen(timeout=...)`` — which would mean
+owning a socket path here.
+
+What each test makes load-bearing:
+
+- SCOPED, NOT LEAKED (both directions): the delegate observes the bound WHILE it runs, and the
+  previous process default is restored afterwards. RED when the scoping is removed, RED when it is
+  applied AFTER the delegate call, and RED when it is set without being restored — a one-way
+  assertion would false-green on an unconditional mutation.
+- RESTORED ON FAILURE: a raising delegate still restores the previous default (the ``finally``).
+- DEFAULT TRANSPORT IS BOUNDED: a real ``materialize`` run with NO injected ``http_get`` reaches
+  the socket path through the wrapper. RED the moment ``materialize`` reverts to handing the
+  library a bare ``None`` (i.e. the unwrapped ``urllib_get``) — which is exactly the pre-S2.4 state.
+- INJECTED TRANSPORT IS CALLER-OWNED: an explicitly injected transport is NOT wrapped. This is the
+  conditional control: it goes RED if the wrapping is made unconditional, and it pins the contract
+  that a caller-supplied transport (``mcp_get``, a canned test stub) keeps its own timeout policy
+  and never has a process-global side effect imposed on it.
+- A REAL HANG IS BOUNDED (the one socket test): the assertions above prove we set the global; only
+  this one proves the global actually bites. It opens a LOOPBACK listener that completes the TCP
+  handshake and then never answers — no egress, no external host, no credential — and asserts the
+  GET fails typed and fast. Run on a daemon thread with a join deadline so that detaching the
+  scoping fails the assertion instead of hanging the suite forever.
+"""
+
+from __future__ import annotations
+
+import json
+import socket
+import threading
+from pathlib import Path
+from typing import Any
+
+import pytest
+
+from portfolio_optimiser.ingest import (
+    HTTP_TIMEOUT_SECONDS,
+    SourceError,
+    _urllib_get,
+    materialize,
+    timeout_get,
+)
+
+_INGESTED_AT = "2026-07-04T12:00:00Z"
+
+
+def _http_manifest(tmp_path: Path) -> Path:
+    manifest = {
+        "manifest_version": 1,
+        "source": {
+            "type": "http",
+            "id": "api",
+            "base_url": "https://api.example.test/v1",
+            "credential_ref": None,
+        },
+        "bundle_summary": "HTTP extracts from a local mock endpoint (http source type).",
+        "extractions": [
+            {
+                "id": "status",
+                "title": "Service status",
+                "query": "status",
+                "okf_type": "dataset",
+                "max_rows": 10,
+            }
+        ],
+    }
+    path = tmp_path / "manifest.json"
+    path.write_text(json.dumps(manifest), encoding="utf-8")
+    return path
+
+
+# --- LOAD-BEARING: the bound is scoped around the delegate, and restored ------------------------
+
+
+def test_timeout_get_scopes_socket_default_around_delegate_and_restores() -> None:
+    observed: list[float | None] = []
+
+    def delegate(url: str, credential: str | None) -> str:
+        observed.append(socket.getdefaulttimeout())
+        return "body"
+
+    before = socket.getdefaulttimeout()
+    assert timeout_get(delegate, timeout=7.5)("https://h.test/x", None) == "body"
+
+    # DURING the call the bound is in force. RED if the scoping is removed, and RED if it is
+    # applied after the delegate returns (both leave the delegate observing `before`).
+    assert observed == [7.5]
+    # AFTER the call the process default is exactly what it was. RED if the wrapper sets the
+    # global without restoring it — the mutation a one-way assertion would false-green on.
+    assert socket.getdefaulttimeout() == before
+
+
+def test_timeout_get_restores_socket_default_when_the_delegate_raises() -> None:
+    def failing(url: str, credential: str | None) -> str:
+        raise SourceError("http GET failed", code="http_transport")
+
+    before = socket.getdefaulttimeout()
+    with pytest.raises(SourceError):
+        timeout_get(failing, timeout=7.5)("https://h.test/x", None)
+    # RED when the restore is not in a `finally`: a transport failure would leak the bound onto
+    # every socket the process opens afterwards.
+    assert socket.getdefaulttimeout() == before
+
+
+# --- LOAD-BEARING: the DEFAULT transport is the bounded one --------------------------------------
+
+
+def test_materialize_default_transport_is_bounded(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    """No ``http_get`` injected → the real socket path must still be reached through the wrapper.
+
+    The socket path itself is stubbed at the module global the wrapper delegates to, so this runs
+    end-to-end through the unchanged library without opening a socket. RED when ``materialize``
+    goes back to passing the library a bare ``None``: the library then resolves its own unwrapped
+    ``urllib_get`` and the recorded timeout is the untouched process default.
+    """
+    observed: list[float | None] = []
+
+    def stub_urllib_get(url: str, credential: str | None) -> str:
+        observed.append(socket.getdefaulttimeout())
+        return "service ok"
+
+    monkeypatch.setattr("portfolio_optimiser.ingest._urllib_get", stub_urllib_get)
+
+    written = materialize(
+        _http_manifest(tmp_path),
+        tmp_path / "b",
+        ingested_at=_INGESTED_AT,
+        allow_network=True,
+        # NOTE: no http_get= — this is the default-path assertion.
+    )
+
+    assert any(p.name == "ingest-status.md" for p in written)
+    assert observed == [HTTP_TIMEOUT_SECONDS]
+
+
+def test_materialize_does_not_wrap_an_injected_transport(tmp_path: Path) -> None:
+    """A caller-supplied transport keeps its own timeout policy (the conditional control).
+
+    ``mcp_get`` fronts a subprocess, not a socket, and already carries its own
+    ``timeout_seconds``; canned test stubs touch nothing. Imposing a process-global socket bound on
+    caller-owned code would be a side effect we do not own. RED if the wrapping is made
+    unconditional — which would also false-green the default-path test above.
+    """
+    observed: list[float | None] = []
+    before = socket.getdefaulttimeout()
+
+    def injected(url: str, credential: str | None) -> str:
+        observed.append(socket.getdefaulttimeout())
+        return "service ok"
+
+    materialize(
+        _http_manifest(tmp_path),
+        tmp_path / "b",
+        ingested_at=_INGESTED_AT,
+        allow_network=True,
+        http_get=injected,
+    )
+
+    assert observed == [before]
+
+
+# --- LOAD-BEARING: the bound actually bites on a real blocked connection -------------------------
+
+
+def test_bounded_transport_fails_fast_against_a_hanging_loopback_server() -> None:
+    """The only socket in this suite: a loopback listener that never answers.
+
+    It binds 127.0.0.1 on an ephemeral port and never calls ``accept``; the OS backlog still
+    completes the handshake, so the client connects and then blocks reading a response that never
+    comes — the exact shape of the hang S2.4 closes. No egress, no external host, no credential.
+
+    Detaching the scoping does not merely change a value here, it removes the only thing that ends
+    the read. The call therefore runs on a daemon thread with a join deadline: a detached seam
+    leaves the thread alive and fails the assertion, instead of hanging the suite forever.
+    """
+    listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    listener.bind(("127.0.0.1", 0))
+    listener.listen(1)
+    host, port = listener.getsockname()
+    before = socket.getdefaulttimeout()
+    outcome: list[Any] = []
+
+    def attempt() -> None:
+        try:
+            outcome.append(
+                timeout_get(_urllib_get, timeout=0.25)(f"http://{host}:{port}/hang", None)
+            )
+        except BaseException as exc:  # noqa: BLE001 - the outcome IS the assertion subject
+            outcome.append(exc)
+
+    worker = threading.Thread(target=attempt, daemon=True)
+    worker.start()
+    worker.join(10.0)
+    try:
+        assert not worker.is_alive(), (
+            "unbounded: the GET was still blocked after 10s — the socket carried no timeout"
+        )
+        assert isinstance(outcome[0], SourceError)
+        # Typed, never a leaked OSError; the library's own transport categorisation rides through.
+        assert outcome[0].code == "http_transport"
+    finally:
+        listener.close()
+
+    assert socket.getdefaulttimeout() == before