open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 1 Activity

feat(ingest): bound the default http transport in time, in front of the pinned library (S2.4)

Browse source 
The hole: `read_http`'s default transport is the library's `urllib_get`, which invokes the
stdlib opener with no `timeout=`. urllib's documented fallback is then the process-wide default
socket timeout — `None` out of the box — so an http source that accepts a connection and never
answers hangs a run indefinitely. That contradicts the invariant that nothing runs unbounded.

The spec text for S2.4 ("a timeout parameter on `_urllib_get`") could NOT be followed literally:
that function is UPSTREAM library code (`llm_ingestion_okf.connectors`, pinned v0.3.1, pull-only),
signature `(url, credential) -> str` — measured, not assumed. Same failure class as S2.2's
"implement it in `ingest.py`": spec text that says "change X" has to be checked against whether
X is ours at all.

So the fix goes in FRONT of the library: `timeout_get` scopes `socket.setdefaulttimeout` around
a delegate call to the library's own `urllib_get`, and `materialize` now hands the library that
wrapped transport instead of letting it resolve its own untimed default. This meets S2.4's own
verification criterion — a bound WITHOUT a second socket path — and avoids duplicating the
credential-header logic. An explicitly injected `http_get` is passed through UNWRAPPED: a
caller-owned transport (MCP fronts a subprocess with its own `timeout_seconds`) keeps its own
policy, and a process-global side effect is not ours to impose on it.

Honest limit, carried in the code comment, the test docstring and `docs/extending.md`, not just
in the commit: the default socket timeout is PROCESS-global. Under `concurrency=k` the runner is
asyncio on one thread, so the scoping holds; driving `read_http` from a thread-pool executor
would make it unsafe.

Half of S2.4's scope was already delivered upstream — transport failures are categorised as
`SourceError(code="http_transport")`. Coarser than the plan envisaged, but not ours to rewrite.

Two pre-existing guards went red on the first pass, both on PROSE only: `ingest.py` must not
contain "urlopen" (no forked connector) or "ingest_mcp" (AST-guarded mcp-free). No code violated
either — my docstrings merely named them. The guards were left exactly as strict as they were and
the prose was reworded; weakening a real guard to save a comment is the trade this repo refuses.

578 -> 583 tests. Five mutations MEASURED red (restored from scratchpad + `shasum -c` each time,
never `git checkout`):
  1. remove the timeout scoping entirely            -> RED
  2. apply the bound AFTER the delegate call        -> RED
  3. set the bound but never restore it (no finally)-> RED  (the unconditional control)
  4. hand the library a bare None again (pre-S2.4)  -> RED  (the wiring)
  5. make the wrapping unconditional                -> RED  (the conditional control)

Mutations 1 and 2 take ~10s to fail rather than failing instantly: that is the loopback test's
join deadline expiring. It is the measurement that the bound actually BITES — a black-hole
listener on 127.0.0.1 that completes the handshake and never answers, run on a daemon thread so
a detached seam fails an assertion instead of hanging the suite forever. Every other assertion
here only proves we set a global; that one proves the global does something.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdLGwd33vqhkToh98Ym34P
This commit is contained in:
Kjell Tore Guttormsen 2026-08-03 14:54:04 +02:00
commit 8910a673ea
3 changed files with 292 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

15
docs/extending.md
View file

@ -98,6 +98,21 @@ implementation (the default `_urllib_get` is the only socket path). The golden c
(`examples/ingest-golden-http/`) and every test inject a canned `get` over committed fixture
payloads, so the suite runs offline against a **local mock** — no live source, no credentials.
**The default transport is time-bounded (S2.4).** The library's `_urllib_get` calls the stdlib
opener without a timeout, which falls back to the process-wide default socket timeout — `None`
out of the box — so a source that accepts a connection and then never answers would hang a run,
contradicting the invariant that nothing runs unbounded. `materialize` therefore hands the library
that same socket path wrapped in `ingest.timeout_get`, which scopes `socket.setdefaulttimeout`
(`HTTP_TIMEOUT_SECONDS`, 30s) around the delegate call: the bound applies **without** a second
socket path and **without** duplicating the credential header, so the pinned library stays
untouched. An explicitly injected `http_get` is passed through **unwrapped** — a caller-supplied
transport owns its own timeout policy.
**Honest limit:** the default socket timeout is *process-global*. Under `concurrency=k` the runner
is asyncio on a single thread, so the scoping holds. Driving `read_http` from a thread-pool
executor would make it unsafe, and the bound would have to move to a per-call timeout argument —
i.e. to owning a socket path locally.
### D7 sibling hook — MCP as an extension of this family
The spec (§4) documents an **MCP-based connector as an extension of the `http` family**, not a new

61
src/portfolio_optimiser/ingest.py
View file

@ -33,6 +33,7 @@ dependencies and imports no ``agent_framework`` / ``mcp``. Guarded by
from __future__ import annotations
import hashlib
import socket
from pathlib import Path
from llm_ingestion_okf import (
@ -64,10 +65,10 @@ from llm_ingestion_okf.render import render_fenced_block, render_table
# Two historical PRIVATE names the repo's existing tests bind to, re-exported so those bindings
# survive the adoption unchanged. `_urllib_get` backs an identity assertion that the default http
# transport is the real socket path rather than a stub (tests/test_ingest_http.py);
# `_sql_value_to_text` is exercised directly for the §5 bool/BLOB refusals
# (tests/test_ingest_sql.py). Not in `__all__` — they are not part of this module's contract.
from llm_ingestion_okf.connectors import urllib_get as _urllib_get # noqa: F401
# transport is the real socket path rather than a stub (tests/test_ingest_http.py), and is the
# delegate `timeout_get` wraps below; `_sql_value_to_text` is exercised directly for the §5
# bool/BLOB refusals (tests/test_ingest_sql.py). Not in `__all__` — not part of this contract.
from llm_ingestion_okf.connectors import urllib_get as _urllib_get
from llm_ingestion_okf.render import sql_value_to_text as _sql_value_to_text # noqa: F401
#: The repo's historical name for the §4 top-level model. The library calls it ``Manifest`` and
@ -75,7 +76,15 @@ from llm_ingestion_okf.render import sql_value_to_text as _sql_value_to_text #
#: repo-local name stable for callers that bound to it before the library adoption.
ManifestV1 = Manifest
#: The wall-clock bound applied to the default http transport (S2.4). A source that accepts a
#: connection and then never answers must not be able to hang a run: the repo requires stop
#: criteria and budget caps at startup precisely so nothing runs unbounded, and an untimed socket
#: is the same failure wearing a different hat. Chosen to match the MCP transport's own
#: `timeout_seconds` default so both transports in the `http` family fail on the same clock.
HTTP_TIMEOUT_SECONDS = 30.0
__all__ = [
"HTTP_TIMEOUT_SECONDS",
"Extraction",
"FileSource",
"HttpGet",
@ -99,9 +108,41 @@ __all__ = [
"read_sql",
"render_fenced_block",
"render_table",
"timeout_get",
]
def timeout_get(get: HttpGet, *, timeout: float = HTTP_TIMEOUT_SECONDS) -> HttpGet:
"""Wrap an http transport so its socket operations are time-bounded (S2.4).
The library's ``urllib_get`` is the only socket path, and it invokes the stdlib opener with no
``timeout=`` argument. urllib's documented fallback in that case is the process-wide default
socket timeout — ``None`` unless set — so an unanswered connection blocks forever. Scoping
``socket.setdefaulttimeout`` around the delegate call therefore bounds the real transport
WITHOUT opening a second socket path and WITHOUT re-implementing the credential header, which
is what S2.4 asks for: the fix belongs in front of the pinned library, not inside it.
The previous default is restored in a ``finally``, so a transport failure cannot leak the
bound onto unrelated sockets.
KNOWN CAVEAT — the default socket timeout is PROCESS-global, not per-call. Under
``concurrency=k`` the runner is asyncio on a single thread, so this scoping holds. If
``read_http`` is ever driven from a thread-pool executor, this is NOT thread-safe and the
bound has to move to a per-call timeout argument on the opener — i.e. to owning a socket
path in this module, which is exactly what this wrapper exists to avoid.
"""
def bounded(url: str, credential: str | None) -> str:
previous = socket.getdefaulttimeout()
socket.setdefaulttimeout(timeout)
try:
return get(url, credential)
finally:
socket.setdefaulttimeout(previous)
return bounded
def load_manifest(path: str | Path) -> tuple[Manifest, str]:
"""Validate a manifest fail-fast, BEFORE any source access (§4, §9).
@ -141,13 +182,21 @@ def materialize(
``http`` source is refused fail-fast unless it is set, so the manifest itself can never
grant network access. ``http_get`` optionally injects the transport seam; both are ignored
for ``file``/``sql`` sources. Call ``materialize_bundle`` directly when the stamp is
wanted alongside the paths."""
wanted alongside the paths.
The DEFAULT transport is time-bounded (S2.4): rather than letting the library resolve its own
untimed ``urllib_get``, this passes that same socket path wrapped in :func:`timeout_get`, so
an unanswered http source fails typed instead of hanging the run. An explicitly injected
``http_get`` is passed through UNWRAPPED — a caller-supplied transport (the MCP one fronts a
subprocess with its own ``timeout_seconds``; test stubs touch nothing) owns its own timeout
policy, and a process-global socket bound is not ours to impose on it."""
transport = http_get if http_get is not None else timeout_get(_urllib_get)
return list(
materialize_bundle(
Path(manifest_path),
Path(bundle_dir),
ingested_at,
allow_network=allow_network,
http_get=http_get,
http_get=transport,
).written
)

222
tests/test_ingest_http_timeout_loadbearing.py Normal file
View file

@ -0,0 +1,222 @@
"""S2.4 load-bearing seam — the ``http`` transport is TIME-BOUNDED (ingest spec §11's detach-RED
regime).
The hole this closes: ``read_http``'s default transport is the library's ``urllib_get``, which
calls ``urlopen(request)`` with no ``timeout=``. With no timeout argument urllib falls back to the
process-wide default socket timeout, which is ``None`` out of the box — so a source that accepts a
connection and then never answers blocks a run forever. That contradicts the repo invariant that
nothing runs unbounded (stop criteria + budget caps are required at startup, never an open loop).
The fix is a wrapper, NOT a fork: ``ingest.timeout_get`` scopes ``socket.setdefaulttimeout`` around
a delegate call to the library's own ``urllib_get``. This is the documented urllib fallback path,
so the bound applies WITHOUT introducing a second socket path and WITHOUT duplicating the
credential-header logic — S2.4's own verification criterion.
KNOWN CAVEAT (also carried in the implementation's comment): the default socket timeout is
PROCESS-global. Under ``concurrency=k`` the runner is asyncio on a single thread, so the scoping
holds. If ``read_http`` is ever driven from a thread-pool executor, this scoping is NOT
thread-safe and the bound must move to a per-call ``urlopen(timeout=...)`` — which would mean
owning a socket path here.
What each test makes load-bearing:
- SCOPED, NOT LEAKED (both directions): the delegate observes the bound WHILE it runs, and the
previous process default is restored afterwards. RED when the scoping is removed, RED when it is
applied AFTER the delegate call, and RED when it is set without being restored — a one-way
assertion would false-green on an unconditional mutation.
- RESTORED ON FAILURE: a raising delegate still restores the previous default (the ``finally``).
- DEFAULT TRANSPORT IS BOUNDED: a real ``materialize`` run with NO injected ``http_get`` reaches
the socket path through the wrapper. RED the moment ``materialize`` reverts to handing the
library a bare ``None`` (i.e. the unwrapped ``urllib_get``) — which is exactly the pre-S2.4 state.
- INJECTED TRANSPORT IS CALLER-OWNED: an explicitly injected transport is NOT wrapped. This is the
conditional control: it goes RED if the wrapping is made unconditional, and it pins the contract
that a caller-supplied transport (``mcp_get``, a canned test stub) keeps its own timeout policy
and never has a process-global side effect imposed on it.
- A REAL HANG IS BOUNDED (the one socket test): the assertions above prove we set the global; only
this one proves the global actually bites. It opens a LOOPBACK listener that completes the TCP
handshake and then never answers — no egress, no external host, no credential — and asserts the
GET fails typed and fast. Run on a daemon thread with a join deadline so that detaching the
scoping fails the assertion instead of hanging the suite forever.
"""
from __future__ import annotations
import json
import socket
import threading
from pathlib import Path
from typing import Any
import pytest
from portfolio_optimiser.ingest import (
HTTP_TIMEOUT_SECONDS,
SourceError,
_urllib_get,
materialize,
timeout_get,
)
_INGESTED_AT = "2026-07-04T12:00:00Z"
def _http_manifest(tmp_path: Path) -> Path:
manifest = {
"manifest_version": 1,
"source": {
"type": "http",
"id": "api",
"base_url": "https://api.example.test/v1",
"credential_ref": None,
},
"bundle_summary": "HTTP extracts from a local mock endpoint (http source type).",
"extractions": [
{
"id": "status",
"title": "Service status",
"query": "status",
"okf_type": "dataset",
"max_rows": 10,
}
],
}
path = tmp_path / "manifest.json"
path.write_text(json.dumps(manifest), encoding="utf-8")
return path
# --- LOAD-BEARING: the bound is scoped around the delegate, and restored ------------------------
def test_timeout_get_scopes_socket_default_around_delegate_and_restores() -> None:
observed: list[float | None] = []
def delegate(url: str, credential: str | None) -> str:
observed.append(socket.getdefaulttimeout())
return "body"
before = socket.getdefaulttimeout()
assert timeout_get(delegate, timeout=7.5)("https://h.test/x", None) == "body"
# DURING the call the bound is in force. RED if the scoping is removed, and RED if it is
# applied after the delegate returns (both leave the delegate observing `before`).
assert observed == [7.5]
# AFTER the call the process default is exactly what it was. RED if the wrapper sets the
# global without restoring it — the mutation a one-way assertion would false-green on.
assert socket.getdefaulttimeout() == before
def test_timeout_get_restores_socket_default_when_the_delegate_raises() -> None:
def failing(url: str, credential: str | None) -> str:
raise SourceError("http GET failed", code="http_transport")
before = socket.getdefaulttimeout()
with pytest.raises(SourceError):
timeout_get(failing, timeout=7.5)("https://h.test/x", None)
# RED when the restore is not in a `finally`: a transport failure would leak the bound onto
# every socket the process opens afterwards.
assert socket.getdefaulttimeout() == before
# --- LOAD-BEARING: the DEFAULT transport is the bounded one --------------------------------------
def test_materialize_default_transport_is_bounded(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""No ``http_get`` injected → the real socket path must still be reached through the wrapper.
The socket path itself is stubbed at the module global the wrapper delegates to, so this runs
end-to-end through the unchanged library without opening a socket. RED when ``materialize``
goes back to passing the library a bare ``None``: the library then resolves its own unwrapped
``urllib_get`` and the recorded timeout is the untouched process default.
"""
observed: list[float | None] = []
def stub_urllib_get(url: str, credential: str | None) -> str:
observed.append(socket.getdefaulttimeout())
return "service ok"
monkeypatch.setattr("portfolio_optimiser.ingest._urllib_get", stub_urllib_get)
written = materialize(
_http_manifest(tmp_path),
tmp_path / "b",
ingested_at=_INGESTED_AT,
allow_network=True,
# NOTE: no http_get= — this is the default-path assertion.
)
assert any(p.name == "ingest-status.md" for p in written)
assert observed == [HTTP_TIMEOUT_SECONDS]
def test_materialize_does_not_wrap_an_injected_transport(tmp_path: Path) -> None:
"""A caller-supplied transport keeps its own timeout policy (the conditional control).
``mcp_get`` fronts a subprocess, not a socket, and already carries its own
``timeout_seconds``; canned test stubs touch nothing. Imposing a process-global socket bound on
caller-owned code would be a side effect we do not own. RED if the wrapping is made
unconditional — which would also false-green the default-path test above.
"""
observed: list[float | None] = []
before = socket.getdefaulttimeout()
def injected(url: str, credential: str | None) -> str:
observed.append(socket.getdefaulttimeout())
return "service ok"
materialize(
_http_manifest(tmp_path),
tmp_path / "b",
ingested_at=_INGESTED_AT,
allow_network=True,
http_get=injected,
)
assert observed == [before]
# --- LOAD-BEARING: the bound actually bites on a real blocked connection -------------------------
def test_bounded_transport_fails_fast_against_a_hanging_loopback_server() -> None:
"""The only socket in this suite: a loopback listener that never answers.
It binds 127.0.0.1 on an ephemeral port and never calls ``accept``; the OS backlog still
completes the handshake, so the client connects and then blocks reading a response that never
comes — the exact shape of the hang S2.4 closes. No egress, no external host, no credential.
Detaching the scoping does not merely change a value here, it removes the only thing that ends
the read. The call therefore runs on a daemon thread with a join deadline: a detached seam
leaves the thread alive and fails the assertion, instead of hanging the suite forever.
"""
listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listener.bind(("127.0.0.1", 0))
listener.listen(1)
host, port = listener.getsockname()
before = socket.getdefaulttimeout()
outcome: list[Any] = []
def attempt() -> None:
try:
outcome.append(
timeout_get(_urllib_get, timeout=0.25)(f"http://{host}:{port}/hang", None)
)
except BaseException as exc: # noqa: BLE001 - the outcome IS the assertion subject
outcome.append(exc)
worker = threading.Thread(target=attempt, daemon=True)
worker.start()
worker.join(10.0)
try:
assert not worker.is_alive(), (
"unbounded: the GET was still blocked after 10s — the socket carried no timeout"
)
assert isinstance(outcome[0], SourceError)
# Typed, never a leaked OSError; the library's own transport categorisation rides through.
assert outcome[0].code == "http_transport"
finally:
listener.close()
assert socket.getdefaulttimeout() == before