open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 1 Activity

feat(ingest): bound the default http transport in time, in front of the pinned library (S2.4)

Browse source 
The hole: `read_http`'s default transport is the library's `urllib_get`, which invokes the
stdlib opener with no `timeout=`. urllib's documented fallback is then the process-wide default
socket timeout — `None` out of the box — so an http source that accepts a connection and never
answers hangs a run indefinitely. That contradicts the invariant that nothing runs unbounded.

The spec text for S2.4 ("a timeout parameter on `_urllib_get`") could NOT be followed literally:
that function is UPSTREAM library code (`llm_ingestion_okf.connectors`, pinned v0.3.1, pull-only),
signature `(url, credential) -> str` — measured, not assumed. Same failure class as S2.2's
"implement it in `ingest.py`": spec text that says "change X" has to be checked against whether
X is ours at all.

So the fix goes in FRONT of the library: `timeout_get` scopes `socket.setdefaulttimeout` around
a delegate call to the library's own `urllib_get`, and `materialize` now hands the library that
wrapped transport instead of letting it resolve its own untimed default. This meets S2.4's own
verification criterion — a bound WITHOUT a second socket path — and avoids duplicating the
credential-header logic. An explicitly injected `http_get` is passed through UNWRAPPED: a
caller-owned transport (MCP fronts a subprocess with its own `timeout_seconds`) keeps its own
policy, and a process-global side effect is not ours to impose on it.

Honest limit, carried in the code comment, the test docstring and `docs/extending.md`, not just
in the commit: the default socket timeout is PROCESS-global. Under `concurrency=k` the runner is
asyncio on one thread, so the scoping holds; driving `read_http` from a thread-pool executor
would make it unsafe.

Half of S2.4's scope was already delivered upstream — transport failures are categorised as
`SourceError(code="http_transport")`. Coarser than the plan envisaged, but not ours to rewrite.

Two pre-existing guards went red on the first pass, both on PROSE only: `ingest.py` must not
contain "urlopen" (no forked connector) or "ingest_mcp" (AST-guarded mcp-free). No code violated
either — my docstrings merely named them. The guards were left exactly as strict as they were and
the prose was reworded; weakening a real guard to save a comment is the trade this repo refuses.

578 -> 583 tests. Five mutations MEASURED red (restored from scratchpad + `shasum -c` each time,
never `git checkout`):
  1. remove the timeout scoping entirely            -> RED
  2. apply the bound AFTER the delegate call        -> RED
  3. set the bound but never restore it (no finally)-> RED  (the unconditional control)
  4. hand the library a bare None again (pre-S2.4)  -> RED  (the wiring)
  5. make the wrapping unconditional                -> RED  (the conditional control)

Mutations 1 and 2 take ~10s to fail rather than failing instantly: that is the loopback test's
join deadline expiring. It is the measurement that the bound actually BITES — a black-hole
listener on 127.0.0.1 that completes the handshake and never answers, run on a daemon thread so
a detached seam fails an assertion instead of hanging the suite forever. Every other assertion
here only proves we set a global; that one proves the global does something.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdLGwd33vqhkToh98Ym34P
This commit is contained in:
Kjell Tore Guttormsen 2026-08-03 14:54:04 +02:00
commit 8910a673ea
3 changed files with 292 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

15
docs/extending.md
View file

@ -98,6 +98,21 @@ implementation (the default `_urllib_get` is the only socket path). The golden c
(`examples/ingest-golden-http/`) and every test inject a canned `get` over committed fixture
payloads, so the suite runs offline against a **local mock** — no live source, no credentials.
**The default transport is time-bounded (S2.4).** The library's `_urllib_get` calls the stdlib
opener without a timeout, which falls back to the process-wide default socket timeout — `None`
out of the box — so a source that accepts a connection and then never answers would hang a run,
contradicting the invariant that nothing runs unbounded. `materialize` therefore hands the library
that same socket path wrapped in `ingest.timeout_get`, which scopes `socket.setdefaulttimeout`
(`HTTP_TIMEOUT_SECONDS`, 30s) around the delegate call: the bound applies **without** a second
socket path and **without** duplicating the credential header, so the pinned library stays
untouched. An explicitly injected `http_get` is passed through **unwrapped** — a caller-supplied
transport owns its own timeout policy.
**Honest limit:** the default socket timeout is *process-global*. Under `concurrency=k` the runner
is asyncio on a single thread, so the scoping holds. Driving `read_http` from a thread-pool
executor would make it unsafe, and the bound would have to move to a per-call timeout argument —
i.e. to owning a socket path locally.
### D7 sibling hook — MCP as an extension of this family
The spec (§4) documents an **MCP-based connector as an extension of the `http` family**, not a new