open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(connectors): add the http connector and wire the network gate

Browse source 
TDD step 7: read_http executes the optional http extension point behind
an injectable transport seam (urllib_get is the only socket path; the
suite runs socket-free against a mock). credential_ref resolves from the
environment at run time, fail-fast before any transport call; the secret
rides in the Authorization header only. Explicit single-slash URL join,
CRLF-to-LF normalization, LF line-count cap (missing trailing newline
still counts), and code-fence-marker rejection. materialize_bundle
renders the body as a verbatim fenced block; the §8 gate test asserts
zero transport calls without the opt-in.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
This commit is contained in:
Kjell Tore Guttormsen 2026-07-16 20:06:21 +02:00
commit ab1fa4d999
3 changed files with 242 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

24
src/llm_ingestion_okf/materialize.py
View file

@ -15,8 +15,8 @@ import re
from dataclasses import dataclass
from pathlib import Path
from .connectors import read_csv, read_sql, safe_resolve
from .errors import ManifestError, MaterializationError, NetworkGateError, SourceError
from .connectors import HttpGet, read_csv, read_http, read_sql, safe_resolve, urllib_get
from .errors import ManifestError, MaterializationError, NetworkGateError
from .manifest import (
Extraction,
FileSource,
@ -26,7 +26,7 @@ from .manifest import (
generated_filename,
load_manifest_bytes,
)
from .render import render_table
from .render import render_fenced_block, render_table
_LOGGER = logging.getLogger(__name__)
@ -148,6 +148,7 @@ def materialize_bundle(
ingested_at: str,
*,
allow_network: bool = False,
http_get: HttpGet | None = None,
) -> IngestResult:
"""Materialize a manifest's extractions into an OKF bundle (§5).
@ -155,7 +156,9 @@ def materialize_bundle(
verbatim) — no wall-clock default; this is what makes golden extractions
bit-deterministic. `allow_network` is the §8 per-run network opt-in: an
`http` source is refused fail-fast unless it is set — the manifest itself
cannot grant network access. Source calls are logged per §8 (which
cannot grant network access. `http_get` optionally injects the transport
seam (default urllib_get, the only socket path; ignored for `file`/`sql`)
so tests run socket-free (§11). Source calls are logged per §8 (which
source, when, row count) — never cell contents, never secrets.
"""
if not _INGESTED_AT_RE.match(ingested_at):
@ -206,8 +209,17 @@ def materialize_bundle(
)
body = render_table(header, rows)
row_count = len(rows)
else: # HttpSource — transport lands with the http connector step.
raise SourceError("http transport is not implemented yet")
else: # HttpSource — the gate above guarantees allow_network here.
get = http_get if http_get is not None else urllib_get
text = read_http(
source.base_url,
extraction.query,
max_rows=extraction.max_rows,
credential_ref=source.credential_ref,
get=get,
)
body = render_fenced_block(text) # verbatim, NOT table-escaped
row_count = len(text.splitlines())
_LOGGER.info(
"source call: source=%s ingested_at=%s rows=%d", source.id, ingested_at, row_count
)