portfolio-optimiser/tests/test_ingest_library_seam.py
Kjell Tore Guttormsen 0a11af74a4 refactor(ingest): adopt shared llm-ingestion-okf v0.3.1 behind a thin adapter
Door A (manifest -> connector -> deterministic materialization -> index) is no
longer implemented here. src/portfolio_optimiser/ingest.py becomes a thin
consumer seam over the shared library, git-pinned to v0.3.1 on the same Forgejo
channel portfolio-optimiser-claude uses. Net -626/+385; ingest.py 599 -> 145 lines.

shared/ingest-spec.md remains the normative spec: the library implements it, it
does not replace it. Spec changes continue to go via commons.

Acceptance criterion met and proven: all three golden bundles (file/sql/http)
are byte-exact before and after, including the idempotence re-run. examples/ and
shared/ carry ZERO modifications -- the fasit was not adjusted to fit.

The rejection set was verified equivalent, not assumed: all 22 malformations the
repo's pydantic models refused are refused by the library, with typed codes
(okf_type_reserved, credential_embedded, extraction_id_duplicate, ...).

Test rebinding (invariants preserved, vehicle changed): the library has zero
runtime dependencies by design, so pydantic is unavailable to it.
ManifestV1.model_validate(dict) -> load_manifest_bytes(bytes); ValidationError ->
ManifestError; model_fields -> dataclasses.fields; PathSecurityError ->
SourceError(path_escape); ValueError -> MaterializationError(ingested_at_invalid).
Tests now also pin the refusal `code`, the library's documented stability
contract -- a sharper assertion than "some validation error was raised".

Two accepted behavioural deltas, recorded rather than silently dropped:
- Title whitespace is stored verbatim instead of collapsed at validation, so the
  frontmatter title and the index label are no longer guaranteed identical for
  irregular whitespace. Both behaviours are spec-conformant (the spec is SILENT;
  the old one was a repo-local pinned decision). Queued as a commons-amendment
  candidate so both stacks pin the same answer. Goldens unaffected.
- The section 8 audit log moves to logger llm_ingestion_okf.materialize. Nothing
  in the repo consumed the old channel.
Also: the `type` discriminator is no longer a dataclass field, so the spec
cross-check asserts it explicitly -- without that line the swap would have
silently narrowed the test.

New tests/test_ingest_library_seam.py pins the seam itself: the restated section 5
stamp formula against the stamp the library actually writes (the one place the
adapter does not purely delegate, since v0.3.1 exposes no stamp helper), the
local-only allow_network default, the list[Path] unwrapping, and a guard that the
adapter never regrows local Door A machinery. All four verified RED when detached,
as were both golden regressions under a byte-level render mutation.

Door A is UNGATED: it calls no guard before writing to disk. Gating untrusted
content remains the caller's responsibility (guard wiring still planned).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B4jNN186eVqfe1x5DnTU6r
2026-07-20 07:47:55 +02:00

127 lines
5.6 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

"""Load-bearing checks on the consumer seam over ``llm-ingestion-okf`` (adopted 2026-07-20).
Door A is no longer implemented here — ``src/portfolio_optimiser/ingest.py`` is a thin adapter
over the shared library, so the spec §4§6 rules are covered by the library's own suite plus the
repo's golden regressions. What is NOT covered by either is the seam itself: the places where the
adapter restates something instead of delegating, and the boundary claims the adapter's docstring
makes. Those are pinned here, because a docstring that no test can falsify is decoration.
"""
from __future__ import annotations
import hashlib
import json
from pathlib import Path
from typing import Any
import pytest
from llm_ingestion_okf import NetworkGateError
from portfolio_optimiser import okf
from portfolio_optimiser.ingest import load_manifest, materialize, materialize_bundle
_INGESTED_AT = "2026-07-03T12:00:00Z"
_MANIFEST: dict[str, Any] = {
"manifest_version": 1,
"source": {"type": "file", "id": "prosjekt-arkiv", "root": "fixture"},
"bundle_summary": "Cost extracts from the project archive.",
"extractions": [
{
"id": "costs",
"title": "Project costs",
"query": "costs.csv",
"okf_type": "dataset",
"max_rows": 100,
}
],
}
def _project(tmp_path: Path, source: dict[str, Any] | None = None) -> tuple[Path, Path]:
fixture = tmp_path / "fixture"
fixture.mkdir()
(fixture / "costs.csv").write_bytes(b"item,cost_nok\nled-retrofit,120000\n")
data = json.loads(json.dumps(_MANIFEST))
if source is not None:
data["source"] = source
manifest_path = tmp_path / "manifest.json"
manifest_path.write_text(json.dumps(data), encoding="utf-8")
return manifest_path, tmp_path / "bundle"
def test_adapter_stamp_equals_library_stamp(tmp_path: Path) -> None:
"""LOAD-BEARING ANTI-DRIFT: the adapter's ``load_manifest`` restates the §5 stamp formula
because the library (v0.3.1) mints the stamp inside ``materialize_bundle`` and exposes no
stamp helper. That is the ONE place the adapter is not purely delegating, so the two
formulas can drift apart silently — this compares the adapter's value against the stamp the
library actually writes into ``ingest_manifest`` frontmatter. RED the moment either side
changes how the stamp is computed."""
manifest_path, bundle_dir = _project(tmp_path)
_, adapter_stamp = load_manifest(manifest_path)
result = materialize_bundle(manifest_path, bundle_dir, _INGESTED_AT)
assert adapter_stamp == result.stamp
# ...and the stamp is what actually landed on disk, not merely an agreeing computation.
written_stamp = okf.parse_frontmatter(result.written[0])["ingest_manifest"]
assert adapter_stamp == written_stamp
# §5 shape, pinned independently of both implementations.
expected = "manifest@" + hashlib.sha256(manifest_path.read_bytes()).hexdigest()[:16]
assert adapter_stamp == expected
def test_adapter_returns_written_paths_in_extraction_order(tmp_path: Path) -> None:
"""The adapter keeps the repo's historical ``list[Path]`` return over the library's
``IngestResult``. RED if the unwrapping is dropped or reordered."""
manifest_path, bundle_dir = _project(tmp_path)
written = materialize(manifest_path, bundle_dir, ingested_at=_INGESTED_AT)
assert isinstance(written, list)
assert [p.name for p in written] == ["ingest-costs.md"]
assert all(p.is_file() for p in written)
def test_local_only_default_holds_at_the_adapter_seam(tmp_path: Path) -> None:
"""LOAD-BEARING BOUNDARY (§8): the adapter's docstring claims an ``http`` source is refused
unless a run explicitly opts in — the manifest can never grant itself network. The library
owns the gate, but the adapter owns the DEFAULT it is called with. RED if the adapter ever
starts passing ``allow_network=True`` (or forwards a manifest-derived value)."""
manifest_path, bundle_dir = _project(
tmp_path, source={"type": "http", "id": "api", "base_url": "https://host/api"}
)
with pytest.raises(NetworkGateError) as exc:
materialize(manifest_path, bundle_dir, ingested_at=_INGESTED_AT)
assert exc.value.code == "network_opt_in_missing"
assert not bundle_dir.exists(), "a refused network source must write NOTHING"
# The opt-in is reachable, so the refusal above is a real default rather than a dead path.
calls: list[str] = []
def fake_get(url: str, credential: str | None) -> str:
calls.append(url)
return "ok\n"
materialize(
manifest_path,
bundle_dir,
ingested_at=_INGESTED_AT,
allow_network=True,
http_get=fake_get,
)
assert calls == ["https://host/api/costs.csv"]
def test_adapter_does_not_reimplement_door_a(tmp_path: Path) -> None:
"""The adoption's point: Door A improves in ONE place. This pins the adapter as thin — it
must not regrow a local connector/renderer/materializer. RED if the module starts carrying
the machinery it delegates (csv/sqlite/urllib reading, table escaping, frontmatter
rendering), which is how a 'temporary local fix' silently forks the shared implementation."""
source = (
Path(__file__).resolve().parents[1] / "src" / "portfolio_optimiser" / "ingest.py"
).read_text(encoding="utf-8")
for forbidden in ("import csv", "import sqlite3", "urlopen", "def render_table", "\\\\|"):
assert forbidden not in source, (
f"ingest.py reimplements Door A machinery ({forbidden!r}) — it must delegate to "
"llm-ingestion-okf, not fork it"
)