refactor(ingest): adopt shared llm-ingestion-okf v0.3.1 behind a thin adapter

Door A (manifest -> connector -> deterministic materialization -> index) is no
longer implemented here. src/portfolio_optimiser/ingest.py becomes a thin
consumer seam over the shared library, git-pinned to v0.3.1 on the same Forgejo
channel portfolio-optimiser-claude uses. Net -626/+385; ingest.py 599 -> 145 lines.

shared/ingest-spec.md remains the normative spec: the library implements it, it
does not replace it. Spec changes continue to go via commons.

Acceptance criterion met and proven: all three golden bundles (file/sql/http)
are byte-exact before and after, including the idempotence re-run. examples/ and
shared/ carry ZERO modifications -- the fasit was not adjusted to fit.

The rejection set was verified equivalent, not assumed: all 22 malformations the
repo's pydantic models refused are refused by the library, with typed codes
(okf_type_reserved, credential_embedded, extraction_id_duplicate, ...).

Test rebinding (invariants preserved, vehicle changed): the library has zero
runtime dependencies by design, so pydantic is unavailable to it.
ManifestV1.model_validate(dict) -> load_manifest_bytes(bytes); ValidationError ->
ManifestError; model_fields -> dataclasses.fields; PathSecurityError ->
SourceError(path_escape); ValueError -> MaterializationError(ingested_at_invalid).
Tests now also pin the refusal `code`, the library's documented stability
contract -- a sharper assertion than "some validation error was raised".

Two accepted behavioural deltas, recorded rather than silently dropped:
- Title whitespace is stored verbatim instead of collapsed at validation, so the
  frontmatter title and the index label are no longer guaranteed identical for
  irregular whitespace. Both behaviours are spec-conformant (the spec is SILENT;
  the old one was a repo-local pinned decision). Queued as a commons-amendment
  candidate so both stacks pin the same answer. Goldens unaffected.
- The section 8 audit log moves to logger llm_ingestion_okf.materialize. Nothing
  in the repo consumed the old channel.
Also: the `type` discriminator is no longer a dataclass field, so the spec
cross-check asserts it explicitly -- without that line the swap would have
silently narrowed the test.

New tests/test_ingest_library_seam.py pins the seam itself: the restated section 5
stamp formula against the stamp the library actually writes (the one place the
adapter does not purely delegate, since v0.3.1 exposes no stamp helper), the
local-only allow_network default, the list[Path] unwrapping, and a guard that the
adapter never regrows local Door A machinery. All four verified RED when detached,
as were both golden regressions under a byte-level render mutation.

Door A is UNGATED: it calls no guard before writing to disk. Gating untrusted
content remains the caller's responsibility (guard wiring still planned).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B4jNN186eVqfe1x5DnTU6r
This commit is contained in:
Kjell Tore Guttormsen 2026-07-20 07:47:55 +02:00
commit 0a11af74a4
9 changed files with 385 additions and 626 deletions

View file

@ -0,0 +1,127 @@
"""Load-bearing checks on the consumer seam over ``llm-ingestion-okf`` (adopted 2026-07-20).
Door A is no longer implemented here ``src/portfolio_optimiser/ingest.py`` is a thin adapter
over the shared library, so the spec §4§6 rules are covered by the library's own suite plus the
repo's golden regressions. What is NOT covered by either is the seam itself: the places where the
adapter restates something instead of delegating, and the boundary claims the adapter's docstring
makes. Those are pinned here, because a docstring that no test can falsify is decoration.
"""
from __future__ import annotations
import hashlib
import json
from pathlib import Path
from typing import Any
import pytest
from llm_ingestion_okf import NetworkGateError
from portfolio_optimiser import okf
from portfolio_optimiser.ingest import load_manifest, materialize, materialize_bundle
_INGESTED_AT = "2026-07-03T12:00:00Z"
_MANIFEST: dict[str, Any] = {
"manifest_version": 1,
"source": {"type": "file", "id": "prosjekt-arkiv", "root": "fixture"},
"bundle_summary": "Cost extracts from the project archive.",
"extractions": [
{
"id": "costs",
"title": "Project costs",
"query": "costs.csv",
"okf_type": "dataset",
"max_rows": 100,
}
],
}
def _project(tmp_path: Path, source: dict[str, Any] | None = None) -> tuple[Path, Path]:
fixture = tmp_path / "fixture"
fixture.mkdir()
(fixture / "costs.csv").write_bytes(b"item,cost_nok\nled-retrofit,120000\n")
data = json.loads(json.dumps(_MANIFEST))
if source is not None:
data["source"] = source
manifest_path = tmp_path / "manifest.json"
manifest_path.write_text(json.dumps(data), encoding="utf-8")
return manifest_path, tmp_path / "bundle"
def test_adapter_stamp_equals_library_stamp(tmp_path: Path) -> None:
"""LOAD-BEARING ANTI-DRIFT: the adapter's ``load_manifest`` restates the §5 stamp formula
because the library (v0.3.1) mints the stamp inside ``materialize_bundle`` and exposes no
stamp helper. That is the ONE place the adapter is not purely delegating, so the two
formulas can drift apart silently this compares the adapter's value against the stamp the
library actually writes into ``ingest_manifest`` frontmatter. RED the moment either side
changes how the stamp is computed."""
manifest_path, bundle_dir = _project(tmp_path)
_, adapter_stamp = load_manifest(manifest_path)
result = materialize_bundle(manifest_path, bundle_dir, _INGESTED_AT)
assert adapter_stamp == result.stamp
# ...and the stamp is what actually landed on disk, not merely an agreeing computation.
written_stamp = okf.parse_frontmatter(result.written[0])["ingest_manifest"]
assert adapter_stamp == written_stamp
# §5 shape, pinned independently of both implementations.
expected = "manifest@" + hashlib.sha256(manifest_path.read_bytes()).hexdigest()[:16]
assert adapter_stamp == expected
def test_adapter_returns_written_paths_in_extraction_order(tmp_path: Path) -> None:
"""The adapter keeps the repo's historical ``list[Path]`` return over the library's
``IngestResult``. RED if the unwrapping is dropped or reordered."""
manifest_path, bundle_dir = _project(tmp_path)
written = materialize(manifest_path, bundle_dir, ingested_at=_INGESTED_AT)
assert isinstance(written, list)
assert [p.name for p in written] == ["ingest-costs.md"]
assert all(p.is_file() for p in written)
def test_local_only_default_holds_at_the_adapter_seam(tmp_path: Path) -> None:
"""LOAD-BEARING BOUNDARY (§8): the adapter's docstring claims an ``http`` source is refused
unless a run explicitly opts in the manifest can never grant itself network. The library
owns the gate, but the adapter owns the DEFAULT it is called with. RED if the adapter ever
starts passing ``allow_network=True`` (or forwards a manifest-derived value)."""
manifest_path, bundle_dir = _project(
tmp_path, source={"type": "http", "id": "api", "base_url": "https://host/api"}
)
with pytest.raises(NetworkGateError) as exc:
materialize(manifest_path, bundle_dir, ingested_at=_INGESTED_AT)
assert exc.value.code == "network_opt_in_missing"
assert not bundle_dir.exists(), "a refused network source must write NOTHING"
# The opt-in is reachable, so the refusal above is a real default rather than a dead path.
calls: list[str] = []
def fake_get(url: str, credential: str | None) -> str:
calls.append(url)
return "ok\n"
materialize(
manifest_path,
bundle_dir,
ingested_at=_INGESTED_AT,
allow_network=True,
http_get=fake_get,
)
assert calls == ["https://host/api/costs.csv"]
def test_adapter_does_not_reimplement_door_a(tmp_path: Path) -> None:
"""The adoption's point: Door A improves in ONE place. This pins the adapter as thin — it
must not regrow a local connector/renderer/materializer. RED if the module starts carrying
the machinery it delegates (csv/sqlite/urllib reading, table escaping, frontmatter
rendering), which is how a 'temporary local fix' silently forks the shared implementation."""
source = (
Path(__file__).resolve().parents[1] / "src" / "portfolio_optimiser" / "ingest.py"
).read_text(encoding="utf-8")
for forbidden in ("import csv", "import sqlite3", "urlopen", "def render_table", "\\\\|"):
assert forbidden not in source, (
f"ingest.py reimplements Door A machinery ({forbidden!r}) — it must delegate to "
"llm-ingestion-okf, not fork it"
)

View file

@ -40,7 +40,7 @@ from pathlib import Path
from typing import Any
import pytest
from pydantic import ValidationError
from llm_ingestion_okf import ManifestError
from portfolio_optimiser import okf
from portfolio_optimiser.ingest import Extraction, materialize
@ -104,8 +104,9 @@ def test_verdict_typed_manifest_is_rejected_and_writes_nothing(tmp_path: Path) -
manifest_path = _write_project(tmp_path, extractions)
bundle_dir = tmp_path / "bundle"
with pytest.raises(ValidationError):
with pytest.raises(ManifestError) as exc:
materialize(manifest_path, bundle_dir, ingested_at=_INGESTED_AT)
assert exc.value.code == "okf_type_reserved"
assert not bundle_dir.exists(), (
"a verdict-typed manifest must write NOTHING (no index, no files)"

View file

@ -4,8 +4,16 @@ Every malformed manifest raises at validation, BEFORE any source access (the sta
discipline of method spec §10 / ingest spec §4 and §9's technical gate). The verdict-layer
reservation (okf_type != verdict, case-insensitive) is enforced here at the contract, never
downstream closing session-plan key assumption 2 (fail-fast manifest validation without
network). Pattern: tests/test_contracts.py (inline dict constants + pytest.raises per
malformation; fail-fast ordering proof mirrors test_no_chat_client_call_on_malformed_contract).
network).
Since the ``llm-ingestion-okf`` adoption (2026-07-20) the contract is enforced by the shared
library rather than by repo-local pydantic models. The INVARIANTS are unchanged every
malformation rejected before was verified to still be rejected but the assertion vehicle
moved: ``ManifestV1.model_validate(dict)`` ``load_manifest_bytes(bytes)``, and
``pydantic.ValidationError`` ``ManifestError``. The library has zero runtime dependencies
BY DESIGN, so pydantic is not available to it. These tests now also pin the refusal ``code``,
which is the library's documented stability contract (the message text explicitly is not) —
a strictly sharper assertion than "some validation error was raised".
"""
import copy
@ -15,7 +23,8 @@ from pathlib import Path
from typing import Any
import pytest
from pydantic import ValidationError
from llm_ingestion_okf import FileSource, HttpSource, ManifestError, SqlSource
from llm_ingestion_okf.manifest import load_manifest_bytes
from portfolio_optimiser.ingest import ManifestV1, load_manifest
@ -54,11 +63,19 @@ def _variant(**overrides: Any) -> dict[str, Any]:
return data
def _validate(data: dict[str, Any]) -> ManifestV1:
"""In-memory validation without touching disk — the library's equivalent of the pydantic
``model_validate`` these tests used before the adoption."""
return load_manifest_bytes(json.dumps(data).encode("utf-8"))
def test_valid_file_manifest_loads_with_stamp(tmp_path: Path) -> None:
path = _write(tmp_path, _MANIFEST)
manifest, stamp = load_manifest(path)
assert isinstance(manifest, ManifestV1)
assert manifest.source.type == "file"
# The library's source models drop the `type` discriminator as a FIELD (it is consumed by
# validation dispatch), so the variant is identified by class rather than by `.type`.
assert isinstance(manifest.source, FileSource)
assert [e.id for e in manifest.extractions] == ["costs", "meta"]
# §5: stamp = {stem}@{first 16 hex of SHA-256 over the manifest file's RAW bytes}.
expected = "manifest@" + hashlib.sha256(path.read_bytes()).hexdigest()[:16]
@ -69,59 +86,62 @@ def test_each_missing_top_level_field_raises(tmp_path: Path) -> None:
for field in ("manifest_version", "source", "bundle_summary", "extractions"):
data = copy.deepcopy(_MANIFEST)
del data[field]
with pytest.raises(ValidationError):
with pytest.raises(ManifestError):
load_manifest(_write(tmp_path, data, name=f"missing-{field}.json"))
def test_manifest_version_other_than_1_rejected(tmp_path: Path) -> None:
with pytest.raises(ValidationError):
with pytest.raises(ManifestError) as exc:
load_manifest(_write(tmp_path, _variant(manifest_version=2)))
assert exc.value.code == "manifest_version_unsupported"
def test_empty_extractions_rejected(tmp_path: Path) -> None:
with pytest.raises(ValidationError):
with pytest.raises(ManifestError):
load_manifest(_write(tmp_path, _variant(extractions=[])))
def test_bad_id_grammar_rejected(tmp_path: Path) -> None:
def test_bad_id_grammar_rejected() -> None:
# §4 grammar for source.id and extraction.id: ^[a-z0-9][a-z0-9-]*$
for bad in ("Upper", "-leading", "", "space id", "æøå"):
data = copy.deepcopy(_MANIFEST)
data["source"]["id"] = bad
with pytest.raises(ValidationError):
ManifestV1.model_validate(data)
with pytest.raises(ManifestError):
_validate(data)
data = copy.deepcopy(_MANIFEST)
data["extractions"][0]["id"] = bad
with pytest.raises(ValidationError):
ManifestV1.model_validate(data)
with pytest.raises(ManifestError):
_validate(data)
def test_duplicate_extraction_ids_rejected(tmp_path: Path) -> None:
data = copy.deepcopy(_MANIFEST)
data["extractions"][1]["id"] = data["extractions"][0]["id"]
with pytest.raises(ValidationError):
with pytest.raises(ManifestError) as exc:
load_manifest(_write(tmp_path, data))
assert exc.value.code == "extraction_id_duplicate"
def test_nonpositive_max_rows_rejected(tmp_path: Path) -> None:
for bad in (0, -1):
data = copy.deepcopy(_MANIFEST)
data["extractions"][0]["max_rows"] = bad
with pytest.raises(ValidationError):
with pytest.raises(ManifestError):
load_manifest(_write(tmp_path, data, name=f"rows-{bad}.json"))
def test_unknown_source_type_rejected(tmp_path: Path) -> None:
data = copy.deepcopy(_MANIFEST)
data["source"] = {"type": "ftp", "id": "x", "root": "fixture"}
with pytest.raises(ValidationError):
with pytest.raises(ManifestError) as exc:
load_manifest(_write(tmp_path, data))
assert exc.value.code == "source_type_unknown"
def test_file_source_missing_root_rejected(tmp_path: Path) -> None:
data = copy.deepcopy(_MANIFEST)
del data["source"]["root"]
with pytest.raises(ValidationError):
with pytest.raises(ManifestError):
load_manifest(_write(tmp_path, data))
@ -132,46 +152,57 @@ def test_verdict_okf_type_rejected_case_insensitively(tmp_path: Path) -> None:
for spelling in ("verdict", "Verdict", "VERDICT"):
data = copy.deepcopy(_MANIFEST)
data["extractions"][0]["okf_type"] = spelling
with pytest.raises(ValidationError):
with pytest.raises(ManifestError) as exc:
load_manifest(_write(tmp_path, data, name=f"verdict-{spelling}.json"))
assert exc.value.code == "okf_type_reserved"
def test_multiline_title_rejected(tmp_path: Path) -> None:
def test_multiline_title_rejected() -> None:
for bad in ("line1\nline2", "line1\rline2", ""):
data = copy.deepcopy(_MANIFEST)
data["extractions"][0]["title"] = bad
with pytest.raises(ValidationError):
ManifestV1.model_validate(data)
with pytest.raises(ManifestError):
_validate(data)
def test_title_whitespace_normalized() -> None:
# Pinned decision: title runs are collapsed at validation so the frontmatter rendering
# (okf.render_frontmatter collapses runs) and the index label are guaranteed identical.
def test_title_whitespace_is_preserved_verbatim() -> None:
"""ACCEPTED DIVERGENCE (2026-07-20), recorded rather than silently dropped.
The repo previously COLLAPSED title whitespace runs at validation, so the frontmatter
title and the index label were guaranteed byte-identical. The library stores the title
verbatim instead: ``_render_frontmatter`` still collapses runs when writing frontmatter,
but the §6 index label is written raw so for a title with irregular internal whitespace
the two now differ. Both behaviours are spec-conformant: ``shared/ingest-spec.md`` is
SILENT on normalization, and the old behaviour was a repo-local pinned decision, not a
spec requirement.
This test pins the CURRENT behaviour so the divergence cannot drift unnoticed. It goes RED
if the library starts normalizing which is the desired end state, and is why the point is
queued as a commons-amendment candidate so both stacks pin the same answer in the spec.
The golden bundles are unaffected (their titles carry no irregular whitespace)."""
data = copy.deepcopy(_MANIFEST)
data["extractions"][0]["title"] = " Project costs "
manifest = ManifestV1.model_validate(data)
assert manifest.extractions[0].title == "Project costs"
manifest = _validate(data)
assert manifest.extractions[0].title == " Project costs "
def test_base_url_embedded_credentials_rejected() -> None:
# §4: base_url MUST NOT embed credentials (userinfo is the URL credential mechanism).
data = copy.deepcopy(_MANIFEST)
data["source"] = {"type": "http", "id": "api", "base_url": "https://user:pw@host/api"}
with pytest.raises(ValidationError):
ManifestV1.model_validate(data)
with pytest.raises(ManifestError) as exc:
_validate(data)
assert exc.value.code == "credential_embedded"
def test_sql_and_http_variants_validate() -> None:
# Schema breadth (brief assumption 1): the polymorphic §4 schema validates all three
# source variants; the `file` and `sql` connectors EXECUTE (I2, I4), http → I6.
sql = ManifestV1.model_validate(
_variant(source={"type": "sql", "id": "db", "connection_ref": "PROJ_DB"})
)
assert sql.source.type == "sql"
http = ManifestV1.model_validate(
_variant(source={"type": "http", "id": "api", "base_url": "https://host/api"})
)
assert http.source.type == "http"
sql = _validate(_variant(source={"type": "sql", "id": "db", "connection_ref": "PROJ_DB"}))
assert isinstance(sql.source, SqlSource)
assert sql.source.connection_ref == "PROJ_DB"
http = _validate(_variant(source={"type": "http", "id": "api", "base_url": "https://host/api"}))
assert isinstance(http.source, HttpSource)
assert http.source.credential_ref is None
@ -183,13 +214,16 @@ def test_failfast_before_source_access(tmp_path: Path) -> None:
data = copy.deepcopy(_MANIFEST)
data["source"]["root"] = str(tmp_path / "does-not-exist")
data["extractions"][0]["max_rows"] = 0
with pytest.raises(ValidationError):
with pytest.raises(ManifestError):
load_manifest(_write(tmp_path, data))
assert not (tmp_path / "does-not-exist").exists()
def test_malformed_json_raises(tmp_path: Path) -> None:
# The library wraps json decoding so EVERY manifest problem surfaces as one typed family
# (previously this leaked a raw json.JSONDecodeError to the caller).
path = tmp_path / "broken.json"
path.write_text("{not json", encoding="utf-8")
with pytest.raises(json.JSONDecodeError):
with pytest.raises(ManifestError) as exc:
load_manifest(path)
assert exc.value.code == "manifest_invalid_json"

View file

@ -19,9 +19,10 @@ from typing import Any
import pytest
from llm_ingestion_okf import MaterializationError, SourceError
from portfolio_optimiser import okf
from portfolio_optimiser.ingest import IngestError, materialize, read_csv, render_table
from portfolio_optimiser.retrieval import PathSecurityError
_INGESTED_AT = "2026-07-03T12:00:00Z"
@ -109,8 +110,11 @@ def test_max_rows_cap_is_an_error_not_truncation(tmp_path: Path) -> None:
def test_path_escape_raises(tmp_path: Path) -> None:
root = _catalogue(tmp_path, {"data.csv": b"a\n"})
(tmp_path / "outside.csv").write_bytes(b"a\n1\n")
with pytest.raises(PathSecurityError):
# Fail-closed containment is unchanged; the library raises its own typed refusal
# (SourceError, code="path_escape") where the repo-local seam raised PathSecurityError.
with pytest.raises(SourceError) as exc:
_read(root, query="../outside.csv")
assert exc.value.code == "path_escape"
def test_missing_root_raises_ingest_error(tmp_path: Path) -> None:
@ -206,13 +210,17 @@ def test_generated_file_is_lf_only_with_one_trailing_newline(tmp_path: Path) ->
assert data.endswith(b"\n") and not data.endswith(b"\n\n")
def test_invalid_ingested_at_raises_value_error(tmp_path: Path) -> None:
def test_invalid_ingested_at_is_refused(tmp_path: Path) -> None:
manifest_path, bundle_dir = _project(tmp_path)
# §5 format is ISO-8601 UTC with a Z suffix — validated by regex (datetime.fromisoformat
# rejects 'Z' on Python 3.10, the repo floor).
# rejects 'Z' on Python 3.10, the repo floor). The refusal is unchanged; its type moved
# from ValueError to the library's MaterializationError.
for bad in ("2026-07-03 12:00:00", "2026-07-03T12:00:00+00:00", "2026-07-03", ""):
with pytest.raises(ValueError):
with pytest.raises(MaterializationError) as exc:
materialize(manifest_path, bundle_dir, ingested_at=bad)
assert exc.value.code == "ingested_at_invalid"
# The refusal is fail-fast: nothing was written before the format was checked.
assert not bundle_dir.exists()
def test_materialize_creates_nonexistent_nested_bundle_dir(tmp_path: Path) -> None:
@ -226,11 +234,16 @@ def test_source_call_logged_with_id_timestamp_rowcount(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
manifest_path, bundle_dir = _project(tmp_path)
with caplog.at_level(logging.INFO, logger="portfolio_optimiser.ingest"):
# The §8 audit log is still emitted per source call, but the CHANNEL moved with the
# implementation: "portfolio_optimiser.ingest" -> "llm_ingestion_okf.materialize"
# (accepted 2026-07-20; nothing in the repo consumed the old logger name).
with caplog.at_level(logging.INFO, logger="llm_ingestion_okf.materialize"):
materialize(manifest_path, bundle_dir, ingested_at=_INGESTED_AT)
# §8: which source, when (the deterministic ingested_at argument), row count.
joined = " ".join(record.getMessage() for record in caplog.records)
assert "prosjekt-arkiv" in joined and _INGESTED_AT in joined and "rows=2" in joined
# §8 also bounds what may be logged: never cell contents.
assert "led-retrofit" not in joined
def test_two_runs_with_identical_inputs_are_byte_identical(tmp_path: Path) -> None:

View file

@ -12,6 +12,7 @@ drifts away from what the spec documents.
from __future__ import annotations
import dataclasses
import json
import re
from pathlib import Path
@ -187,7 +188,7 @@ def test_ingest_spec_documents_every_contract_field() -> None:
def documented(field: str, source: str) -> None:
assert f"`{field}`" in text, f"ingest spec does not document {source} field `{field}`"
for field in ManifestV1.model_fields:
for field in (f.name for f in dataclasses.fields(ManifestV1)):
documented(field, "manifest top-level")
for model, source in (
(FileSource, "file source"),
@ -195,8 +196,13 @@ def test_ingest_spec_documents_every_contract_field() -> None:
(HttpSource, "http source"),
(Extraction, "extraction"),
):
for field in model.model_fields:
for field in (f.name for f in dataclasses.fields(model)):
documented(field, source)
# The library's source models consume the `type` discriminator during validation dispatch
# instead of storing it as a field, so it is no longer reachable by introspection. It is a
# REAL §4 contract field, so it is asserted explicitly — without this line the swap from
# `model_fields` to `dataclasses.fields` would silently drop it from the cross-check.
documented("type", "source discriminator")
# The §5/§7 provenance layer — exactly the keys the materializer stamps.
for key in (