diff --git a/docs/2026-07-04-I3-brief.md b/docs/2026-07-04-I3-brief.md new file mode 100644 index 0000000..bbbb3e2 --- /dev/null +++ b/docs/2026-07-04-I3-brief.md @@ -0,0 +1,69 @@ +# I3 — D7-speil av ingest (filkatalog/CSV) — brief + +> Forankret i hovedrepoets sesjonsplan +> `~/repos/portfolio-optimiser/docs/plan/2026-07-03-sesjonsplan-ingest.md` §I3 og dette +> repoets STATE. Bygget **fra `shared/ingest-spec.md` alene** (subtree-pull fra commons) — +> aldri ved reverse-engineering av MAF-koden. Golden-**bytene** (den delte fasiten) er lest, +> siden spec §11 gjør dem til konformanskravet begge stacker reproduserer. + +## Mål (§I3) + +Samme ingest-kontrakt konsumert **uendret** fra commons i dette søskenrepoet — beviser at +ingest-spec-en faktisk er delt: D7s uavhengige impl reproduserer MAF I2s golden **byte-identisk** +fra spec alene. Scope = `file`-kildetypen (lokal CSV-katalog). `sql` = senere (I5), `http` = +ikke i D7 (målbildet: HTTP/MCP demonstreres MAF-only, I6). + +## To beslutninger (operatør-avgjort denne økta) + +1. **Golden-fasit: repo-lokal kopi.** I2 la ingest-golden repo-lokalt (`examples/ingest-golden-file`, + ulikt method-golden i `shared/examples/`). D7 speiler mønsteret: kopi i dette repoets + `examples/ingest-golden-file/`, speiltest peker dit. Holder I3 som én lukket økt, ingen + commons-runde. (Alternativ — løfte golden til commons — avvist: egen GATET runde + avvik fra + I2-mønsteret.) + +2. **okf relaksert for frontmatterløs index (følg standarden).** Funn: den delte golden-ens + `index.md` har **ingen** frontmatter; D7s `okf.navigate_bundle` krevde det → kunne ikke både + reprodusere golden byte-identisk OG navigere den med uendret okf. **method-spec §3 Step 1 er + avgjørende:** «the rendered read-context is *the index body (the summary)* followed by each + *non-index concept file* as a `## {type}: {title}` section» — standarden behandler index via + sin *body*, ikke som en typet concept-fil. Golden-en er altså spec-konform; D7s okf var + **strengere enn standarden**. Fix: `okf` tolererer nå frontmatterløs index (hele fila = body, + `type` defaulter til `index`); frontmatter-ful kuratert index parses uendret; non-index + concept-filer krever fortsatt `type` (relaksasjonen er scoped, bevist av egen honesty-test). + (Alternativ — «typet index overalt» via commons-RFC + MAF golden-regen — avvist: omskriver + standarden for å romme en D7-quirk, endrer frossen fasit, flerrepo GATET.) + +## Leveranse + +- `examples/ingest-golden-file/` — repo-lokal golden (kopi av I2s, byte-frossen). +- `src/portfolio_optimiser_claude/ingest.py` — `ManifestContract` (pydantic, fail-fast, + `file`-kilde, verdict-reservasjon §3, id-grammatikk, `max_rows`>0), CSV-konnektor + (boundary-checked fail-closed), materialisering (§5-frontmatter i eksakt rekkefølge, + markdown-tabell m/ §5-escaping, LF-only, SHA-256 manifest-stamp), index-generering (§6), + replacement-semantikk (§3/§5). +- `src/…/okf.py` — `_parse_index_entry` (frontmatterløs index-toleranse, method-spec §3). +- Speiltester (I2s load-bearing-sett): `test_ingest_golden.py` (byte-fasit + 3 mutasjonskontroller), + `test_ingest_loadbearing.py` (provenance / navigability / verdict-reservasjon / re-ingest-safety), + `test_ingest.py` (kontrakt fail-fast, max_rows, boundary, kollisjon, escaping), + `test_ingest_spec_loadbearing.py` (spec-integritet §11: fil finnes · navngir ingen toolkit · + dokumenterer hvert kontrakt-felt) — D7-analogen til MAFs I1-guard. + +## SDK-kroker (STATE steg 4 — søk-først) + +I3 (filkatalog/CSV) introduserer **ingen SDK-flate**: ingest er deterministisk, null modellkall. +Søk-først SDK-krok-reverifisering (API-drift på `create_sdk_mcp_server`) er relevant for I6 +(HTTP/MCP), ikke I3 — utsatt dit kroken faktisk brukes. Eksisterende SDK-klient (`sdk_client.py`, +`test_sdk_isolation.py`) uendret og grønn i suiten. + +## Verifisering (kjørt, grønn) + +- Golden **byte-identisk** konsumert (5/5, inkl. idempotens + mutasjonskontroller). +- Speiltester **røde ved detach** — alle fire sømmer bevist load-bearing (detach-bevis-skript). +- Søskensuite **grønn uten nøkkel/nettverk**: 239 passed (189 + 50 nye). +- `ruff check` + `ruff format` rene · `mypy --strict src`: no issues (16 filer). + +## Åpent (forslag, ikke utført — utenfor I3-scope) + +- **Standard-evolusjon:** vil operatøren senere ha «alle bundle-filer er typede concept-filer» + (typet index overalt) som bevisst OKF-endring, er det en egen koordinert commons-RFC (begge + specs, begge golden, begge okf). Dagens permissive okf er forover-kompatibel med det. diff --git a/examples/ingest-golden-file/expected-bundle/index.md b/examples/ingest-golden-file/expected-bundle/index.md new file mode 100644 index 0000000..3c7583e --- /dev/null +++ b/examples/ingest-golden-file/expected-bundle/index.md @@ -0,0 +1,3 @@ +Golden extraction case: two CSV extracts from a local project archive (file source type). +- [Project costs](ingest-costs.md) +- [Escaping edge cases](ingest-edge.md) diff --git a/examples/ingest-golden-file/expected-bundle/ingest-costs.md b/examples/ingest-golden-file/expected-bundle/ingest-costs.md new file mode 100644 index 0000000..fa542d7 --- /dev/null +++ b/examples/ingest-golden-file/expected-bundle/ingest-costs.md @@ -0,0 +1,14 @@ +--- +type: dataset +title: Project costs +source_system: prosjekt-arkiv +source_query: costs.csv +ingested_at: 2026-07-03T12:00:00Z +ingest_manifest: manifest@24109f486b9ef708 +generated: true +--- + +| item | cost_nok | note | +| --- | --- | --- | +| led-retrofit | 120000 | ROI under 2 years | +| hvac-tuning | 80000 | requires vendor quote | diff --git a/examples/ingest-golden-file/expected-bundle/ingest-edge.md b/examples/ingest-golden-file/expected-bundle/ingest-edge.md new file mode 100644 index 0000000..47da3d7 --- /dev/null +++ b/examples/ingest-golden-file/expected-bundle/ingest-edge.md @@ -0,0 +1,18 @@ +--- +type: reference +title: Escaping edge cases +source_system: prosjekt-arkiv +source_query: edge.csv +ingested_at: 2026-07-03T12:00:00Z +ingest_manifest: manifest@24109f486b9ef708 +generated: true +--- + +| id | value | +| --- | --- | +| backslash | a\\b | +| pipe | a\|b | +| newline | x y | +| both | \\\| | +| zeros | 007 | +| decimal | 1.50 | diff --git a/examples/ingest-golden-file/fixture/costs.csv b/examples/ingest-golden-file/fixture/costs.csv new file mode 100644 index 0000000..0dbc341 --- /dev/null +++ b/examples/ingest-golden-file/fixture/costs.csv @@ -0,0 +1,3 @@ +item,cost_nok,note +led-retrofit,120000,ROI under 2 years +hvac-tuning,80000,requires vendor quote diff --git a/examples/ingest-golden-file/fixture/edge.csv b/examples/ingest-golden-file/fixture/edge.csv new file mode 100644 index 0000000..e3fb623 --- /dev/null +++ b/examples/ingest-golden-file/fixture/edge.csv @@ -0,0 +1,8 @@ +id,value +backslash,a\b +pipe,a|b +newline,"x +y" +both,\| +zeros,007 +decimal,1.50 diff --git a/examples/ingest-golden-file/ingested-at.txt b/examples/ingest-golden-file/ingested-at.txt new file mode 100644 index 0000000..9f82d47 --- /dev/null +++ b/examples/ingest-golden-file/ingested-at.txt @@ -0,0 +1 @@ +2026-07-03T12:00:00Z diff --git a/examples/ingest-golden-file/manifest.json b/examples/ingest-golden-file/manifest.json new file mode 100644 index 0000000..5c3a087 --- /dev/null +++ b/examples/ingest-golden-file/manifest.json @@ -0,0 +1,21 @@ +{ + "manifest_version": 1, + "source": {"type": "file", "id": "prosjekt-arkiv", "root": "fixture"}, + "bundle_summary": "Golden extraction case: two CSV extracts from a local project archive (file source type).", + "extractions": [ + { + "id": "costs", + "title": "Project costs", + "query": "costs.csv", + "okf_type": "dataset", + "max_rows": 10 + }, + { + "id": "edge", + "title": "Escaping edge cases", + "query": "edge.csv", + "okf_type": "reference", + "max_rows": 10 + } + ] +} diff --git a/src/portfolio_optimiser_claude/ingest.py b/src/portfolio_optimiser_claude/ingest.py new file mode 100644 index 0000000..d86b6e8 --- /dev/null +++ b/src/portfolio_optimiser_claude/ingest.py @@ -0,0 +1,299 @@ +"""Deterministic ingest layer: manifest → file/CSV connector → OKF bundle (ingest-spec). + +An addition IN FRONT of the loop (ingest-spec §1): a deterministic step that couples the +framework to a real data source and materializes the extract as an OKF knowledge bundle, +which the existing 8-step loop then consumes UNCHANGED. Zero model calls; no network. + +D7 scope (I3, mirror of MAF I2): the ``file`` source type (a local CSV catalogue). The +``sql`` and ``http`` source types are later work — a manifest naming them is rejected +fail-fast at validation, never silently accepted. + +Contract discipline mirrors ``contracts.py``: the manifest is schema-validated fail-fast +BEFORE any source call (§4), and queries are declarative configuration, never evaluated as +code. The provenance frontmatter layer (§7) is ITS OWN contract — separate from the method +spec's §9 proposal provenance, never mixed. +""" + +from __future__ import annotations + +import csv +import hashlib +import json +import logging +import re +from dataclasses import dataclass +from pathlib import Path +from typing import Literal + +from pydantic import BaseModel, Field, field_validator + +_logger = logging.getLogger(__name__) + +_ID_RE = re.compile(r"^[a-z0-9][a-z0-9-]*$") +_VERDICT_TYPE = "verdict" +_INGEST_PREFIX = "ingest-" +_STAMP_HASH_LEN = 16 +# An index cross-link owned by ingest: ``- [label](ingest-{id}.md)`` (§6). Only these +# are managed on re-materialization; curated and promoted links are preserved verbatim. +_INGEST_LINK_RE = re.compile(r"\]\((ingest-[a-z0-9][a-z0-9-]*\.md)\)") +_CROSSLINK_RE = re.compile(r"\]\(([^)]+\.md)\)") + +# Frontmatter keys, in the exact §5 order. +_FRONTMATTER_ORDER = ( + "type", + "title", + "source_system", + "source_query", + "ingested_at", + "ingest_manifest", + "generated", +) + + +class FileSource(BaseModel): + """A local file catalogue (``source.type == "file"``, ingest-spec §4).""" + + type: Literal["file"] + id: str + root: str + + @field_validator("id") + @classmethod + def _id_grammar(cls, value: str) -> str: + if not _ID_RE.match(value): + raise ValueError(f"source.id {value!r} must match [a-z0-9][a-z0-9-]*") + return value + + +class Extraction(BaseModel): + """One extraction description (ingest-spec §4). ``max_rows`` is a required cap (§8).""" + + id: str + title: str + query: str + okf_type: str + max_rows: int = Field(gt=0) + + @field_validator("id") + @classmethod + def _id_grammar(cls, value: str) -> str: + if not _ID_RE.match(value): + raise ValueError(f"extraction.id {value!r} must match [a-z0-9][a-z0-9-]*") + return value + + @field_validator("title") + @classmethod + def _title_single_line(cls, value: str) -> str: + if "\n" in value or "\r" in value: + raise ValueError("extraction.title must be single-line") + return value + + @field_validator("okf_type") + @classmethod + def _okf_type_not_verdict(cls, value: str) -> str: + # §3 verdict-layer reservation: the promotion gate is the ONLY path into the + # verdict layer — an ingest mapping to type: verdict would inject machine-made + # "approved" verdicts around the gate. Case-insensitive, fail-fast. + if value.lower() == _VERDICT_TYPE: + raise ValueError("okf_type MUST NOT be 'verdict' (verdict layer is reserved, §3)") + return value + + +class ManifestContract(BaseModel): + """The ingest manifest (§4) — schema-validated fail-fast before any source call.""" + + manifest_version: Literal[1] + source: FileSource + bundle_summary: str + extractions: list[Extraction] = Field(min_length=1) + + @field_validator("extractions") + @classmethod + def _unique_extraction_ids(cls, value: list[Extraction]) -> list[Extraction]: + ids = [e.id for e in value] + if len(ids) != len(set(ids)): + raise ValueError("extraction ids must be unique within the manifest") + return value + + +@dataclass(frozen=True) +class LoadedManifest: + """A validated manifest plus its provenance stamp and on-disk location.""" + + contract: ManifestContract + stamp: str # ``{manifest stem}@{sha256(raw bytes)[:16]}`` (§5) + path: Path + + +def load_manifest(manifest_path: Path) -> LoadedManifest: + """Validate the manifest fail-fast (§4) and compute its provenance stamp (§5). + + Raises ``pydantic.ValidationError`` on a malformed manifest — never starts a run on a + bad contract. The stamp is over the manifest's RAW bytes, so any edit re-stamps. + """ + raw = manifest_path.read_bytes() + contract = ManifestContract(**json.loads(raw)) + digest = hashlib.sha256(raw).hexdigest()[:_STAMP_HASH_LEN] + stamp = f"{manifest_path.stem}@{digest}" + return LoadedManifest(contract=contract, stamp=stamp, path=manifest_path) + + +def _escape_cell(value: str) -> str: + """Render a cell value: text verbatim with §5 escaping. + + Backslash FIRST (so the pipe escape it introduces is not re-escaped), then pipe, + then any newline collapsed to a single space. For the ``file`` source every cell is + a ``str`` — the integer/float/NULL typing rules (§5) belong to the ``sql`` source. + """ + escaped = value.replace("\\", "\\\\").replace("|", "\\|") + escaped = escaped.replace("\r\n", " ").replace("\r", " ").replace("\n", " ") + return escaped + + +def _render_table(rows: list[list[str]]) -> str: + """Header row, separator, data rows — GitHub-flavoured markdown table (§5).""" + header, *data = rows + lines = [ + "| " + " | ".join(_escape_cell(c) for c in header) + " |", + "| " + " | ".join("---" for _ in header) + " |", + ] + for row in data: + lines.append("| " + " | ".join(_escape_cell(c) for c in row) + " |") + return "\n".join(lines) + + +def _read_csv(csv_path: Path, max_rows: int) -> list[list[str]]: + """Read a CSV (first row = header). Enforce ``max_rows`` fail-fast on DATA rows (§8).""" + with csv_path.open(encoding="utf-8", newline="") as handle: + rows = list(csv.reader(handle)) + data_rows = len(rows) - 1 if rows else 0 + if data_rows > max_rows: + raise ValueError( + f"extraction exceeds max_rows: {data_rows} > {max_rows} for {csv_path.name}" + ) + return rows + + +def _resolve_within(root: Path, relative: str) -> Path: + """Resolve ``relative`` under ``root``, boundary-checked fail-closed (the OKF path rule).""" + resolved = (root / relative).resolve() + if not resolved.is_relative_to(root.resolve()): + raise ValueError(f"extraction path {relative!r} escapes the source root") + return resolved + + +def _frontmatter_block(values: dict[str, str]) -> str: + return "---\n" + "".join(f"{k}: {values[k]}\n" for k in _FRONTMATTER_ORDER) + "---" + + +def _concept_file( + extraction: Extraction, source_id: str, ingested_at: str, stamp: str, body: str +) -> str: + frontmatter = _frontmatter_block( + { + "type": extraction.okf_type, + "title": extraction.title, + "source_system": source_id, + "source_query": _escape_ws(extraction.query), + "ingested_at": ingested_at, + "ingest_manifest": stamp, + "generated": "true", + } + ) + return f"{frontmatter}\n\n{body}\n" + + +def _escape_ws(value: str) -> str: + """Collapse whitespace runs (incl. newlines) to single spaces (§5) — single-line values.""" + return " ".join(value.split()) + + +def _is_ingest_stamped(md_path: Path) -> bool: + """True iff the file carries the ingest stamp: ``generated: true`` + an ``ingest_manifest``.""" + frontmatter: dict[str, str] = {} + lines = md_path.read_text(encoding="utf-8").splitlines() + if not lines or lines[0].strip() != "---": + return False + for line in lines[1:]: + if line.strip() == "---": + break + key, sep, val = line.partition(":") + if sep: + frontmatter[key.strip()] = val.strip() + return frontmatter.get("generated") == "true" and "ingest_manifest" in frontmatter + + +def _update_index(bundle_dir: Path, bundle_summary: str, extractions: list[Extraction]) -> None: + """Create or update ``index.md`` (§6). + + A fresh index carries ``bundle_summary`` as its body; an existing index keeps every + line it does not manage byte for byte. Ingest cross-links (targets ``ingest-*.md``) + are the ONLY managed lines: stale ones are dropped and the current set re-appended in + extraction order, idempotently by target — curated and promoted links are preserved + (so a promoted verdict's index link survives re-ingest, load-bearing §11). + """ + index_path = bundle_dir / "index.md" + if index_path.is_file(): + kept = [ + line + for line in index_path.read_text(encoding="utf-8").splitlines() + if not _INGEST_LINK_RE.search(line) + ] + else: + kept = [bundle_summary] + present = {t for line in kept for t in _CROSSLINK_RE.findall(line)} + lines = list(kept) + for extraction in extractions: + target = f"{_INGEST_PREFIX}{extraction.id}.md" + if target not in present: + lines.append(f"- [{extraction.title}]({target})") + present.add(target) + index_path.write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def materialize(manifest_path: Path, bundle_dir: Path, ingested_at: str) -> list[Path]: + """Materialize the manifest's extractions into ``bundle_dir`` (deterministic, §5). + + ``ingested_at`` is an EXPLICIT required argument (no wall-clock default, §5) — the ISO + timestamp stamped verbatim into every generated file, which is what makes extractions + bit-deterministic. Returns the generated concept-file paths in extraction order. + + Replacement semantics (§3, §5): first every ingest-stamped file is removed, then the + new set is written (a name colliding with a NON-stamped file fails — curated content is + never overwritten), then the index is updated. Curated and promoted files always survive. + """ + loaded = load_manifest(manifest_path) + source = loaded.contract.source + root = manifest_path.parent / source.root + + bundle_dir.mkdir(parents=True, exist_ok=True) + + # 1. Remove exactly the files carrying the ingest stamp (§5) — never curated/promoted. + for md_path in sorted(bundle_dir.glob("*.md")): + if md_path.name != "index.md" and _is_ingest_stamped(md_path): + md_path.unlink() + + # 2. Write the new set. + generated: list[Path] = [] + for extraction in loaded.contract.extractions: + csv_path = _resolve_within(root, extraction.query) + rows = _read_csv(csv_path, extraction.max_rows) + _logger.info( + "ingest source=%s extraction=%s rows=%d", + source.id, + extraction.id, + max(len(rows) - 1, 0), + ) + target = bundle_dir / f"{_INGEST_PREFIX}{extraction.id}.md" + if target.exists(): + # Post-removal, an existing target is a NON-stamped collision (§3): never overwrite. + raise ValueError(f"generated name {target.name} collides with a non-ingest file") + target.write_text( + _concept_file(extraction, source.id, ingested_at, loaded.stamp, _render_table(rows)), + encoding="utf-8", + ) + generated.append(target) + + # 3. Update the index (§6). + _update_index(bundle_dir, loaded.contract.bundle_summary, loaded.contract.extractions) + return generated diff --git a/src/portfolio_optimiser_claude/okf.py b/src/portfolio_optimiser_claude/okf.py index 284f7b7..d563f38 100644 --- a/src/portfolio_optimiser_claude/okf.py +++ b/src/portfolio_optimiser_claude/okf.py @@ -18,6 +18,7 @@ from dataclasses import dataclass from pathlib import Path _INDEX_FILENAME = "index.md" +_INDEX_TYPE = "index" _VERDICT_TYPE = "verdict" # (reference) the intra-bundle cross-link pattern, per §3 Step 1. _CROSSLINK_PATTERN = re.compile(r"\]\(([^)]+\.md)\)") @@ -70,19 +71,40 @@ def parse_concept_file(path: Path) -> ConceptFile: ) +def _parse_index_entry(index_path: Path) -> ConceptFile: + """Parse the bundle entry point, tolerating a frontmatter-less index (§3 Step 1). + + method-spec §3 Step 1 renders the index as "the index body (the summary)" and + treats every OTHER file as a "non-index concept file" (`## {type}: {title}` + section) — so the index is NOT a concept file, and it MAY omit frontmatter: a + generated index carrying only ``bundle_summary`` + cross-links (ingest spec §6, + frozen in the ingest golden) is valid. A frontmatter-ful index (the curated + convention, ``type: index``) still parses normally, its extra fields preserved. + When frontmatter is absent, the whole file is the body and ``type`` defaults to + ``index`` so downstream ``.type`` reads never raise. + """ + text = index_path.read_text(encoding="utf-8") + lines = text.splitlines() + if lines and lines[0].strip() == "---": + return parse_concept_file(index_path) + return ConceptFile(path=index_path, frontmatter={"type": _INDEX_TYPE}, body=text.strip()) + + def navigate_bundle(bundle_dir: Path) -> list[ConceptFile]: """Navigate from ``index.md`` — deterministic order: index first, links first-seen. Targets containing a path separator are out-of-bundle and skipped; resolution is boundary-checked against the bundle directory (fail-closed); broken links are - skipped, never raised. Repeated links are de-duplicated. + skipped, never raised. Repeated links are de-duplicated. The index entry point + may omit frontmatter (``_parse_index_entry``); non-index concept files still + require ``type``. """ index_path = bundle_dir / _INDEX_FILENAME if not index_path.is_file(): raise FileNotFoundError( f"bundle has no entry point: missing {_INDEX_FILENAME} in {bundle_dir}" ) - index = parse_concept_file(index_path) + index = _parse_index_entry(index_path) bundle_root = bundle_dir.resolve() concepts = [index] seen = {_INDEX_FILENAME} diff --git a/tests/test_ingest.py b/tests/test_ingest.py new file mode 100644 index 0000000..adbbc78 --- /dev/null +++ b/tests/test_ingest.py @@ -0,0 +1,138 @@ +"""Ingest unit + fail-fast contract tests (ingest-spec §4, §5, §8). + +The manifest is schema-validated fail-fast BEFORE any source call (§4, the startup-contract +discipline). These tests pin the malformed-manifest rejections, the §5 cell-escaping rules, +and the §8 size cap / fail-closed path resolution / no-overwrite collision. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +from pydantic import ValidationError + +from portfolio_optimiser_claude.ingest import ( + ManifestContract, + _escape_cell, + load_manifest, + materialize, +) + +GOLDEN = Path(__file__).resolve().parents[1] / "examples" / "ingest-golden-file" +INGESTED_AT = "2026-07-03T12:00:00Z" + + +def _valid() -> dict: + return { + "manifest_version": 1, + "source": {"type": "file", "id": "arkiv", "root": "fixture"}, + "bundle_summary": "s", + "extractions": [ + {"id": "e", "title": "T", "query": "e.csv", "okf_type": "dataset", "max_rows": 5} + ], + } + + +def _write_case(tmp_path: Path, manifest: dict, csvs: dict[str, str]) -> Path: + case = tmp_path / "case" + (case / "fixture").mkdir(parents=True) + (case / "manifest.json").write_text(json.dumps(manifest), encoding="utf-8") + for name, text in csvs.items(): + (case / "fixture" / name).write_text(text, encoding="utf-8") + return case + + +class TestManifestValidation: + """§4: a malformed manifest never starts a run (fail-fast).""" + + def test_valid_manifest_loads_and_stamps(self) -> None: + contract = ManifestContract(**_valid()) + assert contract.manifest_version == 1 + assert contract.source.id == "arkiv" + + def test_stamp_is_stem_at_sha256_16(self, tmp_path: Path) -> None: + case = _write_case(tmp_path, _valid(), {"e.csv": "a\n1\n"}) + loaded = load_manifest(case / "manifest.json") + stem, _, digest = loaded.stamp.partition("@") + assert stem == "manifest" + assert len(digest) == 16 and all(c in "0123456789abcdef" for c in digest) + + @pytest.mark.parametrize( + "mutate", + [ + lambda m: m.pop("manifest_version"), + lambda m: m.__setitem__("manifest_version", 2), + lambda m: m.pop("source"), + lambda m: m.pop("bundle_summary"), + lambda m: m.__setitem__("extractions", []), + lambda m: m["source"].__setitem__("id", "Bad_Id"), + lambda m: m["source"].__setitem__("type", "sql"), + lambda m: m["extractions"][0].__setitem__("id", "Bad Id"), + lambda m: m["extractions"][0].__setitem__("title", "two\nlines"), + lambda m: m["extractions"][0].__setitem__("max_rows", 0), + lambda m: m["extractions"][0].__setitem__("max_rows", -1), + ], + ) + def test_malformed_manifest_is_rejected(self, mutate) -> None: + manifest = _valid() + mutate(manifest) + with pytest.raises(ValidationError): + ManifestContract(**manifest) + + def test_duplicate_extraction_ids_are_rejected(self) -> None: + manifest = _valid() + manifest["extractions"].append(dict(manifest["extractions"][0])) + with pytest.raises(ValidationError): + ManifestContract(**manifest) + + +class TestCellEscaping: + """§5: text verbatim with backslash → \\\\, pipe → \\|, newline → single space.""" + + def test_backslash_then_pipe_order(self) -> None: + assert _escape_cell("\\|") == "\\\\\\|" + + def test_pipe_escaped(self) -> None: + assert _escape_cell("a|b") == "a\\|b" + + def test_newline_becomes_single_space(self) -> None: + assert _escape_cell("x\ny") == "x y" + assert _escape_cell("x\r\ny") == "x y" + + def test_plain_text_verbatim(self) -> None: + assert _escape_cell("007") == "007" + assert _escape_cell("1.50") == "1.50" + + +class TestSecurityFrame: + """§8: size cap fail-fast, path resolution fail-closed, curated never overwritten.""" + + def test_extraction_exceeding_max_rows_fails(self, tmp_path: Path) -> None: + manifest = _valid() + manifest["extractions"][0]["max_rows"] = 1 + case = _write_case(tmp_path, manifest, {"e.csv": "col\n1\n2\n"}) # 2 data rows > 1 + with pytest.raises(ValueError, match="max_rows"): + materialize(case / "manifest.json", tmp_path / "bundle", INGESTED_AT) + + def test_query_escaping_root_is_refused(self, tmp_path: Path) -> None: + manifest = _valid() + manifest["extractions"][0]["query"] = "../secret.csv" + case = _write_case(tmp_path, manifest, {"e.csv": "col\n1\n"}) + (case / "secret.csv").write_text("col\nx\n", encoding="utf-8") + with pytest.raises(ValueError, match="escapes"): + materialize(case / "manifest.json", tmp_path / "bundle", INGESTED_AT) + + def test_collision_with_non_ingest_file_fails(self, tmp_path: Path) -> None: + case = _write_case(tmp_path, _valid(), {"e.csv": "col\n1\n"}) + bundle = tmp_path / "bundle" + bundle.mkdir() + # A curated (non-stamped) file already occupies the generated name. + (bundle / "ingest-e.md").write_text( + "---\ntype: reference\ntitle: hand\n---\n\nCurated.\n", encoding="utf-8" + ) + with pytest.raises(ValueError, match="collides"): + materialize(case / "manifest.json", bundle, INGESTED_AT) + # The curated file is untouched — never overwritten. + assert "Curated." in (bundle / "ingest-e.md").read_text(encoding="utf-8") diff --git a/tests/test_ingest_golden.py b/tests/test_ingest_golden.py new file mode 100644 index 0000000..a3f634f --- /dev/null +++ b/tests/test_ingest_golden.py @@ -0,0 +1,86 @@ +"""Ingest golden regression (ingest-spec §11) — the shared byte-for-byte fasit. + +D7 mirror of MAF I2. Consumes the repo-local ``examples/ingest-golden-file/`` +(a byte-frozen copy of MAF I2's golden — the SAME shared extraction, so this test +proves D7's independent implementation reproduces MAF's exact bytes from the shared +spec alone). ``expected-bundle/`` is compared file by file, byte for byte. + +The mutation controls prove the net is taut: one changed determinism input (a CSV +cell, the timestamp, the manifest bytes) must diverge from the golden bytes. +""" + +from __future__ import annotations + +import shutil +from pathlib import Path + +from portfolio_optimiser_claude.ingest import materialize + +GOLDEN = Path(__file__).resolve().parents[1] / "examples" / "ingest-golden-file" + + +def _ingested_at() -> str: + return (GOLDEN / "ingested-at.txt").read_text(encoding="utf-8").strip() + + +def test_materializes_golden_byte_for_byte(tmp_path: Path) -> None: + bundle = tmp_path / "bundle" + bundle.mkdir() + materialize(GOLDEN / "manifest.json", bundle, _ingested_at()) + + expected = GOLDEN / "expected-bundle" + expected_names = sorted(p.name for p in expected.iterdir()) + produced_names = sorted(p.name for p in bundle.iterdir()) + assert produced_names == expected_names # no missing, no extra files + for name in expected_names: + assert (bundle / name).read_bytes() == (expected / name).read_bytes(), name + + +def test_reingest_is_idempotent(tmp_path: Path) -> None: + # ingest-spec §10: same source content, manifest, ingested_at → byte-identical, + # repeated runs idempotent. + bundle = tmp_path / "bundle" + bundle.mkdir() + materialize(GOLDEN / "manifest.json", bundle, _ingested_at()) + first = {p.name: p.read_bytes() for p in bundle.iterdir()} + materialize(GOLDEN / "manifest.json", bundle, _ingested_at()) + second = {p.name: p.read_bytes() for p in bundle.iterdir()} + assert first == second + + +class TestMutationControl: + """One changed determinism input → divergence from the golden bytes.""" + + def _case(self, tmp_path: Path) -> Path: + case = tmp_path / "case" + shutil.copytree(GOLDEN, case) + return case + + def test_changed_cell_diverges(self, tmp_path: Path) -> None: + case = self._case(tmp_path) + csv = case / "fixture" / "costs.csv" + csv.write_bytes(csv.read_bytes().replace(b"120000", b"999999")) + bundle = tmp_path / "bundle" + bundle.mkdir() + materialize(case / "manifest.json", bundle, _ingested_at()) + golden_bytes = (GOLDEN / "expected-bundle" / "ingest-costs.md").read_bytes() + assert (bundle / "ingest-costs.md").read_bytes() != golden_bytes + + def test_changed_timestamp_diverges(self, tmp_path: Path) -> None: + bundle = tmp_path / "bundle" + bundle.mkdir() + materialize(GOLDEN / "manifest.json", bundle, "2099-01-01T00:00:00Z") + golden_bytes = (GOLDEN / "expected-bundle" / "ingest-costs.md").read_bytes() + assert (bundle / "ingest-costs.md").read_bytes() != golden_bytes + + def test_changed_manifest_bytes_change_the_stamp(self, tmp_path: Path) -> None: + # The ingest_manifest stamp is SHA-256 of the manifest's raw bytes (§5) — + # a whitespace-only edit still re-stamps every generated file. + case = self._case(tmp_path) + manifest = case / "manifest.json" + manifest.write_bytes(manifest.read_bytes() + b"\n") + bundle = tmp_path / "bundle" + bundle.mkdir() + materialize(manifest, bundle, _ingested_at()) + golden_bytes = (GOLDEN / "expected-bundle" / "ingest-costs.md").read_bytes() + assert (bundle / "ingest-costs.md").read_bytes() != golden_bytes diff --git a/tests/test_ingest_loadbearing.py b/tests/test_ingest_loadbearing.py new file mode 100644 index 0000000..dcd6b90 --- /dev/null +++ b/tests/test_ingest_loadbearing.py @@ -0,0 +1,178 @@ +"""Load-bearing ingest seams (ingest-spec §11) — D7 mirror of MAF I2's set. + +Each test must go RED when its seam is detached (the method-spec §11 regime): a +grønn-men-død test is the failure mode the rule exists for. The seams mirrored here: + +- Provenance stamping — a generated file carries the §7 provenance layer, in order. +- Navigability — the generated bundle is consumable by the UNCHANGED ``okf`` navigation + (index links included), incl. the frontmatter-less generated index. +- Verdict reservation — a manifest mapping to ``type: verdict`` is rejected fail-fast, + before any source call. +- Re-ingest layer safety — re-materialization over a bundle carrying a promoted verdict + preserves the verdict file AND its index link (curated content always survives). +""" + +from __future__ import annotations + +import json +import shutil +from pathlib import Path + +import pytest +from pydantic import ValidationError + +from portfolio_optimiser_claude import okf +from portfolio_optimiser_claude.ingest import ManifestContract, load_manifest, materialize + +GOLDEN = Path(__file__).resolve().parents[1] / "examples" / "ingest-golden-file" +INGESTED_AT = (GOLDEN / "ingested-at.txt").read_text(encoding="utf-8").strip() + +_PROVENANCE_KEYS = ("source_system", "source_query", "ingested_at", "ingest_manifest", "generated") + + +def _materialized(tmp_path: Path) -> Path: + bundle = tmp_path / "bundle" + bundle.mkdir() + materialize(GOLDEN / "manifest.json", bundle, INGESTED_AT) + return bundle + + +class TestProvenanceStamping: + """Seam: a generated file carries the §7 provenance layer (RED if it stops).""" + + def test_generated_file_carries_the_provenance_layer(self, tmp_path: Path) -> None: + bundle = _materialized(tmp_path) + concept = okf.parse_concept_file(bundle / "ingest-costs.md") + for key in _PROVENANCE_KEYS: + assert key in concept.frontmatter, f"provenance key {key} detached" + assert concept.frontmatter["source_system"] == "prosjekt-arkiv" + assert concept.frontmatter["ingested_at"] == INGESTED_AT + assert concept.frontmatter["generated"] == "true" + stamp = load_manifest(GOLDEN / "manifest.json").stamp + assert concept.frontmatter["ingest_manifest"] == stamp + + def test_provenance_keys_are_in_the_spec_order(self, tmp_path: Path) -> None: + # §5: exactly these keys, in exactly this order — the chain is a contract. + bundle = _materialized(tmp_path) + lines = (bundle / "ingest-costs.md").read_text(encoding="utf-8").splitlines() + keys = [ln.split(":", 1)[0] for ln in lines[1 : lines.index("---", 1)]] + assert keys == [ + "type", + "title", + "source_system", + "source_query", + "ingested_at", + "ingest_manifest", + "generated", + ] + + +class TestNavigability: + """Seam: the generated bundle is consumable by the UNCHANGED okf navigation.""" + + def test_generated_bundle_navigates_via_unchanged_okf(self, tmp_path: Path) -> None: + bundle = _materialized(tmp_path) + names = [c.path.name for c in okf.navigate_bundle(bundle)] + # Every generated ingest file must be REACHABLE via index cross-links — an + # unlinked generated file is unreachable (RED if index linking detaches). + assert names == ["index.md", "ingest-costs.md", "ingest-edge.md"] + + def test_generated_context_renders_the_extracted_rows(self, tmp_path: Path) -> None: + bundle = _materialized(tmp_path) + context = okf.bundle_context(bundle) + assert "led-retrofit" in context and "requires vendor quote" in context + + def test_generated_index_is_frontmatterless_and_still_navigates(self, tmp_path: Path) -> None: + # The generated index carries no frontmatter (§6 shape) — navigation must + # still work, exercising the okf entry-point relaxation end to end. + bundle = _materialized(tmp_path) + assert not (bundle / "index.md").read_text(encoding="utf-8").startswith("---") + assert okf.navigate_bundle(bundle)[0].path.name == "index.md" + + +class TestVerdictReservation: + """Seam: an ingest mapping to the verdict layer is rejected fail-fast (§3).""" + + def _manifest(self, okf_type: str) -> dict: + return { + "manifest_version": 1, + "source": {"type": "file", "id": "x", "root": "fixture"}, + "bundle_summary": "s", + "extractions": [ + {"id": "e", "title": "T", "query": "e.csv", "okf_type": okf_type, "max_rows": 1} + ], + } + + def test_verdict_okf_type_is_rejected(self) -> None: + with pytest.raises(ValidationError): + ManifestContract(**self._manifest("verdict")) + + def test_verdict_reservation_is_case_insensitive(self) -> None: + with pytest.raises(ValidationError): + ManifestContract(**self._manifest("Verdict")) + + def test_rejection_is_fail_fast_before_any_source_call(self, tmp_path: Path) -> None: + # A verdict manifest never touches the source: no bundle is written. + manifest = tmp_path / "manifest.json" + manifest.write_text(json.dumps(self._manifest("verdict")), encoding="utf-8") + bundle = tmp_path / "bundle" + with pytest.raises(ValidationError): + materialize(manifest, bundle, INGESTED_AT) + assert not bundle.exists() or not any(bundle.iterdir()) + + +class TestReingestLayerSafety: + """Seam: re-ingest preserves a promoted verdict AND its index link (§3, §6).""" + + def _promote(self, bundle: Path) -> None: + # Simulate the promotion gate writing a verdict file + its (neutral) index link, + # plus a curated file + link. Neither carries the ingest stamp. + (bundle / "promoted-verdict-led.md").write_text( + "---\ntype: verdict\ndecision: approved\ndescription: LED holds up.\n---\n\nBody.\n", + encoding="utf-8", + ) + (bundle / "curated-note.md").write_text( + "---\ntype: reference\ntitle: Curated\n---\n\nHand-written.\n", encoding="utf-8" + ) + index = bundle / "index.md" + index.write_text( + index.read_text(encoding="utf-8") + + "- [promoted](promoted-verdict-led.md)\n" + + "- [Curated](curated-note.md)\n", + encoding="utf-8", + ) + + def test_promoted_verdict_and_curated_survive_reingest(self, tmp_path: Path) -> None: + bundle = _materialized(tmp_path) + self._promote(bundle) + materialize(GOLDEN / "manifest.json", bundle, INGESTED_AT) # re-ingest + + assert (bundle / "promoted-verdict-led.md").is_file() # verdict file survives + assert (bundle / "curated-note.md").is_file() # curated file survives + index_text = (bundle / "index.md").read_text(encoding="utf-8") + assert "- [promoted](promoted-verdict-led.md)" in index_text # verdict link survives + assert "- [Curated](curated-note.md)" in index_text # curated link survives + # ...and the ingest files were still refreshed. + assert (bundle / "ingest-costs.md").is_file() + assert "](ingest-costs.md)" in index_text + + def test_reingest_drops_a_stale_ingest_file_and_its_link(self, tmp_path: Path) -> None: + # A manifest that no longer generates ingest-edge → the stale file AND its index + # link are removed; the promoted/curated content is untouched (RED if replacement + # over-reaches or under-reaches). + bundle = _materialized(tmp_path) + self._promote(bundle) + # The shrunk manifest must sit beside its own fixture/ (root resolves relative to + # the manifest dir), so copy the golden case and rewrite the manifest there. + case = tmp_path / "case" + shutil.copytree(GOLDEN, case) + shrunk = json.loads((case / "manifest.json").read_text(encoding="utf-8")) + shrunk["extractions"] = [e for e in shrunk["extractions"] if e["id"] == "costs"] + (case / "manifest.json").write_text(json.dumps(shrunk), encoding="utf-8") + materialize(case / "manifest.json", bundle, INGESTED_AT) + + assert not (bundle / "ingest-edge.md").exists() # stale ingest file removed + index_text = (bundle / "index.md").read_text(encoding="utf-8") + assert "](ingest-edge.md)" not in index_text # stale ingest link removed + assert "- [promoted](promoted-verdict-led.md)" in index_text # promoted survives + assert (bundle / "ingest-costs.md").is_file() # kept extraction survives diff --git a/tests/test_ingest_spec_loadbearing.py b/tests/test_ingest_spec_loadbearing.py new file mode 100644 index 0000000..5df1c28 --- /dev/null +++ b/tests/test_ingest_spec_loadbearing.py @@ -0,0 +1,70 @@ +"""Spec-integrity seam for the ingest spec (ingest-spec §11). + +The D7 analog of MAF's I1 framework-guard: this repo consumes ``shared/ingest-spec.md`` +UNCHANGED from commons, and this test keeps the contract honest — it goes RED when the +spec goes missing, names a concrete agent toolkit (the framework-neutrality rule), or +stops documenting a contract field. It is the load-bearing guard the ingest layer relies +on to keep being implementable "from this spec alone". +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +SPEC = Path(__file__).resolve().parents[1] / "shared" / "ingest-spec.md" + +# Concrete agent toolkits / vendor stacks the framework-neutral spec MUST NOT name. +_FORBIDDEN_TOOLKITS = ( + "claude", + "anthropic", + "openai", + "gpt", + "gemini", + "llama", + "langchain", + "autogen", + "crewai", + "semantic kernel", + "microsoft agent framework", + "agent sdk", + "bedrock", + "vertex", + "foundry", + "maf", +) + +# Every field of the machine-readable contracts the D7 implementation depends on — the +# spec's §12 cross-check table must keep documenting each (spec-integrity). +_CONTRACT_FIELDS = ( + "manifest_version", + "source", + "bundle_summary", + "extractions", + "source_system", + "source_query", + "ingested_at", + "ingest_manifest", + "generated", + "okf_type", + "max_rows", + "root", +) + + +def test_spec_is_present() -> None: + # RED if the spec goes missing (the layer stops being implementable from spec alone). + assert SPEC.is_file(), "ingest-spec.md missing — subtree pull the commons contract" + + +def test_spec_names_no_agent_toolkit() -> None: + text = SPEC.read_text(encoding="utf-8").lower() + present = [tok for tok in _FORBIDDEN_TOOLKITS if tok in text] + assert not present, f"framework-neutral spec names a concrete toolkit: {present}" + + +@pytest.mark.parametrize("field", _CONTRACT_FIELDS) +def test_spec_documents_contract_field(field: str) -> None: + text = SPEC.read_text(encoding="utf-8") + assert field in text, f"contract field {field!r} is no longer documented in the spec" diff --git a/tests/test_okf.py b/tests/test_okf.py index 6d5ac97..285f892 100644 --- a/tests/test_okf.py +++ b/tests/test_okf.py @@ -54,6 +54,32 @@ class TestNavigation: names = [c.path.name for c in navigate_bundle(bundle)] assert names == ["index.md", "b.md", "a.md"] + def test_frontmatterless_index_navigates(self, tmp_path: Path) -> None: + # method-spec §3 Step 1: the index contributes "the index body (the summary)" + # and is NOT a "non-index concept file" rendered as a `## {type}: {title}` + # section — so a generated index carrying only the summary + links (ingest + # spec §6 shape, frozen in the ingest golden) is valid. The index MAY omit + # frontmatter; okf must navigate it via its body, never raise. + _write(tmp_path / "index.md", "Summary line.\n- [A](a.md)\n") + _write(tmp_path / "a.md", "---\ntype: project\ntitle: A\n---\nBody A.") + concepts = navigate_bundle(tmp_path) + assert [c.path.name for c in concepts] == ["index.md", "a.md"] + index = concepts[0] + assert index.type == "index" # default type, no KeyError for downstream .type reads + assert index.body == "Summary line.\n- [A](a.md)" # body = the whole file + # The summary flows into the read-context as the leading section. + assert bundle_context(tmp_path).startswith("Summary line.") + + def test_frontmatterless_tolerance_is_scoped_to_the_index(self, tmp_path: Path) -> None: + # LOAD-BEARING (honesty): the relaxation is for the index ENTRY POINT only. + # A NON-index concept file without frontmatter is still malformed and MUST + # raise — else the relaxation has silently weakened the `type`-required rule + # for concept files (method-spec §3 Step 1). RED if the tolerance leaks. + _write(tmp_path / "index.md", "Summary.\n- [A](a.md)\n") + _write(tmp_path / "a.md", "no frontmatter here\n") + with pytest.raises(ValueError, match="frontmatter"): + navigate_bundle(tmp_path) + def test_out_of_bundle_and_escaping_targets_are_skipped(self, tmp_path: Path) -> None: # A target containing a path separator is out-of-bundle — skipped, never # raised; ../-escapes never resolve outside the bundle (fail-closed).