"""P14 — the stress-test context sets, gated (order ``20260912T202210Z``). Each set under ``contexts//`` commissions ONE run against ONE knowledge base: ``mandate.json`` (the ``Mandate`` schema verbatim), ``bundle.txt`` (which base, and the id that base declares) and ``fasit.json`` (what a right answer MUST cite, what the base cannot answer, and what in the set is constructed rather than real). **Five arms, and the split between them is a measurement rather than a taste.** Two are unconditional and can never be silently absent — a mandate that does not load, and a mandate routed at a base the set is not for. Three need the base itself, which lives OUTSIDE this repository (``PORTFOLIO_VEGNORMAL_ROOT``): they SKIP when the root is missing, exactly as MAJOR-3's ceiling gate could not take K2 as a test dependency, and for the same published-package reason — a hard failure would break ``uv run pytest`` for any external recipient of the ``git archive HEAD`` handover. The skip NAMES the root it looked for. **Every bundle-reading arm carries its own denominator.** A scan that sees zero concepts is RED rather than vacuously green: "the anchor was not found" is equally true of a base that was never read (Verifiseringsloven, ansikt 4). **Rule U** — the measurable form of "the base cannot answer this" (documented in ``docs/2026-09-12-p14-kontekstsett.md § 2.3``): each unanswerable question declares >= 1 ``anchor``, a lowercase word of >= 4 characters, and is admitted **iff every anchor is absent — case-insensitive substring — from the WHOLE text (frontmatter + body) of EVERY concept document in the base**. Not "shares no keyword with any title": a tunnel question shares "tunnel" with hundreds of titles and that proves nothing. What makes a question unanswerable is that the base lacks the SUBJECT, and the anchor is that subject. Titles alone would be a proxy the full text costs nothing more to replace (measured: 0.77 s for r761-2025, the largest base). """ from __future__ import annotations import json import os from pathlib import Path import pytest from pydantic import ValidationError from portfolio_optimiser import okf from portfolio_optimiser.mandate import load_mandate _REPO_ROOT = Path(__file__).resolve().parent.parent _CONTEXT_ROOT = _REPO_ROOT / "contexts" #: Where the vegnormal bases are mounted. A SYMBOLIC name in ``bundle.txt`` is resolved against #: this, never an absolute path in the set: this repository is published, and an absolute path #: would pin a set to one machine's home directory and ride out in the handover archive. _DEFAULT_BUNDLE_ROOT = Path.home() / "repos" / "vegnormal-okf" / "build" / "ferdig" #: The concept types the four bases declare. ``index.md`` carries none of them — it is navigation, #: not content — which is why the file count and the concept count differ. _CONCEPT_TYPES = {"Krav", "Prosess", "Kapittel", "Normal", "Håndbok"} _MIN_ANCHOR_CHARS = 4 def own_frontmatter(path: Path) -> dict[str, str]: """The concept's OWN frontmatter: top-level keys only, FIRST occurrence winning. **``okf.parse_frontmatter`` cannot be used for this, and that is measured rather than assumed.** It is linewise and last-write-wins by documented design, so a nested block overwrites a top-level key of the same name. Every vegnormal concept ends its frontmatter with sources: - resource: https://… title: N500:2024 and the indented ``title`` therefore replaces the concept's own. MEASURED on n500-2024: ``okf.navigate_bundle`` yields 270 concept files carrying **1 distinct title** (``N500:2024``, 270 times), and ``okf.directory_listing(bundle, path="krav/N500")`` returns 269 documents whose ``title`` is that same string — so rung 2 and rung 3 of the navigation ladder tell a navigator apart only by an opaque UUID filename and a character count. That is a finding about the PRODUCT (recorded in ``docs/2026-09-12-p14-kontekstsett.md § 5``), not about this test; what it means here is that a fasit assert against ``parse_frontmatter``'s ``title`` would compare every concept to the same constant and be VACUOUS. Uses ``okf._split_frontmatter`` deliberately: it is the module's ONE place ``---`` is compared (B4), and a second delimiter rule here would be the copy that drifts. """ out: dict[str, str] = {} for line in okf._split_frontmatter(path.read_text(encoding="utf-8"))[0]: if not line or line[0].isspace() or line.lstrip().startswith("-"): continue key, sep, value = line.partition(":") if sep and key.strip() not in out: out[key.strip()] = value.strip().strip("'") return out def _bundle_root() -> Path: return Path(os.environ.get("PORTFOLIO_VEGNORMAL_ROOT", str(_DEFAULT_BUNDLE_ROOT))) def read_bundle_txt(path: Path) -> dict[str, str]: """Parse a set's ``bundle.txt``: ``key: value`` lines, nothing else.""" out: dict[str, str] = {} for line in path.read_text(encoding="utf-8").splitlines(): line = line.strip() if not line or line.startswith("#"): continue if ": " not in line: raise ValueError(f"malformed bundle.txt line in {path}: {line!r}") key, value = line.split(": ", 1) out[key.strip()] = value.strip() for required in ("name", "bundle_id"): if required not in out: raise ValueError(f"{path} declares no {required!r}") return out def scan_concepts(base: Path) -> list[tuple[str, dict[str, str], str]]: """Every concept document in a base: bundle-relative path, frontmatter, lowercased full text. Built from the declared ``type``, never from the directory listing: ``index.md`` is navigation and would otherwise be counted as content. """ found: list[tuple[str, dict[str, str], str]] = [] for path in sorted(base.rglob("*.md")): text = path.read_text(encoding="utf-8") frontmatter = own_frontmatter(path) if frontmatter.get("type", "") in _CONCEPT_TYPES: found.append((path.relative_to(base).as_posix(), frontmatter, text.lower())) return found def anchors_are_absent( anchors: list[str], concepts: list[tuple[str, dict[str, str], str]] ) -> list[str]: """Rule U: return the anchors the base DOES carry (empty == the question is admitted). :raises ValueError: an empty scan, or an anchor that is not a usable one. Both are refusals rather than a quiet pass — a rule that cannot fail proves nothing. """ if not concepts: raise ValueError("rule U ran against ZERO concepts: absence here is unmeasured, not false") if not anchors: raise ValueError("an unanswerable question declares no anchors, so nothing was checked") carried = [] for anchor in anchors: if anchor != anchor.lower() or len(anchor) < _MIN_ANCHOR_CHARS: raise ValueError( f"anchor {anchor!r} must be lowercase and at least {_MIN_ANCHOR_CHARS} characters" ) if any(anchor in text for _, _, text in concepts): carried.append(anchor) return carried def context_sets() -> list[Path]: return ( sorted(p for p in _CONTEXT_ROOT.iterdir() if p.is_dir()) if _CONTEXT_ROOT.is_dir() else [] ) _SETS = context_sets() _SET_IDS = [p.name for p in _SETS] def _require_base(declared: dict[str, str]) -> Path: root = _bundle_root() base = root / declared["name"] if not base.is_dir(): pytest.skip( f"knowledge base {declared['name']!r} not mounted under {root} (PORTFOLIO_VEGNORMAL_ROOT)" ) return base # -------------------------------------------------------------------------------------------- # The sets exist at all. Without this, every parametrised arm below would collapse to zero cases # and the file would pass by having nothing to say. # -------------------------------------------------------------------------------------------- def test_the_four_context_sets_are_present() -> None: assert len(_SETS) == 4, f"expected four context sets under {_CONTEXT_ROOT}, found {_SET_IDS}" # -------------------------------------------------------------------------------------------- # (a) + (d): unconditional — no knowledge base needed. # -------------------------------------------------------------------------------------------- @pytest.mark.parametrize("set_dir", _SETS, ids=_SET_IDS) def test_a_mandate_loads_fail_fast(set_dir: Path) -> None: mandate = load_mandate(set_dir / "mandate.json") assert mandate.objective assert mandate.success_criteria, f"{set_dir.name} states no success criteria to judge it by" assert 2 <= len(mandate.approaches) <= 4, "the order asks for 2-4 approaches per set" for approach in mandate.approaches: assert approach.affected_codes, f"{approach.id} names no affected_codes" assert approach.claimed_saving_nok is not None, f"{approach.id} states no estimate" @pytest.mark.parametrize("set_dir", _SETS, ids=_SET_IDS) def test_d_every_approach_is_routed_at_this_sets_own_base(set_dir: Path) -> None: declared = read_bundle_txt(set_dir / "bundle.txt") mandate = load_mandate(set_dir / "mandate.json") for approach in mandate.approaches: assert approach.bundle_id == declared["bundle_id"], ( f"{set_dir.name}/{approach.id} routes at {approach.bundle_id!r} but the set declares " f"{declared['bundle_id']!r}" ) @pytest.mark.parametrize("set_dir", _SETS, ids=_SET_IDS) def test_the_fasit_names_every_commissioned_approach(set_dir: Path) -> None: fasit = json.loads((set_dir / "fasit.json").read_text(encoding="utf-8")) mandate = load_mandate(set_dir / "mandate.json") cited = {row["approach_id"] for row in fasit["must_cite"]} assert cited == {a.id for a in mandate.approaches} assert fasit["honesty"].strip(), "DEL 2(iii): what in this set is constructed must be stated" assert len(fasit["unanswerable"]) >= 2, "the order asks for at least two per set" assert (set_dir / "docs").is_dir(), "the form declares a docs/ directory even when it is empty" # -------------------------------------------------------------------------------------------- # (b) + (c) + (e): these read the base itself and SKIP when it is not mounted. # -------------------------------------------------------------------------------------------- @pytest.mark.parametrize("set_dir", _SETS, ids=_SET_IDS) def test_b_every_fasit_concept_is_in_the_base_as_recorded(set_dir: Path) -> None: declared = read_bundle_txt(set_dir / "bundle.txt") base = _require_base(declared) fasit = json.loads((set_dir / "fasit.json").read_text(encoding="utf-8")) seen = 0 for row in fasit["must_cite"]: assert row["concepts"], f"{row['approach_id']} cites nothing a right answer must reach" for concept in row["concepts"]: path = base / concept["path"] assert path.is_file(), f"{set_dir.name}: {concept['path']} is not in {declared['name']}" frontmatter = own_frontmatter(path) assert frontmatter.get("title", "") == concept["title"], ( f"{concept['path']}: the base's own title has drifted from the fasit" ) if concept.get("ref"): actual = frontmatter.get("req_number") or frontmatter.get("prosessnr", "") assert actual == concept["ref"], ( f"{concept['path']}: the base's own reference has drifted from the fasit" ) seen += 1 assert seen > 0, "the fasit named no concepts at all, so nothing was verified" @pytest.mark.parametrize("set_dir", _SETS, ids=_SET_IDS) def test_c_rule_u_every_unanswerable_question_is_unanswerable(set_dir: Path) -> None: declared = read_bundle_txt(set_dir / "bundle.txt") base = _require_base(declared) concepts = scan_concepts(base) assert len(concepts) >= 100, ( f"{declared['name']} scanned to {len(concepts)} concepts — too few to be the base itself" ) fasit = json.loads((set_dir / "fasit.json").read_text(encoding="utf-8")) for row in fasit["unanswerable"]: carried = anchors_are_absent(row["anchors"], concepts) assert not carried, ( f"{set_dir.name}: {declared['name']} DOES carry {carried} over {len(concepts)} " f"concepts, so {row['question']!r} is not unanswerable by rule U" ) @pytest.mark.parametrize("set_dir", _SETS, ids=_SET_IDS) def test_e_the_declared_bundle_id_is_the_bases_own(set_dir: Path) -> None: declared = read_bundle_txt(set_dir / "bundle.txt") base = _require_base(declared) resolved = okf.reconcile_bundle_id(base) assert resolved.id == declared["bundle_id"], ( f"{set_dir.name}: bundle.txt declares {declared['bundle_id']!r} but the base resolves to " f"{resolved.id!r} (origin {resolved.origin})" ) # -------------------------------------------------------------------------------------------- # KNOWN-POSITIVES (DEL 3): a deliberately broken set must make EXACTLY the arm that guards it red. # Without these, a check that can only pass is indistinguishable from a check that never runs. # -------------------------------------------------------------------------------------------- def _broken_set(tmp_path: Path, *, mandate: dict, bundle: str, fasit: dict) -> Path: set_dir = tmp_path / "broken-set" (set_dir / "docs").mkdir(parents=True) (set_dir / "mandate.json").write_text(json.dumps(mandate), encoding="utf-8") (set_dir / "bundle.txt").write_text(bundle, encoding="utf-8") (set_dir / "fasit.json").write_text(json.dumps(fasit), encoding="utf-8") return set_dir _GOOD_MANDATE = { "objective": "Reduce cost on a synthetic project", "success_criteria": "at least one approach validates", "approaches": [ { "id": "a1", "label": "One", "affected_codes": ["X-1"], "claimed_saving_nok": 1.0, "bundle_id": "vegnormal-n500-2024", }, { "id": "a2", "label": "Two", "affected_codes": ["X-2"], "claimed_saving_nok": 2.0, "bundle_id": "vegnormal-n500-2024", }, ], } _GOOD_BUNDLE_TXT = "name: n500-2024\nbundle_id: vegnormal-n500-2024\n" def test_known_positive_a_a_malformed_mandate_is_refused(tmp_path: Path) -> None: broken = dict(_GOOD_MANDATE) broken["approaches"] = [ dict(_GOOD_MANDATE["approaches"][0]), dict(_GOOD_MANDATE["approaches"][0]), ] set_dir = _broken_set(tmp_path, mandate=broken, bundle=_GOOD_BUNDLE_TXT, fasit={}) with pytest.raises(ValidationError): load_mandate(set_dir / "mandate.json") def test_known_positive_d_a_mandate_routed_at_another_base_is_caught(tmp_path: Path) -> None: broken = json.loads(json.dumps(_GOOD_MANDATE)) broken["approaches"][1]["bundle_id"] = "vegnormal-n100-2023" set_dir = _broken_set(tmp_path, mandate=broken, bundle=_GOOD_BUNDLE_TXT, fasit={}) declared = read_bundle_txt(set_dir / "bundle.txt") mandate = load_mandate(set_dir / "mandate.json") assert any(a.bundle_id != declared["bundle_id"] for a in mandate.approaches) def test_known_positive_c_an_anchor_the_base_carries_is_reported() -> None: concepts = [("a.md", {"type": "Krav"}, "en tunnel med ventilasjon og belysning")] assert anchors_are_absent(["enhetspris"], concepts) == [] assert anchors_are_absent(["ventilasjon"], concepts) == ["ventilasjon"] def test_known_positive_c_an_empty_scan_is_refused_never_vacuously_absent() -> None: with pytest.raises(ValueError, match="ZERO concepts"): anchors_are_absent(["enhetspris"], []) def test_known_positive_c_an_unusable_anchor_is_refused() -> None: concepts = [("a.md", {"type": "Krav"}, "tekst")] with pytest.raises(ValueError, match="at least"): anchors_are_absent(["vei"], concepts) with pytest.raises(ValueError, match="lowercase"): anchors_are_absent(["Enhetspris"], concepts) with pytest.raises(ValueError, match="no anchors"): anchors_are_absent([], concepts) def test_known_positive_b_a_fasit_path_the_base_does_not_carry_is_caught(tmp_path: Path) -> None: base = tmp_path / "base" (base / "krav").mkdir(parents=True) (base / "krav" / "real.md").write_text( "---\ntype: Krav\ntitle: Ekte krav\nreq_number: Krav 1.1—1\n---\n\nkropp\n", encoding="utf-8", ) assert (base / "krav" / "real.md").is_file() assert not (base / "krav" / "invented.md").is_file() assert own_frontmatter(base / "krav" / "real.md")["title"] == "Ekte krav" def test_known_positive_bundle_txt_must_declare_both_keys(tmp_path: Path) -> None: path = tmp_path / "bundle.txt" path.write_text("name: n500-2024\n", encoding="utf-8") with pytest.raises(ValueError, match="bundle_id"): read_bundle_txt(path) @pytest.mark.parametrize("set_dir", _SETS, ids=_SET_IDS) def test_the_fasit_titles_are_distinct_not_the_collapsed_sources_title(set_dir: Path) -> None: """The fasit's recorded titles must tell the cited concepts APART. Paired with ``own_frontmatter``'s measurement, this is what keeps arm (b) from being vacuous: if the recorded titles were ``parse_frontmatter``'s, every one of them would be the base's ``sources`` title and the assert would hold against any concept in the base. **TRIPWIRE, deliberately.** The second half asserts that ``okf.parse_frontmatter`` DOES still collapse them. The day that stops being true — an okf bump, or a decision to read block mappings here — this arm goes red, and whoever sees it should read this docstring, confirm the listing now carries real titles, and DELETE this half rather than weaken it. """ declared = read_bundle_txt(set_dir / "bundle.txt") base = _require_base(declared) fasit = json.loads((set_dir / "fasit.json").read_text(encoding="utf-8")) cited = [c for row in fasit["must_cite"] for c in row["concepts"]] assert len({c["title"] for c in cited}) == len(cited), "recorded titles do not tell them apart" collapsed = {okf.parse_frontmatter(base / c["path"]).get("title", "") for c in cited} assert len(collapsed) == 1, ( "okf.parse_frontmatter no longer collapses these titles onto the sources block — read this " "test's docstring: the finding it guards may be gone, and this half should be deleted" )