open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

feat(p17b): a context set that spans TWO bases, and a judge told which one [skip-docs]

Browse source 
``contexts/dekke-og-kontrakt-lindaas-2027`` is the first set whose approaches
route at more than one knowledge base: a1/a2 at n200-2024 (material requirements)
and a3/a4 at r761-2025 (the rig, and the falsification arm). That is the whole
reason it exists -- P17b measures that ONE commission can be run across several.

``bundle.txt`` grows a block per base; a set naming one base is one block, so the
four pre-P17b files parse byte-identically. The reader now has ONE home
(``stress.read_bundle_declarations``): it used to be a private copy in the P14
gate and a second, looser one inside ``stress.main``, and the multi-base form is
exactly the change that would have let them drift.

Rule U becomes the UNION of every declared base, and that is not a formality.
MEASURED 15.09: ``enhetspris`` is absent from n200-2024 and carried by 70 of
r761-2025's 2 756 concepts, so anchors admitted per base would have admitted a
question the pass as a whole CAN ground. It was dropped from the fifth set's
anchors for that reason.

``score_context_set(bundle_id=...)`` restricts the judgement to the approaches
routed at THIS base. Without it, judging the n200 outbox reports the r761
approach as ``not_evaluated``/``absent`` -- a false finding, because that
approach WAS evaluated, against the other base, under the other run_id. That
defect is pinned by its own arm. The judge's CLI refuses to guess when a set
declares several bases, with an rc-0 control on ``--bundle``.

Arm (d) gained a second half: every DECLARED base must be named by some
approach, because a base no approach names is never run.

The P19/B2 fasit denominator moved 26 -> 32 and is asserted, not dropped: six new
references, two of them bare ``prosessnr`` (12.11, 12.12), so B1's
punctuation-and-digits form is now exercised by a fasit and not only by a
known-positive.

Suite 1774/5, golden byte-unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-15 04:44:37 +02:00
commit da0ccd0489
8 changed files with 496 additions and 57 deletions
Download patch file Download diff file Expand all files Collapse all files

159
tests/test_context_sets_loadbearing.py
View file

@ -47,6 +47,7 @@ from pydantic import ValidationError
from portfolio_optimiser import okf
from portfolio_optimiser.mandate import load_mandate
from portfolio_optimiser.stress import read_bundle_declarations
_REPO_ROOT = Path(__file__).resolve().parent.parent
_CONTEXT_ROOT = _REPO_ROOT / "contexts"
@ -109,21 +110,11 @@ def _bundle_root() -> Path:
return Path(os.environ.get("PORTFOLIO_VEGNORMAL_ROOT", str(_DEFAULT_BUNDLE_ROOT)))
def read_bundle_txt(path: Path) -> dict[str, str]:
"""Parse a set's ``bundle.txt``: ``key: value`` lines, nothing else."""
out: dict[str, str] = {}
for line in path.read_text(encoding="utf-8").splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
if ": " not in line:
raise ValueError(f"malformed bundle.txt line in {path}: {line!r}")
key, value = line.split(": ", 1)
out[key.strip()] = value.strip()
for required in ("name", "bundle_id"):
if required not in out:
raise ValueError(f"{path} declares no {required!r}")
return out
#: The ONE reader, imported from production rather than copied here (P17b). It used to be a
#: private copy in this file and a second, looser one inside ``stress.main`` — and the multi-base
#: form is exactly the change that would have let the two drift into different answers about one
#: set. A set declaring ONE base is one block, so the four pre-P17b files parse unchanged.
read_bundle_txt = read_bundle_declarations
def scan_concepts(base: Path) -> list[tuple[str, dict[str, str], str]]:
@ -184,14 +175,29 @@ def _require_base(declared: dict[str, str]) -> Path:
return base
def _base_by_approach(set_dir: Path) -> dict[str, Path]:
"""Which MOUNTED base each approach was routed at (P17b).
Read off the mandate's ``bundle_id`` and the set's own declarations — the routing has exactly
one home, and a second per-row key in the fasit would be the copy free to drift.
"""
by_id = {block["bundle_id"]: block for block in read_bundle_txt(set_dir / "bundle.txt")}
out: dict[str, Path] = {}
for approach in load_mandate(set_dir / "mandate.json").approaches:
block = by_id.get(approach.bundle_id)
assert block is not None, f"{approach.id} routes at an undeclared base"
out[approach.id] = _require_base(block)
return out
# --------------------------------------------------------------------------------------------
# The sets exist at all. Without this, every parametrised arm below would collapse to zero cases
# and the file would pass by having nothing to say.
# --------------------------------------------------------------------------------------------
def test_the_four_context_sets_are_present() -> None:
assert len(_SETS) == 4, f"expected four context sets under {_CONTEXT_ROOT}, found {_SET_IDS}"
def test_the_five_context_sets_are_present() -> None:
assert len(_SETS) == 5, f"expected five context sets under {_CONTEXT_ROOT}, found {_SET_IDS}"
# --------------------------------------------------------------------------------------------
@ -212,13 +218,26 @@ def test_a_mandate_loads_fail_fast(set_dir: Path) -> None:
@pytest.mark.parametrize("set_dir", _SETS, ids=_SET_IDS)
def test_d_every_approach_is_routed_at_this_sets_own_base(set_dir: Path) -> None:
"""Every approach names ONE of the set's declared bases, and every declared base is named.
Both halves are the claim. The first is the original: an approach routed at a base the set is
not for would be evaluated against a corpus nobody commissioned. The second arrived with the
multi-base form (P17b) and is what keeps the declaration honest the other way — a base listed
in ``bundle.txt`` that no approach names is never run (``route_by_bundle``'s own rule, a run
costs money and the commission ordered nothing for it), so a set declaring it would be
describing a pass wider than the one it commissions.
"""
declared = read_bundle_txt(set_dir / "bundle.txt")
mandate = load_mandate(set_dir / "mandate.json")
for approach in mandate.approaches:
assert approach.bundle_id == declared["bundle_id"], (
f"{set_dir.name}/{approach.id} routes at {approach.bundle_id!r} but the set declares "
f"{declared['bundle_id']!r}"
)
ids = {block["bundle_id"] for block in declared}
routed = {approach.bundle_id for approach in load_mandate(set_dir / "mandate.json").approaches}
assert routed <= ids, (
f"{set_dir.name}: approaches route at {sorted(routed - ids)}, which the set does not "
f"declare (declared: {sorted(ids)})"
)
assert ids <= routed, (
f"{set_dir.name}: declares {sorted(ids - routed)} that no approach names, so the set "
"describes a wider pass than it commissions"
)
@pytest.mark.parametrize("set_dir", _SETS, ids=_SET_IDS)
@ -244,16 +263,23 @@ def test_the_fasit_names_every_commissioned_approach(set_dir: Path) -> None:
@pytest.mark.parametrize("set_dir", _SETS, ids=_SET_IDS)
def test_b_every_fasit_concept_is_in_the_base_as_recorded(set_dir: Path) -> None:
declared = read_bundle_txt(set_dir / "bundle.txt")
base = _require_base(declared)
"""Every cited concept is in the base ITS OWN approach was routed at (P17b).
Resolving per approach rather than per set is the multi-base half: in a set spanning two
bases, checking every path against one of them would fail half the fasit while proving
nothing about the other, and checking against "either" would let a path meant for N200 be
satisfied by a coincidence in R761.
"""
bases = _base_by_approach(set_dir)
fasit = json.loads((set_dir / "fasit.json").read_text(encoding="utf-8"))
seen = 0
for row in fasit["must_cite"]:
assert row["concepts"], f"{row['approach_id']} cites nothing a right answer must reach"
base = bases[row["approach_id"]]
for concept in row["concepts"]:
path = base / concept["path"]
assert path.is_file(), f"{set_dir.name}: {concept['path']} is not in {declared['name']}"
assert path.is_file(), f"{set_dir.name}: {concept['path']} is not in {base.name}"
frontmatter = own_frontmatter(path)
assert frontmatter.get("title", "") == concept["title"], (
f"{concept['path']}: the base's own title has drifted from the fasit"
@ -269,31 +295,43 @@ def test_b_every_fasit_concept_is_in_the_base_as_recorded(set_dir: Path) -> None
@pytest.mark.parametrize("set_dir", _SETS, ids=_SET_IDS)
def test_c_rule_u_every_unanswerable_question_is_unanswerable(set_dir: Path) -> None:
"""Rule U over EVERY base the set declares, as ONE scan.
For a multi-base set "the base cannot answer this" becomes "NEITHER base can", and the union
is the honest reading: an anchor absent from N200 but present in R761 is a question the pass
as a whole CAN reach. MEASURED 15.09 and the reason this is not a formality — ``enhetspris``
is absent from n200-2024 and carried by 70 of r761-2025's 2 756 concepts, so an anchor set
admitted per base would have admitted a question the pass could ground.
"""
declared = read_bundle_txt(set_dir / "bundle.txt")
base = _require_base(declared)
concepts = scan_concepts(base)
concepts: list[tuple[str, dict[str, str], str]] = []
names = []
for block in declared:
base = _require_base(block)
names.append(block["name"])
concepts += scan_concepts(base)
assert len(concepts) >= 100, (
f"{declared['name']} scanned to {len(concepts)} concepts — too few to be the base itself"
f"{', '.join(names)} scanned to {len(concepts)} concepts — too few to be the base(s)"
)
fasit = json.loads((set_dir / "fasit.json").read_text(encoding="utf-8"))
for row in fasit["must_refuse"]:
carried = anchors_are_absent(row["anchors"], concepts)
assert not carried, (
f"{set_dir.name}: {declared['name']} DOES carry {carried} over {len(concepts)} "
f"{set_dir.name}: {', '.join(names)} DOES carry {carried} over {len(concepts)} "
f"concepts, so {row['approach_id']!r} is not un-groundable by rule U"
)
@pytest.mark.parametrize("set_dir", _SETS, ids=_SET_IDS)
def test_e_the_declared_bundle_id_is_the_bases_own(set_dir: Path) -> None:
declared = read_bundle_txt(set_dir / "bundle.txt")
base = _require_base(declared)
resolved = okf.reconcile_bundle_id(base)
assert resolved.id == declared["bundle_id"], (
f"{set_dir.name}: bundle.txt declares {declared['bundle_id']!r} but the base resolves to "
f"{resolved.id!r} (origin {resolved.origin})"
)
for block in read_bundle_txt(set_dir / "bundle.txt"):
base = _require_base(block)
resolved = okf.reconcile_bundle_id(base)
assert resolved.id == block["bundle_id"], (
f"{set_dir.name}: bundle.txt declares {block['bundle_id']!r} for {block['name']} but "
f"the base resolves to {resolved.id!r} (origin {resolved.origin})"
)
# --------------------------------------------------------------------------------------------
@ -350,8 +388,12 @@ def test_known_positive_d_a_mandate_routed_at_another_base_is_caught(tmp_path: P
broken["approaches"][1]["bundle_id"] = "vegnormal-n100-2023"
set_dir = _broken_set(tmp_path, mandate=broken, bundle=_GOOD_BUNDLE_TXT, fasit={})
declared = read_bundle_txt(set_dir / "bundle.txt")
mandate = load_mandate(set_dir / "mandate.json")
assert any(a.bundle_id != declared["bundle_id"] for a in mandate.approaches)
ids = {block["bundle_id"] for block in declared}
routed = {a.bundle_id for a in load_mandate(set_dir / "mandate.json").approaches}
# The SAME two set relations arm (d) asserts, and the broken set must fail the first of them:
# an approach routed at a base the set does not declare.
assert not routed <= ids
assert sorted(routed - ids) == ["vegnormal-n100-2023"]
def test_known_positive_c_an_anchor_the_base_carries_is_reported() -> None:
@ -394,6 +436,34 @@ def test_known_positive_bundle_txt_must_declare_both_keys(tmp_path: Path) -> Non
read_bundle_txt(path)
def test_known_positive_a_second_block_needs_its_own_bundle_id(tmp_path: Path) -> None:
"""P17b: each ``name:`` OPENS a block, and each block closes with its own id.
The half a single-base file cannot exercise: a reader that flattened the file into one
mapping would let the FIRST block's ``bundle_id`` satisfy the second, and the second base
would then be addressed under the first one's name.
"""
path = tmp_path / "bundle.txt"
path.write_text(
"name: n200-2024\nbundle_id: vegnormal-n200-2024\nname: r761-2025\n", encoding="utf-8"
)
with pytest.raises(ValueError, match="bundle_id"):
read_bundle_txt(path)
def test_a_multi_base_bundle_txt_parses_into_one_block_per_base(tmp_path: Path) -> None:
path = tmp_path / "bundle.txt"
path.write_text(
"name: n200-2024\nbundle_id: vegnormal-n200-2024\n"
"name: r761-2025\nbundle_id: vegnormal-r761-2025\n",
encoding="utf-8",
)
assert read_bundle_txt(path) == (
{"name": "n200-2024", "bundle_id": "vegnormal-n200-2024"},
{"name": "r761-2025", "bundle_id": "vegnormal-r761-2025"},
)
@pytest.mark.parametrize("set_dir", _SETS, ids=_SET_IDS)
def test_the_fasit_titles_are_distinct_not_the_collapsed_sources_title(set_dir: Path) -> None:
"""The fasit's recorded titles must tell the cited concepts APART.
@ -409,14 +479,15 @@ def test_the_fasit_titles_are_distinct_not_the_collapsed_sources_title(set_dir:
always wins over a nested one of the same name), so the second half now asserts the opposite —
that ``parse_frontmatter`` agrees with the fasit's own distinct titles — as a live regression
guard against the collapse coming back."""
declared = read_bundle_txt(set_dir / "bundle.txt")
base = _require_base(declared)
bases = _base_by_approach(set_dir)
fasit = json.loads((set_dir / "fasit.json").read_text(encoding="utf-8"))
cited = [c for row in fasit["must_cite"] for c in row["concepts"]]
assert len({c["title"] for c in cited}) == len(cited), "recorded titles do not tell them apart"
cited = [(bases[row["approach_id"]], c) for row in fasit["must_cite"] for c in row["concepts"]]
assert len({c["title"] for _, c in cited}) == len(cited), (
"recorded titles do not tell them apart"
)
titles = {okf.parse_frontmatter(base / c["path"]).get("title", "") for c in cited}
titles = {okf.parse_frontmatter(base / c["path"]).get("title", "") for base, c in cited}
assert len(titles) == len(cited), (
"okf.parse_frontmatter collapsed these titles onto the sources block again — the P15 fix "
"in okf._frontmatter_from_text has regressed"

11
tests/test_prose_code_form_loadbearing.py
View file

@ -140,10 +140,17 @@ def test_a_code_the_baseline_carries_is_never_refused_for_its_shape() -> None:
def test_every_fasit_reference_in_every_context_set_is_an_identifier() -> None:
"""(d) The 26 the order names, with the denominator, plus this repo's own cost code.
"""(d) Every fasit reference across every context set, with the denominator.
One of them — ``Krav 3.3.2—1_1`` — is why the second form grew an optional ``_<n>`` suffix.
Measured, not anticipated: before that it was the single reference the classifier called prose.
**The denominator MOVED 26 -> 32 with P17b's fifth context set**, and it is asserted rather
than dropped for the reason it was written down in the first place: a list comprehension over
``contexts/*/fasit.json`` that quietly found fewer rows would make this arm weaker without
making it red. The six new ones are four ``Krav x.y.z—n`` from n200-2024 and TWO bare
``prosessnr`` from r761-2025 (``12.11``, ``12.12``) — the punctuation-and-digits form B1 added,
now exercised by a fasit and not only by a known-positive.
"""
refs = [
concept["ref"]
@ -151,7 +158,7 @@ def test_every_fasit_reference_in_every_context_set_is_an_identifier() -> None:
for row in json.loads(open(path, encoding="utf-8").read())["must_cite"]
for concept in row["concepts"]
]
assert len(refs) == 26, f"denominator moved: {len(refs)}"
assert len(refs) == 32, f"denominator moved: {len(refs)}"
assert [r for r in refs if not has_identifier_form(r)] == []
assert has_identifier_form("ENERGI-TOTAL-EL"), "this repo's own reference cost code"

143
tests/test_stress_judge_loadbearing.py
View file

@ -486,3 +486,146 @@ def test_k_the_cli_writes_the_verdict_file_and_prints_it(tmp_path: Path) -> None
payload = json.loads(written.read_text(encoding="utf-8"))
assert payload["ferdig"] is True
assert json.loads(proc.stdout)["ferdig"] is True
# --------------------------------------------------------------------------------------------
# P17b DEL 2 — a context set spanning SEVERAL bases is judged ONE base at a time, and the judge
# is told which. Without that restriction the other base's approach is reported
# ``not_evaluated``/``absent``, which is a FALSE finding: that approach WAS evaluated, against the
# other base, under the other ``run_id``.
# --------------------------------------------------------------------------------------------
def _two_base_context(root: Path) -> Path:
ctx = root / "ctx2"
(ctx / "docs").mkdir(parents=True)
(ctx / "bundle.txt").write_text(
"name: minibase\nbundle_id: minibase\nname: otherbase\nbundle_id: otherbase\n",
encoding="utf-8",
)
(ctx / "mandate.json").write_text(
json.dumps(
{
"objective": "o",
"success_criteria": "s",
"approaches": [
{
"id": "a1",
"label": "Here",
"affected_codes": ["CODE-1"],
"claimed_saving_nok": 1000.0,
"bundle_id": "minibase",
},
{
"id": "a2",
"label": "Over there",
"affected_codes": ["CODE-2"],
"claimed_saving_nok": 2000.0,
"bundle_id": "otherbase",
},
],
},
indent=2,
),
encoding="utf-8",
)
(ctx / "fasit.json").write_text(
json.dumps(
{
"project_id": "proj",
"must_cite": [
{
"approach_id": "a1",
"rationale": "why",
"concepts": [{"path": _GOOD, "title": _TITLE, "ref": _REF}],
},
{
"approach_id": "a2",
"rationale": "why",
"concepts": [{"path": _OTHER, "title": "Other", "ref": "Krav 9.9.9-9"}],
},
],
"must_refuse": [],
"honesty": "synthetic",
},
indent=2,
),
encoding="utf-8",
)
return ctx
def test_a_multi_base_set_is_judged_one_base_at_a_time(tmp_path: Path) -> None:
"""Told which base this outbox is for, the judge answers for THAT base's approaches only."""
base = _minibase(tmp_path)
ctx = _two_base_context(tmp_path)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1-minibase", approach_id="a1", tool_calls=_opened(_GOOD))
verdict = stress.score_context_set(ctx, outbox, "r1-minibase", base, bundle_id="minibase")
assert [row.approach_id for row in verdict.approaches] == ["a1"]
assert verdict.bundle_id == "minibase"
assert verdict.approaches[0].status == "validated"
def test_without_the_restriction_the_other_bases_approach_is_falsely_reported_absent(
tmp_path: Path,
) -> None:
"""The defect the restriction removes, stated as a measurement rather than a worry.
This is the UNRESTRICTED call on the same outbox: ``a2`` has no artefact here — it was run
against the other base, under the other ``run_id`` — and the judge reports it as an approach
nobody evaluated, which is exactly the silence ``not_evaluated`` exists to remove.
"""
base = _minibase(tmp_path)
ctx = _two_base_context(tmp_path)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1-minibase", approach_id="a1", tool_calls=_opened(_GOOD))
verdict = stress.score_context_set(ctx, outbox, "r1-minibase", base)
rows = {row.approach_id: row for row in verdict.approaches}
assert set(rows) == {"a1", "a2"}
assert rows["a2"].status == "not_evaluated"
assert rows["a2"].not_evaluated_reason == "absent"
def test_a_base_no_approach_is_routed_at_has_no_denominator(tmp_path: Path) -> None:
base = _minibase(tmp_path)
ctx = _two_base_context(tmp_path)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1-minibase", approach_id="a1")
with pytest.raises(stress.EmptyMeasurement, match="routed at"):
stress.score_context_set(ctx, outbox, "r1-minibase", base, bundle_id="thirdbase")
def test_the_cli_refuses_to_guess_which_base_a_multi_base_outbox_is_for(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""Refused, never guessed: picking would score one base's run against another's fasit rows.
Paired with the rc-0 control below, so "rc 1" cannot be coming from the rest of the argv.
"""
_minibase(tmp_path)
ctx = _two_base_context(tmp_path)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1-minibase", approach_id="a1", tool_calls=_opened(_GOOD))
argv = [
str(ctx),
"--outbox-dir",
str(outbox),
"--run-id",
"r1-minibase",
"--bundle-root",
str(tmp_path),
]
assert stress.main(argv) == 1
assert "--bundle" in capsys.readouterr().err
assert stress.main([*argv, "--bundle", "minibase"]) == 0, "control: naming the base works"
assert stress.main([*argv, "--bundle", "nowhere"]) == 1
assert "nowhere" in capsys.readouterr().err