open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity
portfolio-optimiser/tests/test_stress_judge_loadbearing.py
Kjell Tore Guttormsen 7b4f85d77c feat(p21): the PROJECT carries the price, so a run against a road normal can be anchored 
Four paid stress rounds ran entirely UN-ANCHORED, all of them, because the one file
loader reads cost-baseline.json out of the BUNDLE and no vegnormal ships one: N100,
N200, N500 and R761 are knowledge, and knowledge carries requirements, never amounts.
The validator's stage 0 -- the one stage that tells an invented cost line from a line
this project actually buys -- was skipped in every single run, so "validated" could not
mean what it says. P20 G1/G2 measured real R761 process numbers (12.11 three times on
Soraasen, 1.1.1 on Lindaas) validating with amounts nobody had anywhere.

--cost-baseline FILE is PM decision (e), taken over the three alternatives P20 wrote
down. A LOADED object, never a path (prepass_payload's rule): the CLI owns the file and
loads it ONCE, so the notice, the stamp and every base of an --across-bundle pass all
descend from one read. ONE parse, two doors -- load_cost_baseline delegates to
load_cost_baseline_file -- while safe_resolve stays on the bundle door alone, because a
project's own schedule is legitimately outside every base. No tolerant twin: this path
exists only because an operator NAMED a file.

DEL B: five anchored context sets, a1-a3 with their line and a4 with none, so stage 0 is
what catches the falsification arm. THE ORDER'S OWN ARM (h) WAS FELLED BY MEASUREMENT:
"no baseline code is a requirement number the base declares" is measured 0 of 4 on the
project-coded sets and 5 of 5 on kontrakt-sorasen -- which is what R761 Prosesskoden IS,
a bill of quantities priced BY process code. The complement keeps both, and the order's
own mutation still bites.

DEL B3: the judge reports anchored (off the run's own stamp), priced per row, and WHICH
falsifier caught the falsification arm.

Load-bearing MEASURED, five mutations all red against the WHOLE suite, green control
1850/5 (from 1809/5, superset, 0 removed), golden byte-unchanged:
A3(i) the flag is read but the baseline is unused (3 red) . A3(ii) only the first base
gets it (1) . A3(iii) report_forbidden drops it (1) . B2(i) a4 gets a line (1, arm (g)
alone) . B2(ii) a code swapped to 12.11 (2, arms (f) and (h)).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-15 10:49:10 +02:00

753 lines
30 KiB
Python
Raw Blame History

"""P16 DEL A - the stress judge: "beviselig virkning against the base" made DETERMINISTIC.
**The measured silence.** Session 102's criterion ((a) the run builds on the right fasit concept OR
refuses anchored * (b') it NAMES that concept * (c) zero hallucinations) was adjudicated BY HAND.
A hand-read criterion cannot be repeated four times now and N times later, and nothing in the tree
read ``contexts/<set>/fasit.json`` against an outbox at all (measured 14.09: ``grep -rln fasit src
tests`` hit only unrelated files).
``portfolio_optimiser.stress`` reads the artefacts that already carry the evidence -
``{run_id}[-{approach}]-proposal.json`` (proposal + provenance citations),
``-outcome.json`` (validated / rejected), ``{run_id}-debate.json`` (``tool_calls[]`` with
``name``/``bundle_id``/``path``, S2c) - and returns one typed verdict per commissioned approach.
**THE ORDER'S (a) WAS VACUOUS AS WRITTEN, AND THAT IS MEASURED.** The order defines grounded as
"a must_cite path was OPENED *or* CITED". But on the S2c navigation path ``run_project`` stamps
``citations = bundle_citations(bundle)``, which is ONE CITATION PER CONTEXT FILE - the whole corpus.
Measured on n100-2023: 446 context files, 446 citations, and **6 of 6 fasit paths already "cited"
before a single model call**. A judge honouring the order literally would be a gate that can only
be green, which is the repo's own vacuous-gate class, inside the gate built to stop it. So
``grounded`` counts a CITATION only when the citation list is NARROWER than the base (a declared
pre-pass cut); a whole-base list is reported as such and carries nothing. Both halves are reported
either way, so the operator can read which one fired - the deviation is stated, never silent.
**(b') was checked for the same vacuity and is CLEAN.** ``bundle_citations`` snippets are concept
BODIES, and the ``ref``/``title`` live in FRONTMATTER: measured 0 of 446 n100 bodies contain
``Krav 4.1.2-1``. So the snippet arm can carry (b') without being satisfied by construction, and
the order's definition is kept. Which half fired is still reported.
**A denominator, always** (Verifiseringsloven ansikt 4): every verdict names how many tool calls,
citations and approach rows it saw. An outbox with no artefacts RAISES rather than reporting
"0 hallucinations" - an empty measurement is not a clean bill of health.
**a4 / must_refuse is the falsification half in D-1 form.** ``po`` is not a lookup tool, so an
"unanswerable question" has no runnable form; a commissioned approach whose GROUND the base does
not carry does. It passes iff no ``validated`` row is that approach and no validated proposal
carries its code.
Arms: (a) grounded by an opened path * (b) a whole-base citation list cannot ground * (c) a
narrowed list can * (d) "refuses anchored" counts * (e) (b') by measure and by snippet *
(f) hallucinated citation files, read paths and codes * (g) must_refuse passes and fails *
(h) an empty outbox raises * (i) an empty base raises * (j) not_evaluated is absence of an
artefact * (k) the CLI writes the verdict file.
"""
from __future__ import annotations
import json
import subprocess
import sys
from pathlib import Path
import pytest
from portfolio_optimiser import stress
# --------------------------------------------------------------------------------------------
# A synthetic minibase + a synthetic outbox. Nothing here touches a real bundle or a model.
# --------------------------------------------------------------------------------------------
_GOOD = "krav/N1/id-good.md"
_OTHER = "krav/N1/id-other.md"
_REF = "Krav 1.2.3-4"
_TITLE = "Krav 1.2.3-4 Rundkjoringer"
def _minibase(root: Path) -> Path:
base = root / "minibase"
(base / "krav" / "N1").mkdir(parents=True)
(base / "index.md").write_text(
"---\nbundle_id: minibase\n---\n\n- [good](krav/N1/id-good.md)\n"
"- [other](krav/N1/id-other.md)\n",
encoding="utf-8",
)
(base / _GOOD).write_text(
f'---\ntype: concept\ntitle: "{_TITLE}"\nreq_number: "{_REF}"\n---\n\nBody of the good one.\n',
encoding="utf-8",
)
(base / _OTHER).write_text(
'---\ntype: concept\ntitle: "Other"\nreq_number: "Krav 9.9.9-9"\n---\n\nAnother body.\n',
encoding="utf-8",
)
return base
def _context(
root: Path, *, must_refuse: bool = True, schedule: dict[str, float] | None = None
) -> Path:
ctx = root / "ctx"
(ctx / "docs").mkdir(parents=True)
if schedule is not None:
# P21 B3: the PROJECT's own price schedule, which is the file a run is handed with
# ``--cost-baseline`` and the one the judge measures ``priced`` against.
(ctx / "cost-baseline.json").write_text(
json.dumps(
{
"project_id": "proj",
"items": {
code: {"quantity": 1.0, "unit_cost": unit}
for code, unit in schedule.items()
},
},
indent=2,
),
encoding="utf-8",
)
(ctx / "bundle.txt").write_text("name: minibase\nbundle_id: minibase\n", encoding="utf-8")
approaches = [
{
"id": "a1",
"label": "First approach",
"description": "",
"affected_codes": ["CODE-1"],
"claimed_saving_nok": 1000.0,
"bundle_id": "minibase",
}
]
fasit: dict[str, object] = {
"project_id": "proj",
"bundle": "minibase",
"bundle_id": "minibase",
"must_cite": [
{
"approach_id": "a1",
"rationale": "why",
"concepts": [{"path": _GOOD, "title": _TITLE, "ref": _REF}],
}
],
"must_refuse": [],
"honesty": "synthetic",
}
if must_refuse:
approaches.append(
{
"id": "a4",
"label": "Unit price cut",
"description": "",
"affected_codes": ["CODE-4"],
"claimed_saving_nok": 500.0,
"bundle_id": "minibase",
}
)
fasit["must_refuse"] = [
{"approach_id": "a4", "anchors": ["enhetspris"], "rationale": "base carries no prices"}
]
(ctx / "mandate.json").write_text(
json.dumps(
{"objective": "o", "success_criteria": "s", "approaches": approaches},
ensure_ascii=False,
indent=2,
),
encoding="utf-8",
)
(ctx / "fasit.json").write_text(
json.dumps(fasit, ensure_ascii=False, indent=2), encoding="utf-8"
)
return ctx
def _write_outbox(
outbox: Path,
run_id: str,
*,
approach_id: str,
measure: str = f"Simplify per {_REF}",
codes: list[str] | None = None,
citation_files: list[str] | None = None,
citation_snippet: str = "Body of the good one.",
decision: str = "validated",
tool_calls: list[dict[str, str]] | None = None,
anchored: bool = False,
reason: str = "no",
) -> None:
outbox.mkdir(parents=True, exist_ok=True)
codes = ["CODE-1"] if codes is None else codes
citation_files = [_GOOD, _OTHER] if citation_files is None else citation_files
stem = f"{run_id}-{approach_id}"
(outbox / f"{stem}-proposal.json").write_text(
json.dumps(
{
"run_id": run_id,
"approach_id": approach_id,
"proposal": {
"project_id": "proj",
"measure": measure,
"affected_items": [
{"code": c, "quantity": 1.0, "unit_cost": 2000.0} for c in codes
],
"claimed_saving_nok": 1000.0,
"assumptions": {},
},
"provenance": {
"citations": [
{
"file": f,
"locator": {"start_index": 0, "end_index": 1},
"snippet": citation_snippet,
}
for f in citation_files
],
"model": "m",
"role": "proposer",
"validator_decision": decision,
"token_usage": 10,
"cost_baseline_anchored": anchored,
"bundle_id_source": None,
"external_calls": [],
},
},
indent=2,
),
encoding="utf-8",
)
(outbox / f"{stem}-outcome.json").write_text(
json.dumps(
{
"run_id": run_id,
"approach_id": approach_id,
"outcome_type": "validated" if decision == "validated" else "rejected",
**({"reason": reason} if decision != "validated" else {}),
"checker_verdict": None,
"verdict_id": "vid",
},
indent=2,
),
encoding="utf-8",
)
if tool_calls is not None:
(outbox / f"{run_id}-debate.json").write_text(
json.dumps({"run_id": run_id, "tool_calls": tool_calls}, indent=2), encoding="utf-8"
)
def _opened(path: str) -> list[dict[str, str]]:
return [{"name": "read_file", "bundle_id": "minibase", "path": path}]
def _judge(tmp_path: Path, **kw: object) -> stress.ContextSetVerdict:
base = _minibase(tmp_path)
schedule = kw.pop("schedule", None)
ctx = _context(
tmp_path,
must_refuse=bool(kw.pop("must_refuse", False)),
schedule=schedule, # type: ignore[arg-type]
)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1", approach_id="a1", **kw) # type: ignore[arg-type]
return stress.score_context_set(ctx, outbox, "r1", base)
# --------------------------------------------------------------------------------------------
# (a) grounded by an OPENED path - the non-vacuous half.
# --------------------------------------------------------------------------------------------
def test_a_a_must_cite_path_that_was_opened_grounds_the_approach(tmp_path: Path) -> None:
verdict = _judge(tmp_path, tool_calls=_opened(_GOOD))
row = verdict.approaches[0]
assert row.grounded is True
assert row.opened == (_GOOD,)
def test_a_opening_some_other_document_does_not_ground_it(tmp_path: Path) -> None:
verdict = _judge(tmp_path, tool_calls=_opened(_OTHER))
row = verdict.approaches[0]
assert row.grounded is False
assert row.opened == ()
# --------------------------------------------------------------------------------------------
# (b)+(c) the MEASURED vacuity: a whole-base citation list cannot ground; a narrowed one can.
# --------------------------------------------------------------------------------------------
def test_b_a_whole_base_citation_list_cannot_ground_an_approach(tmp_path: Path) -> None:
"""Measured on n100-2023: 446 context files, 446 citations, 6/6 fasit paths 'cited' before
any model call. Honouring the order literally would make (a) green by construction."""
verdict = _judge(tmp_path, tool_calls=[])
row = verdict.approaches[0]
assert row.cited == (_GOOD,), "the path IS in the citation list"
assert row.citation_scope == "whole-base"
assert row.grounded is False, "a whole-base list is stamped before any model work"
def test_c_a_narrowed_citation_list_does_ground_an_approach(tmp_path: Path) -> None:
verdict = _judge(tmp_path, tool_calls=[], citation_files=[_GOOD])
row = verdict.approaches[0]
assert row.citation_scope == "narrowed"
assert row.grounded is True
# --------------------------------------------------------------------------------------------
# (d) "refuses anchored" counts as (a) - session 102's own second limb.
# --------------------------------------------------------------------------------------------
def test_d_a_rejected_approach_that_opened_the_requirement_is_grounded(tmp_path: Path) -> None:
verdict = _judge(tmp_path, decision="rejected", tool_calls=_opened(_GOOD))
row = verdict.approaches[0]
assert row.status == "rejected"
assert row.grounded is True
# --------------------------------------------------------------------------------------------
# (e) (b') named - by the model's own prose, or by a citation snippet.
# --------------------------------------------------------------------------------------------
def test_e_the_ref_in_the_measure_names_the_concept(tmp_path: Path) -> None:
row = _judge(tmp_path, tool_calls=_opened(_GOOD)).approaches[0]
assert row.named is True
assert row.named_in_measure is True
def test_e_a_measure_that_names_nothing_is_carried_only_by_a_narrowed_snippet(
tmp_path: Path,
) -> None:
"""The snippet arm still carries (b') — but only under a NARROWED scope (P18/C2)."""
row = _judge(
tmp_path,
measure="Do it cheaper",
citation_files=[_GOOD],
citation_snippet=f"see {_REF}",
tool_calls=_opened(_GOOD),
).approaches[0]
assert row.citation_scope == "narrowed"
assert row.named_in_measure is False
assert row.named_in_snippet is True
assert row.named is True
def test_e_a_whole_base_snippet_does_not_name_the_concept(tmp_path: Path) -> None:
"""P18/C2 (PM decision, P16 § 6.2). ``bundle_citations`` stamps EVERY context file before any
model call, so a mark found in a whole-base snippet is evidence about what the base contains,
not about what this run said. Measured on r761: the process number ``12.1`` stands in the
bodies themselves, so that row came back ``named`` for a run that never named it.
The pair with the arm above is the discriminator: the SAME snippet, the same mark, and the only
difference is the scope."""
row = _judge(
tmp_path,
measure="Do it cheaper",
citation_files=[_GOOD, _OTHER],
citation_snippet=f"see {_REF}",
tool_calls=_opened(_GOOD),
).approaches[0]
assert row.citation_scope == "whole-base"
assert row.named_in_snippet is False
assert row.named is False
def test_e_naming_neither_way_fails_b_prime(tmp_path: Path) -> None:
row = _judge(tmp_path, measure="Do it cheaper", tool_calls=_opened(_GOOD)).approaches[0]
assert row.named is False
assert row.ferdig is False
# --------------------------------------------------------------------------------------------
# (f) (c) hallucinations: citation files, read paths, and codes.
# --------------------------------------------------------------------------------------------
def test_f_a_citation_to_a_file_the_base_does_not_carry_is_a_hallucination(
tmp_path: Path,
) -> None:
row = _judge(
tmp_path, citation_files=[_GOOD, "krav/N1/id-invented.md"], tool_calls=_opened(_GOOD)
).approaches[0]
assert row.hallucinations == ("citation:krav/N1/id-invented.md",)
assert row.ferdig is False
def test_f_a_read_path_the_base_does_not_carry_is_a_hallucination(tmp_path: Path) -> None:
verdict = _judge(tmp_path, tool_calls=_opened(_GOOD) + _opened("krav/N1/id-guessed.md"))
assert verdict.hallucinated_reads == ("krav/N1/id-guessed.md",)
assert verdict.approaches[0].ferdig is False
def test_f_a_code_in_neither_the_mandate_nor_the_baseline_is_a_hallucination(
tmp_path: Path,
) -> None:
row = _judge(tmp_path, codes=["CODE-1", "INVENTED-9"], tool_calls=_opened(_GOOD)).approaches[0]
assert row.hallucinations == ("code:INVENTED-9",)
def test_f_a_clean_run_is_ferdig(tmp_path: Path) -> None:
verdict = _judge(tmp_path, tool_calls=_opened(_GOOD))
row = verdict.approaches[0]
assert (row.grounded, row.named, row.hallucinations) == (True, True, ())
assert row.ferdig is True
# --------------------------------------------------------------------------------------------
# (g) must_refuse / a4 - the falsification arm.
# --------------------------------------------------------------------------------------------
def test_g_an_a4_nobody_validated_passes(tmp_path: Path) -> None:
base = _minibase(tmp_path)
ctx = _context(tmp_path, must_refuse=True)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1", approach_id="a1", tool_calls=_opened(_GOOD))
verdict = stress.score_context_set(ctx, outbox, "r1", base)
assert verdict.must_refuse[0].passed is True
assert verdict.must_refuse[0].approach_id == "a4"
def test_g_a_validated_a4_is_a_falsification_failure(tmp_path: Path) -> None:
base = _minibase(tmp_path)
ctx = _context(tmp_path, must_refuse=True)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1", approach_id="a1", tool_calls=_opened(_GOOD))
_write_outbox(outbox, "r1", approach_id="a4", codes=["CODE-4"], decision="validated")
verdict = stress.score_context_set(ctx, outbox, "r1", base)
assert verdict.must_refuse[0].passed is False
assert "VALIDATED" in verdict.must_refuse[0].detail
def test_g_a4s_code_inside_another_validated_proposal_also_fails(tmp_path: Path) -> None:
base = _minibase(tmp_path)
ctx = _context(tmp_path, must_refuse=True)
outbox = tmp_path / "out"
_write_outbox(
outbox, "r1", approach_id="a1", codes=["CODE-1", "CODE-4"], tool_calls=_opened(_GOOD)
)
verdict = stress.score_context_set(ctx, outbox, "r1", base)
assert verdict.must_refuse[0].passed is False
# --------------------------------------------------------------------------------------------
# (h)+(i) denominators: an empty measurement is never a clean bill of health.
# --------------------------------------------------------------------------------------------
def test_h_an_outbox_with_no_artefacts_raises_never_reports_zero_hallucinations(
tmp_path: Path,
) -> None:
base = _minibase(tmp_path)
ctx = _context(tmp_path)
empty = tmp_path / "out"
empty.mkdir()
with pytest.raises(stress.EmptyMeasurement, match="no proposal artefact"):
stress.score_context_set(ctx, empty, "r1", base)
def test_i_a_base_that_scans_to_nothing_raises(tmp_path: Path) -> None:
ctx = _context(tmp_path)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1", approach_id="a1", tool_calls=_opened(_GOOD))
hollow = tmp_path / "hollow"
hollow.mkdir()
(hollow / "index.md").write_text("---\nbundle_id: minibase\n---\n\nnothing\n", encoding="utf-8")
with pytest.raises(stress.EmptyMeasurement, match="concepts"):
stress.score_context_set(ctx, outbox, "r1", hollow)
def test_h_the_denominators_are_always_reported(tmp_path: Path) -> None:
verdict = _judge(tmp_path, tool_calls=_opened(_GOOD))
assert verdict.tool_calls_seen == 1
assert verdict.citations_seen == 2
assert verdict.approach_rows_seen == 1
assert verdict.concepts_in_base == 2
# --------------------------------------------------------------------------------------------
# (j) a commissioned approach with NO artefact is not_evaluated, never omitted.
# --------------------------------------------------------------------------------------------
def test_j_a_commissioned_approach_with_no_artefact_is_not_evaluated(tmp_path: Path) -> None:
base = _minibase(tmp_path)
ctx = _context(tmp_path, must_refuse=True)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1", approach_id="a1", tool_calls=_opened(_GOOD))
verdict = stress.score_context_set(ctx, outbox, "r1", base)
rows = {r.approach_id: r for r in verdict.approaches}
assert rows["a4"].status == "not_evaluated"
assert rows["a4"].ferdig is False
assert len(verdict.approaches) == 2, "an omitted row reads as an approach nobody ordered"
# --------------------------------------------------------------------------------------------
# (k) the CLI writes the verdict beside the artefacts it judged.
# --------------------------------------------------------------------------------------------
def test_k_the_cli_writes_the_verdict_file_and_prints_it(tmp_path: Path) -> None:
base = _minibase(tmp_path)
ctx = _context(tmp_path)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1", approach_id="a1", tool_calls=_opened(_GOOD))
proc = subprocess.run(
[
sys.executable,
"-m",
"portfolio_optimiser.stress",
str(ctx),
"--outbox-dir",
str(outbox),
"--run-id",
"r1",
"--bundle-root",
str(base.parent),
],
capture_output=True,
text=True,
cwd=Path(__file__).resolve().parents[1],
)
assert proc.returncode == 0, proc.stderr
written = outbox / "r1-verdict.json"
assert written.is_file()
payload = json.loads(written.read_text(encoding="utf-8"))
assert payload["ferdig"] is True
assert json.loads(proc.stdout)["ferdig"] is True
# --------------------------------------------------------------------------------------------
# P17b DEL 2 — a context set spanning SEVERAL bases is judged ONE base at a time, and the judge
# is told which. Without that restriction the other base's approach is reported
# ``not_evaluated``/``absent``, which is a FALSE finding: that approach WAS evaluated, against the
# other base, under the other ``run_id``.
# --------------------------------------------------------------------------------------------
def _two_base_context(root: Path) -> Path:
ctx = root / "ctx2"
(ctx / "docs").mkdir(parents=True)
(ctx / "bundle.txt").write_text(
"name: minibase\nbundle_id: minibase\nname: otherbase\nbundle_id: otherbase\n",
encoding="utf-8",
)
(ctx / "mandate.json").write_text(
json.dumps(
{
"objective": "o",
"success_criteria": "s",
"approaches": [
{
"id": "a1",
"label": "Here",
"affected_codes": ["CODE-1"],
"claimed_saving_nok": 1000.0,
"bundle_id": "minibase",
},
{
"id": "a2",
"label": "Over there",
"affected_codes": ["CODE-2"],
"claimed_saving_nok": 2000.0,
"bundle_id": "otherbase",
},
],
},
indent=2,
),
encoding="utf-8",
)
(ctx / "fasit.json").write_text(
json.dumps(
{
"project_id": "proj",
"must_cite": [
{
"approach_id": "a1",
"rationale": "why",
"concepts": [{"path": _GOOD, "title": _TITLE, "ref": _REF}],
},
{
"approach_id": "a2",
"rationale": "why",
"concepts": [{"path": _OTHER, "title": "Other", "ref": "Krav 9.9.9-9"}],
},
],
"must_refuse": [],
"honesty": "synthetic",
},
indent=2,
),
encoding="utf-8",
)
return ctx
def test_a_multi_base_set_is_judged_one_base_at_a_time(tmp_path: Path) -> None:
"""Told which base this outbox is for, the judge answers for THAT base's approaches only."""
base = _minibase(tmp_path)
ctx = _two_base_context(tmp_path)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1-minibase", approach_id="a1", tool_calls=_opened(_GOOD))
verdict = stress.score_context_set(ctx, outbox, "r1-minibase", base, bundle_id="minibase")
assert [row.approach_id for row in verdict.approaches] == ["a1"]
assert verdict.bundle_id == "minibase"
assert verdict.approaches[0].status == "validated"
def test_without_the_restriction_the_other_bases_approach_is_falsely_reported_absent(
tmp_path: Path,
) -> None:
"""The defect the restriction removes, stated as a measurement rather than a worry.
This is the UNRESTRICTED call on the same outbox: ``a2`` has no artefact here — it was run
against the other base, under the other ``run_id`` — and the judge reports it as an approach
nobody evaluated, which is exactly the silence ``not_evaluated`` exists to remove.
"""
base = _minibase(tmp_path)
ctx = _two_base_context(tmp_path)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1-minibase", approach_id="a1", tool_calls=_opened(_GOOD))
verdict = stress.score_context_set(ctx, outbox, "r1-minibase", base)
rows = {row.approach_id: row for row in verdict.approaches}
assert set(rows) == {"a1", "a2"}
assert rows["a2"].status == "not_evaluated"
assert rows["a2"].not_evaluated_reason == "absent"
def test_a_base_no_approach_is_routed_at_has_no_denominator(tmp_path: Path) -> None:
base = _minibase(tmp_path)
ctx = _two_base_context(tmp_path)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1-minibase", approach_id="a1")
with pytest.raises(stress.EmptyMeasurement, match="routed at"):
stress.score_context_set(ctx, outbox, "r1-minibase", base, bundle_id="thirdbase")
def test_the_cli_refuses_to_guess_which_base_a_multi_base_outbox_is_for(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""Refused, never guessed: picking would score one base's run against another's fasit rows.
Paired with the rc-0 control below, so "rc 1" cannot be coming from the rest of the argv.
"""
_minibase(tmp_path)
ctx = _two_base_context(tmp_path)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1-minibase", approach_id="a1", tool_calls=_opened(_GOOD))
argv = [
str(ctx),
"--outbox-dir",
str(outbox),
"--run-id",
"r1-minibase",
"--bundle-root",
str(tmp_path),
]
assert stress.main(argv) == 1
assert "--bundle" in capsys.readouterr().err
assert stress.main([*argv, "--bundle", "minibase"]) == 0, "control: naming the base works"
assert stress.main([*argv, "--bundle", "nowhere"]) == 1
assert "nowhere" in capsys.readouterr().err
# --------------------------------------------------------------------------------------------
# P21 B3: the judge reports what the run was ANCHORED on, which codes the project actually PRICES,
# and WHICH falsifier caught the falsification arm.
#
# The measured reason. Rounds 1-4 all ran un-anchored — a vegnormal ships no ``cost-baseline.json``
# and the only file loader read one out of the bundle — so stage 0 never spoke and the ``a4`` arm
# fell, when it fell, on P7's grounding check. "It was refused" and "the stage that knows what this
# project buys refused it" are different facts, and only the second is what anchoring bought.
# --------------------------------------------------------------------------------------------
def test_p21_priced_is_true_when_the_project_schedule_carries_the_code(tmp_path: Path) -> None:
verdict = _judge(tmp_path, schedule={"CODE-1": 2000.0})
assert verdict.approaches[0].priced is True
def test_p21_priced_is_false_for_a_code_the_project_does_not_buy(tmp_path: Path) -> None:
"""The discriminator: the SAME schedule, a proposal on a code it does not carry."""
verdict = _judge(tmp_path, schedule={"CODE-1": 2000.0}, codes=["CODE-9"])
assert verdict.approaches[0].priced is False
# ... and the control, so the arm cannot be green by measuring nothing.
assert (
_judge(tmp_path / "b", schedule={"CODE-9": 2000.0}, codes=["CODE-9"]).approaches[0].priced
is True
)
def test_p21_priced_is_false_without_a_project_schedule(tmp_path: Path) -> None:
"""Every round before P21: no schedule, so nothing is priced — reported, never guessed."""
assert _judge(tmp_path).approaches[0].priced is False
def test_p21_anchored_follows_the_runs_own_stamp(tmp_path: Path) -> None:
"""Read off ``provenance.cost_baseline_anchored``, not re-derived from the set's files.
Both arms, because a field that is constant is not a measurement: an artefact stamped
un-anchored must report ``False`` EVEN WHEN the set ships a schedule — the judge says what the
run did, and a run that was never given the file is not anchored by the file existing.
"""
assert _judge(tmp_path, anchored=True, schedule={"CODE-1": 2000.0}).anchored is True
assert _judge(tmp_path / "b", anchored=False, schedule={"CODE-1": 2000.0}).anchored is False
def test_p21_the_falsification_arm_reports_which_stage_caught_it(tmp_path: Path) -> None:
"""``stage0-baseline`` is the answer anchoring buys; ``stage0b-grounding`` is what round 4 got.
Driven through the whole judge rather than through ``rejection_stage`` alone, because the
seam being gated is that the judge READS the arm's own outcome artefact — a classifier that
was never called would leave every arm reporting ``""`` and the arms below still green.
"""
base = _minibase(tmp_path)
ctx = _context(tmp_path, must_refuse=True, schedule={"CODE-1": 2000.0})
outbox = tmp_path / "out"
_write_outbox(outbox, "r1", approach_id="a1", tool_calls=_opened(_GOOD))
_write_outbox(
outbox,
"r1",
approach_id="a4",
codes=["CODE-4"],
decision="rejected",
reason=("unknown cost code 'CODE-4': not in project proj's cost baseline (1 known codes)"),
)
verdict = stress.score_context_set(ctx, outbox, "r1", base)
assert [(r.approach_id, r.passed, r.stage) for r in verdict.must_refuse] == [
("a4", True, "stage0-baseline")
]
other = tmp_path / "b"
base2 = _minibase(other)
ctx2 = _context(other, must_refuse=True)
outbox2 = other / "out"
_write_outbox(outbox2, "r1", approach_id="a1", tool_calls=_opened(_GOOD))
_write_outbox(
outbox2,
"r1",
approach_id="a4",
codes=["CODE-4"],
decision="rejected",
reason="ungrounded identifier 'CODE-4': it appears nowhere in the input (10 chars)",
)
assert stress.score_context_set(ctx2, outbox2, "r1", base2).must_refuse[0].stage == (
"stage0b-grounding"
)
def test_p21_a_validated_falsification_arm_reports_no_stage(tmp_path: Path) -> None:
"""``""`` when nothing refused it — an honest absence, never a stage nobody reached."""
base = _minibase(tmp_path)
ctx = _context(tmp_path, must_refuse=True)
outbox = tmp_path / "out"
_write_outbox(outbox, "r1", approach_id="a1", tool_calls=_opened(_GOOD))
_write_outbox(outbox, "r1", approach_id="a4", codes=["CODE-4"])
verdict = stress.score_context_set(ctx, outbox, "r1", base)
assert verdict.must_refuse[0].passed is False
assert verdict.must_refuse[0].stage == ""
Reference in a new issue View git blame Copy permalink