open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

test(round-builder): the stage the report names, and a denominator sentence that counts

Browse source 
RED FIRST, and only one of these can be red in the ordinary way: the ledger's own sentence about
the five artefact types outside the measured seven is UNTRUE, so the new arm that counts the
outboxes fails on ASSERT (1 of 44 arms red, 0 on ImportError/AttributeError/collection). The
other changes fix ARMS, not code -- src/ is untouched -- so their discrimination is shown by the
mutant campaign in the next commit's message: surviving before, felled after.

- `_STAGE_SENTENCES`: stage -> (the clause the refusal heading opens with, a word only that
  stage's explanation carries), written out here the way `_STATUS_WORDS` is. The arm that names
  where an approach fell built its expected heading from `rb.STAGE_PROSE` and then agreed with
  it: PM's mutant P14 (the sentences for `stage4-p90` and `stage0b-grounding` swapped) passed all
  43 arms, and the report then said an approach fell on the grounding when it fell on the
  uncertainty calculation. Both halves are compared against this file's table, on the ONE heading
  line located by prefix, and the whole table is covered -- not only the two stages the fixture
  exercises, because a fixture is not a denominator.
- That each mark belongs to exactly ONE stage is COUNTED in the test, so swapping explanations
  alone is a failure too.
- `_section(report, heading)`: every `split("## ...")[1]` in the file goes through it. Renaming a
  heading failed four arms on 19.09, three of them on IndexError -- a traceback that says a list
  was too short, not that the report lost the section the expert reads first.
- New arm: `multibase` DOES lie in outboxes that also hold coverage (four of them, two coverage
  files each). The arm counts the outboxes and requires the ledger's sentence to say what the
  count says; the false clause is pinned verbatim so its fourth writing is a failing test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 22:33:28 +02:00
commit 93b6def598
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
1 changed files with 157 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

170
tests/test_round_builder_loadbearing.py
View file

@ -141,6 +141,30 @@ _STATUS_WORDS: dict[str, str] = {
"not_evaluated": "ikke vurdert",
}
#: Stadium → (the clause the refusal heading must OPEN with, a word its EXPLANATION must carry).
#: Written out HERE for ``_STATUS_WORDS``' reason, one level deeper: an arm that asks the binder
#: which SENTENCE it chose and then agrees with it cannot tell one stage from another either.
#:
#: Measured 19.09: swapping the sentences for ``stage4-p90`` and ``stage0b-grounding`` in
#: ``round_builder.STAGE_PROSE`` passed all 43 arms of this file. The report then told the expert
#: that «Færre kummer i kryssene» fell on the GROUNDING when it fell on the uncertainty
#: calculation — the one thing that section exists to say, said wrong, with nothing to catch it.
#:
#: Both halves are independent for a reason. The opening clause is what the expert reads as the
#: stage's NAME, so it is compared whole; the mark is a word only that stage's explanation
#: carries, so swapping the explanations alone is a failure too. That the mark belongs to exactly
#: one stage is COUNTED below rather than trusted.
_STAGE_SENTENCES: dict[str, tuple[str, str]] = {
"stage0-baseline": ("prosjektets egen kostnadsbasis", "budsjettert"),
"stage0b-grounding": ("forankringen i kunnskapsbasen", "levert til kjøringen"),
"stage4-p90": ("usikkerhetsberegningen", "gunstigste ti"),
"stage4b-nominal": ("det nominelt mulige", "beste fall"),
"stage5-method-cap": ("metodetaket", "erfaringsmessig"),
"unsupported": ("manglende krav", "uten hjemmel"),
"other": ("en kontroll rapporten ikke kjenner navnet på", "ordrett"),
"not_evaluated": ("ingenting", "stoppet før"),
}
#: Every artefact type OF THE RUNS MEASURED — the denominator, counted here rather than
#: remembered, and deliberately not called «every type a real run leaves». METHOD: each file
#: ``<run_id>-<rest>.json`` is typed as ``proposal``/``outcome`` when ``<rest>`` ends there and
@ -653,6 +677,21 @@ def _report(tmp_path: Path, **kwargs: Any) -> str:
return (_build(tmp_path, **kwargs).round_dir / "report.md").read_text(encoding="utf-8")
def _section(report: str, heading: str) -> str:
"""The body of ONE ``## `` section, cut out by its heading — and REFUSED, with the headings
the report does carry, when that heading is not there.
Measured 19.09: renaming «## Hva ble vurdert» in the builder failed four arms, but three of
them on ``IndexError`` from ``split(...)[1]`` — a traceback that says a list was too short,
not that the report lost the section a domain expert reads first. A guard that cannot say
what it caught costs the next reader the diagnosis."""
assert heading in report, (
f"the report has no «{heading}» section; its headings are "
f"{[line for line in report.splitlines() if line.startswith('## ')]}"
)
return report.split(heading, 1)[1].split("\n## ", 1)[0]
def _overview(tmp_path: Path, **kwargs: Any) -> list[str]:
"""The report's FIRST SCREEN — «Hva ble vurdert», cut out of the report as LINES.
@ -661,8 +700,7 @@ def _overview(tmp_path: Path, **kwargs: Any) -> list[str]:
19.09 over this file's eleven positive substring-assert sites, against the fixture's own
report: FOUR had a needle on more than one line, and THREE of those four were asserts about
one particular line. Only the arm that claims presence and nothing else still says ``in``."""
section = _report(tmp_path, **kwargs).split("## Hva ble vurdert", 1)[1]
return section.split("\n## ", 1)[0].splitlines()
return _section(_report(tmp_path, **kwargs), "## Hva ble vurdert").splitlines()
def test_the_report_names_every_approach_that_was_considered(tmp_path: Path) -> None:
@ -741,24 +779,44 @@ def test_the_report_states_the_validated_saving_per_approach_and_in_total(tmp_pa
def test_the_report_says_where_and_why_each_refused_approach_fell(tmp_path: Path) -> None:
"""(c). MUTANT: drop a refused approach from the report.
"""(c). MUTANT: drop a refused approach from the report. MUTANT: name the wrong stage.
Every approach that fell is named, with the stage in the expert's words AND the validator's
own sentence — the number of refusal sections is counted from the table, not from the text."""
own sentence — the number of refusal sections is counted from the table, not from the text.
WHICH stage is compared against ``_STAGE_SENTENCES``, never against the binder's own
``STAGE_PROSE``: until 19.09 this arm built its expected heading from the binder's table and
then agreed with it, so swapping two stages' sentences left «falt på forankringen i
kunnskapsbasen» standing over an approach that fell on the uncertainty calculation and passed
all 43 arms. The heading line is located by prefix and required to be the ONLY one, so the
two halves compared afterwards are halves of one known line, not needles loose in the
report."""
text = _report(tmp_path)
for aid, label, status, _nok, _claimed, detail in _SPEC:
if status not in ("rejected", "unsupported"):
continue
assert detail in text, aid
stage = "unsupported" if detail == UNSUPPORTED_REASON else "stage4-p90"
assert f"### {label} — falt på {rb.STAGE_PROSE[stage]}" in text.splitlines(), aid
section = text.split("## Hva falt, og hvorfor", 1)[1].split("\n## ", 1)[0]
opening, mark = _STAGE_SENTENCES[stage]
prefix = f"### {label} — falt på "
headings = [line for line in text.splitlines() if line.startswith(prefix)]
assert len(headings) == 1, (aid, headings)
named, _colon, explained = headings[0][len(prefix) :].partition(": ")
assert named == opening, (aid, headings[0])
assert mark in explained, (aid, headings[0])
section = _section(text, "## Hva falt, og hvorfor")
assert section.count("### ") == len(_FELL) == 2
def test_the_stage_vocabulary_covers_every_stage_the_validator_can_name(tmp_path: Path) -> None:
"""The prose table is pinned to the validator's OWN stage list. A stage added there without a
sentence here would otherwise reach a domain expert as a bare identifier."""
sentence here would otherwise reach a domain expert as a bare identifier.
The KEYS are pinned to the validator; WHICH sentence sits under each key is pinned to
``_STAGE_SENTENCES``, this file's own table. Only the first was guarded before 19.09, and the
two are different claims: every stage had a sentence, and two of the sentences were each
other's. The whole table is covered here and not only the two stages the fixture exercises,
because a fixture is not a denominator."""
from portfolio_optimiser import validator
stages = {stage for stage, _markers in validator._REJECTION_STAGES}
@ -768,6 +826,15 @@ def test_the_stage_vocabulary_covers_every_stage_the_validator_can_name(tmp_path
# Both halves are load-bearing: the short name before the colon is what a one-line diff of
# what changed can carry, and without it the diff falls back on the raw stage id.
assert all(": " in text for text in rb.STAGE_PROSE.values())
assert set(_STAGE_SENTENCES) == set(rb.STAGE_PROSE) and len(_STAGE_SENTENCES) == 8
for stage, (opening, mark) in _STAGE_SENTENCES.items():
named, _colon, explained = rb.STAGE_PROSE[stage].partition(": ")
assert named == opening, stage
assert mark in explained, stage
# The mark is only a discriminator if it belongs to ONE stage, and that is counted here
# rather than assumed: a mark two sentences share would let those two swap unnoticed.
assert sum(mark in sentence for sentence in rb.STAGE_PROSE.values()) == 1, mark
assert len({opening for opening, _mark in _STAGE_SENTENCES.values()}) == 8
def test_the_report_carries_no_json_dump_and_no_unexplained_identifier(tmp_path: Path) -> None:
@ -837,8 +904,7 @@ def test_the_report_follows_the_runs_own_order_not_a_sorted_one(tmp_path: Path)
order the expert reads. A reversed coverage must give a reversed report."""
reversed_spec = tuple(reversed(_SPEC))
source = _outbox(tmp_path / "rev", spec=reversed_spec)
considered = _report(tmp_path / "rev", outbox=source).split("## Hva ble vurdert", 1)[1]
considered = considered.split("\n## ", 1)[0]
considered = _section(_report(tmp_path / "rev", outbox=source), "## Hva ble vurdert")
at = [considered.index(label) for _a, label, *_ in reversed_spec]
assert at == sorted(at), "the report does not follow the coverage's own order"
forward = [considered.index(label) for _a, label, *_ in _SPEC]
@ -871,7 +937,7 @@ def test_the_report_lists_every_approach_the_run_never_reached(tmp_path: Path) -
"1 tilnærminger ble aldri vurdert" followed by nothing is a report contradicting itself in
the one section about work that did not happen — and the expert cannot ask for an approach
they were never shown. Counted from ``_NOT_EVALUATED``."""
section = _report(tmp_path).split("## Hva kjøringen aldri rakk", 1)[1].split("\n## ", 1)[0]
section = _section(_report(tmp_path), "## Hva kjøringen aldri rakk")
assert f"{len(_NOT_EVALUATED)} tilnærminger ble aldri vurdert" in section
assert section.count("- **") == len(_NOT_EVALUATED) == 1
for aid, label, status, *_rest in _SPEC:
@ -1019,11 +1085,11 @@ def test_round_one_reports_what_changed_and_admits_when_nothing_explains_it(
feedback=_feedback_file(tmp_path / "f1.json", "f1"),
)
text = (built.round_dir / "report.md").read_text(encoding="utf-8")
section = text.split("## Endret siden forrige runde", 1)[1].split("\n## ", 1)[0]
section = _section(text, "## Endret siden forrige runde")
assert flipped[1] in section
assert "ingen tilbakemelding forklarer dette" in section.lower()
assert "stage4-p90" not in section, "a raw stage id reached the expert"
assert rb.STAGE_PROSE["stage4-p90"].split(":")[0] in section
assert _STAGE_SENTENCES["stage4-p90"][0] in section
def test_a_removed_approach_is_named_by_the_label_the_expert_saw(tmp_path: Path) -> None:
@ -1042,7 +1108,7 @@ def test_a_removed_approach_is_named_by_the_label_the_expert_saw(tmp_path: Path)
feedback=_feedback_file(tmp_path / "f1.json", "f1"),
)
text = (built.round_dir / "report.md").read_text(encoding="utf-8")
section = text.split("## Endret siden forrige runde", 1)[1].split("\n## ", 1)[0]
section = _section(text, "## Endret siden forrige runde")
label = {aid: lab for aid, lab, *_rest in _SPEC}[gone]
assert f"**{label}** ({gone})" in section
assert f"- **{gone}** —" not in section, "the raw id reached the expert without its label"
@ -1206,3 +1272,81 @@ def test_the_help_text_states_the_shape_it_writes(tmp_path: Path) -> None:
)
assert out.returncode == 0
assert "outcome.json" in out.stdout
# ---------------------------------------------------------------------------------------------
# The denominator's own sentence — counted, not remembered
# ---------------------------------------------------------------------------------------------
#: The five artefact types ``run.py`` can write that the SEVEN measured above do not contain.
#: Named here because the claim being pinned is about exactly these five, and a claim about a
#: set has to carry the set.
_TYPES_OUTSIDE_THE_SEVEN = (
"exploration",
"prepass",
"multibase",
"plan-review",
"proposal-reviews",
)
#: The false clause, verbatim, in the form it was written THREE times. Pinned as a needle so the
#: fourth writing of it is a failing test rather than a review finding.
_THE_FALSE_CLAUSE = "ingen av dem ligger i dag i en utboks som også har coverage"
_NUMERAL = {0: "null", 1: "én", 2: "to", 3: "tre", 4: "fire", 5: "fem", 6: "seks"}
def _outboxes_by_type(root: Path) -> dict[str, tuple[int, list[Path]]]:
"""Per artefact type: how many files it has under ``root``, and the outboxes holding both it
and a coverage. The outbox is the DIRECTORY — that is the unit the seven-type union counts."""
measured: dict[str, tuple[int, list[Path]]] = {}
for kind in _TYPES_OUTSIDE_THE_SEVEN:
files = sorted(root.rglob(f"*-{kind}.json"))
with_coverage = sorted({f.parent for f in files if any(f.parent.glob("*-coverage.json"))})
measured[kind] = (len(files), with_coverage)
return measured
def test_the_ledgers_claim_about_the_five_other_types_is_what_the_repo_measures() -> None:
"""The sentence under ``docs/invarianter.md``'s seven-type row has now been rewritten three
times, and the third writing was still untrue: «``exploration``, ``prepass``, ``multibase``,
``plan-review`` og ``proposal-reviews`` … ingen av dem ligger i dag i en utboks som også har
coverage». Measured 19.09 in this working tree: ``multibase`` lies in FOUR outboxes that also
hold coverage (``scratchpad/{p17b-multibase,p20-stress,p21-stress,p22-stress}/lindaas``), and
each of those outboxes holds TWO coverage files — which is why they fall outside the «exactly
one coverage» rule the union of seven is counted over, and at the same time why seven is a
FLOOR and not a ceiling.
This arm counts the outboxes and requires the ledger's sentence to say what the count says.
The count is the independent side: the sentence is not read back into the measurement.
``scratchpad/`` is untracked, so an extract without it cannot measure this and SKIPS — but a
present ``scratchpad/`` with no coverage file at all is a broken query, not a measurement,
and fails."""
repo = Path(rb.__file__).resolve().parents[3]
root = repo / "scratchpad"
if not root.is_dir():
pytest.skip("scratchpad/ is untracked and absent from this checkout")
coverage_files = sorted(root.rglob("*-coverage.json"))
assert coverage_files, "control: scratchpad/ holds no coverage file, so nothing was measured"
measured = _outboxes_by_type(root)
in_a_coverage_outbox = {kind: dirs for kind, (_n, dirs) in measured.items() if dirs}
assert list(in_a_coverage_outbox) == ["multibase"], measured
multibase_outboxes = in_a_coverage_outbox["multibase"]
for outbox in multibase_outboxes:
assert len(list(outbox.glob("*-coverage.json"))) == 2, str(outbox)
row = [
block
for block in (repo / "docs" / "invarianter.md").read_text(encoding="utf-8").split("\n- ")
if all(f"`{kind}`" in block for kind in _TYPES_OUTSIDE_THE_SEVEN)
]
assert len(row) == 1, f"the seven-type row is not one block: {len(row)}"
sentence = row[0]
assert _THE_FALSE_CLAUSE not in sentence, "the false clause was written a fourth time"
assert _THE_FALSE_CLAUSE not in Path(__file__).read_text(encoding="utf-8").replace(
f'_THE_FALSE_CLAUSE = "{_THE_FALSE_CLAUSE}"', ""
), "the false clause is back in this file's own comment"
assert f"{_NUMERAL[len(multibase_outboxes)]} utbokser" in sentence, sentence[-400:]
assert f"{_NUMERAL[2]} coverage-filer" in sentence, sentence[-400:]