fix(tests): blade 2 was testing the wrong defect, and said so out loud

An independent reviewer found it and the claim was verified by measurement
before being accepted, not taken on trust.

test_portfolio_scripted_pass_makes_no_real_client raised from the patched
_default_factory to prove the scripted portfolio pass never builds a production
client. It cannot: the factory is called inside run_project's coroutine, and
run_portfolio gathers with return_exceptions=True, so the AssertionError was
collected into a RunFailure and never escaped. Measured directly -- with BOTH
the client_factory wiring and the rc rule detached, the blade stayed GREEN. It
was going red on rc alone, which means it was testing defect B while claiming
to test defect A. The original seven-mutation sweep did not catch this because
each mutation was applied singly, and dropping the wiring alone still flips rc.

Replaced with a call sentinel: a list appended inside the factory and asserted
in the test body, which the wave handler cannot swallow. Re-measured -- red on
the wiring detach alone, and red on both detaches together.

The docstring now also states what the sweep could not: blades 1 and 8 are
environment-conditional. The local profile points at loopback, so on a machine
running a local model server the wiring detach would make real calls and could
complete the pass. Their red was real on the machine it was measured on and is
not portable; blade 2's is.

Separately, the budget-stop print is marked as defensive and currently
unreachable from main(), because main() never constructs a PortfolioMeter and
every write to budget_stop is gated on one -- the strict=True precedent
directly above says untested future-proofing must be labelled as such. The
README claim that a portfolio pass reports a cap stop is corrected to say the
cap has no CLI flag yet. Noted for whoever wires that door: BudgetRefused is a
RuntimeError and the existing except clause would not catch it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0118noV9rCfrdREH26XqZB5z
This commit is contained in:
Kjell Tore Guttormsen 2026-08-05 11:06:41 +02:00
commit f87d555840
3 changed files with 36 additions and 9 deletions

View file

@ -127,9 +127,12 @@ identical id by design — that key is how a later run finds the earlier judgeme
A portfolio pass reports what happened to every project. Projects that raised are printed to
stderr with their error, and the command exits non-zero; the projects that completed still print
their outcome, because one dead project must not discard the rest of the pass. A pass stopped by
the global token cap says so, separately from a pass stopped because a savings goal was reached —
running out of budget and hitting your target are not the same event.
their outcome, because one dead project must not discard the rest of the pass. A pass stopped
because a savings goal was reached says so too.
The global token cap is reported separately from a goal stop — running out of budget and hitting
your target are not the same event — but note that the cap itself has no command-line flag yet:
only a library caller can install one, so that line is unreachable from the CLI today.
**7 — Report what has actually been realized:**

View file

@ -1352,6 +1352,14 @@ def main(argv: list[str] | None = None) -> int:
# projects, four APIConnectionError, silent success). ``BudgetStop`` is kept apart from
# ``stop_reason`` precisely so a caller can tell exhaustion from success; showing neither
# collapsed the distinction the dataclass was split to preserve.
# The budget-stop arm is DEFENSIVE and currently UNREACHABLE from here — measured, not
# assumed, and said out loud for the same reason ``strict=True`` below is: ``main()`` never
# constructs a ``PortfolioMeter``, and every write to ``budget_stop`` is gated on one, so
# only a LIBRARY caller passing ``portfolio_meter=`` can produce this field today. It is
# printed anyway because the field exists and a CLI door onto the global cap is a natural
# next step; the test that covers it drives a crafted ``PortfolioResult``, and says so.
# TRAP for whoever wires that door: ``BudgetRefused`` is a ``RuntimeError``, so the
# ``except`` above would NOT catch the startup refusal — it needs adding explicitly.
if portfolio_result.budget_stop is not None:
bs = portfolio_result.budget_stop
print(

View file

@ -98,15 +98,31 @@ def test_portfolio_runs_offline_with_scripted_replies(replies_file, capsys) -> N
def test_portfolio_scripted_pass_makes_no_real_client(replies_file, monkeypatch, capsys) -> None:
"""Blade 2 — genuinely model-free. Detonates if the portfolio dispatch falls back to the
production factory (which is exactly what the un-wired flag did)."""
"""Blade 2 — genuinely model-free, and the ONE blade here that proves defect A on its own.
def _boom(_profile: str): # pragma: no cover - the point is that it never runs
It asserts on a CALL SENTINEL, not on an exception. The first version of this blade raised
from the patched factory, and that version was measured GREEN with both the wiring and the rc
rule detached: ``_default_factory`` is called inside ``run_project``'s coroutine (``run.py``),
and ``run_portfolio`` gathers with ``return_exceptions=True``, so the ``AssertionError`` was
collected into a ``RunFailure`` and never escaped the blade only went red because rc had
become 1, i.e. it was testing defect B while claiming to test defect A. A list appended in the
factory and asserted in the test body cannot be swallowed by the wave handler.
This matters because blades 1 and 8 are ENVIRONMENT-CONDITIONAL: the local profile points at
loopback, so on a machine actually running a local model server the A-detach would make real
calls and could complete the pass. Their red is real here and was measured here, but it is not
portable. This blade's red is."""
built: list[str] = []
def _record(profile: str):
built.append(profile)
raise AssertionError("a real client factory was built on the scripted portfolio path")
monkeypatch.setattr(run, "_default_factory", _boom)
monkeypatch.setattr(run, "_default_factory", _record)
rc = run.main(["--portfolio", "--scripted-replies", str(replies_file)])
assert rc == 0, capsys.readouterr().out
out = capsys.readouterr().out
assert built == [], f"the production factory was built {len(built)} time(s): {built}"
assert rc == 0, out
def test_portfolio_scripted_pass_says_so_unmistakably(replies_file, capsys) -> None: