fix(tests): blade 2 was testing the wrong defect, and said so out loud

An independent reviewer found it and the claim was verified by measurement
before being accepted, not taken on trust.

test_portfolio_scripted_pass_makes_no_real_client raised from the patched
_default_factory to prove the scripted portfolio pass never builds a production
client. It cannot: the factory is called inside run_project's coroutine, and
run_portfolio gathers with return_exceptions=True, so the AssertionError was
collected into a RunFailure and never escaped. Measured directly -- with BOTH
the client_factory wiring and the rc rule detached, the blade stayed GREEN. It
was going red on rc alone, which means it was testing defect B while claiming
to test defect A. The original seven-mutation sweep did not catch this because
each mutation was applied singly, and dropping the wiring alone still flips rc.

Replaced with a call sentinel: a list appended inside the factory and asserted
in the test body, which the wave handler cannot swallow. Re-measured -- red on
the wiring detach alone, and red on both detaches together.

The docstring now also states what the sweep could not: blades 1 and 8 are
environment-conditional. The local profile points at loopback, so on a machine
running a local model server the wiring detach would make real calls and could
complete the pass. Their red was real on the machine it was measured on and is
not portable; blade 2's is.

Separately, the budget-stop print is marked as defensive and currently
unreachable from main(), because main() never constructs a PortfolioMeter and
every write to budget_stop is gated on one -- the strict=True precedent
directly above says untested future-proofing must be labelled as such. The
README claim that a portfolio pass reports a cap stop is corrected to say the
cap has no CLI flag yet. Noted for whoever wires that door: BudgetRefused is a
RuntimeError and the existing except clause would not catch it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0118noV9rCfrdREH26XqZB5z
This commit is contained in:
Kjell Tore Guttormsen 2026-08-05 11:06:41 +02:00
commit f87d555840
3 changed files with 36 additions and 9 deletions

View file

@ -98,15 +98,31 @@ def test_portfolio_runs_offline_with_scripted_replies(replies_file, capsys) -> N
def test_portfolio_scripted_pass_makes_no_real_client(replies_file, monkeypatch, capsys) -> None:
"""Blade 2 — genuinely model-free. Detonates if the portfolio dispatch falls back to the
production factory (which is exactly what the un-wired flag did)."""
"""Blade 2 — genuinely model-free, and the ONE blade here that proves defect A on its own.
def _boom(_profile: str): # pragma: no cover - the point is that it never runs
It asserts on a CALL SENTINEL, not on an exception. The first version of this blade raised
from the patched factory, and that version was measured GREEN with both the wiring and the rc
rule detached: ``_default_factory`` is called inside ``run_project``'s coroutine (``run.py``),
and ``run_portfolio`` gathers with ``return_exceptions=True``, so the ``AssertionError`` was
collected into a ``RunFailure`` and never escaped the blade only went red because rc had
become 1, i.e. it was testing defect B while claiming to test defect A. A list appended in the
factory and asserted in the test body cannot be swallowed by the wave handler.
This matters because blades 1 and 8 are ENVIRONMENT-CONDITIONAL: the local profile points at
loopback, so on a machine actually running a local model server the A-detach would make real
calls and could complete the pass. Their red is real here and was measured here, but it is not
portable. This blade's red is."""
built: list[str] = []
def _record(profile: str):
built.append(profile)
raise AssertionError("a real client factory was built on the scripted portfolio path")
monkeypatch.setattr(run, "_default_factory", _boom)
monkeypatch.setattr(run, "_default_factory", _record)
rc = run.main(["--portfolio", "--scripted-replies", str(replies_file)])
assert rc == 0, capsys.readouterr().out
out = capsys.readouterr().out
assert built == [], f"the production factory was built {len(built)} time(s): {built}"
assert rc == 0, out
def test_portfolio_scripted_pass_says_so_unmistakably(replies_file, capsys) -> None: