fix(tests): blade 2 was testing the wrong defect, and said so out loud

An independent reviewer found it and the claim was verified by measurement
before being accepted, not taken on trust.

test_portfolio_scripted_pass_makes_no_real_client raised from the patched
_default_factory to prove the scripted portfolio pass never builds a production
client. It cannot: the factory is called inside run_project's coroutine, and
run_portfolio gathers with return_exceptions=True, so the AssertionError was
collected into a RunFailure and never escaped. Measured directly -- with BOTH
the client_factory wiring and the rc rule detached, the blade stayed GREEN. It
was going red on rc alone, which means it was testing defect B while claiming
to test defect A. The original seven-mutation sweep did not catch this because
each mutation was applied singly, and dropping the wiring alone still flips rc.

Replaced with a call sentinel: a list appended inside the factory and asserted
in the test body, which the wave handler cannot swallow. Re-measured -- red on
the wiring detach alone, and red on both detaches together.

The docstring now also states what the sweep could not: blades 1 and 8 are
environment-conditional. The local profile points at loopback, so on a machine
running a local model server the wiring detach would make real calls and could
complete the pass. Their red was real on the machine it was measured on and is
not portable; blade 2's is.

Separately, the budget-stop print is marked as defensive and currently
unreachable from main(), because main() never constructs a PortfolioMeter and
every write to budget_stop is gated on one -- the strict=True precedent
directly above says untested future-proofing must be labelled as such. The
README claim that a portfolio pass reports a cap stop is corrected to say the
cap has no CLI flag yet. Noted for whoever wires that door: BudgetRefused is a
RuntimeError and the existing except clause would not catch it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0118noV9rCfrdREH26XqZB5z
This commit is contained in:
Kjell Tore Guttormsen 2026-08-05 11:06:41 +02:00
commit f87d555840
3 changed files with 36 additions and 9 deletions

View file

@ -127,9 +127,12 @@ identical id by design — that key is how a later run finds the earlier judgeme
A portfolio pass reports what happened to every project. Projects that raised are printed to
stderr with their error, and the command exits non-zero; the projects that completed still print
their outcome, because one dead project must not discard the rest of the pass. A pass stopped by
the global token cap says so, separately from a pass stopped because a savings goal was reached —
running out of budget and hitting your target are not the same event.
their outcome, because one dead project must not discard the rest of the pass. A pass stopped
because a savings goal was reached says so too.
The global token cap is reported separately from a goal stop — running out of budget and hitting
your target are not the same event — but note that the cap itself has no command-line flag yet:
only a library caller can install one, so that line is unreachable from the CLI today.
**7 — Report what has actually been realized:**