fix(tests): blade 2 was testing the wrong defect, and said so out loud
An independent reviewer found it and the claim was verified by measurement before being accepted, not taken on trust. test_portfolio_scripted_pass_makes_no_real_client raised from the patched _default_factory to prove the scripted portfolio pass never builds a production client. It cannot: the factory is called inside run_project's coroutine, and run_portfolio gathers with return_exceptions=True, so the AssertionError was collected into a RunFailure and never escaped. Measured directly -- with BOTH the client_factory wiring and the rc rule detached, the blade stayed GREEN. It was going red on rc alone, which means it was testing defect B while claiming to test defect A. The original seven-mutation sweep did not catch this because each mutation was applied singly, and dropping the wiring alone still flips rc. Replaced with a call sentinel: a list appended inside the factory and asserted in the test body, which the wave handler cannot swallow. Re-measured -- red on the wiring detach alone, and red on both detaches together. The docstring now also states what the sweep could not: blades 1 and 8 are environment-conditional. The local profile points at loopback, so on a machine running a local model server the wiring detach would make real calls and could complete the pass. Their red was real on the machine it was measured on and is not portable; blade 2's is. Separately, the budget-stop print is marked as defensive and currently unreachable from main(), because main() never constructs a PortfolioMeter and every write to budget_stop is gated on one -- the strict=True precedent directly above says untested future-proofing must be labelled as such. The README claim that a portfolio pass reports a cap stop is corrected to say the cap has no CLI flag yet. Noted for whoever wires that door: BudgetRefused is a RuntimeError and the existing except clause would not catch it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0118noV9rCfrdREH26XqZB5z
This commit is contained in:
parent
51739455ae
commit
f87d555840
3 changed files with 36 additions and 9 deletions
|
|
@ -127,9 +127,12 @@ identical id by design — that key is how a later run finds the earlier judgeme
|
|||
|
||||
A portfolio pass reports what happened to every project. Projects that raised are printed to
|
||||
stderr with their error, and the command exits non-zero; the projects that completed still print
|
||||
their outcome, because one dead project must not discard the rest of the pass. A pass stopped by
|
||||
the global token cap says so, separately from a pass stopped because a savings goal was reached —
|
||||
running out of budget and hitting your target are not the same event.
|
||||
their outcome, because one dead project must not discard the rest of the pass. A pass stopped
|
||||
because a savings goal was reached says so too.
|
||||
|
||||
The global token cap is reported separately from a goal stop — running out of budget and hitting
|
||||
your target are not the same event — but note that the cap itself has no command-line flag yet:
|
||||
only a library caller can install one, so that line is unreachable from the CLI today.
|
||||
|
||||
**7 — Report what has actually been realized:**
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue