fix(tests): blade 2 was testing the wrong defect, and said so out loud
An independent reviewer found it and the claim was verified by measurement before being accepted, not taken on trust. test_portfolio_scripted_pass_makes_no_real_client raised from the patched _default_factory to prove the scripted portfolio pass never builds a production client. It cannot: the factory is called inside run_project's coroutine, and run_portfolio gathers with return_exceptions=True, so the AssertionError was collected into a RunFailure and never escaped. Measured directly -- with BOTH the client_factory wiring and the rc rule detached, the blade stayed GREEN. It was going red on rc alone, which means it was testing defect B while claiming to test defect A. The original seven-mutation sweep did not catch this because each mutation was applied singly, and dropping the wiring alone still flips rc. Replaced with a call sentinel: a list appended inside the factory and asserted in the test body, which the wave handler cannot swallow. Re-measured -- red on the wiring detach alone, and red on both detaches together. The docstring now also states what the sweep could not: blades 1 and 8 are environment-conditional. The local profile points at loopback, so on a machine running a local model server the wiring detach would make real calls and could complete the pass. Their red was real on the machine it was measured on and is not portable; blade 2's is. Separately, the budget-stop print is marked as defensive and currently unreachable from main(), because main() never constructs a PortfolioMeter and every write to budget_stop is gated on one -- the strict=True precedent directly above says untested future-proofing must be labelled as such. The README claim that a portfolio pass reports a cap stop is corrected to say the cap has no CLI flag yet. Noted for whoever wires that door: BudgetRefused is a RuntimeError and the existing except clause would not catch it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0118noV9rCfrdREH26XqZB5z
This commit is contained in:
parent
51739455ae
commit
f87d555840
3 changed files with 36 additions and 9 deletions
|
|
@ -98,15 +98,31 @@ def test_portfolio_runs_offline_with_scripted_replies(replies_file, capsys) -> N
|
|||
|
||||
|
||||
def test_portfolio_scripted_pass_makes_no_real_client(replies_file, monkeypatch, capsys) -> None:
|
||||
"""Blade 2 — genuinely model-free. Detonates if the portfolio dispatch falls back to the
|
||||
production factory (which is exactly what the un-wired flag did)."""
|
||||
"""Blade 2 — genuinely model-free, and the ONE blade here that proves defect A on its own.
|
||||
|
||||
def _boom(_profile: str): # pragma: no cover - the point is that it never runs
|
||||
It asserts on a CALL SENTINEL, not on an exception. The first version of this blade raised
|
||||
from the patched factory, and that version was measured GREEN with both the wiring and the rc
|
||||
rule detached: ``_default_factory`` is called inside ``run_project``'s coroutine (``run.py``),
|
||||
and ``run_portfolio`` gathers with ``return_exceptions=True``, so the ``AssertionError`` was
|
||||
collected into a ``RunFailure`` and never escaped — the blade only went red because rc had
|
||||
become 1, i.e. it was testing defect B while claiming to test defect A. A list appended in the
|
||||
factory and asserted in the test body cannot be swallowed by the wave handler.
|
||||
|
||||
This matters because blades 1 and 8 are ENVIRONMENT-CONDITIONAL: the local profile points at
|
||||
loopback, so on a machine actually running a local model server the A-detach would make real
|
||||
calls and could complete the pass. Their red is real here and was measured here, but it is not
|
||||
portable. This blade's red is."""
|
||||
built: list[str] = []
|
||||
|
||||
def _record(profile: str):
|
||||
built.append(profile)
|
||||
raise AssertionError("a real client factory was built on the scripted portfolio path")
|
||||
|
||||
monkeypatch.setattr(run, "_default_factory", _boom)
|
||||
monkeypatch.setattr(run, "_default_factory", _record)
|
||||
rc = run.main(["--portfolio", "--scripted-replies", str(replies_file)])
|
||||
assert rc == 0, capsys.readouterr().out
|
||||
out = capsys.readouterr().out
|
||||
assert built == [], f"the production factory was built {len(built)} time(s): {built}"
|
||||
assert rc == 0, out
|
||||
|
||||
|
||||
def test_portfolio_scripted_pass_says_so_unmistakably(replies_file, capsys) -> None:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue