test(portfolio): gate the wave handler's catch width and the failed-project ledger (v/t/s)

Three items on one seam — what a FAILED project does to the wave loop — plus the
snapshot copy they sit next to.

(v) The catch is BaseException, not Exception, and that width was ungated. The
existing collect-and-continue test raises RuntimeError, so it stays green when
the handler is narrowed: measured, the whole of test_portfolio_concurrent_
loadbearing.py (13 tests) passes under the narrowing. asyncio.CancelledError is
the one realistic vector that separates the two — probed first, gather(
return_exceptions=True) COLLECTS it, while KeyboardInterrupt propagates
regardless and could never be helped by a wider catch. Narrowed, a cancelled
member is cast into runs as a fake RunResult and the pass dies in _aggregate,
pointing away from its cause. RED measured.

(t) sum_token_usage excludes a failed project's spend, and that is the honest
answer, not a bug: a run that died before producing a stamp has no provenance,
and inventing one is the fabrication RunFailure exists to avoid. What needed
gating is that those tokens still reach the ledger the global cap is enforced
against — otherwise a repeatedly-failing project burns budget while the meter
reads clean. Pins meter.spent as the pass's real cost, sum_token_usage as the
completed-run subtotal, and their difference as exactly the failed spend. RED
measured against the likely "fix" (sourcing sum_token_usage from the meter),
which is wrong because a seeded meter also carries EARLIER passes' spend; 21
existing budget/portfolio tests stay green under it.

(s) _wave_snapshot uses dataclasses.replace, so a field added later is carried
without touching the function. Not cosmetic: measured, dropping retriever by
hand-enumerating left all 585 tests green — the Step-2 coverage its docstring
credited no longer existed, so the S3.1 retriever seam could be downgraded
mid-pass in silence. Now gated by a property test derived from
dataclasses.fields (not a field count, the shape rejected earlier). The explicit
verdicts copy is retained and separately gated: replace(store) alone shares the
caller's list and takes the byte-identical determinism test RED.

strict=True on the zip is documented as deliberately untested — measured green
when dropped, since gather is built from exactly snapshots, so a test could only
go red by manufacturing a mismatch and would exercise zip rather than this pass.

The new double is registered in the S2.5 consolidation guard's delegating-
overrides list rather than the guard being weakened; it already delegates via
super()._inner_get_response, which test_delegating_overrides_call_super now
enforces on it.

583 -> 586 tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MbgTCEZma764i1rHTrzceU
This commit is contained in:
Kjell Tore Guttormsen 2026-08-03 16:09:12 +02:00
commit 873f5fa272
4 changed files with 322 additions and 8 deletions

View file

@ -27,6 +27,7 @@ from __future__ import annotations
import ast
import asyncio
import dataclasses
from collections.abc import Mapping, Sequence
from pathlib import Path
from typing import Any
@ -43,10 +44,17 @@ from test_portfolio_learning_loadbearing import (
)
from portfolio_optimiser import ledger as ledger_mod
from portfolio_optimiser import semretrieval
from portfolio_optimiser import verdicts as verdicts_mod
from portfolio_optimiser.contracts import GoalConfig, GoalContract
from portfolio_optimiser.ledger import SavingsLedger
from portfolio_optimiser.run import PortfolioResult, RunFailure, _waves, run_portfolio
from portfolio_optimiser.run import (
PortfolioResult,
RunFailure,
_wave_snapshot,
_waves,
run_portfolio,
)
from portfolio_optimiser.verdicts import VerdictStore
# The shipped 3-project fixture, in submission order (mirrors tests/test_portfolio.py:57).
@ -223,6 +231,43 @@ def _aggregate_fields(result: PortfolioResult) -> tuple[Any, ...]:
)
def test_wave_snapshot_carries_every_field_except_the_copied_verdicts() -> None:
"""The snapshot copies ``verdicts`` and carries EVERY other ``VerdictStore`` field over.
**Detach point: hand-enumerate the fields (``VerdictStore(verdicts=list(store.verdicts))``)
-> this test goes RED.** MEASURED, and the measurement is why this test exists: before it was
written, dropping ``retriever`` in exactly that way left the whole suite GREEN (585 passed). No
test set ``retriever`` at all, so the S3.1 opt-in seam could be silently downgraded to the
structural default mid-pass a caller-owned semantic retriever would stop being used from the
first wave onward, and nothing would say so. ``_wave_snapshot``'s docstring had claimed this
was caught by a Step-2 contract test; that coverage no longer existed.
This asserts the PROPERTY (every declared field survives), not a field count the shape that
was rejected for going red on every benign addition to ``VerdictStore``. Deriving the expected
set from ``dataclasses.fields`` means a field added later is covered without editing this test,
and ``replace`` in the implementation means it is carried without editing the snapshot either.
"""
# A real ``Retriever``, not a bare sentinel: the field is typed, and the assertion below is
# about identity surviving the copy, which a genuine instance demonstrates without pretence.
marker = semretrieval.StructuralRetriever(verdicts_mod.similarity)
store = VerdictStore(verdicts=[], retriever=marker)
snapshot = _wave_snapshot(store)
carried = [f.name for f in dataclasses.fields(VerdictStore) if f.name != "verdicts"]
assert carried, "VerdictStore declares no field beyond ``verdicts`` — this test guards nothing"
for name in carried:
assert getattr(snapshot, name) is getattr(store, name), (
f"``{name}`` was not carried into the wave snapshot — a project in the wave would run "
f"against a store whose {name} silently differs from the one the caller configured"
)
# The half that must NOT be carried by reference: the verdict list is copied, so a project
# appending to its snapshot cannot touch the shared store. Guarded here too, because the
# tidiest wrong fix for the above (``replace(store)``) breaks exactly this.
assert snapshot.verdicts is not store.verdicts
async def test_concurrent_pass_is_byte_identical_to_sequential() -> None:
"""S3.3's core contract: ``concurrency=3`` produces the SAME store-verdict id SEQUENCE as
``concurrency=1``, under a completion order that genuinely differs from submission order.