fix(fase1): spike B fan-out measures real conversation bleed, not a counter
/trekreview flagged the Spike B(b) fan-out experiment as BROKEN_SUCCESS_CRITERION (BLOCKER): it asserted a per-client call_count reached 3 on a reused instance vs 1 on a fresh one — a tautology true for any un-reset mutable counter, independent of MAF, that never exercised the real G2/B7 shared-Workflow state-corruption footgun. It was a false-confirm of a de-risk assumption. Rebuilt to observe genuine MAF thread state via the messages each participant RECEIVES (new FakeChatClient.received_texts seam): - shared_instance_conversation_bleed: a reused built ConcurrentBuilder Workflow accumulates the conversation across .run() calls — run N's participants receive runs 0..N-1's prompts/replies (measured [[p0],[p0,p1],[p0,p1,p2]], strictly monotonic) => genuine cross-run contamination. - fresh_instance_conversation_isolation: a fresh instance per run gives each a clean thread => each participant sees only its own project ([[p0],[p1],[p2]]). Assumption now CONFIRMED with a meaningful observable. findings-b.md gains a Method note recording why it was rebuilt; README rows updated. Also fixes the MINOR: a_groupchat.run_live now mkdirs the findings dir before write_text so a post-disposal run does not lose the measured result. Gate green: ruff check + format, mypy src, pytest 48 passed / 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
This commit is contained in:
parent
b81e22b637
commit
a2dff210ce
6 changed files with 118 additions and 35 deletions
|
|
@ -7,8 +7,8 @@ Step 2 builder smoke. Pattern: tests/test_backends.py.
|
|||
from portfolio_optimiser.reference_domain import load_reference_projects
|
||||
from spikes.b_footguns import (
|
||||
bounded_magentic_terminates,
|
||||
fresh_instance_call_counts,
|
||||
shared_instance_max_calls,
|
||||
fresh_instance_conversation_isolation,
|
||||
shared_instance_conversation_bleed,
|
||||
unbounded_magentic_self_terminates,
|
||||
)
|
||||
|
||||
|
|
@ -24,16 +24,22 @@ async def test_bounded_magentic_terminates_within_limit() -> None:
|
|||
assert await bounded_magentic_terminates(max_round_count=2) is True
|
||||
|
||||
|
||||
async def test_shared_instance_bleeds_state_across_projects() -> None:
|
||||
async def test_shared_instance_bleeds_conversation_across_projects() -> None:
|
||||
ids = [p.id for p in load_reference_projects()]
|
||||
assert len(ids) == 3
|
||||
bled = await shared_instance_max_calls(ids)
|
||||
# One shared instance reused across all 3 projects -> calls accumulate -> bleed.
|
||||
assert bled == len(ids)
|
||||
seen = await shared_instance_conversation_bleed(ids)
|
||||
# Reusing ONE built workflow accumulates the MAF thread across runs: each run's
|
||||
# participant receives the EARLIER projects' prompts too (genuine G2/B7 bleed),
|
||||
# measured by message CONTENT — not a call counter.
|
||||
assert seen[0] == [ids[0]] # first run is clean — nothing prior to bleed
|
||||
assert set(seen[-1]) == set(ids) # last run saw every project's prompt
|
||||
# Strictly monotonic growth = conversation history accumulating, not coincidence.
|
||||
assert all(set(seen[i]) < set(seen[i + 1]) for i in range(len(seen) - 1))
|
||||
|
||||
|
||||
async def test_fresh_instance_zero_bleed_across_projects() -> None:
|
||||
async def test_fresh_instance_isolates_conversation_across_projects() -> None:
|
||||
ids = [p.id for p in load_reference_projects()]
|
||||
counts = await fresh_instance_call_counts(ids)
|
||||
# A fresh instance per project -> each run sees exactly one call -> zero bleed.
|
||||
assert counts == [1, 1, 1]
|
||||
seen = await fresh_instance_conversation_isolation(ids)
|
||||
# A fresh instance per project gives each run a clean thread -> a participant sees
|
||||
# ONLY its own project -> zero cross-run contamination (the B7 mitigation works).
|
||||
assert seen == [[pid] for pid in ids]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue