fix(fase1): spike B fan-out measures real conversation bleed, not a counter

/trekreview flagged the Spike B(b) fan-out experiment as BROKEN_SUCCESS_CRITERION
(BLOCKER): it asserted a per-client call_count reached 3 on a reused instance vs
1 on a fresh one — a tautology true for any un-reset mutable counter, independent
of MAF, that never exercised the real G2/B7 shared-Workflow state-corruption
footgun. It was a false-confirm of a de-risk assumption.

Rebuilt to observe genuine MAF thread state via the messages each participant
RECEIVES (new FakeChatClient.received_texts seam):
- shared_instance_conversation_bleed: a reused built ConcurrentBuilder Workflow
  accumulates the conversation across .run() calls — run N's participants receive
  runs 0..N-1's prompts/replies (measured [[p0],[p0,p1],[p0,p1,p2]], strictly
  monotonic) => genuine cross-run contamination.
- fresh_instance_conversation_isolation: a fresh instance per run gives each a
  clean thread => each participant sees only its own project ([[p0],[p1],[p2]]).

Assumption now CONFIRMED with a meaningful observable. findings-b.md gains a
Method note recording why it was rebuilt; README rows updated.

Also fixes the MINOR: a_groupchat.run_live now mkdirs the findings dir before
write_text so a post-disposal run does not lose the measured result.

Gate green: ruff check + format, mypy src, pytest 48 passed / 1 skipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
This commit is contained in:
Kjell Tore Guttormsen 2026-06-24 11:09:55 +02:00
commit a2dff210ce
6 changed files with 118 additions and 35 deletions

View file

@ -7,8 +7,8 @@ Step 2 builder smoke. Pattern: tests/test_backends.py.
from portfolio_optimiser.reference_domain import load_reference_projects
from spikes.b_footguns import (
bounded_magentic_terminates,
fresh_instance_call_counts,
shared_instance_max_calls,
fresh_instance_conversation_isolation,
shared_instance_conversation_bleed,
unbounded_magentic_self_terminates,
)
@ -24,16 +24,22 @@ async def test_bounded_magentic_terminates_within_limit() -> None:
assert await bounded_magentic_terminates(max_round_count=2) is True
async def test_shared_instance_bleeds_state_across_projects() -> None:
async def test_shared_instance_bleeds_conversation_across_projects() -> None:
ids = [p.id for p in load_reference_projects()]
assert len(ids) == 3
bled = await shared_instance_max_calls(ids)
# One shared instance reused across all 3 projects -> calls accumulate -> bleed.
assert bled == len(ids)
seen = await shared_instance_conversation_bleed(ids)
# Reusing ONE built workflow accumulates the MAF thread across runs: each run's
# participant receives the EARLIER projects' prompts too (genuine G2/B7 bleed),
# measured by message CONTENT — not a call counter.
assert seen[0] == [ids[0]] # first run is clean — nothing prior to bleed
assert set(seen[-1]) == set(ids) # last run saw every project's prompt
# Strictly monotonic growth = conversation history accumulating, not coincidence.
assert all(set(seen[i]) < set(seen[i + 1]) for i in range(len(seen) - 1))
async def test_fresh_instance_zero_bleed_across_projects() -> None:
async def test_fresh_instance_isolates_conversation_across_projects() -> None:
ids = [p.id for p in load_reference_projects()]
counts = await fresh_instance_call_counts(ids)
# A fresh instance per project -> each run sees exactly one call -> zero bleed.
assert counts == [1, 1, 1]
seen = await fresh_instance_conversation_isolation(ids)
# A fresh instance per project gives each run a clean thread -> a participant sees
# ONLY its own project -> zero cross-run contamination (the B7 mitigation works).
assert seen == [[pid] for pid in ids]