Kjell Tore Guttormsen
a2dff210ce
/trekreview flagged the Spike B(b) fan-out experiment as BROKEN_SUCCESS_CRITERION (BLOCKER): it asserted a per-client call_count reached 3 on a reused instance vs 1 on a fresh one — a tautology true for any un-reset mutable counter, independent of MAF, that never exercised the real G2/B7 shared-Workflow state-corruption footgun. It was a false-confirm of a de-risk assumption. Rebuilt to observe genuine MAF thread state via the messages each participant RECEIVES (new FakeChatClient.received_texts seam): - shared_instance_conversation_bleed: a reused built ConcurrentBuilder Workflow accumulates the conversation across .run() calls — run N's participants receive runs 0..N-1's prompts/replies (measured [[p0],[p0,p1],[p0,p1,p2]], strictly monotonic) => genuine cross-run contamination. - fresh_instance_conversation_isolation: a fresh instance per run gives each a clean thread => each participant sees only its own project ([[p0],[p1],[p2]]). Assumption now CONFIRMED with a meaningful observable. findings-b.md gains a Method note recording why it was rebuilt; README rows updated. Also fixes the MINOR: a_groupchat.run_live now mkdirs the findings dir before write_text so a post-disposal run does not lose the measured result. Gate green: ruff check + format, mypy src, pytest 48 passed / 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
45 lines
2.2 KiB
Python
45 lines
2.2 KiB
Python
"""Spike B tests — Magentic unbounded footgun (G1/B4) + fan-out state isolation (G2/B7).
|
|
|
|
These drive the real GA builders with the fake client (no endpoint), de-risked by the
|
|
Step 2 builder smoke. Pattern: tests/test_backends.py.
|
|
"""
|
|
|
|
from portfolio_optimiser.reference_domain import load_reference_projects
|
|
from spikes.b_footguns import (
|
|
bounded_magentic_terminates,
|
|
fresh_instance_conversation_isolation,
|
|
shared_instance_conversation_bleed,
|
|
unbounded_magentic_self_terminates,
|
|
)
|
|
|
|
|
|
async def test_unbounded_magentic_does_not_self_terminate() -> None:
|
|
# max_round_count=None never finalizes — only the external guard stops it (B4).
|
|
self_terminated = await unbounded_magentic_self_terminates(guard_rounds=8)
|
|
assert self_terminated is False # the guard fired, not a natural stop
|
|
|
|
|
|
async def test_bounded_magentic_terminates_within_limit() -> None:
|
|
# An explicit max_round_count makes the same never-satisfied manager stop cleanly.
|
|
assert await bounded_magentic_terminates(max_round_count=2) is True
|
|
|
|
|
|
async def test_shared_instance_bleeds_conversation_across_projects() -> None:
|
|
ids = [p.id for p in load_reference_projects()]
|
|
assert len(ids) == 3
|
|
seen = await shared_instance_conversation_bleed(ids)
|
|
# Reusing ONE built workflow accumulates the MAF thread across runs: each run's
|
|
# participant receives the EARLIER projects' prompts too (genuine G2/B7 bleed),
|
|
# measured by message CONTENT — not a call counter.
|
|
assert seen[0] == [ids[0]] # first run is clean — nothing prior to bleed
|
|
assert set(seen[-1]) == set(ids) # last run saw every project's prompt
|
|
# Strictly monotonic growth = conversation history accumulating, not coincidence.
|
|
assert all(set(seen[i]) < set(seen[i + 1]) for i in range(len(seen) - 1))
|
|
|
|
|
|
async def test_fresh_instance_isolates_conversation_across_projects() -> None:
|
|
ids = [p.id for p in load_reference_projects()]
|
|
seen = await fresh_instance_conversation_isolation(ids)
|
|
# A fresh instance per project gives each run a clean thread -> a participant sees
|
|
# ONLY its own project -> zero cross-run contamination (the B7 mitigation works).
|
|
assert seen == [[pid] for pid in ids]
|