fix(fase1): spike B fan-out measures real conversation bleed, not a counter

/trekreview flagged the Spike B(b) fan-out experiment as BROKEN_SUCCESS_CRITERION
(BLOCKER): it asserted a per-client call_count reached 3 on a reused instance vs
1 on a fresh one — a tautology true for any un-reset mutable counter, independent
of MAF, that never exercised the real G2/B7 shared-Workflow state-corruption
footgun. It was a false-confirm of a de-risk assumption.

Rebuilt to observe genuine MAF thread state via the messages each participant
RECEIVES (new FakeChatClient.received_texts seam):
- shared_instance_conversation_bleed: a reused built ConcurrentBuilder Workflow
  accumulates the conversation across .run() calls — run N's participants receive
  runs 0..N-1's prompts/replies (measured [[p0],[p0,p1],[p0,p1,p2]], strictly
  monotonic) => genuine cross-run contamination.
- fresh_instance_conversation_isolation: a fresh instance per run gives each a
  clean thread => each participant sees only its own project ([[p0],[p1],[p2]]).

Assumption now CONFIRMED with a meaningful observable. findings-b.md gains a
Method note recording why it was rebuilt; README rows updated.

Also fixes the MINOR: a_groupchat.run_live now mkdirs the findings dir before
write_text so a post-disposal run does not lose the measured result.

Gate green: ruff check + format, mypy src, pytest 48 passed / 1 skipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
This commit is contained in:
Kjell Tore Guttormsen 2026-06-24 11:09:55 +02:00
commit a2dff210ce
6 changed files with 118 additions and 35 deletions

View file

@ -99,6 +99,26 @@ def _word_tokens(text: str) -> int:
return len(text.split())
def message_texts(messages: Sequence[Message]) -> list[str]:
"""Extract the text payloads from a sequence of MAF ``Message`` objects.
MAF content items vary (objects with ``.text``, bare strings, or ``{"text": ...}``
dicts); this normalizes them to a flat list of strings. Used by ``FakeChatClient``
to record exactly what an agent *received* per call the observable Spike B uses to
detect cross-run conversation bleed (G2/B7)."""
out: list[str] = []
for m in messages:
for c in getattr(m, "contents", []) or []:
text = getattr(c, "text", None)
if text is None and isinstance(c, str):
text = c
if text is None and isinstance(c, dict):
text = c.get("text")
if text is not None:
out.append(str(text))
return out
class FakeChatClient(BaseChatClient):
"""A deterministic, network-free ``BaseChatClient`` for driving MAF agents in tests.
@ -116,6 +136,9 @@ class FakeChatClient(BaseChatClient):
self._default = default_reply
self.total_tokens = 0
self.call_count = 0
# One entry per call: the text payloads this client RECEIVED that call. Lets a
# spike observe whether a reused workflow feeds run N+1 the prior runs' history.
self.received_texts: list[list[str]] = []
def _next_reply(self) -> str:
reply = self._scripted[self._idx] if self._idx < len(self._scripted) else self._default
@ -134,6 +157,7 @@ class FakeChatClient(BaseChatClient):
) -> Any:
# Matches the GA BaseChatClient contract: return a ResponseStream when
# streaming, otherwise an awaitable resolving to a ChatResponse.
self.received_texts.append(message_texts(messages))
reply = self._next_reply()
if stream:

View file

@ -176,6 +176,10 @@ async def run_live(
table = render_comparison(mc, single)
if write_findings:
# Guard the parent dir: the spike may run after the documented disposal
# (`rm -rf docs/fase1-spikes`), and a missing dir must not discard the
# just-measured live result with a bare FileNotFoundError.
_FINDINGS.parent.mkdir(parents=True, exist_ok=True)
_FINDINGS.write_text(_findings_doc(table), encoding="utf-8")
return {"mc": mc, "single": single, "verdict": verdict(mc, single), "table": table}

View file

@ -10,9 +10,14 @@ the experiment here — no live LLM, so this whole spike runs in the quality gat
round/iteration guard and confirm the guard is what stops it (it does NOT
self-terminate). With an explicit `max_round_count` it terminates cleanly.
(b) **Fan-out state isolation (G2/B7).** Reusing ONE workflow instance across the three
reference projects bleeds state (the shared fake clients accumulate calls); a FRESH
instance per run via `fresh_workflow()` shows zero bleed.
(b) **Fan-out state isolation (G2/B7).** Reusing ONE built `ConcurrentBuilder` workflow
across the three reference projects bleeds *conversation state*: MAF accumulates the
shared thread across `.run()` calls, so each run's participants receive the PRIOR
projects' prompts and replies (project N contaminates N+1's context). A FRESH instance
per run via `fresh_workflow()` gives each run a clean thread (zero contamination).
The observable is the message history each participant *receives*, captured via
`FakeChatClient.received_texts` NOT a call counter (a counter would rise for any
reused mutable object and prove nothing about MAF state).
Token use: 0 no live LLM (the fake client's "tokens" are word-counts of canned replies).
"""
@ -101,22 +106,41 @@ def fresh_workflow() -> tuple[object, tuple[FakeChatClient, FakeChatClient]]:
return workflow, (c1, c2)
async def shared_instance_max_calls(project_ids: list[str]) -> int:
"""Reuse ONE fan-out instance across every project — state bleeds: the shared clients
accumulate calls across runs. Returns the max per-client call count (== len once
bled)."""
def _projects_seen(received_texts: list[str], project_ids: list[str]) -> list[str]:
"""Which project ids appear anywhere in the messages an agent received on one call.
The observable for genuine state bleed: if a later run's agent sees an EARLIER
project's id, the workflow carried that project's conversation forward."""
blob = " ".join(received_texts)
return [pid for pid in project_ids if pid in blob]
async def shared_instance_conversation_bleed(project_ids: list[str]) -> list[list[str]]:
"""Reuse ONE built fan-out workflow across every project. MAF accumulates the shared
thread across `.run()` calls, so run N's participants also receive runs 0..N-1's
prompts/replies genuine cross-run state corruption (G2/B7).
Returns, per run (in order), which project ids were visible to a participant on that
run. With a reused instance this grows monotonically: ``[[p0], [p0, p1], [p0, p1, p2]]``."""
workflow, clients = fresh_workflow()
for pid in project_ids:
await workflow.run(f"Evaluate project {pid}.")
return max(c.call_count for c in clients)
# One participant is representative: concurrent fan-out feeds every participant the
# same accumulated thread. clients[0] was called once per run, in order.
rep = clients[0]
return [_projects_seen(call_view, project_ids) for call_view in rep.received_texts]
async def fresh_instance_call_counts(project_ids: list[str]) -> list[int]:
"""A FRESH instance per project — zero bleed: every run's clients see exactly one
call. Returns the per-run max call count (each should be 1)."""
counts: list[int] = []
async def fresh_instance_conversation_isolation(project_ids: list[str]) -> list[list[str]]:
"""A FRESH instance per project (the B7 mitigation): each run gets a clean thread, so
a participant sees ONLY its own project zero cross-run contamination.
Returns, per run, the project ids visible to a participant; each should be exactly
its own: ``[[p0], [p1], [p2]]``."""
seen_per_run: list[list[str]] = []
for pid in project_ids:
workflow, clients = fresh_workflow()
await workflow.run(f"Evaluate project {pid}.")
counts.append(max(c.call_count for c in clients))
return counts
# fresh client -> exactly one call this run; read its single received view.
seen_per_run.append(_projects_seen(clients[0].received_texts[0], project_ids))
return seen_per_run