fix(fase1): spike B fan-out measures real conversation bleed, not a counter
/trekreview flagged the Spike B(b) fan-out experiment as BROKEN_SUCCESS_CRITERION (BLOCKER): it asserted a per-client call_count reached 3 on a reused instance vs 1 on a fresh one — a tautology true for any un-reset mutable counter, independent of MAF, that never exercised the real G2/B7 shared-Workflow state-corruption footgun. It was a false-confirm of a de-risk assumption. Rebuilt to observe genuine MAF thread state via the messages each participant RECEIVES (new FakeChatClient.received_texts seam): - shared_instance_conversation_bleed: a reused built ConcurrentBuilder Workflow accumulates the conversation across .run() calls — run N's participants receive runs 0..N-1's prompts/replies (measured [[p0],[p0,p1],[p0,p1,p2]], strictly monotonic) => genuine cross-run contamination. - fresh_instance_conversation_isolation: a fresh instance per run gives each a clean thread => each participant sees only its own project ([[p0],[p1],[p2]]). Assumption now CONFIRMED with a meaningful observable. findings-b.md gains a Method note recording why it was rebuilt; README rows updated. Also fixes the MINOR: a_groupchat.run_live now mkdirs the findings dir before write_text so a post-disposal run does not lose the measured result. Gate green: ruff check + format, mypy src, pytest 48 passed / 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
This commit is contained in:
parent
b81e22b637
commit
a2dff210ce
6 changed files with 118 additions and 35 deletions
|
|
@ -99,6 +99,26 @@ def _word_tokens(text: str) -> int:
|
|||
return len(text.split())
|
||||
|
||||
|
||||
def message_texts(messages: Sequence[Message]) -> list[str]:
|
||||
"""Extract the text payloads from a sequence of MAF ``Message`` objects.
|
||||
|
||||
MAF content items vary (objects with ``.text``, bare strings, or ``{"text": ...}``
|
||||
dicts); this normalizes them to a flat list of strings. Used by ``FakeChatClient``
|
||||
to record exactly what an agent *received* per call — the observable Spike B uses to
|
||||
detect cross-run conversation bleed (G2/B7)."""
|
||||
out: list[str] = []
|
||||
for m in messages:
|
||||
for c in getattr(m, "contents", []) or []:
|
||||
text = getattr(c, "text", None)
|
||||
if text is None and isinstance(c, str):
|
||||
text = c
|
||||
if text is None and isinstance(c, dict):
|
||||
text = c.get("text")
|
||||
if text is not None:
|
||||
out.append(str(text))
|
||||
return out
|
||||
|
||||
|
||||
class FakeChatClient(BaseChatClient):
|
||||
"""A deterministic, network-free ``BaseChatClient`` for driving MAF agents in tests.
|
||||
|
||||
|
|
@ -116,6 +136,9 @@ class FakeChatClient(BaseChatClient):
|
|||
self._default = default_reply
|
||||
self.total_tokens = 0
|
||||
self.call_count = 0
|
||||
# One entry per call: the text payloads this client RECEIVED that call. Lets a
|
||||
# spike observe whether a reused workflow feeds run N+1 the prior runs' history.
|
||||
self.received_texts: list[list[str]] = []
|
||||
|
||||
def _next_reply(self) -> str:
|
||||
reply = self._scripted[self._idx] if self._idx < len(self._scripted) else self._default
|
||||
|
|
@ -134,6 +157,7 @@ class FakeChatClient(BaseChatClient):
|
|||
) -> Any:
|
||||
# Matches the GA BaseChatClient contract: return a ResponseStream when
|
||||
# streaming, otherwise an awaitable resolving to a ChatResponse.
|
||||
self.received_texts.append(message_texts(messages))
|
||||
reply = self._next_reply()
|
||||
|
||||
if stream:
|
||||
|
|
|
|||
|
|
@ -176,6 +176,10 @@ async def run_live(
|
|||
|
||||
table = render_comparison(mc, single)
|
||||
if write_findings:
|
||||
# Guard the parent dir: the spike may run after the documented disposal
|
||||
# (`rm -rf docs/fase1-spikes`), and a missing dir must not discard the
|
||||
# just-measured live result with a bare FileNotFoundError.
|
||||
_FINDINGS.parent.mkdir(parents=True, exist_ok=True)
|
||||
_FINDINGS.write_text(_findings_doc(table), encoding="utf-8")
|
||||
return {"mc": mc, "single": single, "verdict": verdict(mc, single), "table": table}
|
||||
|
||||
|
|
|
|||
|
|
@ -10,9 +10,14 @@ the experiment here — no live LLM, so this whole spike runs in the quality gat
|
|||
round/iteration guard and confirm the guard is what stops it (it does NOT
|
||||
self-terminate). With an explicit `max_round_count` it terminates cleanly.
|
||||
|
||||
(b) **Fan-out state isolation (G2/B7).** Reusing ONE workflow instance across the three
|
||||
reference projects bleeds state (the shared fake clients accumulate calls); a FRESH
|
||||
instance per run — via `fresh_workflow()` — shows zero bleed.
|
||||
(b) **Fan-out state isolation (G2/B7).** Reusing ONE built `ConcurrentBuilder` workflow
|
||||
across the three reference projects bleeds *conversation state*: MAF accumulates the
|
||||
shared thread across `.run()` calls, so each run's participants receive the PRIOR
|
||||
projects' prompts and replies (project N contaminates N+1's context). A FRESH instance
|
||||
per run — via `fresh_workflow()` — gives each run a clean thread (zero contamination).
|
||||
The observable is the message history each participant *receives*, captured via
|
||||
`FakeChatClient.received_texts` — NOT a call counter (a counter would rise for any
|
||||
reused mutable object and prove nothing about MAF state).
|
||||
|
||||
Token use: 0 — no live LLM (the fake client's "tokens" are word-counts of canned replies).
|
||||
"""
|
||||
|
|
@ -101,22 +106,41 @@ def fresh_workflow() -> tuple[object, tuple[FakeChatClient, FakeChatClient]]:
|
|||
return workflow, (c1, c2)
|
||||
|
||||
|
||||
async def shared_instance_max_calls(project_ids: list[str]) -> int:
|
||||
"""Reuse ONE fan-out instance across every project — state bleeds: the shared clients
|
||||
accumulate calls across runs. Returns the max per-client call count (== len once
|
||||
bled)."""
|
||||
def _projects_seen(received_texts: list[str], project_ids: list[str]) -> list[str]:
|
||||
"""Which project ids appear anywhere in the messages an agent received on one call.
|
||||
|
||||
The observable for genuine state bleed: if a later run's agent sees an EARLIER
|
||||
project's id, the workflow carried that project's conversation forward."""
|
||||
blob = " ".join(received_texts)
|
||||
return [pid for pid in project_ids if pid in blob]
|
||||
|
||||
|
||||
async def shared_instance_conversation_bleed(project_ids: list[str]) -> list[list[str]]:
|
||||
"""Reuse ONE built fan-out workflow across every project. MAF accumulates the shared
|
||||
thread across `.run()` calls, so run N's participants also receive runs 0..N-1's
|
||||
prompts/replies — genuine cross-run state corruption (G2/B7).
|
||||
|
||||
Returns, per run (in order), which project ids were visible to a participant on that
|
||||
run. With a reused instance this grows monotonically: ``[[p0], [p0, p1], [p0, p1, p2]]``."""
|
||||
workflow, clients = fresh_workflow()
|
||||
for pid in project_ids:
|
||||
await workflow.run(f"Evaluate project {pid}.")
|
||||
return max(c.call_count for c in clients)
|
||||
# One participant is representative: concurrent fan-out feeds every participant the
|
||||
# same accumulated thread. clients[0] was called once per run, in order.
|
||||
rep = clients[0]
|
||||
return [_projects_seen(call_view, project_ids) for call_view in rep.received_texts]
|
||||
|
||||
|
||||
async def fresh_instance_call_counts(project_ids: list[str]) -> list[int]:
|
||||
"""A FRESH instance per project — zero bleed: every run's clients see exactly one
|
||||
call. Returns the per-run max call count (each should be 1)."""
|
||||
counts: list[int] = []
|
||||
async def fresh_instance_conversation_isolation(project_ids: list[str]) -> list[list[str]]:
|
||||
"""A FRESH instance per project (the B7 mitigation): each run gets a clean thread, so
|
||||
a participant sees ONLY its own project — zero cross-run contamination.
|
||||
|
||||
Returns, per run, the project ids visible to a participant; each should be exactly
|
||||
its own: ``[[p0], [p1], [p2]]``."""
|
||||
seen_per_run: list[list[str]] = []
|
||||
for pid in project_ids:
|
||||
workflow, clients = fresh_workflow()
|
||||
await workflow.run(f"Evaluate project {pid}.")
|
||||
counts.append(max(c.call_count for c in clients))
|
||||
return counts
|
||||
# fresh client -> exactly one call this run; read its single received view.
|
||||
seen_per_run.append(_projects_seen(clients[0].received_texts[0], project_ids))
|
||||
return seen_per_run
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue