fix(fase1): spike B fan-out measures real conversation bleed, not a counter
/trekreview flagged the Spike B(b) fan-out experiment as BROKEN_SUCCESS_CRITERION (BLOCKER): it asserted a per-client call_count reached 3 on a reused instance vs 1 on a fresh one — a tautology true for any un-reset mutable counter, independent of MAF, that never exercised the real G2/B7 shared-Workflow state-corruption footgun. It was a false-confirm of a de-risk assumption. Rebuilt to observe genuine MAF thread state via the messages each participant RECEIVES (new FakeChatClient.received_texts seam): - shared_instance_conversation_bleed: a reused built ConcurrentBuilder Workflow accumulates the conversation across .run() calls — run N's participants receive runs 0..N-1's prompts/replies (measured [[p0],[p0,p1],[p0,p1,p2]], strictly monotonic) => genuine cross-run contamination. - fresh_instance_conversation_isolation: a fresh instance per run gives each a clean thread => each participant sees only its own project ([[p0],[p1],[p2]]). Assumption now CONFIRMED with a meaningful observable. findings-b.md gains a Method note recording why it was rebuilt; README rows updated. Also fixes the MINOR: a_groupchat.run_live now mkdirs the findings dir before write_text so a post-disposal run does not lose the measured result. Gate green: ruff check + format, mypy src, pytest 48 passed / 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
This commit is contained in:
parent
b81e22b637
commit
a2dff210ce
6 changed files with 118 additions and 35 deletions
|
|
@ -10,9 +10,14 @@ the experiment here — no live LLM, so this whole spike runs in the quality gat
|
|||
round/iteration guard and confirm the guard is what stops it (it does NOT
|
||||
self-terminate). With an explicit `max_round_count` it terminates cleanly.
|
||||
|
||||
(b) **Fan-out state isolation (G2/B7).** Reusing ONE workflow instance across the three
|
||||
reference projects bleeds state (the shared fake clients accumulate calls); a FRESH
|
||||
instance per run — via `fresh_workflow()` — shows zero bleed.
|
||||
(b) **Fan-out state isolation (G2/B7).** Reusing ONE built `ConcurrentBuilder` workflow
|
||||
across the three reference projects bleeds *conversation state*: MAF accumulates the
|
||||
shared thread across `.run()` calls, so each run's participants receive the PRIOR
|
||||
projects' prompts and replies (project N contaminates N+1's context). A FRESH instance
|
||||
per run — via `fresh_workflow()` — gives each run a clean thread (zero contamination).
|
||||
The observable is the message history each participant *receives*, captured via
|
||||
`FakeChatClient.received_texts` — NOT a call counter (a counter would rise for any
|
||||
reused mutable object and prove nothing about MAF state).
|
||||
|
||||
Token use: 0 — no live LLM (the fake client's "tokens" are word-counts of canned replies).
|
||||
"""
|
||||
|
|
@ -101,22 +106,41 @@ def fresh_workflow() -> tuple[object, tuple[FakeChatClient, FakeChatClient]]:
|
|||
return workflow, (c1, c2)
|
||||
|
||||
|
||||
async def shared_instance_max_calls(project_ids: list[str]) -> int:
|
||||
"""Reuse ONE fan-out instance across every project — state bleeds: the shared clients
|
||||
accumulate calls across runs. Returns the max per-client call count (== len once
|
||||
bled)."""
|
||||
def _projects_seen(received_texts: list[str], project_ids: list[str]) -> list[str]:
|
||||
"""Which project ids appear anywhere in the messages an agent received on one call.
|
||||
|
||||
The observable for genuine state bleed: if a later run's agent sees an EARLIER
|
||||
project's id, the workflow carried that project's conversation forward."""
|
||||
blob = " ".join(received_texts)
|
||||
return [pid for pid in project_ids if pid in blob]
|
||||
|
||||
|
||||
async def shared_instance_conversation_bleed(project_ids: list[str]) -> list[list[str]]:
|
||||
"""Reuse ONE built fan-out workflow across every project. MAF accumulates the shared
|
||||
thread across `.run()` calls, so run N's participants also receive runs 0..N-1's
|
||||
prompts/replies — genuine cross-run state corruption (G2/B7).
|
||||
|
||||
Returns, per run (in order), which project ids were visible to a participant on that
|
||||
run. With a reused instance this grows monotonically: ``[[p0], [p0, p1], [p0, p1, p2]]``."""
|
||||
workflow, clients = fresh_workflow()
|
||||
for pid in project_ids:
|
||||
await workflow.run(f"Evaluate project {pid}.")
|
||||
return max(c.call_count for c in clients)
|
||||
# One participant is representative: concurrent fan-out feeds every participant the
|
||||
# same accumulated thread. clients[0] was called once per run, in order.
|
||||
rep = clients[0]
|
||||
return [_projects_seen(call_view, project_ids) for call_view in rep.received_texts]
|
||||
|
||||
|
||||
async def fresh_instance_call_counts(project_ids: list[str]) -> list[int]:
|
||||
"""A FRESH instance per project — zero bleed: every run's clients see exactly one
|
||||
call. Returns the per-run max call count (each should be 1)."""
|
||||
counts: list[int] = []
|
||||
async def fresh_instance_conversation_isolation(project_ids: list[str]) -> list[list[str]]:
|
||||
"""A FRESH instance per project (the B7 mitigation): each run gets a clean thread, so
|
||||
a participant sees ONLY its own project — zero cross-run contamination.
|
||||
|
||||
Returns, per run, the project ids visible to a participant; each should be exactly
|
||||
its own: ``[[p0], [p1], [p2]]``."""
|
||||
seen_per_run: list[list[str]] = []
|
||||
for pid in project_ids:
|
||||
workflow, clients = fresh_workflow()
|
||||
await workflow.run(f"Evaluate project {pid}.")
|
||||
counts.append(max(c.call_count for c in clients))
|
||||
return counts
|
||||
# fresh client -> exactly one call this run; read its single received view.
|
||||
seen_per_run.append(_projects_seen(clients[0].received_texts[0], project_ids))
|
||||
return seen_per_run
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue