fix(fase1): spike B fan-out measures real conversation bleed, not a counter

/trekreview flagged the Spike B(b) fan-out experiment as BROKEN_SUCCESS_CRITERION
(BLOCKER): it asserted a per-client call_count reached 3 on a reused instance vs
1 on a fresh one — a tautology true for any un-reset mutable counter, independent
of MAF, that never exercised the real G2/B7 shared-Workflow state-corruption
footgun. It was a false-confirm of a de-risk assumption.

Rebuilt to observe genuine MAF thread state via the messages each participant
RECEIVES (new FakeChatClient.received_texts seam):
- shared_instance_conversation_bleed: a reused built ConcurrentBuilder Workflow
  accumulates the conversation across .run() calls — run N's participants receive
  runs 0..N-1's prompts/replies (measured [[p0],[p0,p1],[p0,p1,p2]], strictly
  monotonic) => genuine cross-run contamination.
- fresh_instance_conversation_isolation: a fresh instance per run gives each a
  clean thread => each participant sees only its own project ([[p0],[p1],[p2]]).

Assumption now CONFIRMED with a meaningful observable. findings-b.md gains a
Method note recording why it was rebuilt; README rows updated.

Also fixes the MINOR: a_groupchat.run_live now mkdirs the findings dir before
write_text so a post-disposal run does not lose the measured result.

Gate green: ruff check + format, mypy src, pytest 48 passed / 1 skipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
This commit is contained in:
Kjell Tore Guttormsen 2026-06-24 11:09:55 +02:00
commit a2dff210ce
6 changed files with 118 additions and 35 deletions

View file

@ -29,16 +29,41 @@ in-flight fake response surfaces a `RuntimeWarning: coroutine was never awaited`
cosmetic (no functional effect); the spike silences the OTel context logger and filters the
warning. Worth knowing if Fase 2 observes streamed orchestrations.
## (b) Fan-out state isolation — CONFIRMED
## (b) Fan-out state isolation — CONFIRMED (conversation-history bleed)
- **Shared instance:** one `ConcurrentBuilder` workflow reused across the three reference
projects → the shared fake clients accumulate calls (max call count == 3) → **state
bleed**.
- **Fresh instance:** `fresh_workflow()` builds a new workflow + fresh clients per run →
each run's clients see exactly one call (`[1, 1, 1]`) → **zero bleed** across all 3
projects (B7).
The observable is the **message history each participant receives** (captured via
`FakeChatClient.received_texts`), NOT a call counter. A counter rises for any reused
mutable object and proves nothing about MAF; message content proves the workflow carried
state across runs. (An earlier version of this spike measured only a call counter and was
a tautology — caught and rebuilt; see the Method note below.)
- **Shared instance:** one built `ConcurrentBuilder` workflow reused across the three
reference projects. MAF **accumulates the shared conversation thread across `.run()`
calls**: each run's participants receive the PRIOR projects' prompts and replies too.
Measured (project ids visible per run, in order):
`[[FV42-GSV-E1], [FV42-GSV-E1, RV13-RAS-TP], [FV42-GSV-E1, RV13-RAS-TP, BRU-LAKS-REHAB]]`
— strictly monotonic growth → run N is contaminated by runs 0..N-1 → **genuine state
bleed (G2/B7)**.
- **Fresh instance:** `fresh_workflow()` builds a new workflow + clean thread per run →
each participant sees ONLY its own project: `[[FV42-GSV-E1], [RV13-RAS-TP],
[BRU-LAKS-REHAB]]` → **zero cross-run contamination** (B7 mitigation works).
- **Implication:** Fase 2 fan-out must build a fresh workflow/executor instance per
project run (a factory like `fresh_workflow()`), never reuse a shared instance.
project run (a factory like `fresh_workflow()`), never reuse a shared instance — a
reused `Workflow` leaks one project's context into the next, which would corrupt
per-project cost analyses. This is a property of MAF's `Workflow` thread state, not of
the participants.
### Method note (why this was rebuilt)
The first cut of this experiment asserted that a reused instance's per-client `call_count`
reached 3 while a fresh instance's stayed at 1. That is a tautology: any un-reset mutable
counter rises across reuse, independent of MAF, so it never demonstrated the footgun.
`/trekreview` flagged it as a `BROKEN_SUCCESS_CRITERION` (false-confirm of a de-risk
assumption). The rebuilt experiment observes real MAF thread state via the messages each
participant receives, so a passing test now genuinely means "the reused workflow carried
project N's conversation into project N+1." Lesson carried into Fase 2: **a de-risk
assertion must observe the mechanism it claims to test, not a proxy that moves for unrelated
reasons.**
## Token use