Kjell Tore Guttormsen a2dff210ce fix(fase1): spike B fan-out measures real conversation bleed, not a counter

/trekreview flagged the Spike B(b) fan-out experiment as BROKEN_SUCCESS_CRITERION
(BLOCKER): it asserted a per-client call_count reached 3 on a reused instance vs
1 on a fresh one — a tautology true for any un-reset mutable counter, independent
of MAF, that never exercised the real G2/B7 shared-Workflow state-corruption
footgun. It was a false-confirm of a de-risk assumption.

Rebuilt to observe genuine MAF thread state via the messages each participant
RECEIVES (new FakeChatClient.received_texts seam):
- shared_instance_conversation_bleed: a reused built ConcurrentBuilder Workflow
  accumulates the conversation across .run() calls — run N's participants receive
  runs 0..N-1's prompts/replies (measured [[p0],[p0,p1],[p0,p1,p2]], strictly
  monotonic) => genuine cross-run contamination.
- fresh_instance_conversation_isolation: a fresh instance per run gives each a
  clean thread => each participant sees only its own project ([[p0],[p1],[p2]]).

Assumption now CONFIRMED with a meaningful observable. findings-b.md gains a
Method note recording why it was rebuilt; README rows updated.

Also fixes the MINOR: a_groupchat.run_live now mkdirs the findings dir before
write_text so a post-disposal run does not lose the measured result.

Gate green: ruff check + format, mypy src, pytest 48 passed / 1 skipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH

2026-06-24 11:09:55 +02:00

4.6 KiB

Raw Blame History

Spike B findings — Magentic unbounded footgun (G1/B4) + fan-out state isolation (G2/B7)

Assumptions: (G1/B4) Magentic with limits=None does NOT self-terminate, so an explicit limit is mandatory; (G2/B7) reusing a shared fan-out builder/workflow instance corrupts state, while a fresh-instance helper prevents it.

This spike drives the real GA builders with the fake client (de-risked by the Step 2 builder smoke), so it runs entirely in the quality gate — no live LLM.

(a) Magentic unbounded termination — CONFIRMED

A StandardMagenticManager whose progress ledger always reports not satisfied / progress being made never finalizes.
With max_round_count=None the workflow streams orchestration events indefinitely; only the shared harness round/iteration guard (Budget + TokenMeter) stops it. unbounded_magentic_self_terminates() returns False → it does not self-terminate.
With an explicit max_round_count=2 the same manager stops cleanly ("Magentic Orchestrator: Max round count reached") and yields an output.
Implication (B4): a stop-criterion/round cap is mandatory at startup for any Magentic use — never rely on natural termination. (Magentic stays experimental and is exercised here ONLY to confirm the footgun, never as the debate default — that remains Group Chat maker-checker, A2/G8.)

Minor MAF observation (recorded)

Cutting an instrumented Magentic stream short — the only way to observe an unbounded run — makes OpenTelemetry log a benign Failed to detach context on generator close, and the in-flight fake response surfaces a RuntimeWarning: coroutine was never awaited. Both are cosmetic (no functional effect); the spike silences the OTel context logger and filters the warning. Worth knowing if Fase 2 observes streamed orchestrations.

(b) Fan-out state isolation — CONFIRMED (conversation-history bleed)

The observable is the message history each participant receives (captured via FakeChatClient.received_texts), NOT a call counter. A counter rises for any reused mutable object and proves nothing about MAF; message content proves the workflow carried state across runs. (An earlier version of this spike measured only a call counter and was a tautology — caught and rebuilt; see the Method note below.)

Shared instance: one built ConcurrentBuilder workflow reused across the three reference projects. MAF accumulates the shared conversation thread across .run() calls: each run's participants receive the PRIOR projects' prompts and replies too. Measured (project ids visible per run, in order): [[FV42-GSV-E1], [FV42-GSV-E1, RV13-RAS-TP], [FV42-GSV-E1, RV13-RAS-TP, BRU-LAKS-REHAB]] — strictly monotonic growth → run N is contaminated by runs 0..N-1 → genuine state bleed (G2/B7).
Fresh instance: fresh_workflow() builds a new workflow + clean thread per run → each participant sees ONLY its own project: [[FV42-GSV-E1], [RV13-RAS-TP], [BRU-LAKS-REHAB]] → zero cross-run contamination (B7 mitigation works).
Implication: Fase 2 fan-out must build a fresh workflow/executor instance per project run (a factory like fresh_workflow()), never reuse a shared instance — a reused Workflow leaks one project's context into the next, which would corrupt per-project cost analyses. This is a property of MAF's Workflow thread state, not of the participants.

Method note (why this was rebuilt)

The first cut of this experiment asserted that a reused instance's per-client call_count reached 3 while a fresh instance's stayed at 1. That is a tautology: any un-reset mutable counter rises across reuse, independent of MAF, so it never demonstrated the footgun. /trekreview flagged it as a BROKEN_SUCCESS_CRITERION (false-confirm of a de-risk assumption). The rebuilt experiment observes real MAF thread state via the messages each participant receives, so a passing test now genuinely means "the reused workflow carried project N's conversation into project N+1." Lesson carried into Fase 2: a de-risk assertion must observe the mechanism it claims to test, not a proxy that moves for unrelated reasons.

Token use

0 — no live LLM. All behavior is driven by the deterministic FakeChatClient; its "tokens" are word-counts of canned replies and are not meaningful cost numbers here.

Implication for Fase 2

Both footguns behave exactly as the §15 register predicted. The framework MUST: (1) require explicit round/stop caps for any Magentic-style loop (fail-fast at startup, D6/B4); and (2) use a fresh-instance-per-run factory for fan-out to avoid state corruption (B7).

4.6 KiB Raw Blame History