Kjell Tore Guttormsen
a2dff210ce
/trekreview flagged the Spike B(b) fan-out experiment as BROKEN_SUCCESS_CRITERION (BLOCKER): it asserted a per-client call_count reached 3 on a reused instance vs 1 on a fresh one — a tautology true for any un-reset mutable counter, independent of MAF, that never exercised the real G2/B7 shared-Workflow state-corruption footgun. It was a false-confirm of a de-risk assumption. Rebuilt to observe genuine MAF thread state via the messages each participant RECEIVES (new FakeChatClient.received_texts seam): - shared_instance_conversation_bleed: a reused built ConcurrentBuilder Workflow accumulates the conversation across .run() calls — run N's participants receive runs 0..N-1's prompts/replies (measured [[p0],[p0,p1],[p0,p1,p2]], strictly monotonic) => genuine cross-run contamination. - fresh_instance_conversation_isolation: a fresh instance per run gives each a clean thread => each participant sees only its own project ([[p0],[p1],[p2]]). Assumption now CONFIRMED with a meaningful observable. findings-b.md gains a Method note recording why it was rebuilt; README rows updated. Also fixes the MINOR: a_groupchat.run_live now mkdirs the findings dir before write_text so a post-disposal run does not lose the measured result. Gate green: ruff check + format, mypy src, pytest 48 passed / 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
77 lines
4.6 KiB
Markdown
77 lines
4.6 KiB
Markdown
# Spike B findings — Magentic unbounded footgun (G1/B4) + fan-out state isolation (G2/B7)
|
|
|
|
**Assumptions:** (G1/B4) Magentic with `limits=None` does NOT self-terminate, so an
|
|
explicit limit is mandatory; (G2/B7) reusing a shared fan-out builder/workflow instance
|
|
corrupts state, while a fresh-instance helper prevents it.
|
|
|
|
This spike drives the **real** GA builders with the fake client (de-risked by the Step 2
|
|
builder smoke), so it runs entirely in the quality gate — **no live LLM**.
|
|
|
|
## (a) Magentic unbounded termination — CONFIRMED
|
|
|
|
- A `StandardMagenticManager` whose progress ledger always reports *not satisfied /
|
|
progress being made* never finalizes.
|
|
- With **`max_round_count=None`** the workflow streams orchestration events indefinitely;
|
|
only the shared harness round/iteration guard (`Budget` + `TokenMeter`) stops it.
|
|
`unbounded_magentic_self_terminates()` returns **False** → it does **not** self-terminate.
|
|
- With an explicit **`max_round_count=2`** the same manager stops cleanly
|
|
("Magentic Orchestrator: Max round count reached") and yields an output.
|
|
- **Implication (B4):** a stop-criterion/round cap is mandatory at startup for any
|
|
Magentic use — never rely on natural termination. (Magentic stays experimental and is
|
|
exercised here ONLY to confirm the footgun, never as the debate default — that remains
|
|
Group Chat maker-checker, A2/G8.)
|
|
|
|
### Minor MAF observation (recorded)
|
|
|
|
Cutting an *instrumented* Magentic stream short — the only way to observe an unbounded run
|
|
— makes OpenTelemetry log a benign `Failed to detach context` on generator close, and the
|
|
in-flight fake response surfaces a `RuntimeWarning: coroutine was never awaited`. Both are
|
|
cosmetic (no functional effect); the spike silences the OTel context logger and filters the
|
|
warning. Worth knowing if Fase 2 observes streamed orchestrations.
|
|
|
|
## (b) Fan-out state isolation — CONFIRMED (conversation-history bleed)
|
|
|
|
The observable is the **message history each participant receives** (captured via
|
|
`FakeChatClient.received_texts`), NOT a call counter. A counter rises for any reused
|
|
mutable object and proves nothing about MAF; message content proves the workflow carried
|
|
state across runs. (An earlier version of this spike measured only a call counter and was
|
|
a tautology — caught and rebuilt; see the Method note below.)
|
|
|
|
- **Shared instance:** one built `ConcurrentBuilder` workflow reused across the three
|
|
reference projects. MAF **accumulates the shared conversation thread across `.run()`
|
|
calls**: each run's participants receive the PRIOR projects' prompts and replies too.
|
|
Measured (project ids visible per run, in order):
|
|
`[[FV42-GSV-E1], [FV42-GSV-E1, RV13-RAS-TP], [FV42-GSV-E1, RV13-RAS-TP, BRU-LAKS-REHAB]]`
|
|
— strictly monotonic growth → run N is contaminated by runs 0..N-1 → **genuine state
|
|
bleed (G2/B7)**.
|
|
- **Fresh instance:** `fresh_workflow()` builds a new workflow + clean thread per run →
|
|
each participant sees ONLY its own project: `[[FV42-GSV-E1], [RV13-RAS-TP],
|
|
[BRU-LAKS-REHAB]]` → **zero cross-run contamination** (B7 mitigation works).
|
|
- **Implication:** Fase 2 fan-out must build a fresh workflow/executor instance per
|
|
project run (a factory like `fresh_workflow()`), never reuse a shared instance — a
|
|
reused `Workflow` leaks one project's context into the next, which would corrupt
|
|
per-project cost analyses. This is a property of MAF's `Workflow` thread state, not of
|
|
the participants.
|
|
|
|
### Method note (why this was rebuilt)
|
|
|
|
The first cut of this experiment asserted that a reused instance's per-client `call_count`
|
|
reached 3 while a fresh instance's stayed at 1. That is a tautology: any un-reset mutable
|
|
counter rises across reuse, independent of MAF, so it never demonstrated the footgun.
|
|
`/trekreview` flagged it as a `BROKEN_SUCCESS_CRITERION` (false-confirm of a de-risk
|
|
assumption). The rebuilt experiment observes real MAF thread state via the messages each
|
|
participant receives, so a passing test now genuinely means "the reused workflow carried
|
|
project N's conversation into project N+1." Lesson carried into Fase 2: **a de-risk
|
|
assertion must observe the mechanism it claims to test, not a proxy that moves for unrelated
|
|
reasons.**
|
|
|
|
## Token use
|
|
|
|
**0 — no live LLM.** All behavior is driven by the deterministic `FakeChatClient`; its
|
|
"tokens" are word-counts of canned replies and are not meaningful cost numbers here.
|
|
|
|
## Implication for Fase 2
|
|
|
|
Both footguns behave exactly as the §15 register predicted. The framework MUST: (1) require
|
|
explicit round/stop caps for any Magentic-style loop (fail-fast at startup, D6/B4); and
|
|
(2) use a fresh-instance-per-run factory for fan-out to avoid state corruption (B7).
|