52 lines
2.9 KiB
Markdown
52 lines
2.9 KiB
Markdown
# Spike B findings — Magentic unbounded footgun (G1/B4) + fan-out state isolation (G2/B7)
|
|
|
|
**Assumptions:** (G1/B4) Magentic with `limits=None` does NOT self-terminate, so an
|
|
explicit limit is mandatory; (G2/B7) reusing a shared fan-out builder/workflow instance
|
|
corrupts state, while a fresh-instance helper prevents it.
|
|
|
|
This spike drives the **real** GA builders with the fake client (de-risked by the Step 2
|
|
builder smoke), so it runs entirely in the quality gate — **no live LLM**.
|
|
|
|
## (a) Magentic unbounded termination — CONFIRMED
|
|
|
|
- A `StandardMagenticManager` whose progress ledger always reports *not satisfied /
|
|
progress being made* never finalizes.
|
|
- With **`max_round_count=None`** the workflow streams orchestration events indefinitely;
|
|
only the shared harness round/iteration guard (`Budget` + `TokenMeter`) stops it.
|
|
`unbounded_magentic_self_terminates()` returns **False** → it does **not** self-terminate.
|
|
- With an explicit **`max_round_count=2`** the same manager stops cleanly
|
|
("Magentic Orchestrator: Max round count reached") and yields an output.
|
|
- **Implication (B4):** a stop-criterion/round cap is mandatory at startup for any
|
|
Magentic use — never rely on natural termination. (Magentic stays experimental and is
|
|
exercised here ONLY to confirm the footgun, never as the debate default — that remains
|
|
Group Chat maker-checker, A2/G8.)
|
|
|
|
### Minor MAF observation (recorded)
|
|
|
|
Cutting an *instrumented* Magentic stream short — the only way to observe an unbounded run
|
|
— makes OpenTelemetry log a benign `Failed to detach context` on generator close, and the
|
|
in-flight fake response surfaces a `RuntimeWarning: coroutine was never awaited`. Both are
|
|
cosmetic (no functional effect); the spike silences the OTel context logger and filters the
|
|
warning. Worth knowing if Fase 2 observes streamed orchestrations.
|
|
|
|
## (b) Fan-out state isolation — CONFIRMED
|
|
|
|
- **Shared instance:** one `ConcurrentBuilder` workflow reused across the three reference
|
|
projects → the shared fake clients accumulate calls (max call count == 3) → **state
|
|
bleed**.
|
|
- **Fresh instance:** `fresh_workflow()` builds a new workflow + fresh clients per run →
|
|
each run's clients see exactly one call (`[1, 1, 1]`) → **zero bleed** across all 3
|
|
projects (B7).
|
|
- **Implication:** Fase 2 fan-out must build a fresh workflow/executor instance per
|
|
project run (a factory like `fresh_workflow()`), never reuse a shared instance.
|
|
|
|
## Token use
|
|
|
|
**0 — no live LLM.** All behavior is driven by the deterministic `FakeChatClient`; its
|
|
"tokens" are word-counts of canned replies and are not meaningful cost numbers here.
|
|
|
|
## Implication for Fase 2
|
|
|
|
Both footguns behave exactly as the §15 register predicted. The framework MUST: (1) require
|
|
explicit round/stop caps for any Magentic-style loop (fail-fast at startup, D6/B4); and
|
|
(2) use a fresh-instance-per-run factory for fan-out to avoid state corruption (B7).
|