Kjell Tore Guttormsen
37547fe292
The context sets, the packaged knowledge bases and the example bundles are replaced by one fictitious example set about IT operations in an invented organisation: three context sets (serverrom-2027, driftsavtale-2027 and the two-base drift-og-avtale-2027), two synthetic knowledge bases under src/portfolio_optimiser/data/kunnskapsbaser and two example bundles under src/portfolio_optimiser/data/bundles. Numbers, codes and structural values in tests and fixtures are kept; names, ids and wording change. Dated measurement documents that only recorded runs on the replaced material are deleted. Gate figures measured on the new set are not comparable with earlier ones. The exclusion gate from the previous commit is green: 0 tracked files hit outside the shared/ subtree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
77 lines
4.6 KiB
Markdown
77 lines
4.6 KiB
Markdown
# Spike B findings — Magentic unbounded footgun (G1/B4) + fan-out state isolation (G2/B7)
|
|
|
|
**Assumptions:** (G1/B4) Magentic with `limits=None` does NOT self-terminate, so an
|
|
explicit limit is mandatory; (G2/B7) reusing a shared fan-out builder/workflow instance
|
|
corrupts state, while a fresh-instance helper prevents it.
|
|
|
|
This spike drives the **real** GA builders with the fake client (de-risked by the Step 2
|
|
builder smoke), so it runs entirely in the quality gate — **no live LLM**.
|
|
|
|
## (a) Magentic unbounded termination — CONFIRMED
|
|
|
|
- A `StandardMagenticManager` whose progress ledger always reports *not satisfied /
|
|
progress being made* never finalizes.
|
|
- With **`max_round_count=None`** the workflow streams orchestration events indefinitely;
|
|
only the shared harness round/iteration guard (`Budget` + `TokenMeter`) stops it.
|
|
`unbounded_magentic_self_terminates()` returns **False** → it does **not** self-terminate.
|
|
- With an explicit **`max_round_count=2`** the same manager stops cleanly
|
|
("Magentic Orchestrator: Max round count reached") and yields an output.
|
|
- **Implication (B4):** a stop-criterion/round cap is mandatory at startup for any
|
|
Magentic use — never rely on natural termination. (Magentic stays experimental and is
|
|
exercised here ONLY to confirm the footgun, never as the debate default — that remains
|
|
Group Chat maker-checker, A2/G8.)
|
|
|
|
### Minor MAF observation (recorded)
|
|
|
|
Cutting an *instrumented* Magentic stream short — the only way to observe an unbounded run
|
|
— makes OpenTelemetry log a benign `Failed to detach context` on generator close, and the
|
|
in-flight fake response surfaces a `RuntimeWarning: coroutine was never awaited`. Both are
|
|
cosmetic (no functional effect); the spike silences the OTel context logger and filters the
|
|
warning. Worth knowing if Fase 2 observes streamed orchestrations.
|
|
|
|
## (b) Fan-out state isolation — CONFIRMED (conversation-history bleed)
|
|
|
|
The observable is the **message history each participant receives** (captured via
|
|
`FakeChatClient.received_texts`), NOT a call counter. A counter rises for any reused
|
|
mutable object and proves nothing about MAF; message content proves the workflow carried
|
|
state across runs. (An earlier version of this spike measured only a call counter and was
|
|
a tautology — caught and rebuilt; see the Method note below.)
|
|
|
|
- **Shared instance:** one built `ConcurrentBuilder` workflow reused across the three
|
|
reference projects. MAF **accumulates the shared conversation thread across `.run()`
|
|
calls**: each run's participants receive the PRIOR projects' prompts and replies too.
|
|
Measured (project ids visible per run, in order):
|
|
`[[KONTOR-IT-E1], [KONTOR-IT-E1, NETT-SIKR-TP], [KONTOR-IT-E1, NETT-SIKR-TP, ARKIV-LAGR-MIGR]]`
|
|
— strictly monotonic growth → run N is contaminated by runs 0..N-1 → **genuine state
|
|
bleed (G2/B7)**.
|
|
- **Fresh instance:** `fresh_workflow()` builds a new workflow + clean thread per run →
|
|
each participant sees ONLY its own project: `[[KONTOR-IT-E1], [NETT-SIKR-TP],
|
|
[ARKIV-LAGR-MIGR]]` → **zero cross-run contamination** (B7 mitigation works).
|
|
- **Implication:** Fase 2 fan-out must build a fresh workflow/executor instance per
|
|
project run (a factory like `fresh_workflow()`), never reuse a shared instance — a
|
|
reused `Workflow` leaks one project's context into the next, which would corrupt
|
|
per-project cost analyses. This is a property of MAF's `Workflow` thread state, not of
|
|
the participants.
|
|
|
|
### Method note (why this was rebuilt)
|
|
|
|
The first cut of this experiment asserted that a reused instance's per-client `call_count`
|
|
reached 3 while a fresh instance's stayed at 1. That is a tautology: any un-reset mutable
|
|
counter rises across reuse, independent of MAF, so it never demonstrated the footgun.
|
|
`/trekreview` flagged it as a `BROKEN_SUCCESS_CRITERION` (false-confirm of a de-risk
|
|
assumption). The rebuilt experiment observes real MAF thread state via the messages each
|
|
participant receives, so a passing test now genuinely means "the reused workflow carried
|
|
project N's conversation into project N+1." Lesson carried into Fase 2: **a de-risk
|
|
assertion must observe the mechanism it claims to test, not a proxy that moves for unrelated
|
|
reasons.**
|
|
|
|
## Token use
|
|
|
|
**0 — no live LLM.** All behavior is driven by the deterministic `FakeChatClient`; its
|
|
"tokens" are word-counts of canned replies and are not meaningful cost numbers here.
|
|
|
|
## Implication for Fase 2
|
|
|
|
Both footguns behave exactly as the §15 register predicted. The framework MUST: (1) require
|
|
explicit round/stop caps for any Magentic-style loop (fail-fast at startup, D6/B4); and
|
|
(2) use a fresh-instance-per-run factory for fan-out to avoid state corruption (B7).
|