refactor(examples): replace sector-specific example material with generic, fictitious examples

The context sets, the packaged knowledge bases and the example bundles are
replaced by one fictitious example set about IT operations in an invented
organisation: three context sets (serverrom-2027, driftsavtale-2027 and the
two-base drift-og-avtale-2027), two synthetic knowledge bases under
src/portfolio_optimiser/data/kunnskapsbaser and two example bundles under
src/portfolio_optimiser/data/bundles. Numbers, codes and structural values in
tests and fixtures are kept; names, ids and wording change. Dated measurement
documents that only recorded runs on the replaced material are deleted.

Gate figures measured on the new set are not comparable with earlier ones.
The exclusion gate from the previous commit is green: 0 tracked files hit
outside the shared/ subtree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-09-23 15:04:21 +02:00

4.6 KiB

Raw Blame History

Spike B findings — Magentic unbounded footgun (G1/B4) + fan-out state isolation (G2/B7)

Assumptions: (G1/B4) Magentic with limits=None does NOT self-terminate, so an explicit limit is mandatory; (G2/B7) reusing a shared fan-out builder/workflow instance corrupts state, while a fresh-instance helper prevents it.

This spike drives the real GA builders with the fake client (de-risked by the Step 2 builder smoke), so it runs entirely in the quality gate — no live LLM.

(a) Magentic unbounded termination — CONFIRMED

A StandardMagenticManager whose progress ledger always reports not satisfied / progress being made never finalizes.
With max_round_count=None the workflow streams orchestration events indefinitely; only the shared harness round/iteration guard (Budget + TokenMeter) stops it. unbounded_magentic_self_terminates() returns False → it does not self-terminate.
With an explicit max_round_count=2 the same manager stops cleanly ("Magentic Orchestrator: Max round count reached") and yields an output.
Implication (B4): a stop-criterion/round cap is mandatory at startup for any Magentic use — never rely on natural termination. (Magentic stays experimental and is exercised here ONLY to confirm the footgun, never as the debate default — that remains Group Chat maker-checker, A2/G8.)

Minor MAF observation (recorded)

Cutting an instrumented Magentic stream short — the only way to observe an unbounded run — makes OpenTelemetry log a benign Failed to detach context on generator close, and the in-flight fake response surfaces a RuntimeWarning: coroutine was never awaited. Both are cosmetic (no functional effect); the spike silences the OTel context logger and filters the warning. Worth knowing if Fase 2 observes streamed orchestrations.

(b) Fan-out state isolation — CONFIRMED (conversation-history bleed)

The observable is the message history each participant receives (captured via FakeChatClient.received_texts), NOT a call counter. A counter rises for any reused mutable object and proves nothing about MAF; message content proves the workflow carried state across runs. (An earlier version of this spike measured only a call counter and was a tautology — caught and rebuilt; see the Method note below.)

Shared instance: one built ConcurrentBuilder workflow reused across the three reference projects. MAF accumulates the shared conversation thread across .run() calls: each run's participants receive the PRIOR projects' prompts and replies too. Measured (project ids visible per run, in order): [[KONTOR-IT-E1], [KONTOR-IT-E1, NETT-SIKR-TP], [KONTOR-IT-E1, NETT-SIKR-TP, ARKIV-LAGR-MIGR]] — strictly monotonic growth → run N is contaminated by runs 0..N-1 → genuine state bleed (G2/B7).
Fresh instance: fresh_workflow() builds a new workflow + clean thread per run → each participant sees ONLY its own project: [[KONTOR-IT-E1], [NETT-SIKR-TP], [ARKIV-LAGR-MIGR]] → zero cross-run contamination (B7 mitigation works).
Implication: Fase 2 fan-out must build a fresh workflow/executor instance per project run (a factory like fresh_workflow()), never reuse a shared instance — a reused Workflow leaks one project's context into the next, which would corrupt per-project cost analyses. This is a property of MAF's Workflow thread state, not of the participants.

Method note (why this was rebuilt)

The first cut of this experiment asserted that a reused instance's per-client call_count reached 3 while a fresh instance's stayed at 1. That is a tautology: any un-reset mutable counter rises across reuse, independent of MAF, so it never demonstrated the footgun. /trekreview flagged it as a BROKEN_SUCCESS_CRITERION (false-confirm of a de-risk assumption). The rebuilt experiment observes real MAF thread state via the messages each participant receives, so a passing test now genuinely means "the reused workflow carried project N's conversation into project N+1." Lesson carried into Fase 2: a de-risk assertion must observe the mechanism it claims to test, not a proxy that moves for unrelated reasons.

Token use

0 — no live LLM. All behavior is driven by the deterministic FakeChatClient; its "tokens" are word-counts of canned replies and are not meaningful cost numbers here.

Implication for Fase 2

Both footguns behave exactly as the §15 register predicted. The framework MUST: (1) require explicit round/stop caps for any Magentic-style loop (fail-fast at startup, D6/B4); and (2) use a fresh-instance-per-run factory for fan-out to avoid state corruption (B7).

4.6 KiB Raw Blame History