open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases Activity

fix(fase1): spike B fan-out measures real conversation bleed, not a counter

Browse source 
/trekreview flagged the Spike B(b) fan-out experiment as BROKEN_SUCCESS_CRITERION
(BLOCKER): it asserted a per-client call_count reached 3 on a reused instance vs
1 on a fresh one — a tautology true for any un-reset mutable counter, independent
of MAF, that never exercised the real G2/B7 shared-Workflow state-corruption
footgun. It was a false-confirm of a de-risk assumption.

Rebuilt to observe genuine MAF thread state via the messages each participant
RECEIVES (new FakeChatClient.received_texts seam):
- shared_instance_conversation_bleed: a reused built ConcurrentBuilder Workflow
  accumulates the conversation across .run() calls — run N's participants receive
  runs 0..N-1's prompts/replies (measured [[p0],[p0,p1],[p0,p1,p2]], strictly
  monotonic) => genuine cross-run contamination.
- fresh_instance_conversation_isolation: a fresh instance per run gives each a
  clean thread => each participant sees only its own project ([[p0],[p1],[p2]]).

Assumption now CONFIRMED with a meaningful observable. findings-b.md gains a
Method note recording why it was rebuilt; README rows updated.

Also fixes the MINOR: a_groupchat.run_live now mkdirs the findings dir before
write_text so a post-disposal run does not lose the measured result.

Gate green: ruff check + format, mypy src, pytest 48 passed / 1 skipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
This commit is contained in:
Kjell Tore Guttormsen 2026-06-24 11:09:55 +02:00
commit a2dff210ce
6 changed files with 118 additions and 35 deletions
Download patch file Download diff file Expand all files Collapse all files

4
docs/fase1-spikes/README.md
View file

@ -13,7 +13,7 @@ the four assumptions that — if wrong — force a redesign:
| Spike | Assumption (register ref) | What it measures |
|-------|---------------------------|------------------|
| **A** | Group Chat maker-checker beats a single-agent baseline by enough to justify its multiplicative token cost (U3 / G7) | convergence rounds, stall frequency, token use — maker-checker vs single-agent, with a cheaper/better verdict |
| **B** | The known MAF footguns behave as predicted and our guards hold: Magentic unbounded termination when `limits=None` (G1/B4); shared-builder / fan-out state corruption (G2/B7) | guard fires on unbounded Magentic; zero state-bleed with the fresh-instance helper |
| **B** | The known MAF footguns behave as predicted and our guards hold: Magentic unbounded termination when `limits=None` (G1/B4); shared-builder / fan-out state corruption (G2/B7) | guard fires on unbounded Magentic; a reused `Workflow` accumulates conversation across runs (project N contaminates N+1) while the fresh-instance helper gives each run a clean thread — measured by received-message content, not a call counter |
| **C** | A blocking deterministic hybrid-validator (B1) can *structurally* block an out-of-range proposal | structural rejection of an out-of-range proposal; P10/P50/P90 for a valid one; capped self-repair |
| **D** | ExpeL retrieval (B2) surfaces a relevant prior verdict for a similar new proposal | top-K retrieval returns the structurally-similar verdict over surface-text decoys |
@ -50,7 +50,7 @@ The gate stays green from the logic layer alone.
| Spike | Assumption | Result | Verdict | Token use | Implication for Fase 2 |
|-------|-----------|--------|---------|-----------|------------------------|
| **A** | maker-checker > single-agent (U3/G7) | verdict logic green; `GroupChatBuilder` drivable; cheaper/better is endpoint-dependent | **CONFIRMED (logic)** — empirical arm gated | word-count proxy; live arm not run (no endpoint) | Keep the codified `verdict` (better ∧ ≤3× tokens); measure the empirical cost/benefit on a LOCAL endpoint before locking the debate default |
| **B** | Magentic unbounded + fan-out bleed (G1/G2) | unbounded `max_round_count=None` needs an external guard; shared fan-out instance bleeds, fresh does not | **CONFIRMED** | 0 — no live LLM | Require explicit round/stop caps for any Magentic loop (fail-fast); use a fresh-instance-per-run factory for fan-out (B7) |
| **B** | Magentic unbounded + fan-out bleed (G1/G2) | unbounded `max_round_count=None` needs an external guard; a reused `Workflow` accumulates the conversation thread across runs (project N's prompts/replies leak into N+1), a fresh instance per run does not — measured by received-message content | **CONFIRMED** | 0 — no live LLM | Require explicit round/stop caps for any Magentic loop (fail-fast); use a fresh-instance-per-run factory for fan-out (B7) |
| **C** | blocking hybrid-validator (B1) | typed IR + real CBC solve + Monte-Carlo P10/P50/P90; out-of-range → `Rejection` (distinct type, no percentiles) | **CONFIRMED** | 0 — deterministic; live gen gated | Keep the `Rejection`/`ValidatedProposal` type split (structural block) + CBC-absent escalate; migrate to `pulp[cbc]`/`COIN_CMD` for PuLP 4.0 |
| **D** | ExpeL retrieval (B2) | structural similarity (codes + measure + magnitude) returns the true match as top-1 over surface-text decoys; deterministic | **CONFIRMED** | 0 — deterministic retrieval | Keep structured similarity as the baseline; add embeddings only if it proves insufficient on real data |