fix(fase1): spike B fan-out measures real conversation bleed, not a counter
/trekreview flagged the Spike B(b) fan-out experiment as BROKEN_SUCCESS_CRITERION (BLOCKER): it asserted a per-client call_count reached 3 on a reused instance vs 1 on a fresh one — a tautology true for any un-reset mutable counter, independent of MAF, that never exercised the real G2/B7 shared-Workflow state-corruption footgun. It was a false-confirm of a de-risk assumption. Rebuilt to observe genuine MAF thread state via the messages each participant RECEIVES (new FakeChatClient.received_texts seam): - shared_instance_conversation_bleed: a reused built ConcurrentBuilder Workflow accumulates the conversation across .run() calls — run N's participants receive runs 0..N-1's prompts/replies (measured [[p0],[p0,p1],[p0,p1,p2]], strictly monotonic) => genuine cross-run contamination. - fresh_instance_conversation_isolation: a fresh instance per run gives each a clean thread => each participant sees only its own project ([[p0],[p1],[p2]]). Assumption now CONFIRMED with a meaningful observable. findings-b.md gains a Method note recording why it was rebuilt; README rows updated. Also fixes the MINOR: a_groupchat.run_live now mkdirs the findings dir before write_text so a post-disposal run does not lose the measured result. Gate green: ruff check + format, mypy src, pytest 48 passed / 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
This commit is contained in:
parent
b81e22b637
commit
a2dff210ce
6 changed files with 118 additions and 35 deletions
|
|
@ -13,7 +13,7 @@ the four assumptions that — if wrong — force a redesign:
|
|||
| Spike | Assumption (register ref) | What it measures |
|
||||
|-------|---------------------------|------------------|
|
||||
| **A** | Group Chat maker-checker beats a single-agent baseline by enough to justify its multiplicative token cost (U3 / G7) | convergence rounds, stall frequency, token use — maker-checker vs single-agent, with a cheaper/better verdict |
|
||||
| **B** | The known MAF footguns behave as predicted and our guards hold: Magentic unbounded termination when `limits=None` (G1/B4); shared-builder / fan-out state corruption (G2/B7) | guard fires on unbounded Magentic; zero state-bleed with the fresh-instance helper |
|
||||
| **B** | The known MAF footguns behave as predicted and our guards hold: Magentic unbounded termination when `limits=None` (G1/B4); shared-builder / fan-out state corruption (G2/B7) | guard fires on unbounded Magentic; a reused `Workflow` accumulates conversation across runs (project N contaminates N+1) while the fresh-instance helper gives each run a clean thread — measured by received-message content, not a call counter |
|
||||
| **C** | A blocking deterministic hybrid-validator (B1) can *structurally* block an out-of-range proposal | structural rejection of an out-of-range proposal; P10/P50/P90 for a valid one; capped self-repair |
|
||||
| **D** | ExpeL retrieval (B2) surfaces a relevant prior verdict for a similar new proposal | top-K retrieval returns the structurally-similar verdict over surface-text decoys |
|
||||
|
||||
|
|
@ -50,7 +50,7 @@ The gate stays green from the logic layer alone.
|
|||
| Spike | Assumption | Result | Verdict | Token use | Implication for Fase 2 |
|
||||
|-------|-----------|--------|---------|-----------|------------------------|
|
||||
| **A** | maker-checker > single-agent (U3/G7) | verdict logic green; `GroupChatBuilder` drivable; cheaper/better is endpoint-dependent | **CONFIRMED (logic)** — empirical arm gated | word-count proxy; live arm not run (no endpoint) | Keep the codified `verdict` (better ∧ ≤3× tokens); measure the empirical cost/benefit on a LOCAL endpoint before locking the debate default |
|
||||
| **B** | Magentic unbounded + fan-out bleed (G1/G2) | unbounded `max_round_count=None` needs an external guard; shared fan-out instance bleeds, fresh does not | **CONFIRMED** | 0 — no live LLM | Require explicit round/stop caps for any Magentic loop (fail-fast); use a fresh-instance-per-run factory for fan-out (B7) |
|
||||
| **B** | Magentic unbounded + fan-out bleed (G1/G2) | unbounded `max_round_count=None` needs an external guard; a reused `Workflow` accumulates the conversation thread across runs (project N's prompts/replies leak into N+1), a fresh instance per run does not — measured by received-message content | **CONFIRMED** | 0 — no live LLM | Require explicit round/stop caps for any Magentic loop (fail-fast); use a fresh-instance-per-run factory for fan-out (B7) |
|
||||
| **C** | blocking hybrid-validator (B1) | typed IR + real CBC solve + Monte-Carlo P10/P50/P90; out-of-range → `Rejection` (distinct type, no percentiles) | **CONFIRMED** | 0 — deterministic; live gen gated | Keep the `Rejection`/`ValidatedProposal` type split (structural block) + CBC-absent escalate; migrate to `pulp[cbc]`/`COIN_CMD` for PuLP 4.0 |
|
||||
| **D** | ExpeL retrieval (B2) | structural similarity (codes + measure + magnitude) returns the true match as top-1 over surface-text decoys; deterministic | **CONFIRMED** | 0 — deterministic retrieval | Keep structured similarity as the baseline; add embeddings only if it proves insufficient on real data |
|
||||
|
||||
|
|
|
|||
|
|
@ -29,16 +29,41 @@ in-flight fake response surfaces a `RuntimeWarning: coroutine was never awaited`
|
|||
cosmetic (no functional effect); the spike silences the OTel context logger and filters the
|
||||
warning. Worth knowing if Fase 2 observes streamed orchestrations.
|
||||
|
||||
## (b) Fan-out state isolation — CONFIRMED
|
||||
## (b) Fan-out state isolation — CONFIRMED (conversation-history bleed)
|
||||
|
||||
- **Shared instance:** one `ConcurrentBuilder` workflow reused across the three reference
|
||||
projects → the shared fake clients accumulate calls (max call count == 3) → **state
|
||||
bleed**.
|
||||
- **Fresh instance:** `fresh_workflow()` builds a new workflow + fresh clients per run →
|
||||
each run's clients see exactly one call (`[1, 1, 1]`) → **zero bleed** across all 3
|
||||
projects (B7).
|
||||
The observable is the **message history each participant receives** (captured via
|
||||
`FakeChatClient.received_texts`), NOT a call counter. A counter rises for any reused
|
||||
mutable object and proves nothing about MAF; message content proves the workflow carried
|
||||
state across runs. (An earlier version of this spike measured only a call counter and was
|
||||
a tautology — caught and rebuilt; see the Method note below.)
|
||||
|
||||
- **Shared instance:** one built `ConcurrentBuilder` workflow reused across the three
|
||||
reference projects. MAF **accumulates the shared conversation thread across `.run()`
|
||||
calls**: each run's participants receive the PRIOR projects' prompts and replies too.
|
||||
Measured (project ids visible per run, in order):
|
||||
`[[FV42-GSV-E1], [FV42-GSV-E1, RV13-RAS-TP], [FV42-GSV-E1, RV13-RAS-TP, BRU-LAKS-REHAB]]`
|
||||
— strictly monotonic growth → run N is contaminated by runs 0..N-1 → **genuine state
|
||||
bleed (G2/B7)**.
|
||||
- **Fresh instance:** `fresh_workflow()` builds a new workflow + clean thread per run →
|
||||
each participant sees ONLY its own project: `[[FV42-GSV-E1], [RV13-RAS-TP],
|
||||
[BRU-LAKS-REHAB]]` → **zero cross-run contamination** (B7 mitigation works).
|
||||
- **Implication:** Fase 2 fan-out must build a fresh workflow/executor instance per
|
||||
project run (a factory like `fresh_workflow()`), never reuse a shared instance.
|
||||
project run (a factory like `fresh_workflow()`), never reuse a shared instance — a
|
||||
reused `Workflow` leaks one project's context into the next, which would corrupt
|
||||
per-project cost analyses. This is a property of MAF's `Workflow` thread state, not of
|
||||
the participants.
|
||||
|
||||
### Method note (why this was rebuilt)
|
||||
|
||||
The first cut of this experiment asserted that a reused instance's per-client `call_count`
|
||||
reached 3 while a fresh instance's stayed at 1. That is a tautology: any un-reset mutable
|
||||
counter rises across reuse, independent of MAF, so it never demonstrated the footgun.
|
||||
`/trekreview` flagged it as a `BROKEN_SUCCESS_CRITERION` (false-confirm of a de-risk
|
||||
assumption). The rebuilt experiment observes real MAF thread state via the messages each
|
||||
participant receives, so a passing test now genuinely means "the reused workflow carried
|
||||
project N's conversation into project N+1." Lesson carried into Fase 2: **a de-risk
|
||||
assertion must observe the mechanism it claims to test, not a proxy that moves for unrelated
|
||||
reasons.**
|
||||
|
||||
## Token use
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue