feat(fase1): spike A - group chat maker-checker vs single-agent [skip-docs]

This commit is contained in:
Kjell Tore Guttormsen 2026-06-24 10:13:23 +02:00
commit 9b9a17e2ed
3 changed files with 305 additions and 0 deletions

View file

@ -0,0 +1,44 @@
# Spike A findings — Group Chat maker-checker vs single-agent (U3 / G7)
**Assumption (U3 / G7):** a Group Chat maker-checker debate (proposer · critic ·
validator) beats a single-agent baseline by enough to justify its *multiplicative*
token cost.
## Verdict logic (proven by the quality gate — always green)
The falsifiable decision lives in `spikes/a_groupchat.py::verdict(mc, single)`:
- **better** = maker-checker caught the planted flaw AND the single agent did not.
- **affordable** = maker-checker tokens ≤ **3×** the single-agent tokens (G7 discipline).
- **passed** = better AND affordable.
`tests/spikes/test_a_groupchat.py` exercises this with *varied* inputs (non-tautological):
passes when better & within 3×; fails when better but 5× (unaffordable); fails when both
caught the flaw (not better). `make_termination(3)` stops at 3 rounds. **Result: logic
layer CONFIRMED green.**
## Builder de-risk (from Step 2)
The Step 2 builder smoke confirmed `FakeChatClient` can drive the GA `GroupChatBuilder`
(`selection_func` + `with_max_rounds`) — so the maker-checker arm is buildable. The round
cap terminates reliably ("reached max_rounds=N; forcing completion").
## Token use
The **empirical** better/cheaper numbers (convergence rounds, stall frequency, token use
for BOTH arms) are produced by the gated `run_live` arm, which counts tokens as a
word-count proxy over assistant outputs. They are **endpoint-dependent**: only measured
when a LOCAL OpenAI-compatible endpoint is configured (`PORTFOLIO_LOCAL_BASE_URL` +
`PORTFOLIO_LOCAL_MODEL`).
- **Live arm token use this session:** _not run — no LOCAL endpoint configured._ The
logic layer (verdict function) is what the quality gate proves; the cheaper/better
verdict is honestly reported as endpoint-dependent and will be filled in by `run_live`
when an endpoint is available.
## Implication for Fase 2
The maker-checker machinery is buildable and its cheaper/better decision is codified and
tested. Whether the debate is *worth its cost* on real models is the one open empirical
question — to be measured with a LOCAL endpoint before committing the debate default in
the Fase 2 vertical slice.