2.1 KiB
Spike A findings — Group Chat maker-checker vs single-agent (U3 / G7)
Assumption (U3 / G7): a Group Chat maker-checker debate (proposer · critic · validator) beats a single-agent baseline by enough to justify its multiplicative token cost.
Verdict logic (proven by the quality gate — always green)
The falsifiable decision lives in spikes/a_groupchat.py::verdict(mc, single):
- better = maker-checker caught the planted flaw AND the single agent did not.
- affordable = maker-checker tokens ≤ 3× the single-agent tokens (G7 discipline).
- passed = better AND affordable.
tests/spikes/test_a_groupchat.py exercises this with varied inputs (non-tautological):
passes when better & within 3×; fails when better but 5× (unaffordable); fails when both
caught the flaw (not better). make_termination(3) stops at 3 rounds. Result: logic
layer CONFIRMED green.
Builder de-risk (from Step 2)
The Step 2 builder smoke confirmed FakeChatClient can drive the GA GroupChatBuilder
(selection_func + with_max_rounds) — so the maker-checker arm is buildable. The round
cap terminates reliably ("reached max_rounds=N; forcing completion").
Token use
The empirical better/cheaper numbers (convergence rounds, stall frequency, token use
for BOTH arms) are produced by the gated run_live arm, which counts tokens as a
word-count proxy over assistant outputs. They are endpoint-dependent: only measured
when a LOCAL OpenAI-compatible endpoint is configured (PORTFOLIO_LOCAL_BASE_URL +
PORTFOLIO_LOCAL_MODEL).
- Live arm token use this session: not run — no LOCAL endpoint configured. The
logic layer (verdict function) is what the quality gate proves; the cheaper/better
verdict is honestly reported as endpoint-dependent and will be filled in by
run_livewhen an endpoint is available.
Implication for Fase 2
The maker-checker machinery is buildable and its cheaper/better decision is codified and tested. Whether the debate is worth its cost on real models is the one open empirical question — to be measured with a LOCAL endpoint before committing the debate default in the Fase 2 vertical slice.