44 lines
2.1 KiB
Markdown
44 lines
2.1 KiB
Markdown
# Spike A findings — Group Chat maker-checker vs single-agent (U3 / G7)
|
||
|
||
**Assumption (U3 / G7):** a Group Chat maker-checker debate (proposer · critic ·
|
||
validator) beats a single-agent baseline by enough to justify its *multiplicative*
|
||
token cost.
|
||
|
||
## Verdict logic (proven by the quality gate — always green)
|
||
|
||
The falsifiable decision lives in `spikes/a_groupchat.py::verdict(mc, single)`:
|
||
|
||
- **better** = maker-checker caught the planted flaw AND the single agent did not.
|
||
- **affordable** = maker-checker tokens ≤ **3×** the single-agent tokens (G7 discipline).
|
||
- **passed** = better AND affordable.
|
||
|
||
`tests/spikes/test_a_groupchat.py` exercises this with *varied* inputs (non-tautological):
|
||
passes when better & within 3×; fails when better but 5× (unaffordable); fails when both
|
||
caught the flaw (not better). `make_termination(3)` stops at 3 rounds. **Result: logic
|
||
layer CONFIRMED green.**
|
||
|
||
## Builder de-risk (from Step 2)
|
||
|
||
The Step 2 builder smoke confirmed `FakeChatClient` can drive the GA `GroupChatBuilder`
|
||
(`selection_func` + `with_max_rounds`) — so the maker-checker arm is buildable. The round
|
||
cap terminates reliably ("reached max_rounds=N; forcing completion").
|
||
|
||
## Token use
|
||
|
||
The **empirical** better/cheaper numbers (convergence rounds, stall frequency, token use
|
||
for BOTH arms) are produced by the gated `run_live` arm, which counts tokens as a
|
||
word-count proxy over assistant outputs. They are **endpoint-dependent**: only measured
|
||
when a LOCAL OpenAI-compatible endpoint is configured (`PORTFOLIO_LOCAL_BASE_URL` +
|
||
`PORTFOLIO_LOCAL_MODEL`).
|
||
|
||
- **Live arm token use this session:** _not run — no LOCAL endpoint configured._ The
|
||
logic layer (verdict function) is what the quality gate proves; the cheaper/better
|
||
verdict is honestly reported as endpoint-dependent and will be filled in by `run_live`
|
||
when an endpoint is available.
|
||
|
||
## Implication for Fase 2
|
||
|
||
The maker-checker machinery is buildable and its cheaper/better decision is codified and
|
||
tested. Whether the debate is *worth its cost* on real models is the one open empirical
|
||
question — to be measured with a LOCAL endpoint before committing the debate default in
|
||
the Fase 2 vertical slice.
|