feat(fase1): spike A - group chat maker-checker vs single-agent [skip-docs]
This commit is contained in:
parent
432476346c
commit
9b9a17e2ed
3 changed files with 305 additions and 0 deletions
44
docs/fase1-spikes/findings-a.md
Normal file
44
docs/fase1-spikes/findings-a.md
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
# Spike A findings — Group Chat maker-checker vs single-agent (U3 / G7)
|
||||
|
||||
**Assumption (U3 / G7):** a Group Chat maker-checker debate (proposer · critic ·
|
||||
validator) beats a single-agent baseline by enough to justify its *multiplicative*
|
||||
token cost.
|
||||
|
||||
## Verdict logic (proven by the quality gate — always green)
|
||||
|
||||
The falsifiable decision lives in `spikes/a_groupchat.py::verdict(mc, single)`:
|
||||
|
||||
- **better** = maker-checker caught the planted flaw AND the single agent did not.
|
||||
- **affordable** = maker-checker tokens ≤ **3×** the single-agent tokens (G7 discipline).
|
||||
- **passed** = better AND affordable.
|
||||
|
||||
`tests/spikes/test_a_groupchat.py` exercises this with *varied* inputs (non-tautological):
|
||||
passes when better & within 3×; fails when better but 5× (unaffordable); fails when both
|
||||
caught the flaw (not better). `make_termination(3)` stops at 3 rounds. **Result: logic
|
||||
layer CONFIRMED green.**
|
||||
|
||||
## Builder de-risk (from Step 2)
|
||||
|
||||
The Step 2 builder smoke confirmed `FakeChatClient` can drive the GA `GroupChatBuilder`
|
||||
(`selection_func` + `with_max_rounds`) — so the maker-checker arm is buildable. The round
|
||||
cap terminates reliably ("reached max_rounds=N; forcing completion").
|
||||
|
||||
## Token use
|
||||
|
||||
The **empirical** better/cheaper numbers (convergence rounds, stall frequency, token use
|
||||
for BOTH arms) are produced by the gated `run_live` arm, which counts tokens as a
|
||||
word-count proxy over assistant outputs. They are **endpoint-dependent**: only measured
|
||||
when a LOCAL OpenAI-compatible endpoint is configured (`PORTFOLIO_LOCAL_BASE_URL` +
|
||||
`PORTFOLIO_LOCAL_MODEL`).
|
||||
|
||||
- **Live arm token use this session:** _not run — no LOCAL endpoint configured._ The
|
||||
logic layer (verdict function) is what the quality gate proves; the cheaper/better
|
||||
verdict is honestly reported as endpoint-dependent and will be filled in by `run_live`
|
||||
when an endpoint is available.
|
||||
|
||||
## Implication for Fase 2
|
||||
|
||||
The maker-checker machinery is buildable and its cheaper/better decision is codified and
|
||||
tested. Whether the debate is *worth its cost* on real models is the one open empirical
|
||||
question — to be measured with a LOCAL endpoint before committing the debate default in
|
||||
the Fase 2 vertical slice.
|
||||
Loading…
Add table
Add a link
Reference in a new issue