Kjell Tore Guttormsen
a2dff210ce
/trekreview flagged the Spike B(b) fan-out experiment as BROKEN_SUCCESS_CRITERION (BLOCKER): it asserted a per-client call_count reached 3 on a reused instance vs 1 on a fresh one — a tautology true for any un-reset mutable counter, independent of MAF, that never exercised the real G2/B7 shared-Workflow state-corruption footgun. It was a false-confirm of a de-risk assumption. Rebuilt to observe genuine MAF thread state via the messages each participant RECEIVES (new FakeChatClient.received_texts seam): - shared_instance_conversation_bleed: a reused built ConcurrentBuilder Workflow accumulates the conversation across .run() calls — run N's participants receive runs 0..N-1's prompts/replies (measured [[p0],[p0,p1],[p0,p1,p2]], strictly monotonic) => genuine cross-run contamination. - fresh_instance_conversation_isolation: a fresh instance per run gives each a clean thread => each participant sees only its own project ([[p0],[p1],[p2]]). Assumption now CONFIRMED with a meaningful observable. findings-b.md gains a Method note recording why it was rebuilt; README rows updated. Also fixes the MINOR: a_groupchat.run_live now mkdirs the findings dir before write_text so a post-disposal run does not lose the measured result. Gate green: ruff check + format, mypy src, pytest 48 passed / 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
81 lines
5.9 KiB
Markdown
81 lines
5.9 KiB
Markdown
# Fase 1 — De-risk spikes (A–D)
|
||
|
||
> **Throwaway** spikes. Their only job is to turn the research §15 register's most
|
||
> dangerous *documented-but-unverified* assumptions into **measured facts** before
|
||
> the Fase 2 vertical slice. Code here is expected to be discarded once the
|
||
> findings are recorded — see [Disposal](#disposal).
|
||
|
||
## Why these four
|
||
|
||
Before committing to the full architecture in **Fase 2**, we empirically de-risk
|
||
the four assumptions that — if wrong — force a redesign:
|
||
|
||
| Spike | Assumption (register ref) | What it measures |
|
||
|-------|---------------------------|------------------|
|
||
| **A** | Group Chat maker-checker beats a single-agent baseline by enough to justify its multiplicative token cost (U3 / G7) | convergence rounds, stall frequency, token use — maker-checker vs single-agent, with a cheaper/better verdict |
|
||
| **B** | The known MAF footguns behave as predicted and our guards hold: Magentic unbounded termination when `limits=None` (G1/B4); shared-builder / fan-out state corruption (G2/B7) | guard fires on unbounded Magentic; a reused `Workflow` accumulates conversation across runs (project N contaminates N+1) while the fresh-instance helper gives each run a clean thread — measured by received-message content, not a call counter |
|
||
| **C** | A blocking deterministic hybrid-validator (B1) can *structurally* block an out-of-range proposal | structural rejection of an out-of-range proposal; P10/P50/P90 for a valid one; capped self-repair |
|
||
| **D** | ExpeL retrieval (B2) surfaces a relevant prior verdict for a similar new proposal | top-K retrieval returns the structurally-similar verdict over surface-text decoys |
|
||
|
||
Each spike produces a short findings note (`findings-{a,b,c,d}.md`) with a
|
||
**confirmed/refuted** verdict and a **token-use line** (including `0 — no live LLM`
|
||
where the spike runs without an endpoint).
|
||
|
||
## Resolved version gate (premise-verified at planning time)
|
||
|
||
- Installed `agent-framework-core` = **1.9.0** (introspection / `uv pip show`).
|
||
- The orchestration builders (`GroupChatBuilder`, `ConcurrentBuilder`,
|
||
`MagenticBuilder`, `StandardMagenticManager`, `TerminationCondition`) live in the
|
||
separate package **`agent-framework-orchestrations`**, GA **1.0.0**
|
||
(`requires agent-framework-core<2,>=1.9.0` — exactly our installed core).
|
||
- Because the spikes are **throwaway**, this package + `pulp` (Spike C solver) +
|
||
`pytest-asyncio` (MAF orchestrations are async) are pinned in the **`dev`**
|
||
optional-dependency group — **not** core. Promotion to core is a Fase 2 decision.
|
||
|
||
## Gate-green contract
|
||
|
||
Each spike splits into:
|
||
- a **logic layer** — pure functions/classes we author, always exercised by the
|
||
quality gate (no live endpoint, no full MAF workflow run); and
|
||
- an **integration/live layer** — drives the real MAF builders and/or a real LOCAL
|
||
LLM; runs when available, otherwise `skip`s (except Spike B, where driving the
|
||
builders with a fake client IS the de-risk).
|
||
|
||
The headline empirical claims (Spike A's better/cheaper verdict; live token numbers)
|
||
live in the integration/live layer and are honestly reported as **endpoint-dependent**.
|
||
The gate stays green from the logic layer alone.
|
||
|
||
## Status — all four de-risked (gate green)
|
||
|
||
| Spike | Assumption | Result | Verdict | Token use | Implication for Fase 2 |
|
||
|-------|-----------|--------|---------|-----------|------------------------|
|
||
| **A** | maker-checker > single-agent (U3/G7) | verdict logic green; `GroupChatBuilder` drivable; cheaper/better is endpoint-dependent | **CONFIRMED (logic)** — empirical arm gated | word-count proxy; live arm not run (no endpoint) | Keep the codified `verdict` (better ∧ ≤3× tokens); measure the empirical cost/benefit on a LOCAL endpoint before locking the debate default |
|
||
| **B** | Magentic unbounded + fan-out bleed (G1/G2) | unbounded `max_round_count=None` needs an external guard; a reused `Workflow` accumulates the conversation thread across runs (project N's prompts/replies leak into N+1), a fresh instance per run does not — measured by received-message content | **CONFIRMED** | 0 — no live LLM | Require explicit round/stop caps for any Magentic loop (fail-fast); use a fresh-instance-per-run factory for fan-out (B7) |
|
||
| **C** | blocking hybrid-validator (B1) | typed IR + real CBC solve + Monte-Carlo P10/P50/P90; out-of-range → `Rejection` (distinct type, no percentiles) | **CONFIRMED** | 0 — deterministic; live gen gated | Keep the `Rejection`/`ValidatedProposal` type split (structural block) + CBC-absent escalate; migrate to `pulp[cbc]`/`COIN_CMD` for PuLP 4.0 |
|
||
| **D** | ExpeL retrieval (B2) | structural similarity (codes + measure + magnitude) returns the true match as top-1 over surface-text decoys; deterministic | **CONFIRMED** | 0 — deterministic retrieval | Keep structured similarity as the baseline; add embeddings only if it proves insufficient on real data |
|
||
|
||
**Builder de-risk (Step 2):** `FakeChatClient` drives the GA `GroupChatBuilder`,
|
||
`ConcurrentBuilder`, and `MagenticBuilder` — confirmed by the builder smoke + probes, so
|
||
Spikes A/B never needed the escalate fallback. Per-spike detail: `findings-{a,b,c,d}.md`.
|
||
|
||
**Quality gate at Fase 1 close:** `ruff check .` ✓, `ruff format --check .` ✓,
|
||
`mypy src` ✓, `pytest` ✓ (48 passed, 1 skipped = the gated live arm). `src/` untouched.
|
||
|
||
## Disposal
|
||
|
||
These spikes are throwaway. To remove them completely after the findings are recorded:
|
||
|
||
```bash
|
||
rm -rf spikes tests/spikes docs/fase1-spikes
|
||
```
|
||
|
||
Then revert the three `pyproject.toml` edits that supported them:
|
||
|
||
1. Remove the dev deps `agent-framework-orchestrations`, `pulp`, `pytest-asyncio`
|
||
from `[project.optional-dependencies].dev`.
|
||
2. Restore `[tool.ruff] src = ["src", "tests"]` (drop `"spikes"`).
|
||
3. Restore `[tool.pytest.ini_options] pythonpath = ["src"]` and remove
|
||
`asyncio_mode = "auto"`.
|
||
|
||
Then `uv sync --extra dev` to regenerate `uv.lock`. Nothing in `src/` is touched by
|
||
the spikes, so disposal leaves the framework core exactly as Fase 0 left it.
|