open/portfolio-optimiser
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity Actions

docs(fase1): Voyage brief + plan for de-risk spikes (A-D)

Browse source 
Planning-only artifacts (no code yet). Plan A- after adversarial review
(critic REVISE -> revised; scope MIXED -> addressed; 19 findings, 0 overlap).
Ground truth: agent-framework-orchestrations is a separate GA 1.0.0 pkg
(-> dev dep); core is 1.9.0; MAF orchestrations are async. Next: /trekexecute.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9FyyENxebxVThjrn9et8C
This commit is contained in:
Kjell Tore Guttormsen 2026-06-24 01:09:35 +02:00
commit 8b1ff745b9
2 changed files with 595 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

122
.claude/projects/2026-06-23-fase1-derisk-spikes/brief.md Normal file
View file

@ -0,0 +1,122 @@
---
type: trekbrief
brief_version: "2.2"
created: 2026-06-23
task: "Fase 1 de-risk spikes (A–D): empirically validate the framework's most dangerous assumptions before the MVP vertical slice"
slug: fase1-derisk-spikes
project_dir: .claude/projects/2026-06-23-fase1-derisk-spikes/
research_topics: 0
research_status: skipped
auto_research: false
interview_turns: 3
source: interview
framing: refine
phase_signals:
- phase: research
effort: standard
- phase: plan
effort: standard
- phase: execute
effort: standard
- phase: review
effort: standard
---
# Task: Fase 1 — De-risk spikes (A–D)
> Generated by `/trekbrief` on 2026-06-23.
> This brief is the contract between requirements and planning. `/trekplan`
> reads it to produce the implementation plan. Every decision in the plan must
> trace back to content in this brief.
## TL;DR
**Refine** of the locked plan ([§Fase 1](docs/plan/2026-06-23-incremental-plan.md)) — same intent, narrowed to executable spike scope. Build four **throwaway** de-risk spikes (A–D) that convert the framework's most dangerous *documented-but-unverified* assumptions into *measured facts* before the Fase 2 vertical slice. Each spike has a falsifiable pass/fail from the plan's verification block. No new external research — reuse the research report's §15 implementation register. Runtime agent calls default to the **LOCAL profile** with hard token/round caps per D6.
## Intent
Fase 0 delivered the skeleton, locked decisions (D1–D6), and a synthetic reference domain (D4). Before committing to the full architecture in Fase 2 (vertical slice), we must empirically de-risk the four assumptions that — if wrong — would force a redesign. Specifically: (1) that a Group Chat maker-checker debate beats a single-agent baseline by enough to justify its multiplicative token cost (U3 / G7); (2) that the known MAF footguns behave as the research predicts and our guards hold — Magentic's unbounded termination when `limits=None` (G1/B4) and shared-`WorkflowBuilder` state corruption in fan-out (G2/B7); (3) that a blocking deterministic hybrid-validator (B1) can *structurally* block an out-of-range proposal from ever reaching the expert; and (4) that ExpeL retrieval (B2) actually surfaces a relevant prior verdict for a similar new proposal. These are throwaway spikes — code we expect to discard — whose only job is to turn §15 register assumptions into evidence. Getting each wrong *cheaply now* is vastly cheaper than discovering it mid-Fase-2.
## Goal
Four runnable spike modules (isolated so they are trivial to discard), each producing a **measurable pass/fail** against the plan §Fase 1 verification criteria, each running on the LOCAL profile by default with hard token and round/iteration caps, and each emitting a short findings note with the measured numbers (convergence rounds, stall frequency, token use, state-bleed observations, validator block/pass behavior, retrieval hit/miss). At the end, the framework's four most dangerous assumptions are each **confirmed or refuted with evidence**, and the findings are written down so they directly inform the Fase 2 design. The repo's quality gate (ruff + mypy(src) + pytest) stays green.
## Non-Goals
- Production code for Fase 2 (the vertical slice) — these spikes are throwaway and may be deleted after their findings are recorded.
- Compliance functions (D3 — the deployer owns DPIA/ROS/behandlingsformål; we build only technical preconditions).
- Chasing the last 10% (D5) — spikes prove the generic core, not edge cases or polish.
- Heavy Foundry/Azure runs (D6) — LOCAL profile is default; Foundry is used only for targeted, minimal verification if at all in Fase 1.
- A production sandbox for user-supplied skill scripts (B8 / G3) — out of Fase 1 scope.
- A full production VerdictStore — Spike D is a minimal 10–20 synthetic-verdict retrieval test only, not the durable store design.
- Resolving the second-rank open questions (plan §Risiko: review-latency, project topology, schema ownership, Foundry-memory Preview) — these do not block Fase 1.
## Constraints
- **D6 cost-discipline:** LOCAL profile default (OpenAI-compatible endpoint); cheapest models; tiny synthetic data; hard token + round caps; no heavy test runs. Foundry/Azure only for targeted, minimal verification.
- **Deterministic validator is obligatory and blocking** (never an optional plugin) — Spike C must demonstrate *structural* blocking, not advisory warning.
- **Stop-criteria + budget caps required at startup** — fail-fast if missing; never an unbounded loop.
- **Group Chat maker-checker is the debate default**, NOT Magentic (which is experimental, G8/A2).
- **Stack:** Python ≥3.10, MAF (`agent-framework`), `uv`, `ruff`, `mypy`, `pytest`. Type hints throughout; Pydantic for IR/validation.
- Spikes reuse the existing synthetic domain (D4: `data/reference_projects.json`, `reference_domain.py`) and the backend-profile seam (D2: `backends.py`).
## Preferences
- **90%-principle (D5):** generic core + clear extension points; do not over-fit.
- Per-agent model selection via chat-client (`FoundryChatClient` / `OpenAIChatClient`, common base `BaseChatClient`).
- Keep spikes physically isolated (e.g. a `spikes/` package or `tests/spikes/`) so discarding them leaves the core untouched.
- Findings recorded as short markdown notes inside the project dir (`.claude/projects/2026-06-23-fase1-derisk-spikes/`), not scattered.
- Prefer reusing `mcp-solver` / OR-Tools / PuLP / Z3 (R1/R2) for Spike C's solver step rather than hand-rolling.
## Non-Functional Requirements
- Each spike enforces a **hard token-budget cap** and a **round/iteration cap**; exceeding either yields a structured stop event, never a silent hang.
- Each spike **reports measured token usage** (per the plan's "every spike measures and reports token consumption").
- Runtime defaults to the LOCAL profile; **no silent egress** to Azure/Foundry without explicit profile selection.
- Spikes must not require real/sensitive data — synthetic only.
## Success Criteria
*Falsifiable, mapped 1:1 to the plan §Fase 1 verification block.*
- **Spike A (U3 / G7):** Maker-checker (proposer · critic · validator) converges in ≤ N rounds (N fixed at spike-design time) AND the hard cap is respected; a findings note documents convergence rate, stall frequency, and token use for BOTH maker-checker and single-agent baseline, with an explicit cheaper/better verdict. Verify: the spike command/test exits 0 and prints the comparison table.
- **Spike B (G1 / G2):** Unbounded Magentic (`limits=None`) does NOT self-terminate → confirms an explicit limit is required (B4); `ConcurrentBuilder` fan-out from a shared builder shows **zero state-bleed** when the fresh-instance helper (B7) is used. Verify: the spike asserts both observations and exits 0.
- **Spike C (B1):** An out-of-range proposal is **structurally blocked** (never reaches the expert/output); a valid proposal passes and yields P10/P50/P90 from the Monte Carlo step; self-repair retries are capped at N. Verify: a test asserts the blocked case raises/returns a structured rejection and the valid case returns the percentiles.
- **Spike D (B2):** Top-K retrieval fetches the relevant historical verdict for a similar new proposal from a store of 10–20 synthetic verdicts. Verify: a test asserts the expected verdict id is in the top-K for a crafted similar proposal.
- **Quality gate:** `uv run ruff check .` exits 0, `uv run ruff format --check .` clean, `uv run mypy src` exits 0, `uv run pytest` exits 0.
## Research Plan
No external research needed — the codebase and the existing research report ([docs/research/2026-06-23-prior-art-platform.md](docs/research/2026-06-23-prior-art-platform.md) §13 architecture, §15 implementation register, §15.3 footguns) plus this brief contain sufficient context for planning. The spikes themselves ARE the empirical de-risking. Any MAF API specifics (e.g. the exact current-version Python Group Chat termination API, U3) are confirmed inline via the `microsoft-learn` MCP at coding time rather than as a separate research pass.
## Open Questions / Assumptions
- **[ASSUMPTION]** The LOCAL profile resolves to an OpenAI-compatible endpoint reachable during spikes (per D6 and the `local` profile in `backends.py`). If no local endpoint is available at run time, Spike A/C/D live agent calls fall back to a minimal, capped Foundry run.
- **[ASSUMPTION — verify before Spike A]** The installed `agent-framework` version's Python Group Chat termination API matches research §15 (U3: `termination_condition` lambda). Note a version ambiguity to resolve first: project `CLAUDE.md` states `agent-framework` 1.8.0 while `STATE.md` references `agent-framework-core` 1.9.0 — confirm the actual installed version (`uv pip show agent-framework-core`) and the current termination API via the `microsoft-learn` MCP before coding Spike A.
- **[ASSUMPTION]** Spike A's convergence target "N rounds" will be fixed at spike-design time (candidate ≤ 3–5 rounds); the plan deliberately leaves N open.
- Magentic is experimental (G8); Spike B exercises it ONLY to confirm the footgun (G1), never to build core flow on it (A2).
## Prior Attempts
None for the spikes themselves — fresh. Substrate from Fase 0 is complete and committed: repo scaffold, locked decisions D1–D6, GA-slimmed dependencies (`uv.lock`), the D4 synthetic "anleggskostnad" domain (`reference_domain.py` + `data/reference_projects.json`, 3 fictional projects), and the D2 backend-profile skeleton (`backends.py`: `Profile` azure|local + `ChatBackend` protocol + stubs raising `NotImplementedError` until Fase 1). Quality gate was green at Fase 0 close (ruff + mypy(src) + 12 pytest passed).
## Metadata
- **Created:** 2026-06-23
- **Interview turns:** 3
- **Auto-research opted in:** no
- **Source:** trekbrief interview
---
## How to continue
Manual (default):
```bash
# No research topics — go straight to planning:
/trekplan --project .claude/projects/2026-06-23-fase1-derisk-spikes
# Then execute:
/trekexecute --project .claude/projects/2026-06-23-fase1-derisk-spikes
```

473
.claude/projects/2026-06-23-fase1-derisk-spikes/plan.md Normal file
View file

@ -0,0 +1,473 @@
# Fase 1 — De-risk spikes (A–D)
> **Plan quality: A-** (post adversarial review + revision) — APPROVE_WITH_NOTES
>
> Generated by trekplan v2.0 on 2026-06-23 — `plan_version: 1.7`
## Context
Fase 0 delivered the skeleton, locked decisions (D1–D6), and a synthetic reference domain (D4). Before committing to the full architecture in Fase 2, we empirically de-risk the four assumptions that — if wrong — force a redesign (brief Intent): (A) Group Chat maker-checker beats a single-agent baseline by enough to justify its multiplicative token cost (U3 / G7); (B) the known MAF footguns behave as predicted and our guards hold — Magentic unbounded termination when `limits=None` (G1/B4) and shared-builder/fan-out state corruption (G2/B7); (C) a blocking deterministic hybrid-validator (B1) can *structurally* block an out-of-range proposal; (D) ExpeL retrieval (B2) surfaces a relevant prior verdict. These are **throwaway** spikes whose only job is to turn §15-register assumptions into measured facts, cheaply, under cost-discipline (D6: LOCAL profile default, hard token/round caps).
**Decisive planning finding (premise-verification):** the GA-slimmed Fase 0 install deliberately dropped the `agent-framework[all]` meta, so the orchestration builders (`GroupChatBuilder`, `ConcurrentBuilder`, `MagenticBuilder`) are **not importable** — they live in the separate package `agent-framework-orchestrations`, which has a clean **GA release `1.0.0`** depending on `agent-framework-core<2,>=1.9.0` (exactly our installed core). Because the spikes are **throwaway**, this package is added to the **`dev`** dependency group (not core) — exactly like the PuLP solver — deferring any core-runtime commitment to Fase 2 (revised after scope review). Also resolved: the installed `agent-framework-core` is **1.9.0** (STATE.md correct; project `CLAUDE.md` says 1.8.0 and is corrected in Step 1; the `project-core` user-memory also says 1.8.0 — that lives outside the repo and is a session-end follow-up).
**Gate-green contract (the central design choice).** Each spike is split into a **logic layer** (pure functions/classes we author — always exercised by the quality gate, no live endpoint and no full MAF workflow run) and an **integration/live layer** (drives the actual MAF builders and/or a real LOCAL LLM — runs when available, otherwise `skip`s, except Spike B where driving the builders IS the de-risk; see Step 2/Step 4). The headline empirical claims (Spike A's better/cheaper verdict; live token numbers) live in the integration/live layer and are honestly reported as endpoint-dependent. The gate stays green from the logic layer alone.
## Architecture Diagram
```mermaid
graph TD
subgraph "Reused (Fase 0, unchanged — src/ never touched)"
RD["reference_domain.py<br/>(D4 synthetic projects)"]
BC["agent_framework.BaseChatClient<br/>(subclassed by FakeChatClient)"]
end
subgraph "New — throwaway spikes/ package (dev-only)"
H["_harness.py<br/>Budget+RoundCap (B4)<br/>FakeChatClient + minimal Agent<br/>live_local_client_or_skip() -> OpenAIChatClient"]
SM["builder smoke (Step 2)<br/>proves FakeChatClient drives the GA builders"]
A["a_groupchat.py — Spike A (U3/G7)"]
B["b_footguns.py — Spike B (G1, G2/B7)"]
C["c_validator.py — Spike C (B1)"]
D["d_verdictstore.py — Spike D (B2)"]
end
subgraph "Deliverables (tracked)"
F["docs/fase1-spikes/<br/>README + findings-{a,b,c,d}.md"]
end
H --> SM --> A & B & C & D
RD --> A & C & D
BC --> H
A --> F
B --> F
C --> F
D --> F
```
## Codebase Analysis
- **Tech stack:** Python ≥3.10 (venv on 3.12), MAF GA (`agent-framework-core` 1.9.0, `-foundry` 1.8.2, `-openai` 1.8.2), `pydantic` 2.13.4 (stable), `uv`, `ruff` 0.15.18, `mypy` 2.1.0, `pytest` 9.1.1. Build: hatchling (wheel packages `src/portfolio_optimiser` only — top-level `spikes/` is never shipped).
- **Key patterns:** small typed package; `from __future__ import annotations`; frozen dataclasses for the domain; `Protocol` + `runtime_checkable` for the D2 seam; fail-fast (`ValueError`/`NotImplementedError`); tests are plain pytest functions with `-> None`, `parametrize`, `pytest.raises`, module-scope fixtures, importing from `portfolio_optimiser`.
- **Relevant files:** `src/portfolio_optimiser/backends.py` (D2 seam — `Profile`, `ChatBackend`, `get_backend`, `LocalBackend`; live-wiring of the seam is a **Fase 2** concern, NOT touched here), `src/portfolio_optimiser/reference_domain.py` (D4 — `Project`, `CostItem`, `load_reference_projects`, `.total_cost`), `src/portfolio_optimiser/data/reference_projects.json` (3 synthetic projects), `tests/test_*.py` (test style), `pyproject.toml`.
- **Reusable code:** `load_reference_projects()` → spike fixtures (A/C/D); `Project.total_cost` / `CostItem.total_cost` → savings-IR constraint source (C); `agent_framework.BaseChatClient` → subclassed by the harness `FakeChatClient`; `agent_framework.openai.OpenAIChatClient` → built **directly** by the harness for the live arm (the D2 `LocalBackend` seam is intentionally left un-wired until Fase 2, so `src/` stays untouched and the throwaway spikes add no core surface).
- **External tech (verified by introspection + PyPI, not docs):** orchestration builders in `agent_framework.orchestrations` (need `agent-framework-orchestrations`); chat clients in `agent_framework.openai` (`OpenAIChatClient`); `ContextProvider`/`MemoryContextProvider` top-level. **MAF workflows/orchestrations are async** (`await workflow.run(...)`), so the spike tests use `pytest-asyncio` (added in Step 1). Exact builder method signatures are confirmed by introspection at coding time (brief-sanctioned) and de-risked once up-front by the Step 2 builder smoke.
- **Recent git activity:** `b57aa83` Fase 0 (D4+D2+tests), `491a746` deps GA-fix + lockfile. Clean tree on `main`, synced to origin.
## Research Sources
*Ground-truth API/dependency verification — introspection + PyPI metadata are authoritative here (no web docs needed).*
| Topic | Source | Key Findings | Confidence |
|-------|--------|--------------|------------|
| Orchestration builders' real import path | `import agent_framework.orchestrations` (introspection) | `GroupChatBuilder`, `ConcurrentBuilder`, `MagenticBuilder`, `StandardMagenticManager`, `SequentialBuilder`, `TerminationCondition`, `MagenticPlanReviewRequest` live there; not top-level | high |
| Orchestrations package availability/stability | PyPI JSON | GA `1.0.0` (uploaded 2026-06-18, not yanked); `requires_dist: agent-framework-core<2,>=1.9.0`; rest b/rc | high |
| Installed core version (resolves brief ambiguity) | `uv pip show agent-framework-core` | **1.9.0** (STATE correct; CLAUDE.md 1.8.0 → corrected Step 1) | high |
| MAF orchestrations are async | introspection (`WorkflowBuilder`/orchestration `.run`) | tests need `pytest-asyncio`; integration arm is async | high |
| Solver lib for Spike C | PyPI / project knowledge | `pulp` (PuLP) MIT, pure-Python, **bundles a CBC binary in its wheel** → faithful solver-in-the-loop (R2) | high |
## Implementation Plan
Each step targets one focused change and follows the repo's TDD-ish style. Per the **gate-green contract** above, the deterministic logic layer is always exercised; integration/live layers are gated.
### Step 1: Scaffold throwaway spikes package and pin dev-only orchestration + solver + async deps
- **Files:** `pyproject.toml`, `CLAUDE.md`, `spikes/__init__.py`, `tests/spikes/__init__.py`, `tests/spikes/test_imports.py`, `docs/fase1-spikes/README.md`, `uv.lock`
- **Changes:** In `pyproject.toml` add to `[project.optional-dependencies].dev` (NOT core — spikes are throwaway): `agent-framework-orchestrations>=1.0.0` (GA; resolves with core 1.9.0), `pulp>=2.8` (CBC solver), `pytest-asyncio>=0.24`. Set `[tool.pytest.ini_options] pythonpath = ["src", "."]` (repo-root added so `import spikes` resolves; `spikes/` has no name-collision with any installed dist, so it cannot shadow them) and `asyncio_mode = "auto"`. Set `[tool.ruff] src = ["src", "tests", "spikes"]`. Create `spikes/__init__.py` and `tests/spikes/__init__.py` (docstring: "Throwaway Fase 1 de-risk spikes — dev-only, never shipped in the wheel; safe to delete after findings recorded"). Create `docs/fase1-spikes/README.md` (the four spikes, the resolved version gate, a **Disposal** subsection: `rm -rf spikes tests/spikes docs/fase1-spikes` + revert the three `dev`/`tool` pyproject edits). In `CLAUDE.md` correct the stale stack line `MAF (`agent-framework` 1.8.0)` → `MAF (`agent-framework-core` 1.9.0)` (honors the brief's Spike-A version precondition). Run `uv sync --extra dev`, then re-confirm `uv pip show agent-framework-orchestrations` resolves cleanly (premise re-verification). (new files: all except `pyproject.toml`, `CLAUDE.md`)
- **Reuses:** existing `pyproject.toml` / `CLAUDE.md` structure (Fase 0).
- **Test first:**
- File: `tests/spikes/test_imports.py` (new)
- Verifies: `import spikes`; `from agent_framework.orchestrations import GroupChatBuilder, ConcurrentBuilder, MagenticBuilder, StandardMagenticManager, TerminationCondition`; `import pulp`; `import pytest_asyncio`.
- Pattern: `tests/test_smoke.py`
- **Verify:** `uv run --extra dev pytest tests/spikes/test_imports.py -q` → expected: passed; `uv pip show agent-framework-orchestrations` → `Version: 1.0.0`; `uv run ruff check .` → exit 0
- **On failure:** retry — if the orchestrations import fails after sync, confirm the lock resolved `agent-framework-orchestrations==1.0.0`; if anything pulls a pre-release, pin tighter. Revert with `git checkout -- pyproject.toml CLAUDE.md uv.lock` if unrecoverable.
- **Checkpoint:** `git commit -m "build(fase1): add dev orchestration + solver + async deps, scaffold spikes"`
- **Manifest:**
```yaml
manifest:
expected_paths:
- pyproject.toml
- CLAUDE.md
- spikes/__init__.py
- tests/spikes/__init__.py
- tests/spikes/test_imports.py
- docs/fase1-spikes/README.md
- uv.lock
min_file_count: 7
commit_message_pattern: "^build\\(fase1\\): add dev orchestration \\+ solver \\+ async deps"
bash_syntax_check: []
forbidden_paths:
- src/portfolio_optimiser/backends.py
- src/portfolio_optimiser/reference_domain.py
must_contain:
- path: pyproject.toml
pattern: "agent-framework-orchestrations"
- path: pyproject.toml
pattern: "pytest-asyncio"
- path: CLAUDE.md
pattern: "agent-framework-core` 1.9.0"
```
### Step 2: Shared spike harness + builder smoke (budget caps, fake client, live gate, async builder de-risk)
- **Files:** `spikes/_harness.py`, `tests/spikes/test_harness.py`
- **Changes:** Implement the cost/stop invariant (B4) and the test seams shared by all spikes. (1) `@dataclass Budget(max_tokens: int, max_rounds: int)` — constructor raises `ValueError` for non-positive caps (refuse to start without caps, A4); a `TokenMeter` plus `charge(tokens)`/`tick_round()` raise `BudgetExceeded(kind, limit, observed)` the moment a cap is crossed. (Two exception types is intentional and documented in the module: `ValueError` = bad construction argument; `BudgetExceeded` = runtime breach.) (2) `FakeChatClient(BaseChatClient)` returning scripted deterministic responses and counting tokens by word-count; plus `fake_agent(client, name, instructions)` building a minimal real `agent_framework.Agent` backed by the fake client so the orchestration builders get genuine participants. (3) `live_local_client_or_skip()` → builds `agent_framework.openai.OpenAIChatClient(base_url=env PORTFOLIO_LOCAL_BASE_URL, model=env PORTFOLIO_LOCAL_MODEL)` **directly**, or `pytest.skip(...)` when env is unset (the D2 `LocalBackend` seam is deliberately not used — its live-wiring is Fase 2). (4) A **builder smoke** that front-loads the biggest risk: construct a `GroupChatBuilder` and a `ConcurrentBuilder` with two `fake_agent`s and `await` a trivial run, asserting they execute — this is the one place that proves `FakeChatClient` can drive the GA builders. If it cannot, that is itself a recorded de-risk finding (escalate), surfaced before any spike is built on the assumption. (new file)
- **Reuses:** `agent_framework.BaseChatClient`, `agent_framework.Agent`; `agent_framework.openai.OpenAIChatClient`; `agent_framework.orchestrations.{GroupChatBuilder, ConcurrentBuilder}`; the D6 cost contract.
- **Test first:**
- File: `tests/spikes/test_harness.py` (new)
- Verifies: `Budget(0, 5)` and `Budget(5, 0)` raise `ValueError`; `charge`/`tick_round` raise `BudgetExceeded` past their caps; `FakeChatClient` returns scripted replies in order and accumulates tokens; `live_local_client_or_skip()` skips when env unset; **builder smoke** (async test) constructs and runs `GroupChatBuilder`+`ConcurrentBuilder` with fake agents and asserts completion.
- Pattern: `tests/test_backends.py` (`parametrize` + `pytest.raises`)
- **Verify:** `uv run --extra dev pytest tests/spikes/test_harness.py -q` → expected: passed; `uv run ruff check spikes tests/spikes` → exit 0
- **On failure:** escalate — if the builder smoke proves `FakeChatClient` cannot drive the GA builders, STOP and record it in `docs/fase1-spikes/README.md` as a primary de-risk finding (it changes Spikes A/B); decide with the operator whether to back the builders with a tiny canned-LLM stub or restructure. Otherwise revert with `git checkout -- spikes/_harness.py tests/spikes/test_harness.py`.
- **Checkpoint:** `git commit -m "feat(fase1): shared spike harness + builder smoke (budget caps, fake client, live gate)"`
- **Manifest:**
```yaml
manifest:
expected_paths:
- spikes/_harness.py
- tests/spikes/test_harness.py
min_file_count: 2
commit_message_pattern: "^feat\\(fase1\\): shared spike harness \\+ builder smoke"
bash_syntax_check: []
forbidden_paths:
- src/portfolio_optimiser/backends.py
must_contain:
- path: spikes/_harness.py
pattern: "class BudgetExceeded"
- path: spikes/_harness.py
pattern: "def live_local_client_or_skip"
- path: tests/spikes/test_harness.py
pattern: "GroupChatBuilder"
```
### Step 3: Spike A — Group Chat maker-checker vs single-agent (U3 / G7)
- **Files:** `spikes/a_groupchat.py`, `tests/spikes/test_a_groupchat.py`, `docs/fase1-spikes/findings-a.md`
- **Changes:** **Logic layer (always green):** `Arm` result dataclass (rounds, stalls, tokens, caught_flaw); `make_termination(n_rounds=3)` (N fixed at 3 per reviewer refinement #1); `verdict(mc: Arm, single: Arm) -> dict` computing `better = mc.caught_flaw and not single.caught_flaw` and `affordable = mc.tokens <= 3 * single.tokens` and `passed = better and affordable`, plus a comparison table renderer. **Integration/live layer (gated):** `run_live(project)` builds the 3-role maker-checker (proposer · critic · validator) via `GroupChatBuilder` + `make_termination()` and a single-agent baseline, runs both against `load_reference_projects()[0]` carrying a **planted flaw** (a savings candidate violating an obvious constraint, e.g. negative residual quantity), measures real numbers, writes them + the computed verdict to `findings-a.md`. (new file)
- **Reuses:** `spikes._harness` (Budget, FakeChatClient, fake_agent, live gate); `reference_domain.load_reference_projects`; `agent_framework.orchestrations.GroupChatBuilder`/`TerminationCondition`.
- **Test first:**
- File: `tests/spikes/test_a_groupchat.py` (new)
- Verifies (logic, non-tautological — inputs are **varied**, not a single scripted constant): `verdict()` returns `passed=True` when mc caught & single missed & tokens within 3×; `passed=False` when mc tokens = 5× single (affordable=False) even if better; `passed=False` when both caught the flaw (better=False); `make_termination(3)` stops at 3 rounds. A `@pytest.mark.skipif(no endpoint)` live test runs `run_live` and asserts ≤3 rounds, cap respected, a findings table emitted.
- Pattern: `tests/test_reference_domain.py` + `tests/test_backends.py`
- **Verify:** `uv run --extra dev pytest tests/spikes/test_a_groupchat.py -q` → expected: logic tests passed (live test skipped without endpoint)
- **On failure:** retry — confirm `GroupChatBuilder` method names by introspection and adjust `run_live`; the logic layer is independent of the MAF surface. Revert with `git checkout -- spikes/a_groupchat.py tests/spikes/test_a_groupchat.py docs/fase1-spikes/findings-a.md` if unrecoverable.
- **Checkpoint:** `git commit -m "feat(fase1): spike A - group chat maker-checker vs single-agent"`
- **Manifest:**
```yaml
manifest:
expected_paths:
- spikes/a_groupchat.py
- tests/spikes/test_a_groupchat.py
- docs/fase1-spikes/findings-a.md
min_file_count: 3
commit_message_pattern: "^feat\\(fase1\\): spike A"
bash_syntax_check: []
forbidden_paths:
- src/portfolio_optimiser/reference_domain.py
must_contain:
- path: spikes/a_groupchat.py
pattern: "GroupChatBuilder"
- path: tests/spikes/test_a_groupchat.py
pattern: "skipif"
- path: docs/fase1-spikes/findings-a.md
pattern: "token"
```
### Step 4: Spike B — Magentic limits=None footgun + ConcurrentBuilder state isolation (G1 / G2)
- **Files:** `spikes/b_footguns.py`, `tests/spikes/test_b_footguns.py`, `docs/fase1-spikes/findings-b.md`
- **Changes:** Driving the builders **is** the de-risk here, so these tests run in the gate (no live LLM — `FakeChatClient`/`fake_agent`), having been de-risked by the Step 2 builder smoke. (a) **Magentic unbounded (G1):** `MagenticBuilder` + `StandardMagenticManager` with `max_round_count=None` over a fake agent that never finalizes; drive it under the harness **round/iteration guard** (the deterministic assertion — wall-clock is only a secondary safety net, not asserted) and assert it does NOT self-terminate before the guard fires → confirms an explicit limit is required (B4). With explicit limits → terminates cleanly. (b) **Fan-out state isolation (G2/B7):** `fresh_workflow()` factory + a stateful executor; assert a SHARED instance bleeds state across the 3 projects and a FRESH instance per run shows zero bleed (B7). Writes observations + a token-use line ("0 — no live LLM") to `findings-b.md`. (new file)
- **Reuses:** `spikes._harness` (round guard, FakeChatClient, fake_agent); `reference_domain.load_reference_projects`; `agent_framework.orchestrations.{MagenticBuilder, StandardMagenticManager, ConcurrentBuilder}`.
- **Test first:**
- File: `tests/spikes/test_b_footguns.py` (new)
- Verifies: unbounded Magentic hits the round guard without self-terminating (asserts guard fired, not a natural stop); bounded Magentic stops within its limit; shared-instance fan-out leaks state (assertion of bleed); fresh-instance fan-out asserts no bleed across all 3 projects.
- Pattern: `tests/test_backends.py`
- **Verify:** `uv run --extra dev pytest tests/spikes/test_b_footguns.py -q` → expected: passed
- **On failure:** escalate — if `MagenticBuilder`/`ConcurrentBuilder` cannot be driven by a fake agent despite the Step 2 smoke, record it as a genuine de-risk finding in `findings-b.md` (it answers the footgun question by other means) before reverting `git checkout -- spikes/b_footguns.py tests/spikes/test_b_footguns.py docs/fase1-spikes/findings-b.md`.
- **Checkpoint:** `git commit -m "feat(fase1): spike B - magentic unbounded + concurrent state isolation"`
- **Manifest:**
```yaml
manifest:
expected_paths:
- spikes/b_footguns.py
- tests/spikes/test_b_footguns.py
- docs/fase1-spikes/findings-b.md
min_file_count: 3
commit_message_pattern: "^feat\\(fase1\\): spike B"
bash_syntax_check: []
forbidden_paths: []
must_contain:
- path: spikes/b_footguns.py
pattern: "ConcurrentBuilder"
- path: spikes/b_footguns.py
pattern: "fresh_workflow"
- path: docs/fase1-spikes/findings-b.md
pattern: "token"
```
### Step 5: Spike C — blocking hybrid validator (IR→PuLP→Monte Carlo, self-repair) (B1)
- **Files:** `spikes/c_validator.py`, `tests/spikes/test_c_validator.py`, `docs/fase1-spikes/findings-c.md`
- **Changes:** Fully deterministic — no MAF builders. Pydantic IR `SavingsProposal` (project_id, measure, affected cost codes, claimed saving NOK, assumptions) with field validators + a cross-field `@model_validator` (claimed saving ≤ affected items' total; quantities ≥ 0). `validate_proposal(proposal) -> ValidatedProposal | Rejection`: (1) Pydantic schema validation; (2) **PuLP** feasibility/optimization over the project's cost items (uses PuLP's **bundled CBC** binary — if CBC is genuinely absent the step **escalates**; no silent LP-relaxation fallback); (3) **Monte Carlo** over uncertain unit-costs → P10/P50/P90 (stdlib `random` seeded + `statistics.quantiles`); (4) **structural block**: an out-of-range/infeasible proposal returns a `Rejection` that can never be consumed as a `ValidatedProposal`. `self_repair(generate, max_attempts=N)` capped, then hard-stops. The LLM-generation entry is a thin wrapper gated by `live_local_client_or_skip()`; all validator/solver/MC logic is tested with crafted IR directly. Writes a token-use line ("0 — validator deterministic; live generation gated") + percentiles to `findings-c.md`. (new file)
- **Reuses:** `pydantic.BaseModel`/`model_validator`; `pulp`; `reference_domain.Project`/`CostItem`/`load_reference_projects`; `spikes._harness` (Budget for the self-repair cap, live gate).
- **Test first:**
- File: `tests/spikes/test_c_validator.py` (new)
- Verifies: an out-of-range proposal is structurally blocked — `validate_proposal` returns `Rejection` and raises if treated as validated; a valid proposal returns `ValidatedProposal` with P10 ≤ P50 ≤ P90; Monte Carlo reproducible under the fixed seed; `self_repair` stops at `max_attempts`.
- Pattern: `tests/test_reference_domain.py` + `pytest.raises`
- **Verify:** `uv run --extra dev pytest tests/spikes/test_c_validator.py -q` → expected: passed
- **On failure:** escalate (if CBC absent — record platform finding) / otherwise revert `git checkout -- spikes/c_validator.py tests/spikes/test_c_validator.py docs/fase1-spikes/findings-c.md`.
- **Checkpoint:** `git commit -m "feat(fase1): spike C - blocking hybrid validator (IR/solver/monte-carlo)"`
- **Manifest:**
```yaml
manifest:
expected_paths:
- spikes/c_validator.py
- tests/spikes/test_c_validator.py
- docs/fase1-spikes/findings-c.md
min_file_count: 3
commit_message_pattern: "^feat\\(fase1\\): spike C"
bash_syntax_check: []
forbidden_paths: []
must_contain:
- path: spikes/c_validator.py
pattern: "model_validator"
- path: spikes/c_validator.py
pattern: "class Rejection"
- path: tests/spikes/test_c_validator.py
pattern: "P10"
```
### Step 6: Spike D — VerdictStore + ExpeL retrieval (B2)
- **Files:** `spikes/d_verdictstore.py`, `tests/spikes/test_d_verdictstore.py`, `docs/fase1-spikes/findings-d.md`
- **Changes:** Minimal in-memory VerdictStore: a list of `Verdict(id, proposal_features, decision, rationale)` seeded with 10–20 synthetic verdicts from the reference domain. **Similarity defined operationally (reviewer refinement #2):** a weighted score over *structured* fields — Jaccard on the affected cost-code set + match on `savings_measure_type` + a magnitude-bucket match on claimed saving — **not** raw description text. `retrieve(proposal, k) -> list[Verdict]` ranks by that score (this is the always-tested unit that satisfies SC-D). A thin custom `ContextProvider` subclass wraps `retrieve()` for ExpeL few-shot injection; its injection is asserted only against the introspected interface (if the abstract surface differs at coding time, the `retrieve()` ranking remains the deliverable and the SC-D assertion). The embedding-based similarity path is **removed** as out of scope for a throwaway spike (noted as a Fase-2 option in `findings-d.md`). Writes a token-use line ("0 — deterministic retrieval") + a top-K example to `findings-d.md`. (new file)
- **Reuses:** `agent_framework.ContextProvider` (top-level); `reference_domain.load_reference_projects`; `spikes._harness`.
- **Test first:**
- File: `tests/spikes/test_d_verdictstore.py` (new)
- Verifies (non-tautological by construction): seed a store where the true match shares the *structured* similarity fields with the query but uses **different description text**, while 2–3 **decoys** share surface description text but differ in the structured fields; assert the true match is top-1 and within top-K; assert deterministic ordering. (ContextProvider-injection assertion included only if the interface is confirmed; `retrieve()` ranking is the guaranteed SC-D check.)
- Pattern: `tests/test_reference_domain.py`
- **Verify:** `uv run --extra dev pytest tests/spikes/test_d_verdictstore.py -q` → expected: passed
- **On failure:** retry — confirm `ContextProvider` abstract methods by introspection; keep `retrieve()` ranking separable. Revert `git checkout -- spikes/d_verdictstore.py tests/spikes/test_d_verdictstore.py docs/fase1-spikes/findings-d.md` if unrecoverable.
- **Checkpoint:** `git commit -m "feat(fase1): spike D - verdictstore + expel retrieval"`
- **Manifest:**
```yaml
manifest:
expected_paths:
- spikes/d_verdictstore.py
- tests/spikes/test_d_verdictstore.py
- docs/fase1-spikes/findings-d.md
min_file_count: 3
commit_message_pattern: "^feat\\(fase1\\): spike D"
bash_syntax_check: []
forbidden_paths: []
must_contain:
- path: spikes/d_verdictstore.py
pattern: "def retrieve"
- path: tests/spikes/test_d_verdictstore.py
pattern: "decoy"
```
### Step 7: Consolidate findings, document disposal, confirm the green quality gate
- **Files:** `docs/fase1-spikes/README.md`
- **Changes:** Update `docs/fase1-spikes/README.md` to a consolidated summary table (spike → assumption → result → confirmed/refuted → implication for Fase 2), pulling each per-spike note's verdict and token-use line (each `findings-{a,b,c,d}.md` already exists from Steps 3–6). Confirm the **Disposal** subsection (from Step 1) is complete. Run the full quality gate across the whole repo. (no new findings files here — they are owned by their spike steps)
- **Reuses:** the per-spike findings written in Steps 3–6.
- **Test first:** n/a (documentation + gate; the gate IS the verification).
- **Verify:** `uv run ruff check .` → exit 0; `uv run ruff format --check .` → exit 0; `uv run mypy src` → exit 0; `uv run --extra dev pytest -q` → expected: all passed (originals + spike logic layers; live arms skipped without endpoint)
- **On failure:** escalate — a red gate means a prior step regressed; bisect by step and fix before consolidating.
- **Checkpoint:** `git commit -m "docs(fase1): consolidate spike findings + confirm green quality gate"`
- **Manifest:**
```yaml
manifest:
expected_paths:
- docs/fase1-spikes/README.md
- docs/fase1-spikes/findings-a.md
- docs/fase1-spikes/findings-b.md
- docs/fase1-spikes/findings-c.md
- docs/fase1-spikes/findings-d.md
min_file_count: 5
commit_message_pattern: "^docs\\(fase1\\): consolidate spike findings"
bash_syntax_check: []
forbidden_paths:
- src/portfolio_optimiser/backends.py
must_contain:
- path: docs/fase1-spikes/README.md
pattern: "Fase 2"
- path: docs/fase1-spikes/README.md
pattern: "Disposal"
```
### Failure recovery rules
- **revert** — undo this step's changes (`git checkout -- {files}`), do not proceed.
- **retry** — confirm the real MAF method signature by introspection, adjust, then revert if still failing.
- **escalate** — stop entirely; needs human judgment (Step 2 builder smoke fails / Step 4 builder un-fakeable / Step 5 CBC absent / Step 7 gate regression). For Steps 2 and 4, an escalate that *records the MAF limitation* is itself a valid de-risk outcome.
- **Checkpoint** — commit after each green step.
## Alternatives Considered
| Approach | Pros | Cons | Why rejected |
|----------|------|------|--------------|
| Spikes under `src/portfolio_optimiser/spikes/` | Single import root | Ships throwaway code in the wheel | Top-level `spikes/` keeps the wheel clean (hatchling packages only `src/portfolio_optimiser`) |
| `agent-framework-orchestrations` in **core** deps | One home for the locked debate default | Permanent Fase-2 commitment made under a throwaway task (scope creep, flagged in review) | Put it in `dev` like PuLP; promote to core in Fase 2 when the real flow is built |
| Wire `LocalBackend.create_chat_client()` in Step 2 for the live arm | Fills the D2 seam | Touches `src/` under a throwaway task; forces editing `test_backends.py`; non-throwaway commitment | Build `OpenAIChatClient` directly in the throwaway harness; defer seam-wiring to Fase 2; `src/` stays untouched |
| Require a live LOCAL endpoint for all spike tests | Tests the real thing | Gate red without an endpoint; violates SC-gate + D6 | Logic layer always green; integration/live arm gated |
| numpy for Monte Carlo | Familiar | New dep for trivial sampling | stdlib `random` + `statistics.quantiles` — zero new dep |
| `agent-framework[all]` to get orchestrations | One line | Re-introduces the beta/`--prerelease`/alpha-pydantic problem Fase 0 removed (`491a746`) | Pin the GA `agent-framework-orchestrations>=1.0.0` only |
| Embedding similarity (Spike D) | "Real" RAG | Needs live endpoint; non-deterministic; serves no SC; over-builds throwaway | Deterministic structured-field similarity; embeddings = Fase-2 note only |
| Hand-rolled feasibility instead of PuLP | No dep | Not a faithful solver-in-the-loop (B1/R2) | PuLP (bundled CBC) keeps the spike honest |
## Test Strategy
- **Framework:** pytest 9.1.1 + `pytest-asyncio` (`asyncio_mode = "auto"`); plain functions with `-> None`, `parametrize`, `pytest.raises`; fixtures from `load_reference_projects()`. Async tests for the MAF-builder integration.
- **Existing patterns:** `tests/test_backends.py`, `tests/test_reference_domain.py`, `tests/test_smoke.py`.
- **New tests in this plan:** 6 modules (imports, harness+builder-smoke, A, B, C, D). Determinism via `FakeChatClient` + fixed seeds; live arms isolated behind `@pytest.mark.skipif(no endpoint)`. Spike B drives the builders in-gate (no endpoint) — de-risked by the Step 2 smoke.
### Tests to write
| Type | File | Verifies | Model test |
|------|------|----------|------------|
| Import | `tests/spikes/test_imports.py` | orchestrations + pulp + pytest-asyncio + `spikes` import | `tests/test_smoke.py` |
| Unit+async | `tests/spikes/test_harness.py` | Budget caps; FakeChatClient; skip gate; builder smoke | `tests/test_backends.py` |
| Unit | `tests/spikes/test_a_groupchat.py` | varied-input verdict logic; ≤3-round termination; gated live arm | `tests/test_reference_domain.py` |
| Unit+async | `tests/spikes/test_b_footguns.py` | unbounded Magentic needs guard; zero fan-out bleed | `tests/test_backends.py` |
| Unit | `tests/spikes/test_c_validator.py` | structural block; P10≤P50≤P90; self-repair cap | `tests/test_reference_domain.py` |
| Unit | `tests/spikes/test_d_verdictstore.py` | non-tautological top-K retrieval | `tests/test_reference_domain.py` |
## Risks and Mitigations
| Priority | Risk | Location | Impact | Mitigation |
|----------|------|----------|--------|------------|
| High | `FakeChatClient` cannot drive the async GA builders | `spikes/_harness.py` (smoke), A/B | Spikes A/B integration blocked | Step 2 builder smoke front-loads this; failure is a recorded de-risk finding (escalate), not a late surprise; logic layers (A verdict, B isolation) stay green regardless |
| High | Real builder method signatures differ from planned wiring | A/B integration | Integration code won't run as written | Confirm by introspection at coding time (brief-sanctioned); per-step retry; logic layers independent of the MAF surface |
| Medium | No LOCAL endpoint during the session | A/C live arms | Live numbers not captured | Logic layers prove the machinery; live arms `skip`; findings note records "live arm not run" honestly |
| Medium | PuLP CBC binary missing on Intel mac | `spikes/c_validator.py` | Solver call fails | PuLP wheels bundle CBC; if genuinely absent → **escalate** (recorded), no silent fallback |
| Low | Async-mode misconfig breaks existing sync tests | `pyproject.toml` | Original 12 tests affected | `asyncio_mode="auto"` only auto-marks coroutine tests; sync tests unaffected — Step 7 gate catches any regression |
| Low | 2 `ExperimentalWarning`s (SKILLS/HARNESS) at import (G8) | all imports | Noise only | Harmless; do not enable `filterwarnings=error` |
| Low | `.claude/projects/**` committed to a to-be-public repo | repo | Planning scratch in history | Out of scope here; flag at Fase 4 release prep |
## Assumptions
| # | Assumption | Why unverifiable now | Impact if wrong |
|---|-----------|-----------------|-----------------|
| 1 | LOCAL profile = reachable OpenAI-compatible endpoint via `PORTFOLIO_LOCAL_*` env | Depends on operator's running Ollama/LM Studio | Live arms skip; logic layers still pass — low impact |
| 2 | `FakeChatClient` can drive the GA builders | Not yet installed/introspected to method level | Step 2 smoke decides early; failure is a documented finding, not silent breakage |
| 3 | PuLP's bundled CBC works on the Intel-mac venv | Not installed yet | Step 5 escalates with a recorded platform finding |
| 4 | N = 3 rounds is a fair maker-checker bound | Judgment from G7 token-discipline | If too tight to converge live, bump to ≤5 and re-record (documented range) |
## Verification
*Per-step manifests are checked automatically by trekexecute. These map to the brief Success Criteria.*
- [ ] Spike A: `uv run --extra dev pytest tests/spikes/test_a_groupchat.py -q` → varied-input verdict logic passes; gated live arm (when run) converges ≤3 rounds, cap respected, comparison table emitted (SC-A; empirical better/cheaper verdict is endpoint-dependent and so reported)
- [ ] Spike B: `uv run --extra dev pytest tests/spikes/test_b_footguns.py -q` → unbounded Magentic needs an external guard; fresh-instance fan-out zero state-bleed (SC-B)
- [ ] Spike C: `uv run --extra dev pytest tests/spikes/test_c_validator.py -q` → out-of-range blocked structurally; valid proposal yields P10/P50/P90; self-repair capped (SC-C)
- [ ] Spike D: `uv run --extra dev pytest tests/spikes/test_d_verdictstore.py -q` → top-K retrieval returns the structurally-similar verdict over surface-text decoys (SC-D)
- [ ] Gate: `uv run ruff check .` exit 0; `uv run ruff format --check .` clean; `uv run mypy src` exit 0; `uv run --extra dev pytest -q` all passed (SC-gate)
- [ ] Findings: `docs/fase1-spikes/` has README (summary table + Disposal) + findings-{a,b,c,d}.md, each with a confirmed/refuted verdict and a token-use line (brief Goal + NFR)
## Estimated Scope
- **Files to modify:** 2 (`pyproject.toml`, `CLAUDE.md`; `uv.lock` regenerated)
- **Files to create:** ~19 (4 spike modules + `_harness.py` + 6 test modules + 2 `__init__.py` + 5 docs notes + README owned by Step 1/7)
- **Complexity:** medium (logic is modest; the real unknown — MAF builder ergonomics with a fake client — is de-risked up-front by the Step 2 smoke)
## Execution Strategy
### Session 1: Foundation
- **Steps:** 1, 2
- **Wave:** 1
- **Depends on:** none
- **Scope fence:**
- Touch: `pyproject.toml`, `CLAUDE.md`, `spikes/__init__.py`, `spikes/_harness.py`, `tests/spikes/{__init__,test_imports,test_harness}.py`, `docs/fase1-spikes/README.md`, `uv.lock`
- Never touch: `src/**`, the four spike modules
### Session 2: Spike A
- **Steps:** 3
- **Wave:** 2
- **Depends on:** Session 1
- **Scope fence:**
- Touch: `spikes/a_groupchat.py`, `tests/spikes/test_a_groupchat.py`, `docs/fase1-spikes/findings-a.md`
- Never touch: other spike modules, `src/**`, `pyproject.toml`, `CLAUDE.md`
### Session 3: Spike B
- **Steps:** 4
- **Wave:** 2
- **Depends on:** Session 1
- **Scope fence:**
- Touch: `spikes/b_footguns.py`, `tests/spikes/test_b_footguns.py`, `docs/fase1-spikes/findings-b.md`
- Never touch: other spike modules, `src/**`, `pyproject.toml`, `CLAUDE.md`
### Session 4: Spike C
- **Steps:** 5
- **Wave:** 2
- **Depends on:** Session 1
- **Scope fence:**
- Touch: `spikes/c_validator.py`, `tests/spikes/test_c_validator.py`, `docs/fase1-spikes/findings-c.md`
- Never touch: other spike modules, `src/**`, `pyproject.toml`, `CLAUDE.md`
### Session 5: Spike D
- **Steps:** 6
- **Wave:** 2
- **Depends on:** Session 1
- **Scope fence:**
- Touch: `spikes/d_verdictstore.py`, `tests/spikes/test_d_verdictstore.py`, `docs/fase1-spikes/findings-d.md`
- Never touch: other spike modules, `src/**`, `pyproject.toml`, `CLAUDE.md`
### Session 6: Consolidation
- **Steps:** 7
- **Wave:** 3
- **Depends on:** Sessions 2–5
- **Scope fence:**
- Touch: `docs/fase1-spikes/README.md`
- Never touch: `spikes/**`, `src/**`, `pyproject.toml`, `CLAUDE.md`
### Execution Order
- **Wave 1:** Session 1 (foundation — includes the builder smoke that de-risks all spikes)
- **Wave 2:** Sessions 2, 3, 4, 5 (parallel — independent spike modules sharing only the read-only harness)
- **Wave 3:** Session 6 (after Wave 2)
### Grouping rules applied
- Steps sharing files → same session (1+2 = foundation).
- Independent spike modules → separate sessions (parallelizable in Wave 2).
- Consolidation depends on all spikes → Wave 3.
## Plan Quality Score
| Dimension | Weight | Score | Notes |
|-----------|--------|-------|-------|
| Structural integrity | 0.15 | 90 | Foundation→spikes→consolidation; src/ untouched so scope fences are accurate |
| Step quality | 0.20 | 88 | TDD-first; false LocalBackend reuse removed; findings owned per step |
| Coverage completeness | 0.20 | 90 | SC-A/B/C/D + gate + findings mapped; A's empirical verdict honestly endpoint-dependent |
| Specification quality | 0.15 | 86 | Real import paths; builder ergonomics de-risked by Step 2 smoke; CBC fallback decided |
| Risk & pre-mortem | 0.15 | 90 | Biggest risk (fake-drives-builders) front-loaded with a green-keeping logic layer |
| Headless readiness | 0.10 | 92 | On-failure + Checkpoint + Manifest per step; expected_paths ⊆ Files |
| Manifest quality | 0.05 | 90 | All manifests checkable; orphan paths fixed |
| **Weighted total** | **1.00** | **89** | **Grade: A-** |
**Adversarial review:**
- **Plan critic:** REVISE → addressed (2 blockers + 8 major + 4 minor); see Revisions.
- **Scope guardian:** MIXED → addressed (orchestrations moved to dev; embedding path dropped; token-reporting + findings-location ratified); see Revisions.
## Revisions
*Added after Phase 9 adversarial review (plan-critic + scope-guardian, deduped: 19 findings, 0 overlap).*
| # | Finding (rule_key) | Severity | Resolution |
|---|--------------------|----------|------------|
| 1 | live-arm/impossible-localbackend-wiring | blocker | Harness builds `OpenAIChatClient` directly; dropped the false `LocalBackend` reuse; D2 seam-wiring deferred to Fase 2; `src/` untouched |
| 2 | fragile-assumption/fakeclient-drives-async-builders | blocker | Added Step 2 builder smoke (front-loads the risk); added `pytest-asyncio` + `asyncio_mode=auto`; split each spike into always-green logic layer + gated integration/live layer; failure is a recorded escalate |
| 3 | structural/scope-fence-vs-live-arm | major | Resolved by #1 — `src/**` is now genuinely never touched; fences accurate |
| 4 | missing-setup/async-test-runner | major | `pytest-asyncio>=0.24` added (Step 1); async integration tests |
| 5 | verification/spike-a-verdict-tautological | major | Logic test now varies inputs to prove the verdict function; empirical U3/G7 verdict explicitly endpoint-dependent (live arm) |
| 6 | underspecified/contextprovider-inject-no-target | major | `retrieve()` ranking is the guaranteed SC-D unit; ContextProvider injection asserted only against the introspected interface |
| 7 | manifest/expected-paths-not-subset-of-files | major | All `Files:` lines now include test/findings/`uv.lock`/`CLAUDE.md` paths |
| 8 | ordering/findings-files-creation-ambiguous | major | Each spike step (3–6) owns and creates its `findings-X.md`; Step 7 only consolidates the README |
| 9 | missing-error-handling/cbc-fallback-deferred | major | Committed to PuLP's bundled CBC; absence → escalate (recorded), no silent LP-relaxation |
| 10 | coverage/spike-a-precondition-downgraded | major | Step 1 corrects `CLAUDE.md` version; installed version confirmed 1.9.0; memory fix noted as session follow-up |
| 11 | creep — orchestrations in core deps | major | Moved `agent-framework-orchestrations` to `dev` (throwaway scope), like PuLP |
| 12 | creep — Spike D embedding path | minor | Removed; deterministic similarity only; embeddings = Fase-2 note |
| 13 | gap — token reporting B/C/D | minor | Each findings note carries a token-use line (incl. "0 — no live LLM") |
| 14 | gap — findings location vs brief | minor | Kept `docs/fase1-spikes/` (tracked, survives, consistent with research/plan) — ratified as a deliberate refinement; operator can object |
| 15 | dependency — re-confirm orchestrations resolves | minor | Step 1 runs `uv pip show agent-framework-orchestrations` post-sync |
| 16 | minor/pythonpath-root-shadowing | minor | Documented: repo-root added for `import spikes`; no name collision with installed dists |
| 17 | minor/budget-two-exception-types | minor | Documented intent: `ValueError` = bad ctor arg, `BudgetExceeded` = runtime breach |
| 18 | minor/wall-clock-flaky-assertion | minor | Spike B asserts the round/iteration guard; wall-clock is a secondary safety net only |
| 19 | minor/no-teardown-for-throwaway-spikes | minor | Step 1/7 README **Disposal** subsection: `rm -rf` + revert the pyproject dev/tool edits |