diff --git a/.claude/projects/2026-06-23-fase1-derisk-spikes/brief.md b/.claude/projects/2026-06-23-fase1-derisk-spikes/brief.md new file mode 100644 index 0000000..493cea8 --- /dev/null +++ b/.claude/projects/2026-06-23-fase1-derisk-spikes/brief.md @@ -0,0 +1,122 @@ +--- +type: trekbrief +brief_version: "2.2" +created: 2026-06-23 +task: "Fase 1 de-risk spikes (A–D): empirically validate the framework's most dangerous assumptions before the MVP vertical slice" +slug: fase1-derisk-spikes +project_dir: .claude/projects/2026-06-23-fase1-derisk-spikes/ +research_topics: 0 +research_status: skipped +auto_research: false +interview_turns: 3 +source: interview +framing: refine +phase_signals: + - phase: research + effort: standard + - phase: plan + effort: standard + - phase: execute + effort: standard + - phase: review + effort: standard +--- + +# Task: Fase 1 — De-risk spikes (A–D) + +> Generated by `/trekbrief` on 2026-06-23. +> This brief is the contract between requirements and planning. `/trekplan` +> reads it to produce the implementation plan. Every decision in the plan must +> trace back to content in this brief. + +## TL;DR + +**Refine** of the locked plan ([§Fase 1](docs/plan/2026-06-23-incremental-plan.md)) — same intent, narrowed to executable spike scope. Build four **throwaway** de-risk spikes (A–D) that convert the framework's most dangerous *documented-but-unverified* assumptions into *measured facts* before the Fase 2 vertical slice. Each spike has a falsifiable pass/fail from the plan's verification block. No new external research — reuse the research report's §15 implementation register. Runtime agent calls default to the **LOCAL profile** with hard token/round caps per D6. + +## Intent + +Fase 0 delivered the skeleton, locked decisions (D1–D6), and a synthetic reference domain (D4). Before committing to the full architecture in Fase 2 (vertical slice), we must empirically de-risk the four assumptions that — if wrong — would force a redesign. Specifically: (1) that a Group Chat maker-checker debate beats a single-agent baseline by enough to justify its multiplicative token cost (U3 / G7); (2) that the known MAF footguns behave as the research predicts and our guards hold — Magentic's unbounded termination when `limits=None` (G1/B4) and shared-`WorkflowBuilder` state corruption in fan-out (G2/B7); (3) that a blocking deterministic hybrid-validator (B1) can *structurally* block an out-of-range proposal from ever reaching the expert; and (4) that ExpeL retrieval (B2) actually surfaces a relevant prior verdict for a similar new proposal. These are throwaway spikes — code we expect to discard — whose only job is to turn §15 register assumptions into evidence. Getting each wrong *cheaply now* is vastly cheaper than discovering it mid-Fase-2. + +## Goal + +Four runnable spike modules (isolated so they are trivial to discard), each producing a **measurable pass/fail** against the plan §Fase 1 verification criteria, each running on the LOCAL profile by default with hard token and round/iteration caps, and each emitting a short findings note with the measured numbers (convergence rounds, stall frequency, token use, state-bleed observations, validator block/pass behavior, retrieval hit/miss). At the end, the framework's four most dangerous assumptions are each **confirmed or refuted with evidence**, and the findings are written down so they directly inform the Fase 2 design. The repo's quality gate (ruff + mypy(src) + pytest) stays green. + +## Non-Goals + +- Production code for Fase 2 (the vertical slice) — these spikes are throwaway and may be deleted after their findings are recorded. +- Compliance functions (D3 — the deployer owns DPIA/ROS/behandlingsformål; we build only technical preconditions). +- Chasing the last 10% (D5) — spikes prove the generic core, not edge cases or polish. +- Heavy Foundry/Azure runs (D6) — LOCAL profile is default; Foundry is used only for targeted, minimal verification if at all in Fase 1. +- A production sandbox for user-supplied skill scripts (B8 / G3) — out of Fase 1 scope. +- A full production VerdictStore — Spike D is a minimal 10–20 synthetic-verdict retrieval test only, not the durable store design. +- Resolving the second-rank open questions (plan §Risiko: review-latency, project topology, schema ownership, Foundry-memory Preview) — these do not block Fase 1. + +## Constraints + +- **D6 cost-discipline:** LOCAL profile default (OpenAI-compatible endpoint); cheapest models; tiny synthetic data; hard token + round caps; no heavy test runs. Foundry/Azure only for targeted, minimal verification. +- **Deterministic validator is obligatory and blocking** (never an optional plugin) — Spike C must demonstrate *structural* blocking, not advisory warning. +- **Stop-criteria + budget caps required at startup** — fail-fast if missing; never an unbounded loop. +- **Group Chat maker-checker is the debate default**, NOT Magentic (which is experimental, G8/A2). +- **Stack:** Python ≥3.10, MAF (`agent-framework`), `uv`, `ruff`, `mypy`, `pytest`. Type hints throughout; Pydantic for IR/validation. +- Spikes reuse the existing synthetic domain (D4: `data/reference_projects.json`, `reference_domain.py`) and the backend-profile seam (D2: `backends.py`). + +## Preferences + +- **90%-principle (D5):** generic core + clear extension points; do not over-fit. +- Per-agent model selection via chat-client (`FoundryChatClient` / `OpenAIChatClient`, common base `BaseChatClient`). +- Keep spikes physically isolated (e.g. a `spikes/` package or `tests/spikes/`) so discarding them leaves the core untouched. +- Findings recorded as short markdown notes inside the project dir (`.claude/projects/2026-06-23-fase1-derisk-spikes/`), not scattered. +- Prefer reusing `mcp-solver` / OR-Tools / PuLP / Z3 (R1/R2) for Spike C's solver step rather than hand-rolling. + +## Non-Functional Requirements + +- Each spike enforces a **hard token-budget cap** and a **round/iteration cap**; exceeding either yields a structured stop event, never a silent hang. +- Each spike **reports measured token usage** (per the plan's "every spike measures and reports token consumption"). +- Runtime defaults to the LOCAL profile; **no silent egress** to Azure/Foundry without explicit profile selection. +- Spikes must not require real/sensitive data — synthetic only. + +## Success Criteria + +*Falsifiable, mapped 1:1 to the plan §Fase 1 verification block.* + +- **Spike A (U3 / G7):** Maker-checker (proposer · critic · validator) converges in ≤ N rounds (N fixed at spike-design time) AND the hard cap is respected; a findings note documents convergence rate, stall frequency, and token use for BOTH maker-checker and single-agent baseline, with an explicit cheaper/better verdict. Verify: the spike command/test exits 0 and prints the comparison table. +- **Spike B (G1 / G2):** Unbounded Magentic (`limits=None`) does NOT self-terminate → confirms an explicit limit is required (B4); `ConcurrentBuilder` fan-out from a shared builder shows **zero state-bleed** when the fresh-instance helper (B7) is used. Verify: the spike asserts both observations and exits 0. +- **Spike C (B1):** An out-of-range proposal is **structurally blocked** (never reaches the expert/output); a valid proposal passes and yields P10/P50/P90 from the Monte Carlo step; self-repair retries are capped at N. Verify: a test asserts the blocked case raises/returns a structured rejection and the valid case returns the percentiles. +- **Spike D (B2):** Top-K retrieval fetches the relevant historical verdict for a similar new proposal from a store of 10–20 synthetic verdicts. Verify: a test asserts the expected verdict id is in the top-K for a crafted similar proposal. +- **Quality gate:** `uv run ruff check .` exits 0, `uv run ruff format --check .` clean, `uv run mypy src` exits 0, `uv run pytest` exits 0. + +## Research Plan + +No external research needed — the codebase and the existing research report ([docs/research/2026-06-23-prior-art-platform.md](docs/research/2026-06-23-prior-art-platform.md) §13 architecture, §15 implementation register, §15.3 footguns) plus this brief contain sufficient context for planning. The spikes themselves ARE the empirical de-risking. Any MAF API specifics (e.g. the exact current-version Python Group Chat termination API, U3) are confirmed inline via the `microsoft-learn` MCP at coding time rather than as a separate research pass. + +## Open Questions / Assumptions + +- **[ASSUMPTION]** The LOCAL profile resolves to an OpenAI-compatible endpoint reachable during spikes (per D6 and the `local` profile in `backends.py`). If no local endpoint is available at run time, Spike A/C/D live agent calls fall back to a minimal, capped Foundry run. +- **[ASSUMPTION — verify before Spike A]** The installed `agent-framework` version's Python Group Chat termination API matches research §15 (U3: `termination_condition` lambda). Note a version ambiguity to resolve first: project `CLAUDE.md` states `agent-framework` 1.8.0 while `STATE.md` references `agent-framework-core` 1.9.0 — confirm the actual installed version (`uv pip show agent-framework-core`) and the current termination API via the `microsoft-learn` MCP before coding Spike A. +- **[ASSUMPTION]** Spike A's convergence target "N rounds" will be fixed at spike-design time (candidate ≤ 3–5 rounds); the plan deliberately leaves N open. +- Magentic is experimental (G8); Spike B exercises it ONLY to confirm the footgun (G1), never to build core flow on it (A2). + +## Prior Attempts + +None for the spikes themselves — fresh. Substrate from Fase 0 is complete and committed: repo scaffold, locked decisions D1–D6, GA-slimmed dependencies (`uv.lock`), the D4 synthetic "anleggskostnad" domain (`reference_domain.py` + `data/reference_projects.json`, 3 fictional projects), and the D2 backend-profile skeleton (`backends.py`: `Profile` azure|local + `ChatBackend` protocol + stubs raising `NotImplementedError` until Fase 1). Quality gate was green at Fase 0 close (ruff + mypy(src) + 12 pytest passed). + +## Metadata + +- **Created:** 2026-06-23 +- **Interview turns:** 3 +- **Auto-research opted in:** no +- **Source:** trekbrief interview + +--- + +## How to continue + +Manual (default): + +```bash +# No research topics — go straight to planning: +/trekplan --project .claude/projects/2026-06-23-fase1-derisk-spikes + +# Then execute: +/trekexecute --project .claude/projects/2026-06-23-fase1-derisk-spikes +``` diff --git a/.claude/projects/2026-06-23-fase1-derisk-spikes/plan.md b/.claude/projects/2026-06-23-fase1-derisk-spikes/plan.md new file mode 100644 index 0000000..08f8d24 --- /dev/null +++ b/.claude/projects/2026-06-23-fase1-derisk-spikes/plan.md @@ -0,0 +1,473 @@ +# Fase 1 — De-risk spikes (A–D) + +> **Plan quality: A-** (post adversarial review + revision) — APPROVE_WITH_NOTES +> +> Generated by trekplan v2.0 on 2026-06-23 — `plan_version: 1.7` + +## Context + +Fase 0 delivered the skeleton, locked decisions (D1–D6), and a synthetic reference domain (D4). Before committing to the full architecture in Fase 2, we empirically de-risk the four assumptions that — if wrong — force a redesign (brief Intent): (A) Group Chat maker-checker beats a single-agent baseline by enough to justify its multiplicative token cost (U3 / G7); (B) the known MAF footguns behave as predicted and our guards hold — Magentic unbounded termination when `limits=None` (G1/B4) and shared-builder/fan-out state corruption (G2/B7); (C) a blocking deterministic hybrid-validator (B1) can *structurally* block an out-of-range proposal; (D) ExpeL retrieval (B2) surfaces a relevant prior verdict. These are **throwaway** spikes whose only job is to turn §15-register assumptions into measured facts, cheaply, under cost-discipline (D6: LOCAL profile default, hard token/round caps). + +**Decisive planning finding (premise-verification):** the GA-slimmed Fase 0 install deliberately dropped the `agent-framework[all]` meta, so the orchestration builders (`GroupChatBuilder`, `ConcurrentBuilder`, `MagenticBuilder`) are **not importable** — they live in the separate package `agent-framework-orchestrations`, which has a clean **GA release `1.0.0`** depending on `agent-framework-core<2,>=1.9.0` (exactly our installed core). Because the spikes are **throwaway**, this package is added to the **`dev`** dependency group (not core) — exactly like the PuLP solver — deferring any core-runtime commitment to Fase 2 (revised after scope review). Also resolved: the installed `agent-framework-core` is **1.9.0** (STATE.md correct; project `CLAUDE.md` says 1.8.0 and is corrected in Step 1; the `project-core` user-memory also says 1.8.0 — that lives outside the repo and is a session-end follow-up). + +**Gate-green contract (the central design choice).** Each spike is split into a **logic layer** (pure functions/classes we author — always exercised by the quality gate, no live endpoint and no full MAF workflow run) and an **integration/live layer** (drives the actual MAF builders and/or a real LOCAL LLM — runs when available, otherwise `skip`s, except Spike B where driving the builders IS the de-risk; see Step 2/Step 4). The headline empirical claims (Spike A's better/cheaper verdict; live token numbers) live in the integration/live layer and are honestly reported as endpoint-dependent. The gate stays green from the logic layer alone. + +## Architecture Diagram + +```mermaid +graph TD + subgraph "Reused (Fase 0, unchanged — src/ never touched)" + RD["reference_domain.py
(D4 synthetic projects)"] + BC["agent_framework.BaseChatClient
(subclassed by FakeChatClient)"] + end + subgraph "New — throwaway spikes/ package (dev-only)" + H["_harness.py
Budget+RoundCap (B4)
FakeChatClient + minimal Agent
live_local_client_or_skip() -> OpenAIChatClient"] + SM["builder smoke (Step 2)
proves FakeChatClient drives the GA builders"] + A["a_groupchat.py — Spike A (U3/G7)"] + B["b_footguns.py — Spike B (G1, G2/B7)"] + C["c_validator.py — Spike C (B1)"] + D["d_verdictstore.py — Spike D (B2)"] + end + subgraph "Deliverables (tracked)" + F["docs/fase1-spikes/
README + findings-{a,b,c,d}.md"] + end + H --> SM --> A & B & C & D + RD --> A & C & D + BC --> H + A --> F + B --> F + C --> F + D --> F +``` + +## Codebase Analysis + +- **Tech stack:** Python ≥3.10 (venv on 3.12), MAF GA (`agent-framework-core` 1.9.0, `-foundry` 1.8.2, `-openai` 1.8.2), `pydantic` 2.13.4 (stable), `uv`, `ruff` 0.15.18, `mypy` 2.1.0, `pytest` 9.1.1. Build: hatchling (wheel packages `src/portfolio_optimiser` only — top-level `spikes/` is never shipped). +- **Key patterns:** small typed package; `from __future__ import annotations`; frozen dataclasses for the domain; `Protocol` + `runtime_checkable` for the D2 seam; fail-fast (`ValueError`/`NotImplementedError`); tests are plain pytest functions with `-> None`, `parametrize`, `pytest.raises`, module-scope fixtures, importing from `portfolio_optimiser`. +- **Relevant files:** `src/portfolio_optimiser/backends.py` (D2 seam — `Profile`, `ChatBackend`, `get_backend`, `LocalBackend`; live-wiring of the seam is a **Fase 2** concern, NOT touched here), `src/portfolio_optimiser/reference_domain.py` (D4 — `Project`, `CostItem`, `load_reference_projects`, `.total_cost`), `src/portfolio_optimiser/data/reference_projects.json` (3 synthetic projects), `tests/test_*.py` (test style), `pyproject.toml`. +- **Reusable code:** `load_reference_projects()` → spike fixtures (A/C/D); `Project.total_cost` / `CostItem.total_cost` → savings-IR constraint source (C); `agent_framework.BaseChatClient` → subclassed by the harness `FakeChatClient`; `agent_framework.openai.OpenAIChatClient` → built **directly** by the harness for the live arm (the D2 `LocalBackend` seam is intentionally left un-wired until Fase 2, so `src/` stays untouched and the throwaway spikes add no core surface). +- **External tech (verified by introspection + PyPI, not docs):** orchestration builders in `agent_framework.orchestrations` (need `agent-framework-orchestrations`); chat clients in `agent_framework.openai` (`OpenAIChatClient`); `ContextProvider`/`MemoryContextProvider` top-level. **MAF workflows/orchestrations are async** (`await workflow.run(...)`), so the spike tests use `pytest-asyncio` (added in Step 1). Exact builder method signatures are confirmed by introspection at coding time (brief-sanctioned) and de-risked once up-front by the Step 2 builder smoke. +- **Recent git activity:** `b57aa83` Fase 0 (D4+D2+tests), `491a746` deps GA-fix + lockfile. Clean tree on `main`, synced to origin. + +## Research Sources + +*Ground-truth API/dependency verification — introspection + PyPI metadata are authoritative here (no web docs needed).* + +| Topic | Source | Key Findings | Confidence | +|-------|--------|--------------|------------| +| Orchestration builders' real import path | `import agent_framework.orchestrations` (introspection) | `GroupChatBuilder`, `ConcurrentBuilder`, `MagenticBuilder`, `StandardMagenticManager`, `SequentialBuilder`, `TerminationCondition`, `MagenticPlanReviewRequest` live there; not top-level | high | +| Orchestrations package availability/stability | PyPI JSON | GA `1.0.0` (uploaded 2026-06-18, not yanked); `requires_dist: agent-framework-core<2,>=1.9.0`; rest b/rc | high | +| Installed core version (resolves brief ambiguity) | `uv pip show agent-framework-core` | **1.9.0** (STATE correct; CLAUDE.md 1.8.0 → corrected Step 1) | high | +| MAF orchestrations are async | introspection (`WorkflowBuilder`/orchestration `.run`) | tests need `pytest-asyncio`; integration arm is async | high | +| Solver lib for Spike C | PyPI / project knowledge | `pulp` (PuLP) MIT, pure-Python, **bundles a CBC binary in its wheel** → faithful solver-in-the-loop (R2) | high | + +## Implementation Plan + +Each step targets one focused change and follows the repo's TDD-ish style. Per the **gate-green contract** above, the deterministic logic layer is always exercised; integration/live layers are gated. + +### Step 1: Scaffold throwaway spikes package and pin dev-only orchestration + solver + async deps + +- **Files:** `pyproject.toml`, `CLAUDE.md`, `spikes/__init__.py`, `tests/spikes/__init__.py`, `tests/spikes/test_imports.py`, `docs/fase1-spikes/README.md`, `uv.lock` +- **Changes:** In `pyproject.toml` add to `[project.optional-dependencies].dev` (NOT core — spikes are throwaway): `agent-framework-orchestrations>=1.0.0` (GA; resolves with core 1.9.0), `pulp>=2.8` (CBC solver), `pytest-asyncio>=0.24`. Set `[tool.pytest.ini_options] pythonpath = ["src", "."]` (repo-root added so `import spikes` resolves; `spikes/` has no name-collision with any installed dist, so it cannot shadow them) and `asyncio_mode = "auto"`. Set `[tool.ruff] src = ["src", "tests", "spikes"]`. Create `spikes/__init__.py` and `tests/spikes/__init__.py` (docstring: "Throwaway Fase 1 de-risk spikes — dev-only, never shipped in the wheel; safe to delete after findings recorded"). Create `docs/fase1-spikes/README.md` (the four spikes, the resolved version gate, a **Disposal** subsection: `rm -rf spikes tests/spikes docs/fase1-spikes` + revert the three `dev`/`tool` pyproject edits). In `CLAUDE.md` correct the stale stack line `MAF (`agent-framework` 1.8.0)` → `MAF (`agent-framework-core` 1.9.0)` (honors the brief's Spike-A version precondition). Run `uv sync --extra dev`, then re-confirm `uv pip show agent-framework-orchestrations` resolves cleanly (premise re-verification). (new files: all except `pyproject.toml`, `CLAUDE.md`) +- **Reuses:** existing `pyproject.toml` / `CLAUDE.md` structure (Fase 0). +- **Test first:** + - File: `tests/spikes/test_imports.py` (new) + - Verifies: `import spikes`; `from agent_framework.orchestrations import GroupChatBuilder, ConcurrentBuilder, MagenticBuilder, StandardMagenticManager, TerminationCondition`; `import pulp`; `import pytest_asyncio`. + - Pattern: `tests/test_smoke.py` +- **Verify:** `uv run --extra dev pytest tests/spikes/test_imports.py -q` → expected: passed; `uv pip show agent-framework-orchestrations` → `Version: 1.0.0`; `uv run ruff check .` → exit 0 +- **On failure:** retry — if the orchestrations import fails after sync, confirm the lock resolved `agent-framework-orchestrations==1.0.0`; if anything pulls a pre-release, pin tighter. Revert with `git checkout -- pyproject.toml CLAUDE.md uv.lock` if unrecoverable. +- **Checkpoint:** `git commit -m "build(fase1): add dev orchestration + solver + async deps, scaffold spikes"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - pyproject.toml + - CLAUDE.md + - spikes/__init__.py + - tests/spikes/__init__.py + - tests/spikes/test_imports.py + - docs/fase1-spikes/README.md + - uv.lock + min_file_count: 7 + commit_message_pattern: "^build\\(fase1\\): add dev orchestration \\+ solver \\+ async deps" + bash_syntax_check: [] + forbidden_paths: + - src/portfolio_optimiser/backends.py + - src/portfolio_optimiser/reference_domain.py + must_contain: + - path: pyproject.toml + pattern: "agent-framework-orchestrations" + - path: pyproject.toml + pattern: "pytest-asyncio" + - path: CLAUDE.md + pattern: "agent-framework-core` 1.9.0" + ``` + +### Step 2: Shared spike harness + builder smoke (budget caps, fake client, live gate, async builder de-risk) + +- **Files:** `spikes/_harness.py`, `tests/spikes/test_harness.py` +- **Changes:** Implement the cost/stop invariant (B4) and the test seams shared by all spikes. (1) `@dataclass Budget(max_tokens: int, max_rounds: int)` — constructor raises `ValueError` for non-positive caps (refuse to start without caps, A4); a `TokenMeter` plus `charge(tokens)`/`tick_round()` raise `BudgetExceeded(kind, limit, observed)` the moment a cap is crossed. (Two exception types is intentional and documented in the module: `ValueError` = bad construction argument; `BudgetExceeded` = runtime breach.) (2) `FakeChatClient(BaseChatClient)` returning scripted deterministic responses and counting tokens by word-count; plus `fake_agent(client, name, instructions)` building a minimal real `agent_framework.Agent` backed by the fake client so the orchestration builders get genuine participants. (3) `live_local_client_or_skip()` → builds `agent_framework.openai.OpenAIChatClient(base_url=env PORTFOLIO_LOCAL_BASE_URL, model=env PORTFOLIO_LOCAL_MODEL)` **directly**, or `pytest.skip(...)` when env is unset (the D2 `LocalBackend` seam is deliberately not used — its live-wiring is Fase 2). (4) A **builder smoke** that front-loads the biggest risk: construct a `GroupChatBuilder` and a `ConcurrentBuilder` with two `fake_agent`s and `await` a trivial run, asserting they execute — this is the one place that proves `FakeChatClient` can drive the GA builders. If it cannot, that is itself a recorded de-risk finding (escalate), surfaced before any spike is built on the assumption. (new file) +- **Reuses:** `agent_framework.BaseChatClient`, `agent_framework.Agent`; `agent_framework.openai.OpenAIChatClient`; `agent_framework.orchestrations.{GroupChatBuilder, ConcurrentBuilder}`; the D6 cost contract. +- **Test first:** + - File: `tests/spikes/test_harness.py` (new) + - Verifies: `Budget(0, 5)` and `Budget(5, 0)` raise `ValueError`; `charge`/`tick_round` raise `BudgetExceeded` past their caps; `FakeChatClient` returns scripted replies in order and accumulates tokens; `live_local_client_or_skip()` skips when env unset; **builder smoke** (async test) constructs and runs `GroupChatBuilder`+`ConcurrentBuilder` with fake agents and asserts completion. + - Pattern: `tests/test_backends.py` (`parametrize` + `pytest.raises`) +- **Verify:** `uv run --extra dev pytest tests/spikes/test_harness.py -q` → expected: passed; `uv run ruff check spikes tests/spikes` → exit 0 +- **On failure:** escalate — if the builder smoke proves `FakeChatClient` cannot drive the GA builders, STOP and record it in `docs/fase1-spikes/README.md` as a primary de-risk finding (it changes Spikes A/B); decide with the operator whether to back the builders with a tiny canned-LLM stub or restructure. Otherwise revert with `git checkout -- spikes/_harness.py tests/spikes/test_harness.py`. +- **Checkpoint:** `git commit -m "feat(fase1): shared spike harness + builder smoke (budget caps, fake client, live gate)"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - spikes/_harness.py + - tests/spikes/test_harness.py + min_file_count: 2 + commit_message_pattern: "^feat\\(fase1\\): shared spike harness \\+ builder smoke" + bash_syntax_check: [] + forbidden_paths: + - src/portfolio_optimiser/backends.py + must_contain: + - path: spikes/_harness.py + pattern: "class BudgetExceeded" + - path: spikes/_harness.py + pattern: "def live_local_client_or_skip" + - path: tests/spikes/test_harness.py + pattern: "GroupChatBuilder" + ``` + +### Step 3: Spike A — Group Chat maker-checker vs single-agent (U3 / G7) + +- **Files:** `spikes/a_groupchat.py`, `tests/spikes/test_a_groupchat.py`, `docs/fase1-spikes/findings-a.md` +- **Changes:** **Logic layer (always green):** `Arm` result dataclass (rounds, stalls, tokens, caught_flaw); `make_termination(n_rounds=3)` (N fixed at 3 per reviewer refinement #1); `verdict(mc: Arm, single: Arm) -> dict` computing `better = mc.caught_flaw and not single.caught_flaw` and `affordable = mc.tokens <= 3 * single.tokens` and `passed = better and affordable`, plus a comparison table renderer. **Integration/live layer (gated):** `run_live(project)` builds the 3-role maker-checker (proposer · critic · validator) via `GroupChatBuilder` + `make_termination()` and a single-agent baseline, runs both against `load_reference_projects()[0]` carrying a **planted flaw** (a savings candidate violating an obvious constraint, e.g. negative residual quantity), measures real numbers, writes them + the computed verdict to `findings-a.md`. (new file) +- **Reuses:** `spikes._harness` (Budget, FakeChatClient, fake_agent, live gate); `reference_domain.load_reference_projects`; `agent_framework.orchestrations.GroupChatBuilder`/`TerminationCondition`. +- **Test first:** + - File: `tests/spikes/test_a_groupchat.py` (new) + - Verifies (logic, non-tautological — inputs are **varied**, not a single scripted constant): `verdict()` returns `passed=True` when mc caught & single missed & tokens within 3×; `passed=False` when mc tokens = 5× single (affordable=False) even if better; `passed=False` when both caught the flaw (better=False); `make_termination(3)` stops at 3 rounds. A `@pytest.mark.skipif(no endpoint)` live test runs `run_live` and asserts ≤3 rounds, cap respected, a findings table emitted. + - Pattern: `tests/test_reference_domain.py` + `tests/test_backends.py` +- **Verify:** `uv run --extra dev pytest tests/spikes/test_a_groupchat.py -q` → expected: logic tests passed (live test skipped without endpoint) +- **On failure:** retry — confirm `GroupChatBuilder` method names by introspection and adjust `run_live`; the logic layer is independent of the MAF surface. Revert with `git checkout -- spikes/a_groupchat.py tests/spikes/test_a_groupchat.py docs/fase1-spikes/findings-a.md` if unrecoverable. +- **Checkpoint:** `git commit -m "feat(fase1): spike A - group chat maker-checker vs single-agent"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - spikes/a_groupchat.py + - tests/spikes/test_a_groupchat.py + - docs/fase1-spikes/findings-a.md + min_file_count: 3 + commit_message_pattern: "^feat\\(fase1\\): spike A" + bash_syntax_check: [] + forbidden_paths: + - src/portfolio_optimiser/reference_domain.py + must_contain: + - path: spikes/a_groupchat.py + pattern: "GroupChatBuilder" + - path: tests/spikes/test_a_groupchat.py + pattern: "skipif" + - path: docs/fase1-spikes/findings-a.md + pattern: "token" + ``` + +### Step 4: Spike B — Magentic limits=None footgun + ConcurrentBuilder state isolation (G1 / G2) + +- **Files:** `spikes/b_footguns.py`, `tests/spikes/test_b_footguns.py`, `docs/fase1-spikes/findings-b.md` +- **Changes:** Driving the builders **is** the de-risk here, so these tests run in the gate (no live LLM — `FakeChatClient`/`fake_agent`), having been de-risked by the Step 2 builder smoke. (a) **Magentic unbounded (G1):** `MagenticBuilder` + `StandardMagenticManager` with `max_round_count=None` over a fake agent that never finalizes; drive it under the harness **round/iteration guard** (the deterministic assertion — wall-clock is only a secondary safety net, not asserted) and assert it does NOT self-terminate before the guard fires → confirms an explicit limit is required (B4). With explicit limits → terminates cleanly. (b) **Fan-out state isolation (G2/B7):** `fresh_workflow()` factory + a stateful executor; assert a SHARED instance bleeds state across the 3 projects and a FRESH instance per run shows zero bleed (B7). Writes observations + a token-use line ("0 — no live LLM") to `findings-b.md`. (new file) +- **Reuses:** `spikes._harness` (round guard, FakeChatClient, fake_agent); `reference_domain.load_reference_projects`; `agent_framework.orchestrations.{MagenticBuilder, StandardMagenticManager, ConcurrentBuilder}`. +- **Test first:** + - File: `tests/spikes/test_b_footguns.py` (new) + - Verifies: unbounded Magentic hits the round guard without self-terminating (asserts guard fired, not a natural stop); bounded Magentic stops within its limit; shared-instance fan-out leaks state (assertion of bleed); fresh-instance fan-out asserts no bleed across all 3 projects. + - Pattern: `tests/test_backends.py` +- **Verify:** `uv run --extra dev pytest tests/spikes/test_b_footguns.py -q` → expected: passed +- **On failure:** escalate — if `MagenticBuilder`/`ConcurrentBuilder` cannot be driven by a fake agent despite the Step 2 smoke, record it as a genuine de-risk finding in `findings-b.md` (it answers the footgun question by other means) before reverting `git checkout -- spikes/b_footguns.py tests/spikes/test_b_footguns.py docs/fase1-spikes/findings-b.md`. +- **Checkpoint:** `git commit -m "feat(fase1): spike B - magentic unbounded + concurrent state isolation"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - spikes/b_footguns.py + - tests/spikes/test_b_footguns.py + - docs/fase1-spikes/findings-b.md + min_file_count: 3 + commit_message_pattern: "^feat\\(fase1\\): spike B" + bash_syntax_check: [] + forbidden_paths: [] + must_contain: + - path: spikes/b_footguns.py + pattern: "ConcurrentBuilder" + - path: spikes/b_footguns.py + pattern: "fresh_workflow" + - path: docs/fase1-spikes/findings-b.md + pattern: "token" + ``` + +### Step 5: Spike C — blocking hybrid validator (IR→PuLP→Monte Carlo, self-repair) (B1) + +- **Files:** `spikes/c_validator.py`, `tests/spikes/test_c_validator.py`, `docs/fase1-spikes/findings-c.md` +- **Changes:** Fully deterministic — no MAF builders. Pydantic IR `SavingsProposal` (project_id, measure, affected cost codes, claimed saving NOK, assumptions) with field validators + a cross-field `@model_validator` (claimed saving ≤ affected items' total; quantities ≥ 0). `validate_proposal(proposal) -> ValidatedProposal | Rejection`: (1) Pydantic schema validation; (2) **PuLP** feasibility/optimization over the project's cost items (uses PuLP's **bundled CBC** binary — if CBC is genuinely absent the step **escalates**; no silent LP-relaxation fallback); (3) **Monte Carlo** over uncertain unit-costs → P10/P50/P90 (stdlib `random` seeded + `statistics.quantiles`); (4) **structural block**: an out-of-range/infeasible proposal returns a `Rejection` that can never be consumed as a `ValidatedProposal`. `self_repair(generate, max_attempts=N)` capped, then hard-stops. The LLM-generation entry is a thin wrapper gated by `live_local_client_or_skip()`; all validator/solver/MC logic is tested with crafted IR directly. Writes a token-use line ("0 — validator deterministic; live generation gated") + percentiles to `findings-c.md`. (new file) +- **Reuses:** `pydantic.BaseModel`/`model_validator`; `pulp`; `reference_domain.Project`/`CostItem`/`load_reference_projects`; `spikes._harness` (Budget for the self-repair cap, live gate). +- **Test first:** + - File: `tests/spikes/test_c_validator.py` (new) + - Verifies: an out-of-range proposal is structurally blocked — `validate_proposal` returns `Rejection` and raises if treated as validated; a valid proposal returns `ValidatedProposal` with P10 ≤ P50 ≤ P90; Monte Carlo reproducible under the fixed seed; `self_repair` stops at `max_attempts`. + - Pattern: `tests/test_reference_domain.py` + `pytest.raises` +- **Verify:** `uv run --extra dev pytest tests/spikes/test_c_validator.py -q` → expected: passed +- **On failure:** escalate (if CBC absent — record platform finding) / otherwise revert `git checkout -- spikes/c_validator.py tests/spikes/test_c_validator.py docs/fase1-spikes/findings-c.md`. +- **Checkpoint:** `git commit -m "feat(fase1): spike C - blocking hybrid validator (IR/solver/monte-carlo)"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - spikes/c_validator.py + - tests/spikes/test_c_validator.py + - docs/fase1-spikes/findings-c.md + min_file_count: 3 + commit_message_pattern: "^feat\\(fase1\\): spike C" + bash_syntax_check: [] + forbidden_paths: [] + must_contain: + - path: spikes/c_validator.py + pattern: "model_validator" + - path: spikes/c_validator.py + pattern: "class Rejection" + - path: tests/spikes/test_c_validator.py + pattern: "P10" + ``` + +### Step 6: Spike D — VerdictStore + ExpeL retrieval (B2) + +- **Files:** `spikes/d_verdictstore.py`, `tests/spikes/test_d_verdictstore.py`, `docs/fase1-spikes/findings-d.md` +- **Changes:** Minimal in-memory VerdictStore: a list of `Verdict(id, proposal_features, decision, rationale)` seeded with 10–20 synthetic verdicts from the reference domain. **Similarity defined operationally (reviewer refinement #2):** a weighted score over *structured* fields — Jaccard on the affected cost-code set + match on `savings_measure_type` + a magnitude-bucket match on claimed saving — **not** raw description text. `retrieve(proposal, k) -> list[Verdict]` ranks by that score (this is the always-tested unit that satisfies SC-D). A thin custom `ContextProvider` subclass wraps `retrieve()` for ExpeL few-shot injection; its injection is asserted only against the introspected interface (if the abstract surface differs at coding time, the `retrieve()` ranking remains the deliverable and the SC-D assertion). The embedding-based similarity path is **removed** as out of scope for a throwaway spike (noted as a Fase-2 option in `findings-d.md`). Writes a token-use line ("0 — deterministic retrieval") + a top-K example to `findings-d.md`. (new file) +- **Reuses:** `agent_framework.ContextProvider` (top-level); `reference_domain.load_reference_projects`; `spikes._harness`. +- **Test first:** + - File: `tests/spikes/test_d_verdictstore.py` (new) + - Verifies (non-tautological by construction): seed a store where the true match shares the *structured* similarity fields with the query but uses **different description text**, while 2–3 **decoys** share surface description text but differ in the structured fields; assert the true match is top-1 and within top-K; assert deterministic ordering. (ContextProvider-injection assertion included only if the interface is confirmed; `retrieve()` ranking is the guaranteed SC-D check.) + - Pattern: `tests/test_reference_domain.py` +- **Verify:** `uv run --extra dev pytest tests/spikes/test_d_verdictstore.py -q` → expected: passed +- **On failure:** retry — confirm `ContextProvider` abstract methods by introspection; keep `retrieve()` ranking separable. Revert `git checkout -- spikes/d_verdictstore.py tests/spikes/test_d_verdictstore.py docs/fase1-spikes/findings-d.md` if unrecoverable. +- **Checkpoint:** `git commit -m "feat(fase1): spike D - verdictstore + expel retrieval"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - spikes/d_verdictstore.py + - tests/spikes/test_d_verdictstore.py + - docs/fase1-spikes/findings-d.md + min_file_count: 3 + commit_message_pattern: "^feat\\(fase1\\): spike D" + bash_syntax_check: [] + forbidden_paths: [] + must_contain: + - path: spikes/d_verdictstore.py + pattern: "def retrieve" + - path: tests/spikes/test_d_verdictstore.py + pattern: "decoy" + ``` + +### Step 7: Consolidate findings, document disposal, confirm the green quality gate + +- **Files:** `docs/fase1-spikes/README.md` +- **Changes:** Update `docs/fase1-spikes/README.md` to a consolidated summary table (spike → assumption → result → confirmed/refuted → implication for Fase 2), pulling each per-spike note's verdict and token-use line (each `findings-{a,b,c,d}.md` already exists from Steps 3–6). Confirm the **Disposal** subsection (from Step 1) is complete. Run the full quality gate across the whole repo. (no new findings files here — they are owned by their spike steps) +- **Reuses:** the per-spike findings written in Steps 3–6. +- **Test first:** n/a (documentation + gate; the gate IS the verification). +- **Verify:** `uv run ruff check .` → exit 0; `uv run ruff format --check .` → exit 0; `uv run mypy src` → exit 0; `uv run --extra dev pytest -q` → expected: all passed (originals + spike logic layers; live arms skipped without endpoint) +- **On failure:** escalate — a red gate means a prior step regressed; bisect by step and fix before consolidating. +- **Checkpoint:** `git commit -m "docs(fase1): consolidate spike findings + confirm green quality gate"` +- **Manifest:** + ```yaml + manifest: + expected_paths: + - docs/fase1-spikes/README.md + - docs/fase1-spikes/findings-a.md + - docs/fase1-spikes/findings-b.md + - docs/fase1-spikes/findings-c.md + - docs/fase1-spikes/findings-d.md + min_file_count: 5 + commit_message_pattern: "^docs\\(fase1\\): consolidate spike findings" + bash_syntax_check: [] + forbidden_paths: + - src/portfolio_optimiser/backends.py + must_contain: + - path: docs/fase1-spikes/README.md + pattern: "Fase 2" + - path: docs/fase1-spikes/README.md + pattern: "Disposal" + ``` + +### Failure recovery rules + +- **revert** — undo this step's changes (`git checkout -- {files}`), do not proceed. +- **retry** — confirm the real MAF method signature by introspection, adjust, then revert if still failing. +- **escalate** — stop entirely; needs human judgment (Step 2 builder smoke fails / Step 4 builder un-fakeable / Step 5 CBC absent / Step 7 gate regression). For Steps 2 and 4, an escalate that *records the MAF limitation* is itself a valid de-risk outcome. +- **Checkpoint** — commit after each green step. + +## Alternatives Considered + +| Approach | Pros | Cons | Why rejected | +|----------|------|------|--------------| +| Spikes under `src/portfolio_optimiser/spikes/` | Single import root | Ships throwaway code in the wheel | Top-level `spikes/` keeps the wheel clean (hatchling packages only `src/portfolio_optimiser`) | +| `agent-framework-orchestrations` in **core** deps | One home for the locked debate default | Permanent Fase-2 commitment made under a throwaway task (scope creep, flagged in review) | Put it in `dev` like PuLP; promote to core in Fase 2 when the real flow is built | +| Wire `LocalBackend.create_chat_client()` in Step 2 for the live arm | Fills the D2 seam | Touches `src/` under a throwaway task; forces editing `test_backends.py`; non-throwaway commitment | Build `OpenAIChatClient` directly in the throwaway harness; defer seam-wiring to Fase 2; `src/` stays untouched | +| Require a live LOCAL endpoint for all spike tests | Tests the real thing | Gate red without an endpoint; violates SC-gate + D6 | Logic layer always green; integration/live arm gated | +| numpy for Monte Carlo | Familiar | New dep for trivial sampling | stdlib `random` + `statistics.quantiles` — zero new dep | +| `agent-framework[all]` to get orchestrations | One line | Re-introduces the beta/`--prerelease`/alpha-pydantic problem Fase 0 removed (`491a746`) | Pin the GA `agent-framework-orchestrations>=1.0.0` only | +| Embedding similarity (Spike D) | "Real" RAG | Needs live endpoint; non-deterministic; serves no SC; over-builds throwaway | Deterministic structured-field similarity; embeddings = Fase-2 note only | +| Hand-rolled feasibility instead of PuLP | No dep | Not a faithful solver-in-the-loop (B1/R2) | PuLP (bundled CBC) keeps the spike honest | + +## Test Strategy + +- **Framework:** pytest 9.1.1 + `pytest-asyncio` (`asyncio_mode = "auto"`); plain functions with `-> None`, `parametrize`, `pytest.raises`; fixtures from `load_reference_projects()`. Async tests for the MAF-builder integration. +- **Existing patterns:** `tests/test_backends.py`, `tests/test_reference_domain.py`, `tests/test_smoke.py`. +- **New tests in this plan:** 6 modules (imports, harness+builder-smoke, A, B, C, D). Determinism via `FakeChatClient` + fixed seeds; live arms isolated behind `@pytest.mark.skipif(no endpoint)`. Spike B drives the builders in-gate (no endpoint) — de-risked by the Step 2 smoke. + +### Tests to write + +| Type | File | Verifies | Model test | +|------|------|----------|------------| +| Import | `tests/spikes/test_imports.py` | orchestrations + pulp + pytest-asyncio + `spikes` import | `tests/test_smoke.py` | +| Unit+async | `tests/spikes/test_harness.py` | Budget caps; FakeChatClient; skip gate; builder smoke | `tests/test_backends.py` | +| Unit | `tests/spikes/test_a_groupchat.py` | varied-input verdict logic; ≤3-round termination; gated live arm | `tests/test_reference_domain.py` | +| Unit+async | `tests/spikes/test_b_footguns.py` | unbounded Magentic needs guard; zero fan-out bleed | `tests/test_backends.py` | +| Unit | `tests/spikes/test_c_validator.py` | structural block; P10≤P50≤P90; self-repair cap | `tests/test_reference_domain.py` | +| Unit | `tests/spikes/test_d_verdictstore.py` | non-tautological top-K retrieval | `tests/test_reference_domain.py` | + +## Risks and Mitigations + +| Priority | Risk | Location | Impact | Mitigation | +|----------|------|----------|--------|------------| +| High | `FakeChatClient` cannot drive the async GA builders | `spikes/_harness.py` (smoke), A/B | Spikes A/B integration blocked | Step 2 builder smoke front-loads this; failure is a recorded de-risk finding (escalate), not a late surprise; logic layers (A verdict, B isolation) stay green regardless | +| High | Real builder method signatures differ from planned wiring | A/B integration | Integration code won't run as written | Confirm by introspection at coding time (brief-sanctioned); per-step retry; logic layers independent of the MAF surface | +| Medium | No LOCAL endpoint during the session | A/C live arms | Live numbers not captured | Logic layers prove the machinery; live arms `skip`; findings note records "live arm not run" honestly | +| Medium | PuLP CBC binary missing on Intel mac | `spikes/c_validator.py` | Solver call fails | PuLP wheels bundle CBC; if genuinely absent → **escalate** (recorded), no silent fallback | +| Low | Async-mode misconfig breaks existing sync tests | `pyproject.toml` | Original 12 tests affected | `asyncio_mode="auto"` only auto-marks coroutine tests; sync tests unaffected — Step 7 gate catches any regression | +| Low | 2 `ExperimentalWarning`s (SKILLS/HARNESS) at import (G8) | all imports | Noise only | Harmless; do not enable `filterwarnings=error` | +| Low | `.claude/projects/**` committed to a to-be-public repo | repo | Planning scratch in history | Out of scope here; flag at Fase 4 release prep | + +## Assumptions + +| # | Assumption | Why unverifiable now | Impact if wrong | +|---|-----------|-----------------|-----------------| +| 1 | LOCAL profile = reachable OpenAI-compatible endpoint via `PORTFOLIO_LOCAL_*` env | Depends on operator's running Ollama/LM Studio | Live arms skip; logic layers still pass — low impact | +| 2 | `FakeChatClient` can drive the GA builders | Not yet installed/introspected to method level | Step 2 smoke decides early; failure is a documented finding, not silent breakage | +| 3 | PuLP's bundled CBC works on the Intel-mac venv | Not installed yet | Step 5 escalates with a recorded platform finding | +| 4 | N = 3 rounds is a fair maker-checker bound | Judgment from G7 token-discipline | If too tight to converge live, bump to ≤5 and re-record (documented range) | + +## Verification + +*Per-step manifests are checked automatically by trekexecute. These map to the brief Success Criteria.* + +- [ ] Spike A: `uv run --extra dev pytest tests/spikes/test_a_groupchat.py -q` → varied-input verdict logic passes; gated live arm (when run) converges ≤3 rounds, cap respected, comparison table emitted (SC-A; empirical better/cheaper verdict is endpoint-dependent and so reported) +- [ ] Spike B: `uv run --extra dev pytest tests/spikes/test_b_footguns.py -q` → unbounded Magentic needs an external guard; fresh-instance fan-out zero state-bleed (SC-B) +- [ ] Spike C: `uv run --extra dev pytest tests/spikes/test_c_validator.py -q` → out-of-range blocked structurally; valid proposal yields P10/P50/P90; self-repair capped (SC-C) +- [ ] Spike D: `uv run --extra dev pytest tests/spikes/test_d_verdictstore.py -q` → top-K retrieval returns the structurally-similar verdict over surface-text decoys (SC-D) +- [ ] Gate: `uv run ruff check .` exit 0; `uv run ruff format --check .` clean; `uv run mypy src` exit 0; `uv run --extra dev pytest -q` all passed (SC-gate) +- [ ] Findings: `docs/fase1-spikes/` has README (summary table + Disposal) + findings-{a,b,c,d}.md, each with a confirmed/refuted verdict and a token-use line (brief Goal + NFR) + +## Estimated Scope + +- **Files to modify:** 2 (`pyproject.toml`, `CLAUDE.md`; `uv.lock` regenerated) +- **Files to create:** ~19 (4 spike modules + `_harness.py` + 6 test modules + 2 `__init__.py` + 5 docs notes + README owned by Step 1/7) +- **Complexity:** medium (logic is modest; the real unknown — MAF builder ergonomics with a fake client — is de-risked up-front by the Step 2 smoke) + +## Execution Strategy + +### Session 1: Foundation +- **Steps:** 1, 2 +- **Wave:** 1 +- **Depends on:** none +- **Scope fence:** + - Touch: `pyproject.toml`, `CLAUDE.md`, `spikes/__init__.py`, `spikes/_harness.py`, `tests/spikes/{__init__,test_imports,test_harness}.py`, `docs/fase1-spikes/README.md`, `uv.lock` + - Never touch: `src/**`, the four spike modules + +### Session 2: Spike A +- **Steps:** 3 +- **Wave:** 2 +- **Depends on:** Session 1 +- **Scope fence:** + - Touch: `spikes/a_groupchat.py`, `tests/spikes/test_a_groupchat.py`, `docs/fase1-spikes/findings-a.md` + - Never touch: other spike modules, `src/**`, `pyproject.toml`, `CLAUDE.md` + +### Session 3: Spike B +- **Steps:** 4 +- **Wave:** 2 +- **Depends on:** Session 1 +- **Scope fence:** + - Touch: `spikes/b_footguns.py`, `tests/spikes/test_b_footguns.py`, `docs/fase1-spikes/findings-b.md` + - Never touch: other spike modules, `src/**`, `pyproject.toml`, `CLAUDE.md` + +### Session 4: Spike C +- **Steps:** 5 +- **Wave:** 2 +- **Depends on:** Session 1 +- **Scope fence:** + - Touch: `spikes/c_validator.py`, `tests/spikes/test_c_validator.py`, `docs/fase1-spikes/findings-c.md` + - Never touch: other spike modules, `src/**`, `pyproject.toml`, `CLAUDE.md` + +### Session 5: Spike D +- **Steps:** 6 +- **Wave:** 2 +- **Depends on:** Session 1 +- **Scope fence:** + - Touch: `spikes/d_verdictstore.py`, `tests/spikes/test_d_verdictstore.py`, `docs/fase1-spikes/findings-d.md` + - Never touch: other spike modules, `src/**`, `pyproject.toml`, `CLAUDE.md` + +### Session 6: Consolidation +- **Steps:** 7 +- **Wave:** 3 +- **Depends on:** Sessions 2–5 +- **Scope fence:** + - Touch: `docs/fase1-spikes/README.md` + - Never touch: `spikes/**`, `src/**`, `pyproject.toml`, `CLAUDE.md` + +### Execution Order + +- **Wave 1:** Session 1 (foundation — includes the builder smoke that de-risks all spikes) +- **Wave 2:** Sessions 2, 3, 4, 5 (parallel — independent spike modules sharing only the read-only harness) +- **Wave 3:** Session 6 (after Wave 2) + +### Grouping rules applied + +- Steps sharing files → same session (1+2 = foundation). +- Independent spike modules → separate sessions (parallelizable in Wave 2). +- Consolidation depends on all spikes → Wave 3. + +## Plan Quality Score + +| Dimension | Weight | Score | Notes | +|-----------|--------|-------|-------| +| Structural integrity | 0.15 | 90 | Foundation→spikes→consolidation; src/ untouched so scope fences are accurate | +| Step quality | 0.20 | 88 | TDD-first; false LocalBackend reuse removed; findings owned per step | +| Coverage completeness | 0.20 | 90 | SC-A/B/C/D + gate + findings mapped; A's empirical verdict honestly endpoint-dependent | +| Specification quality | 0.15 | 86 | Real import paths; builder ergonomics de-risked by Step 2 smoke; CBC fallback decided | +| Risk & pre-mortem | 0.15 | 90 | Biggest risk (fake-drives-builders) front-loaded with a green-keeping logic layer | +| Headless readiness | 0.10 | 92 | On-failure + Checkpoint + Manifest per step; expected_paths ⊆ Files | +| Manifest quality | 0.05 | 90 | All manifests checkable; orphan paths fixed | +| **Weighted total** | **1.00** | **89** | **Grade: A-** | + +**Adversarial review:** +- **Plan critic:** REVISE → addressed (2 blockers + 8 major + 4 minor); see Revisions. +- **Scope guardian:** MIXED → addressed (orchestrations moved to dev; embedding path dropped; token-reporting + findings-location ratified); see Revisions. + +## Revisions + +*Added after Phase 9 adversarial review (plan-critic + scope-guardian, deduped: 19 findings, 0 overlap).* + +| # | Finding (rule_key) | Severity | Resolution | +|---|--------------------|----------|------------| +| 1 | live-arm/impossible-localbackend-wiring | blocker | Harness builds `OpenAIChatClient` directly; dropped the false `LocalBackend` reuse; D2 seam-wiring deferred to Fase 2; `src/` untouched | +| 2 | fragile-assumption/fakeclient-drives-async-builders | blocker | Added Step 2 builder smoke (front-loads the risk); added `pytest-asyncio` + `asyncio_mode=auto`; split each spike into always-green logic layer + gated integration/live layer; failure is a recorded escalate | +| 3 | structural/scope-fence-vs-live-arm | major | Resolved by #1 — `src/**` is now genuinely never touched; fences accurate | +| 4 | missing-setup/async-test-runner | major | `pytest-asyncio>=0.24` added (Step 1); async integration tests | +| 5 | verification/spike-a-verdict-tautological | major | Logic test now varies inputs to prove the verdict function; empirical U3/G7 verdict explicitly endpoint-dependent (live arm) | +| 6 | underspecified/contextprovider-inject-no-target | major | `retrieve()` ranking is the guaranteed SC-D unit; ContextProvider injection asserted only against the introspected interface | +| 7 | manifest/expected-paths-not-subset-of-files | major | All `Files:` lines now include test/findings/`uv.lock`/`CLAUDE.md` paths | +| 8 | ordering/findings-files-creation-ambiguous | major | Each spike step (3–6) owns and creates its `findings-X.md`; Step 7 only consolidates the README | +| 9 | missing-error-handling/cbc-fallback-deferred | major | Committed to PuLP's bundled CBC; absence → escalate (recorded), no silent LP-relaxation | +| 10 | coverage/spike-a-precondition-downgraded | major | Step 1 corrects `CLAUDE.md` version; installed version confirmed 1.9.0; memory fix noted as session follow-up | +| 11 | creep — orchestrations in core deps | major | Moved `agent-framework-orchestrations` to `dev` (throwaway scope), like PuLP | +| 12 | creep — Spike D embedding path | minor | Removed; deterministic similarity only; embeddings = Fase-2 note | +| 13 | gap — token reporting B/C/D | minor | Each findings note carries a token-use line (incl. "0 — no live LLM") | +| 14 | gap — findings location vs brief | minor | Kept `docs/fase1-spikes/` (tracked, survives, consistent with research/plan) — ratified as a deliberate refinement; operator can object | +| 15 | dependency — re-confirm orchestrations resolves | minor | Step 1 runs `uv pip show agent-framework-orchestrations` post-sync | +| 16 | minor/pythonpath-root-shadowing | minor | Documented: repo-root added for `import spikes`; no name collision with installed dists | +| 17 | minor/budget-two-exception-types | minor | Documented intent: `ValueError` = bad ctor arg, `BudgetExceeded` = runtime breach | +| 18 | minor/wall-clock-flaky-assertion | minor | Spike B asserts the round/iteration guard; wall-clock is a secondary safety net only | +| 19 | minor/no-teardown-for-throwaway-spikes | minor | Step 1/7 README **Disposal** subsection: `rm -rf` + revert the pyproject dev/tool edits |