Kjell Tore Guttormsen
9973d9fe58
14-step plan composing the four Fase 1 spikes into a src/ vertical slice (debate -> blocking validator -> two-layer HITL + provenance -> ExpeL learning), deterministic-core-first, on real chat clients in both profiles. Grounded in the 3 research briefs + installed-1.9.0-source introspection (7 exploration agents). Adversarial review: plan-critic REVISE (3 blockers/7 major/4 minor) -> all addressed; scope-guardian ALIGNED (0 creep, 9/9 criteria mapped, 6/6 Non-Goals honored). Key revisions: in-process retriever-as-tool MVP path (mcp dep conditional/GA-only); two-layer HITL capture + stable Verdict.id minting; extend_instructions retired by a REAL SessionContext test; TextSpan ownership + wave re-ordering; budget None-as-hard-fail with synthetic-usage test double; self_repair token-bound in the generate loop (validator.py frozen); global stop-on-failure rule. gemini-bridge Pass 2 unavailable (MCP SDK broke). plan-validator strict 0 errors. brief research_status pending to complete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
686 lines
53 KiB
Markdown
686 lines
53 KiB
Markdown
# Fase 2 — MVP Vertical Slice: one synthetic project end-to-end on MAF 1.9.0
|
||
|
||
> **Plan quality: A−** (88/100) — APPROVE_WITH_NOTES (revised after adversarial review)
|
||
>
|
||
> Generated by trekplan v2.0 on 2026-06-24 — `plan_version: 1.7`
|
||
|
||
## Context
|
||
|
||
Fase 1 proved the four riskiest assumptions in isolation with throwaway spikes
|
||
(maker-checker convergence, fan-out state-bleed, the hybrid blocking validator,
|
||
ExpeL retrieval) and produced a verified MAF 1.9.0 capability map. Fase 2 builds
|
||
the first thing that is actually a *system*: ONE synthetic "anleggskostnad"
|
||
project flowing through the whole method — candidate measures debated (Group Chat
|
||
maker-checker), a deterministic validator deciding the value, a domain expert
|
||
judging via HITL, and the judgment fed back so the next run is smarter — on
|
||
**real** (not faked) chat clients, in both profiles. The danger being bought down
|
||
is **integration risk**: each spike worked alone; the open question is whether they
|
||
compose into one fail-fast, budget-bounded, provenance-stamped, learning loop. This
|
||
slice is the MVP backbone every later phase builds on (Fase 3 fan-out, Fase 4
|
||
open-source, the D7 Claude-SDK sibling), so getting the seams right matters more
|
||
than breadth (brief §Intent).
|
||
|
||
Three research briefs (installed-1.9.0-source-verified) correct the brief's earlier
|
||
leanings and are binding on this plan:
|
||
- **HITL (research 01):** capture the durable learned verdict **out-of-band** in the
|
||
VerdictStore; use the GA in-run gate (`GroupChatBuilder.with_request_info`) only
|
||
for optional synchronous Layer-1 review; **defer checkpointing off the MVP path**.
|
||
- **Data access (research 02):** the official `@modelcontextprotocol/server-filesystem`
|
||
cannot carry citations → the data source is a **framework-agnostic in-process
|
||
retriever** returning `{file, locator, snippet, score}` (the D7-portable seam),
|
||
exposed to the agents as a GA tool/ContextProvider; provenance is first-class
|
||
Pydantic data, not MAF's `Annotation` (open streaming-drop bug #4316).
|
||
- **Local client (research 03):** use **`OpenAIChatCompletionClient(base_url=…)`
|
||
non-streaming** (NOT `OpenAIChatClient`/Responses); `UsageDetails` is populated
|
||
None-safely non-streaming; the blocking validator becomes the **retry/repair**
|
||
reliability mechanism for weak local models.
|
||
|
||
## Architecture Diagram
|
||
|
||
```mermaid
|
||
graph TD
|
||
subgraph "src/portfolio_optimiser/ — Fase 2 vertical slice"
|
||
RUN[run.py orchestrator<br/>single-command entry] --> CONTRACTS[contracts.py<br/>fail-fast loaders]
|
||
RUN --> WF[workflow.py<br/>fresh_workflow + GroupChat maker-checker]
|
||
RUN --> CAP[verdicts.py<br/>capture_verdict Layer-2 + VerdictStore + ExpeL]
|
||
WF --> BK[backends.py<br/>LocalBackend OpenAIChatCompletionClient / FoundryChatClient + model-map]
|
||
WF --> BUD[budget.py<br/>TokenMeter + ChatMiddleware off UsageDetails]
|
||
WF --> TOOL[retrieval.py exposed as GA tool/ContextProvider]
|
||
TOOL --> RET[retrieval.py<br/>in-process retriever core + path-security - D7 seam]
|
||
WF --> GEN[generate.py<br/>LLM->IR + validator-as-retry]
|
||
GEN --> VAL[validator.py + ir.py<br/>blocking: IR->CBC->Monte-Carlo]
|
||
VAL --> PROV[provenance.py<br/>first-class Pydantic stamp]
|
||
PROV --> CAP
|
||
CAP -.next run.-> WF
|
||
end
|
||
REF[reference_domain.py + data/*.json<br/>Fase 0, settled] --> RUN
|
||
```
|
||
|
||
## Codebase Analysis
|
||
|
||
- **Tech stack:** Python ≥3.10, `uv`, hatchling, `ruff` (line-length 100), `mypy`,
|
||
`pytest`+`pytest-asyncio` (`asyncio_mode="auto"`). Installed (ground-truth-verified):
|
||
`agent-framework-core` 1.9.0, `-openai` 1.8.2, `-foundry` 1.8.2, `-orchestrations`
|
||
1.0.0; underlying `openai` SDK **2.43.0**; `pulp` **3.3.2**; `pydantic` 2.13.4. CBC
|
||
binary present (`.venv/.../pulp/solverdir/cbc/osx/i64/cbc`). `mcp` SDK is **NOT
|
||
installed** (governs Step 7's build-vs-defer decision).
|
||
- **Key patterns:** backend profiles = Strategy+Factory (`backends.py:26-84`,
|
||
`ChatBackend` `@runtime_checkable` Protocol + `Profile` enum + `get_backend()`,
|
||
fail-fast `ValueError`); validator = staged pipeline returning a type-discriminated
|
||
`ValidatedProposal | Rejection` (frozen dataclasses); debate = `GroupChatBuilder`
|
||
round-robin with deterministic termination; learning = retrieval + `ContextProvider`
|
||
injection. Tests: `tests/` (production) + `tests/spikes/` (mirror); gated live arms
|
||
via `@pytest.mark.skipif(_NO_ENDPOINT)`; determinism asserted over fixed inputs.
|
||
- **Relevant files (verified):** `src/portfolio_optimiser/backends.py` (D2 seam, both
|
||
`create_chat_client` raise `NotImplementedError` `:50-61,64-76`),
|
||
`reference_domain.py` (`Project`/`CostItem` frozen + `load_reference_projects()`),
|
||
`data/reference_projects.json` (3 projects, project[0]=`FV42-GSV-E1`). Real
|
||
injection seam for tests: `SessionContext` (exported, `_sessions.py:154`;
|
||
`extend_instructions(source_id, instructions)` two-arg `:253`).
|
||
- **Reusable code (promote from spikes — REUSE near-verbatim unless noted):**
|
||
`spikes/c_validator.py:58-205` (Pydantic IR + validate_proposal + CBC + Monte-Carlo
|
||
seed `_MC_SEED=20260624` + `self_repair` attempts-bounded + `proposal_for`;
|
||
`generate_via_llm` `:207` is the stub); `spikes/d_verdictstore.py:29-227`
|
||
(ProposalFeatures/Verdict/similarity/VerdictStore/`ExpeLContextProvider`/seed_store —
|
||
**bug `:115`** one-arg `extend_instructions`, masked by single-arg fake
|
||
`test_d_verdictstore.py:99`); `spikes/b_footguns.py:100-106` (`fresh_workflow()`);
|
||
`spikes/a_groupchat.py:48-63,130-163` (termination + maker-checker builder);
|
||
`spikes/_harness.py:41-94` (Budget/TokenMeter — drop `_word_tokens` `:97-99`).
|
||
- **External tech (researched):** see Research Sources.
|
||
- **Recent git activity:** `src/` written in Fase 0 (`b57aa83`), cold/settled. Spike
|
||
cluster most-recently churned by the round-2 fan-out rebuild (`a2dff21`).
|
||
Conventions: Conventional Commits, `feat(fase2):`, `[skip-docs]` on doc commits.
|
||
|
||
## Research Sources
|
||
|
||
| Technology | Source | Key Findings | Confidence |
|
||
|-----------|--------|--------------|------------|
|
||
| MAF native HITL | research/01 | `ctx.request_info`/`@response_handler`/`run(responses=)`; `GroupChatBuilder.with_request_info(agents=[checker])`; durable checkpoint-resume fragile → out-of-band verdict, defer checkpointing | high |
|
||
| Local-folder MCP + citations | research/02 | official filesystem server can't cite → in-process retriever (D7 seam) + provenance as own Pydantic (route around #4316); `SessionContext.extend_instructions(source_id,…)` confirmed | high |
|
||
| Local chat client + UsageDetails | research/03 | `OpenAIChatCompletionClient(base_url)` non-streaming; `UsageDetails` None-safe (`_chat_completion_client.py:705,757`); `add_usage_details()`; qwen3:4b; Intel-CPU = plumbing only | high |
|
||
| Solver | Spike C + installed | `pulp` 3.3.2, bundled CBC works Intel mac; PULP_CBC_CMD deprecation; PuLP 4.0 → `pulp[cbc]`/`COIN_CMD` | high |
|
||
|
||
## Implementation Plan
|
||
|
||
Deterministic-core-first ordering (MAF-free modules before MAF-coupled wiring). TDD
|
||
throughout. **Dependency note:** `TextSpan` is owned by `retrieval.py` (Step 5) and
|
||
imported by `provenance.py` (Step 6) — so Step 6 follows Step 5. **Headless halt
|
||
rule (applies to every step): on any failure, perform the step's On-failure action
|
||
then HALT the session — never run a dependent step on a failed predecessor.**
|
||
|
||
### Step 1: Promote orchestrations + pulp to core; resolve the mcp dependency decision
|
||
|
||
- **Files:** `pyproject.toml`
|
||
- **Changes:** Move `agent-framework-orchestrations>=1.0.0` and `pulp>=2.8` from `dev` into `[project.dependencies]` (now MVP-runtime). Update the pulp comment to note installed 3.3.2 + the PuLP-4.0 `pulp[cbc]`/`COIN_CMD` migration. **mcp decision:** attempt `uv add mcp` — if it resolves a GA (non-pre) release, add it to core deps (enables the optional thin MCP-server wrapper in Step 7); if it resolves only as a pre-release, DO NOT add it (GA-pin discipline) and record that the MVP data source is the in-process retriever-as-tool (research 02 fallback), with the MCP-server wrapper deferred as a documented fast-follow. Re-lock. (Per brief Constraints.)
|
||
- **Reuses:** existing dependency block `pyproject.toml:7-26`.
|
||
- **Test first:**
|
||
- File: `tests/test_imports_core.py` (new)
|
||
- Verifies: `agent_framework_orchestrations` + `pulp` import; `PULP_CBC_CMD().available()` is True.
|
||
- Pattern: `tests/spikes/test_imports.py`
|
||
- **Verify:** `uv lock && uv sync && uv run pytest tests/test_imports_core.py` → expected: `1 passed`
|
||
- **On failure:** retry once pinning exact installed versions; if still failing, escalate — then HALT.
|
||
- **Checkpoint:** `git commit -m "build(fase2): promote orchestrations + pulp to core deps"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- pyproject.toml
|
||
- tests/test_imports_core.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^build\\(fase2\\): promote orchestrations \\+ pulp to core deps$"
|
||
bash_syntax_check: []
|
||
forbidden_paths:
|
||
- src/portfolio_optimiser/backends.py
|
||
must_contain:
|
||
- path: pyproject.toml
|
||
pattern: "agent-framework-orchestrations"
|
||
```
|
||
|
||
### Step 2: Promote the Pydantic IR + blocking validator to src/
|
||
|
||
- **Files:** `src/portfolio_optimiser/ir.py` (new), `src/portfolio_optimiser/validator.py` (new)
|
||
- **Changes:** Move `AffectedItem`/`SavingsProposal` (+ validators) into `ir.py`. Move `validate_proposal`, `ValidatedProposal`, `Rejection`, `_solve_max_feasible`, `_monte_carlo` (keep `_MC_SEED=20260624`), `self_repair` (attempts-bounded — promoted verbatim; token-budget bounding is added by the generate loop in Step 10, NOT here), `proposal_for`, `CbcUnavailable`, `_quiet_pulp` into `validator.py`. Pure modules — NO `agent_framework` import (D7-portable core). Import `Project` from `reference_domain`. (new files)
|
||
- **Reuses:** `spikes/c_validator.py:44-205`; `reference_domain.Project`.
|
||
- **Test first:**
|
||
- File: `tests/test_validator.py` (new)
|
||
- Verifies: valid IR → `ValidatedProposal` (`p10<=p50<=p90`); out-of-range → `Rejection` (reason, no percentiles); same fixed input twice → identical `(p10,p50,p90)`; Pydantic blocks negative qty / claim>affected-total.
|
||
- Pattern: `tests/spikes/test_c_validator.py:24-78` (determinism over FIXED inputs, never an LLM)
|
||
- **Verify:** `uv run pytest tests/test_validator.py` → expected: `5 passed`
|
||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/ir.py src/portfolio_optimiser/validator.py tests/test_validator.py`; HALT.
|
||
- **Checkpoint:** `git commit -m "feat(fase2): promote blocking validator + IR to src"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/ir.py
|
||
- src/portfolio_optimiser/validator.py
|
||
- tests/test_validator.py
|
||
min_file_count: 3
|
||
commit_message_pattern: "^feat\\(fase2\\): promote blocking validator \\+ IR to src$"
|
||
bash_syntax_check: []
|
||
forbidden_paths: []
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/validator.py
|
||
pattern: "class Rejection"
|
||
- path: src/portfolio_optimiser/validator.py
|
||
pattern: "_MC_SEED"
|
||
```
|
||
|
||
### Step 3: Promote VerdictStore + ExpeL + verdict capture (two-arg fix; real-SessionContext test)
|
||
|
||
- **Files:** `src/portfolio_optimiser/verdicts.py` (new)
|
||
- **Changes:** Move `ProposalFeatures`, `Verdict`, `similarity`, `VerdictStore` (in-memory, deterministic top-k retrieve), `ExpeLContextProvider(ContextProvider)`, `seed_store()`. **Fix the Fase 1 bug:** `before_run` must call `context.extend_instructions(self.source_id, [self.format_fewshot()])` — TWO args (`_sessions.py:253`). Add **`Verdict.id` minting** = stable content hash `sha256(canonical(ProposalFeatures))[:16]` (so a structurally identical proposal maps to a stable id) and **`capture_verdict(features, decision, rationale) -> Verdict`** (the Layer-2 out-of-band constructor). VerdictStore is **in-memory only for the MVP** — no JSON/file persistence (durable persistence deferred to Fase 3; the brief's learning-loop criterion is satisfied within a session via `seed_store()` + in-memory state). (new file)
|
||
- **Reuses:** `spikes/d_verdictstore.py:29-227` (with the `:115` two-arg correction); `agent_framework.ContextProvider`, `agent_framework.SessionContext`.
|
||
- **Test first:**
|
||
- File: `tests/test_verdicts.py` (new)
|
||
- Verifies: structural match beats text decoys (`hits[0].id=="TRUE"`); retrieval deterministic; `k<=0` raises; `capture_verdict` mints a stable id (same features → same id); **`before_run` populates a REAL `SessionContext` via two-arg `extend_instructions(source_id, instructions)`** — construct `from agent_framework import SessionContext`, call `await provider.before_run(...)` with it, assert the retrieved verdict text landed in `ctx.instructions` (NOT a single-arg fake — this exercises the genuine GA signature and retires the Critical risk).
|
||
- Pattern: `tests/spikes/test_d_verdictstore.py:67-89` (retrieval) + real `SessionContext`
|
||
- **Verify:** `uv run pytest tests/test_verdicts.py` → expected: `5 passed`
|
||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/verdicts.py tests/test_verdicts.py`; HALT.
|
||
- **Checkpoint:** `git commit -m "feat(fase2): VerdictStore + ExpeL + capture_verdict (two-arg, real SessionContext test)"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/verdicts.py
|
||
- tests/test_verdicts.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(fase2\\): VerdictStore \\+ ExpeL \\+ capture_verdict \\(two-arg, real SessionContext test\\)$"
|
||
bash_syntax_check: []
|
||
forbidden_paths: []
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/verdicts.py
|
||
pattern: "extend_instructions\\(self.source_id"
|
||
- path: src/portfolio_optimiser/verdicts.py
|
||
pattern: "def capture_verdict"
|
||
```
|
||
|
||
### Step 4: Token meter + budget ChatMiddleware off real UsageDetails
|
||
|
||
- **Files:** `src/portfolio_optimiser/budget.py` (new)
|
||
- **Changes:** Promote `Budget`/`BudgetExceeded`/`TokenMeter` (fail-fast positive caps, structured `BudgetExceeded.kind/limit/observed`) fed from real `UsageDetails`: a `ChatMiddleware` reads `response.usage_details["total_token_count"]` (None-safe via `add_usage_details`) and short-circuits when the cap is crossed. **`usage_details is None` handling:** a `strict_usage: bool` flag (default True in dev/prod) makes a `None` usage a HARD FAIL (a usage regression must never silently disable the cap, research 03 Rec 3); test doubles that legitimately supply synthetic usage do not trip it. NO `len(...split())` proxy. (new file)
|
||
- **Reuses:** `spikes/_harness.py:41-94` (drop `_word_tokens`); `agent_framework.UsageDetails`/`add_usage_details` (`_types.py:417`), `agent_framework.ChatMiddleware`.
|
||
- **Test first:**
|
||
- File: `tests/test_budget.py` (new)
|
||
- Verifies (hand-built `UsageDetails`, no LLM): meter reads `total_token_count` + accumulates; cap crossed → `BudgetExceeded`; non-positive cap rejected; with `strict_usage=True`, `usage_details=None` raises. Meta-guard: `grep -rE "\.split\(\)" src/` finds zero token proxies.
|
||
- Pattern: `tests/spikes/test_harness.py:27-60`
|
||
- **Verify:** `uv run pytest tests/test_budget.py` → expected: `5 passed`
|
||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/budget.py tests/test_budget.py`; HALT.
|
||
- **Checkpoint:** `git commit -m "feat(fase2): real-UsageDetails token meter + budget middleware"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/budget.py
|
||
- tests/test_budget.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(fase2\\): real-UsageDetails token meter \\+ budget middleware$"
|
||
bash_syntax_check: []
|
||
forbidden_paths: []
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/budget.py
|
||
pattern: "usage_details"
|
||
```
|
||
|
||
### Step 5: Framework-agnostic local-folder retriever core (owns TextSpan + path-security)
|
||
|
||
- **Files:** `src/portfolio_optimiser/retrieval.py` (new)
|
||
- **Changes:** Define `TextSpan` (`start_index`,`end_index` — the single owner; imported by provenance in Step 6), `RetrievedChunk` (`file`, `locator: TextSpan`, `snippet`, `score`), and `retrieve(query, docs_dir, top_k) -> list[RetrievedChunk]`. Chunk-at-ingest with EXACT char-span locators (citations exact by construction). Keyword/substring scoring (no heavy vector deps — D5/D6). **Path-security (this module reads the docs folder):** canonicalise + `os.path.realpath` + boundary-check every accessed path against `docs_dir`, fail closed on prefix-collision and symlink-escape (the EscapeRoute CVE class, research 02 Dim 5); native file APIs only. NO `agent_framework`/`mcp` imports — D7-portable seam. (new file)
|
||
- **Reuses:** stdlib only; `TextSpan` aligns with `provenance.Citation.locator` (Step 6).
|
||
- **Test first:**
|
||
- File: `tests/test_retrieval.py` (new)
|
||
- Verifies: `retrieve` returns chunks whose `locator` char-span exactly slices `snippet`; deterministic for fixed query+dir; top_k respected; **a `../` traversal and a symlink pointing outside `docs_dir` are both rejected (fail-closed); a sibling dir sharing a name prefix is rejected**.
|
||
- Pattern: `tests/test_reference_domain.py:8-38` + `tests/test_backends.py:30-32` (raises)
|
||
- **Verify:** `uv run pytest tests/test_retrieval.py` → expected: `7 passed`
|
||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/retrieval.py tests/test_retrieval.py`; HALT.
|
||
- **Checkpoint:** `git commit -m "feat(fase2): local-folder retriever core with path-security"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/retrieval.py
|
||
- tests/test_retrieval.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(fase2\\): local-folder retriever core with path-security$"
|
||
bash_syntax_check: []
|
||
forbidden_paths: []
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/retrieval.py
|
||
pattern: "class RetrievedChunk"
|
||
- path: src/portfolio_optimiser/retrieval.py
|
||
pattern: "realpath"
|
||
```
|
||
|
||
### Step 6: First-class Pydantic provenance stamp
|
||
|
||
- **Files:** `src/portfolio_optimiser/provenance.py` (new)
|
||
- **Changes:** Define `Citation` (`file`, `locator: TextSpan` imported from `retrieval`, `snippet`) and `ProvenanceStamp` (`citations: list[Citation]` `min_length=1`, `model`, `role`, `validator_decision: Literal["validated","rejected"]`, `token_usage: int`). Authoritative provenance — independent of MAF `Annotation` (#4316). Provide `to_annotations()` mapping to MAF `Annotation(type="citation", …)` for display only. (new file; imports `TextSpan` from Step 5)
|
||
- **Reuses:** `retrieval.TextSpan` (Step 5); `agent_framework._types.Annotation`/`TextSpanRegion` (display adapter); `pydantic.BaseModel`.
|
||
- **Test first:**
|
||
- File: `tests/test_provenance.py` (new)
|
||
- Verifies: 0 citations → `ValidationError`; valid stamp exposes ≥1 citation + model/role + decision + token usage; `to_annotations()` yields `type=="citation"` dicts.
|
||
- Pattern: `tests/spikes/test_c_validator.py:66-78`
|
||
- **Verify:** `uv run pytest tests/test_provenance.py` → expected: `3 passed`
|
||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/provenance.py tests/test_provenance.py`; HALT.
|
||
- **Checkpoint:** `git commit -m "feat(fase2): first-class Pydantic provenance stamp"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/provenance.py
|
||
- tests/test_provenance.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(fase2\\): first-class Pydantic provenance stamp$"
|
||
bash_syntax_check: []
|
||
forbidden_paths: []
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/provenance.py
|
||
pattern: "class ProvenanceStamp"
|
||
```
|
||
|
||
### Step 7: Expose the retriever to the agents as a citation-bearing data source
|
||
|
||
- **Files:** `src/portfolio_optimiser/datasource.py` (new)
|
||
- **Changes:** Wire `retrieval.retrieve()` into the debate as a citation-bearing data source. **MVP (GA-safe) path:** a GA `@tool`/`FunctionTool` (or a `ContextProvider`) that calls `retrieve()` and returns chunks whose `{file, locator, snippet}` the orchestrator maps into `provenance.Citation` — no new dependency, D7-portable. **Conditional MCP wrapper:** IF Step 1 added `mcp` as a GA dependency, ALSO expose a thin custom stdio MCP server (FastMCP) returning the same chunks as `structuredContent`, consumed via `MCPStdioTool(parse_tool_results=…)`, to honor the CLAUDE.md "data access via MCP" convention; if `mcp` was not GA-addable, the MCP wrapper is a documented fast-follow and the MVP uses the in-process tool. Either way the path-security lives in `retrieval.py` (Step 5). (new file)
|
||
- **Reuses:** `retrieval.retrieve` (Step 5); `provenance.Citation` (Step 6); `agent_framework` `@tool`/`FunctionTool`; (conditional) `MCPStdioTool` + `mcp`.
|
||
- **Test first:**
|
||
- File: `tests/test_datasource.py` (new)
|
||
- Verifies: the tool/ContextProvider returns citation-ready chunks for a query over the synthetic docs dir; chunks carry exact locators usable to build a `Citation`; (if `mcp` present) the MCP wrapper returns the same `structuredContent` shape.
|
||
- Pattern: `tests/test_retrieval.py` (Step 5) + `tests/test_backends.py`
|
||
- **Verify:** `uv run pytest tests/test_datasource.py` → expected: `all passed`
|
||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/datasource.py tests/test_datasource.py`; HALT.
|
||
- **Checkpoint:** `git commit -m "feat(fase2): expose retriever as citation-bearing data source"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/datasource.py
|
||
- tests/test_datasource.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(fase2\\): expose retriever as citation-bearing data source$"
|
||
bash_syntax_check: []
|
||
forbidden_paths: []
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/datasource.py
|
||
pattern: "retrieve"
|
||
```
|
||
|
||
### Step 8: Wire the two backend profiles + role→model map + env template
|
||
|
||
- **Files:** `src/portfolio_optimiser/backends.py`, `src/portfolio_optimiser/data/model_map.json` (new), `.env.template` (new), `tests/test_backends.py`
|
||
- **Changes:** Implement `LocalBackend.create_chat_client` → `OpenAIChatCompletionClient(base_url=<PORTFOLIO_LOCAL_BASE_URL>, api_key=<PORTFOLIO_LOCAL_API_KEY default "ollama">, model_id=<resolved>)` configured **non-streaming** (research 03 — NOT `OpenAIChatClient`). Implement `AzureFoundryBackend.create_chat_client` → `FoundryChatClient` (deployment names from env). Add a `data/model_map.json` (the role→model/deployment map artifact, the SAME file Step 11's contract validates) loaded into a small resolver feeding `create_chat_client(model=…)`. Add `.env.template` documenting `PORTFOLIO_LOCAL_BASE_URL`, `PORTFOLIO_LOCAL_API_KEY`, Foundry `project_endpoint`/credential + the research-03 no-egress notes (Ollama bind `127.0.0.1`, pin ≥0.17.1, `OLLAMA_DEBUG` unset, model-pull = explicit egress). Keep `get_backend()` fail-fast. Invert the skeleton test.
|
||
- **Reuses:** `backends.py:26-84`; `spikes/_harness.py:191-215` (env pattern, switch client class); `agent_framework_openai.OpenAIChatCompletionClient`, `agent_framework_foundry.FoundryChatClient`.
|
||
- **Test first:**
|
||
- File: `tests/test_backends.py` (existing — modify)
|
||
- Verifies: both backends satisfy `ChatBackend`; unknown profile raises `ValueError`; the model_map resolves a role→model id; `create_chat_client` returns a client object (no network) — replace `test_create_chat_client_is_skeleton` (`:36`).
|
||
- Pattern: `tests/test_backends.py:14-39`
|
||
- **Verify:** `uv run pytest tests/test_backends.py` → expected: `all passed`
|
||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/backends.py src/portfolio_optimiser/data/model_map.json .env.template tests/test_backends.py`; HALT.
|
||
- **Checkpoint:** `git commit -m "feat(fase2): wire local + Foundry profiles + model-map + env template"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/backends.py
|
||
- src/portfolio_optimiser/data/model_map.json
|
||
- .env.template
|
||
- tests/test_backends.py
|
||
min_file_count: 4
|
||
commit_message_pattern: "^feat\\(fase2\\): wire local \\+ Foundry profiles \\+ model-map \\+ env template$"
|
||
bash_syntax_check: []
|
||
forbidden_paths: []
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/backends.py
|
||
pattern: "OpenAIChatCompletionClient"
|
||
```
|
||
|
||
### Step 9: fresh_workflow isolation factory + maker-checker GroupChat (Layer-1 HITL gate)
|
||
|
||
- **Files:** `src/portfolio_optimiser/workflow.py` (new)
|
||
- **Changes:** Promote `fresh_workflow()` as the core factory — FRESH `GroupChatBuilder` + FRESH chat clients per project run (zero cross-run bleed, B7). Maker-checker debate: proposer + checker, deterministic termination (`make_termination`), `with_max_rounds(n)` AND the external budget-middleware guard (B4: `max_round_count=None` does NOT self-terminate). **Layer-1 HITL** = optionally enable `with_request_info(agents=[checker])` (in-process synchronous review, no checkpoint). Non-streaming. (new file)
|
||
- **Reuses:** `spikes/b_footguns.py:100-106`; `spikes/a_groupchat.py:48-63,130-163`; `budget.ChatMiddleware` (Step 4); `backends` (Step 8); `agent_framework_orchestrations.GroupChatBuilder`.
|
||
- **Test first:**
|
||
- File: `tests/test_workflow.py` (new)
|
||
- Verifies (FakeChatClient): fresh workflow per run → each run a clean thread; a participant sees ONLY its own project (assert on received-message CONTENT, not a call counter — the Fase 1 round-2 fix); a never-converging debate with `with_max_rounds(3)` halts at **exactly 3 rounds** (pin the observed count, retiring the 1-turn=1-round off-by-one assumption — not just `<=`).
|
||
- Pattern: `tests/spikes/test_b_footguns.py:40-45` + `tests/spikes/test_harness.py:103-126`
|
||
- **Verify:** `uv run pytest tests/test_workflow.py` → expected: `all passed`
|
||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/workflow.py tests/test_workflow.py`; HALT.
|
||
- **Checkpoint:** `git commit -m "feat(fase2): fresh_workflow factory + maker-checker GroupChat"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/workflow.py
|
||
- tests/test_workflow.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(fase2\\): fresh_workflow factory \\+ maker-checker GroupChat$"
|
||
bash_syntax_check: []
|
||
forbidden_paths: []
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/workflow.py
|
||
pattern: "with_max_rounds"
|
||
```
|
||
|
||
### Step 10: LLM→IR generation wired to validator-as-retry (token budget in the generate loop)
|
||
|
||
- **Files:** `src/portfolio_optimiser/generate.py` (new)
|
||
- **Changes:** Implement `generate_via_llm(chat_client, project, context, meter) -> SavingsProposal` (replaces the stub `c_validator.py:207`): a NON-STREAMING chat call requesting structured output, parsed into the IR. On `ValidationError` (small models emit text-leaked/wrong-typed tool calls — research 03 Dim 3), invoke `validator.self_repair` — bounded by `self_repair`'s own `max_attempts` AND by the **budget `meter` checked here in the generate loop between attempts** (NO edit to `validator.py` — the validator stays the verbatim-promoted module from Step 2; the token bound lives in this loop). (new file)
|
||
- **Reuses:** `validator.self_repair`/`ir.SavingsProposal` (Step 2); `budget` meter (Step 4); a chat client from `backends` (Step 8).
|
||
- **Test first:**
|
||
- File: `tests/test_generate.py` (new)
|
||
- Verifies (FakeChatClient scripted): well-formed reply → `SavingsProposal`; malformed/text-leaked reply → bounded `self_repair` (retried, not silently accepted); exhausting attempts OR crossing the meter cap → typed failure (never a malformed proposal); validator.py is unmodified (no diff).
|
||
- Pattern: `tests/spikes/test_harness.py:66-76` + `test_c_validator.py`
|
||
- **Verify:** `uv run pytest tests/test_generate.py` → expected: `3 passed`
|
||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/generate.py tests/test_generate.py`; HALT.
|
||
- **Checkpoint:** `git commit -m "feat(fase2): LLM->IR generation with validator-as-retry"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/generate.py
|
||
- tests/test_generate.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(fase2\\): LLM->IR generation with validator-as-retry$"
|
||
bash_syntax_check: []
|
||
forbidden_paths:
|
||
- src/portfolio_optimiser/validator.py
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/generate.py
|
||
pattern: "self_repair"
|
||
```
|
||
|
||
### Step 11: Fail-fast contract loaders
|
||
|
||
- **Files:** `src/portfolio_optimiser/contracts.py` (new)
|
||
- **Changes:** Pydantic models + `load_contracts(...)` for the four contracts: data-source config, **model-map (validates `data/model_map.json` from Step 8)**, termination contract, feedback schema. Validate ALL at startup and raise before any chat-client is constructed or called (brief NFR fail-fast). Pydantic gives JSON-Schema-grade validation (CLAUDE.md convention). (new file)
|
||
- **Reuses:** `pydantic.BaseModel`; `data/model_map.json` (Step 8); `backends.Profile`.
|
||
- **Test first:**
|
||
- File: `tests/test_contracts.py` (new)
|
||
- Verifies: malformed data-source / model-map / termination / feedback → `ValidationError` at load; AND **no chat-client call was made** (inject a FakeChatClient spy, assert `call_count==0`).
|
||
- Pattern: `tests/test_backends.py:30-32` + FakeChatClient `call_count` (`spikes/_harness.py:138`)
|
||
- **Verify:** `uv run pytest tests/test_contracts.py` → expected: `all passed`
|
||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/contracts.py tests/test_contracts.py`; HALT.
|
||
- **Checkpoint:** `git commit -m "feat(fase2): fail-fast contract loaders"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/contracts.py
|
||
- tests/test_contracts.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(fase2\\): fail-fast contract loaders$"
|
||
bash_syntax_check: []
|
||
forbidden_paths: []
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/contracts.py
|
||
pattern: "def load_contracts"
|
||
```
|
||
|
||
### Step 12: Orchestrator + single-command entry (two-layer HITL wiring)
|
||
|
||
- **Files:** `src/portfolio_optimiser/run.py` (new), `src/portfolio_optimiser/__init__.py`
|
||
- **Changes:** `run_project(project_id, profile, verdict_input=None)` composes the slice: `load_contracts` (fail-fast) → load project + retrieve cited chunks via the Step-7 data source → `fresh_workflow` maker-checker debate (budget middleware + round cap; **Layer-1** = the optional in-run `with_request_info` gate) → `generate_via_llm` candidate → blocking `validate_proposal` → `ValidatedProposal | Rejection` → attach `ProvenanceStamp` → **Layer-2 (out-of-band)**: `capture_verdict(features, decision, rationale)` where `decision` comes from `verdict_input` (function arg / CLI prompt / fixture in tests; the **B11 notification is a stub** in Fase 2) → `VerdictStore` persist → (next run) `ExpeLContextProvider` retrieval. Document the two layers explicitly in a module docstring. Console entry for the single-command run. Export public API from `__init__.py`. (new file + modify `__init__.py`)
|
||
- **Reuses:** every Step 2–11 module.
|
||
- **Test first:**
|
||
- File: `tests/test_run_smoke.py` (new)
|
||
- Verifies: `run_project` importable; runs end-to-end on an injected FakeChatClient (with a fixture `verdict_input`) and returns a `ValidatedProposal` with a `ProvenanceStamp`.
|
||
- Pattern: `tests/test_smoke.py`
|
||
- **Verify:** `uv run python -c "from portfolio_optimiser.run import run_project"` → no error; `uv run pytest tests/test_run_smoke.py` → `1 passed`
|
||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/run.py src/portfolio_optimiser/__init__.py tests/test_run_smoke.py`; HALT.
|
||
- **Checkpoint:** `git commit -m "feat(fase2): vertical-slice orchestrator + two-layer HITL wiring"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- src/portfolio_optimiser/run.py
|
||
- tests/test_run_smoke.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^feat\\(fase2\\): vertical-slice orchestrator \\+ two-layer HITL wiring$"
|
||
bash_syntax_check: []
|
||
forbidden_paths: []
|
||
must_contain:
|
||
- path: src/portfolio_optimiser/run.py
|
||
pattern: "def run_project"
|
||
```
|
||
|
||
### Step 13: End-to-end integration test (success criteria, on FakeChatClient with synthetic usage)
|
||
|
||
- **Files:** `tests/test_vertical_slice_e2e.py` (new), `tests/conftest.py` (new)
|
||
- **Changes:** A deterministic CI integration test (scripted FakeChatClient — no real LLM). **The test double emits a synthetic `UsageDetails`** (e.g. `{"total_token_count": N}`) so the budget middleware (Step 4, `strict_usage=True`) does NOT hard-fail and the provenance stamp's `token_usage` is a positive number sourced from `UsageDetails` (this is the in-CI accounting check; the REAL-provider populated-usage assertion lives in Step 14). Add a shared `tests/conftest.py` providing the scripted FakeChatClient + synthetic-usage factory + `seed_store` fixtures. Criteria covered: (a) one valid proposal e2e → exactly one `ValidatedProposal` with populated provenance (≥1 citation, model/role, validator decision, token usage>0 from synthetic UsageDetails); (b) out-of-range → `Rejection` with reason; (c) Layer-2 verdict captured via `capture_verdict` → written to VerdictStore (record exists); (d) second run on a structurally similar proposal retrieves the prior verdict via `ExpeLContextProvider` (assert `retrieved.id == persisted.id`, exercising two-arg `extend_instructions`); (e) tiny budget/round cap → halts via middleware/round-cap (assert cap not exceeded); (f) malformed contract → raises before any chat call. (new files)
|
||
- **Reuses:** `run.run_project`; `tests/conftest.py` FakeChatClient + synthetic-usage; `seed_store()`.
|
||
- **Test first:** this IS the test step.
|
||
- File: `tests/test_vertical_slice_e2e.py` (new)
|
||
- Verifies: all six assertions above.
|
||
- Pattern: `tests/spikes/test_harness.py:103-126` + `test_c_validator.py` + `test_d_verdictstore.py`
|
||
- **Verify:** `uv run pytest && uv run ruff check . && uv run mypy src` → expected: all pass, ruff clean, mypy clean
|
||
- **On failure:** escalate — an e2e failure means a composition seam is wrong; diagnose against the failing criterion; HALT.
|
||
- **Checkpoint:** `git commit -m "test(fase2): end-to-end vertical-slice integration suite"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- tests/test_vertical_slice_e2e.py
|
||
- tests/conftest.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^test\\(fase2\\): end-to-end vertical-slice integration suite$"
|
||
bash_syntax_check: []
|
||
forbidden_paths: []
|
||
must_contain:
|
||
- path: tests/test_vertical_slice_e2e.py
|
||
pattern: "ValidatedProposal"
|
||
```
|
||
|
||
### Step 14: Gated live-profile checks (real UsageDetails + no-egress)
|
||
|
||
- **Files:** `tests/test_local_profile_live.py` (new), `tests/test_foundry_profile_live.py` (new)
|
||
- **Changes:** Separately-gated tests (`@pytest.mark.skipif`, NOT default CI). **Local:** a REAL `OpenAIChatCompletionClient(base_url)` non-streaming call returns a response AND the meter is populated from real `UsageDetails` — positive, load-bearing `total_token_count > 0` (this is the criterion FakeChatClient cannot prove). Assert the call goes only to the configured loopback `base_url` (no-egress: no outbound to `api.openai.com`). **Foundry:** a trivial agent responds on `FoundryChatClient` (cheapest model, hard cap — D6). (new files)
|
||
- **Reuses:** `backends` (Step 8); `live_local_client_or_skip` skip pattern (`spikes/_harness.py:191-215`).
|
||
- **Test first:** these ARE gated tests.
|
||
- File: `tests/test_local_profile_live.py`, `tests/test_foundry_profile_live.py` (new)
|
||
- Verifies: skip cleanly without env; when env present, a successful response + `total_token_count > 0` from each path; local call targets only the loopback endpoint.
|
||
- Pattern: `tests/spikes/test_a_groupchat.py:14-16,65`
|
||
- **Verify:** `uv run pytest tests/test_local_profile_live.py tests/test_foundry_profile_live.py` → expected: `2 skipped` (no endpoint in CI) — or `passed` with env set
|
||
- **On failure:** skip — no endpoint is the expected CI outcome; a non-skip failure with env set escalates; HALT.
|
||
- **Checkpoint:** `git commit -m "test(fase2): gated live local + Foundry profile checks"`
|
||
- **Manifest:**
|
||
```yaml
|
||
manifest:
|
||
expected_paths:
|
||
- tests/test_local_profile_live.py
|
||
- tests/test_foundry_profile_live.py
|
||
min_file_count: 2
|
||
commit_message_pattern: "^test\\(fase2\\): gated live local \\+ Foundry profile checks$"
|
||
bash_syntax_check: []
|
||
forbidden_paths: []
|
||
must_contain:
|
||
- path: tests/test_local_profile_live.py
|
||
pattern: "skipif"
|
||
```
|
||
|
||
## Alternatives Considered
|
||
|
||
| Approach | Pros | Cons | Why rejected |
|
||
|----------|------|------|--------------|
|
||
| Reuse `@modelcontextprotocol/server-filesystem` | No custom code; official | Cannot produce citations; npx/Node dep; CVE/supply-chain surface (research 02) | Citation NFR is load-bearing; in-process retriever produces exact locators |
|
||
| Standalone thin MCP server on the MVP critical path | Honors "data access via MCP" convention fully | Requires `mcp` (not installed; possibly `--pre` → violates GA-pin); extra process | In-process retriever-as-tool for MVP (GA-safe); MCP wrapper conditional on `mcp` GA (Step 7) |
|
||
| Native `OllamaChatClient` for local | MS-recommended; sidesteps `/v1` edges | `--pre` beta + new `ollama` dep — violates GA-pin | `OpenAIChatCompletionClient` non-streaming; native is spike-gated fallback |
|
||
| MAF checkpoint-resume for durable verdict | Native durable pause | Fragile (open bugs into 1.9.0; pickle vs Pydantic IR) + D7 lock-in | Out-of-band VerdictStore; defer checkpointing |
|
||
| MAF `Annotation` propagation for provenance | Native | Python streaming-drop bug #4316 → silent empty citations | First-class own Pydantic provenance |
|
||
| JSON-persistent VerdictStore in MVP | Cross-process durability | Not required by any Fase 2 criterion (in-session learning suffices) | In-memory for MVP; durable persistence deferred to Fase 3 |
|
||
|
||
## Test Strategy
|
||
|
||
- **Framework:** `pytest` + `pytest-asyncio` (`asyncio_mode="auto"`). `pythonpath=["src","."]`, `testpaths=["tests"]`. New shared `tests/conftest.py` (Step 13) provides the scripted FakeChatClient + synthetic-`UsageDetails` factory + `seed_store` fixtures.
|
||
- **Existing patterns:** deterministic validator/Monte-Carlo over FIXED inputs (`test_c_validator.py:59-63`, seed `20260624`); FakeChatClient driving real GA builders with round caps (`test_harness.py:103-126`); gated live arm via `skipif` (`test_a_groupchat.py:14-16`); real-`SessionContext` exercise for the two-arg `extend_instructions` (Step 3).
|
||
- **Determinism discipline (research 03 Dim 5):** determinism asserted ONLY over the validator + Monte-Carlo with fixed inputs; NO test asserts LLM output. The propose (non-deterministic) → validate (deterministic) boundary is explicit.
|
||
- **Usage-accounting split:** in-CI accounting uses the FakeChatClient's synthetic `UsageDetails` (Step 13); the REAL-provider `total_token_count > 0` assertion is gated/live (Step 14) — this is the only place "real UsageDetails" is provable.
|
||
|
||
### Tests to write
|
||
|
||
| Type | File | Verifies | Model test |
|
||
|------|------|----------|------------|
|
||
| Unit | `tests/test_validator.py` | validate/reject/determinism/Pydantic-block | `tests/spikes/test_c_validator.py` |
|
||
| Unit | `tests/test_verdicts.py` | retrieval + REAL-SessionContext two-arg extend_instructions + id minting | `tests/spikes/test_d_verdictstore.py` |
|
||
| Unit | `tests/test_budget.py` | UsageDetails meter, strict None hard-fail, cap, no-proxy guard | `tests/spikes/test_harness.py` |
|
||
| Unit | `tests/test_retrieval.py` | exact-locator chunks + EscapeRoute path rejection | `tests/test_reference_domain.py` |
|
||
| Unit | `tests/test_provenance.py` | ≥1 citation, fields, annotation adapter | `tests/spikes/test_c_validator.py` |
|
||
| Unit | `tests/test_datasource.py` | citation-ready chunks via tool/(opt) MCP | `tests/test_retrieval.py` |
|
||
| Unit | `tests/test_backends.py` | profiles wired, model-map, no-NotImplementedError | `tests/test_backends.py` (existing) |
|
||
| Integration | `tests/test_workflow.py` | fresh-instance isolation (content), exact round cap | `tests/spikes/test_b_footguns.py` |
|
||
| Unit | `tests/test_generate.py` | LLM→IR + bounded self_repair (attempts+budget); validator.py unmodified | `tests/spikes/test_harness.py` |
|
||
| Unit | `tests/test_contracts.py` | fail-fast before any chat call | `tests/test_backends.py` |
|
||
| Integration | `tests/test_vertical_slice_e2e.py` | all 6 e2e criteria (synthetic usage) | `tests/spikes/test_harness.py` |
|
||
| Live (gated) | `tests/test_local_profile_live.py` | real client + real UsageDetails>0 + loopback-only | `tests/spikes/test_a_groupchat.py` |
|
||
| Live (gated) | `tests/test_foundry_profile_live.py` | trivial Foundry response | `tests/spikes/test_a_groupchat.py` |
|
||
|
||
## Risks and Mitigations
|
||
|
||
| Priority | Risk | Location | Impact | Mitigation |
|
||
|----------|------|----------|--------|------------|
|
||
| Critical | `extend_instructions` single-arg bug masked by a tautological fake | `spikes/d_verdictstore.py:115`, `test_d_verdictstore.py:99` | Learning-loop close crashes against a real `SessionContext` | Step 3: two-arg call + a test using a REAL `SessionContext` (not a fake), retiring the risk in CI |
|
||
| Critical | Budget enforcement word-count theatre / `usage=None` silently disables cap | `spikes/_harness.py:97-99` | Hard-cap NFR fake; unbounded run | Step 4: middleware off `UsageDetails`; `strict_usage` None=hard fail; non-streaming; src grep-guard |
|
||
| Critical | Round cap not self-terminating | Fase 1 B4 | Unbounded debate on a real client | Step 9: explicit `with_max_rounds` + external budget guard; test pins EXACT round count |
|
||
| High | `usage=None` on FakeChatClient breaks e2e under strict middleware | Steps 4/13 | e2e cannot run | Step 13: test double emits synthetic `UsageDetails`; real-usage assertion moved to gated Step 14 |
|
||
| High | `mcp` not installed / possibly `--pre` | Steps 1/7 | Step 7 import fails; GA-pin violated | Step 1 conditional add; Step 7 MVP uses in-process tool (no dep); MCP wrapper only if `mcp` GA |
|
||
| High | Wrong OpenAI client (Responses vs Chat Completions) | `spikes/_harness.py:210`, `backends.py:65` | Local profile fails / UsageDetails not populated | Step 8: `OpenAIChatCompletionClient` non-streaming |
|
||
| High | Small-model tool-calling unreliability | research 03 Dim 3 | Malformed candidates crash/pass | Step 10: validator-as-retry (attempts+budget); qwen3:4b; num_ctx≥32k |
|
||
| High | Provenance via MAF annotation (#4316) | research 02 Dim 2 | Citations silently drop on streaming | Step 6: own Pydantic provenance; non-streaming |
|
||
| High | Retriever owns path/symlink security (EscapeRoute) | Step 5 | Out-of-folder read → silent exfil | Step 5: canonicalise+realpath+boundary fail-closed; TDD bypass patterns |
|
||
| Medium | self_repair token-budget vs frozen validator | Steps 2/10 | Scope-fence violation | Step 10: budget checked in generate loop; `validator.py` in `forbidden_paths` |
|
||
| Medium | HITL verdict on checkpoint-resume | research 01 | Silent loss / pickle rejection | Out-of-band `capture_verdict`; checkpointing deferred |
|
||
| Medium | Determinism asserted through the LLM | research 03 Dim 5 | Flaky tests | Pin validator+MC inputs only |
|
||
| Medium | Intel-CPU latency makes a real debate minutes-long | research 03 Dim 4 | Live test hangs | Tiny runs (1 maker+1 checker, ≤3 rounds, hard caps); gate live; representative runs on Foundry |
|
||
| Medium | `fresh_workflow` reuse → B7 bleed returns | `spikes/b_footguns.py:100` | Silent cross-project contamination | Step 9: fresh workflow + clients per run; B7 content-level test |
|
||
| Medium | Headless cascade on dependency-chained failure | all steps | Downstream steps run on a failed predecessor | Global halt rule + per-step "HALT" in On-failure |
|
||
| Low | `#1772` double system prompt (OpenAIChatClient+Ollama+middleware) | research 03 Dim 3 | Wasted tokens | Verify vs 1.9.0 in client spike-gate; native client avoids it |
|
||
| Low | no-op `uv.lock` diff on re-lock | Step 1 | Manifest min_file_count misfire | Step 1 manifest gates only pyproject.toml + test (min_file_count 2) |
|
||
|
||
## Assumptions
|
||
|
||
| # | Assumption | Why unverifiable | Impact if wrong |
|
||
|---|-----------|-----------------|-----------------|
|
||
| 1 | Research is COMPLETE (3 valid briefs on disk) | — (verified) | Would re-run research |
|
||
| 2 | Follow research over brief prose (HITL split, in-process retriever, OpenAIChatCompletionClient non-streaming) | research post-dates brief prose | Planning against superseded premises |
|
||
| 3 | Operator supplies a working local OpenAI-compatible endpoint + model (Ollama qwen3:4b on `127.0.0.1:11434/v1/`) | machine-specific (brief [OPEN]) | Local full-run + "both profiles" criterion can't execute; local-only run still satisfies the other 8 |
|
||
| 4 | Operator supplies Foundry deployment names | tenant-specific (brief [OPEN]) | Foundry minimal check skips |
|
||
| 5 | `mcp` GA availability decides Step 7's MCP wrapper (else in-process tool only) | not installed; resolved in Step 1 | If `--pre`-only, the "data access via MCP" convention is met by a fast-follow, not the MVP |
|
||
| 6 | Synthetic Fase 0 domain reused as the single project | brief [ASSUMPTION] | — |
|
||
| 7 | Solver stays PuLP/CBC (installed 3.3.2, CBC present); PuLP 4.0 → `pulp[cbc]`/`COIN_CMD` | — (verified) | Future PuLP bump breaks bundled CBC |
|
||
|
||
## Verification
|
||
|
||
End-to-end integration checks (per-step manifests verify each step):
|
||
|
||
- [ ] `uv run pytest` → all pass (live arms skip without env)
|
||
- [ ] `uv run ruff check .` → clean
|
||
- [ ] `uv run mypy src` → clean
|
||
- [ ] `uv run python -c "from portfolio_optimiser.run import run_project"` then a FakeChatClient e2e → exactly one `ValidatedProposal` with a populated `ProvenanceStamp` (≥1 citation, model/role, validator decision, token usage>0)
|
||
- [ ] Out-of-range candidate → `Rejection` (type-asserted, reason)
|
||
- [ ] Second run on a structurally similar proposal → `retrieved.id == persisted.id`
|
||
- [ ] Malformed contract → raises at startup; no chat-client call made
|
||
- [ ] Tiny budget/round cap → halts; loop did not exceed cap (exact round count pinned)
|
||
- [ ] `grep -rE "\.split\(\)" src/` → no word-count token proxy
|
||
- [ ] No silent egress: gated live local call targets only the loopback `base_url` (Step 14); telemetry env vars unset
|
||
- [ ] Gated live (when env set): real `UsageDetails.total_token_count > 0` on the local profile; trivial Foundry response
|
||
|
||
## Estimated Scope
|
||
|
||
- **Files to modify:** 2 existing (`pyproject.toml`, `backends.py`, `__init__.py`, `tests/test_backends.py` — 4 touch points)
|
||
- **Files to create:** ~25 (11 `src/` modules incl. `datasource.py` + `data/model_map.json` + `.env.template` + ~12 test files incl. `conftest.py`)
|
||
- **Complexity:** high (integration of four subsystems on real clients, two profiles)
|
||
|
||
## Execution Strategy
|
||
|
||
14 steps > 5 → grouped into sessions/waves. **Dependencies are explicit; sessions in
|
||
the same wave touch disjoint files AND have no cross-session import edge.** Global
|
||
headless rule: on any step failure, perform the On-failure action then HALT — never
|
||
run a dependent step on a failed predecessor.
|
||
|
||
### Session 1: Dependencies
|
||
- **Steps:** 1
|
||
- **Wave:** 1
|
||
- **Depends on:** none
|
||
- **Scope fence:** Touch `pyproject.toml`, `uv.lock`, `tests/test_imports_core.py`; Never touch `src/**`
|
||
|
||
### Session 2: Deterministic core (no MAF), dependency-ordered
|
||
- **Steps:** 2, 5, 6, 3, 4 (note: 5 retrieval before 6 provenance — TextSpan ownership; all internally independent except 6→5)
|
||
- **Wave:** 2
|
||
- **Depends on:** Session 1
|
||
- **Scope fence:** Touch `src/portfolio_optimiser/{ir,validator,retrieval,provenance,verdicts,budget}.py` + tests; Never touch `backends.py`, `workflow.py`, `datasource.py`
|
||
|
||
### Session 3: Data source + backend wiring
|
||
- **Steps:** 7, 8
|
||
- **Wave:** 3 (Step 7 depends on Steps 5+6; Step 8 independent of Session 2 internals)
|
||
- **Depends on:** Session 2 (Step 7 → retrieval/provenance)
|
||
- **Scope fence:** Touch `src/portfolio_optimiser/{datasource,backends}.py`, `data/model_map.json`, `.env.template` + tests; Never touch the core modules' internals (import only)
|
||
|
||
### Session 4: Composition
|
||
- **Steps:** 9, 10, 11, 12
|
||
- **Wave:** 4
|
||
- **Depends on:** Sessions 2, 3
|
||
- **Scope fence:** Touch `src/portfolio_optimiser/{workflow,generate,contracts,run,__init__}.py` + tests; Never touch core/datasource/backends internals (incl. `validator.py` — Step 10 forbidden_path)
|
||
|
||
### Session 5: Integration + live
|
||
- **Steps:** 13, 14
|
||
- **Wave:** 5
|
||
- **Depends on:** Session 4
|
||
- **Scope fence:** Touch `tests/test_vertical_slice_e2e.py`, `tests/conftest.py`, `tests/test_local_profile_live.py`, `tests/test_foundry_profile_live.py`; Never touch `src/**`
|
||
|
||
### Execution Order
|
||
- **Wave 1:** Session 1
|
||
- **Wave 2:** Session 2 (internal order 2 → 5 → 6 → 3 → 4)
|
||
- **Wave 3:** Session 3 (after Wave 2)
|
||
- **Wave 4:** Session 4 (after Wave 3)
|
||
- **Wave 5:** Session 5 (after Wave 4)
|
||
|
||
### Grouping rules applied
|
||
- Steps sharing files / import edges → same session, sequenced; 3–5 steps/session; sessions ordered by dependency. No two sessions in the same wave share an import edge (the Step 7→6 edge moved Step 7 to Wave 3).
|
||
|
||
## Plan Quality Score
|
||
|
||
| Dimension | Weight | Score | Notes |
|
||
|-----------|--------|-------|-------|
|
||
| Structural integrity | 0.15 | 90 | dependency-ordered; TextSpan ownership fixed; Step 7 moved to its own wave; halt rule added |
|
||
| Step quality | 0.20 | 88 | each step 1–2 files, TDD, concrete reuse; verdict capture + id minting now explicit |
|
||
| Coverage completeness | 0.20 | 92 | all 9 criteria → steps + verification; two-layer HITL mapped; no-egress verified |
|
||
| Specification quality | 0.15 | 86 | model-map artifact + env template defined; some new-module internals left to impl |
|
||
| Risk & pre-mortem | 0.15 | 92 | 17 risks incl. the 3 premise corrections; extend_instructions retired by a real-session test |
|
||
| Headless readiness | 0.10 | 88 | On-failure + HALT + Checkpoint + Manifest per step |
|
||
| Manifest quality | 0.05 | 86 | checkable manifests; strengthened retrieval/datasource predicates; lock-step gating fixed |
|
||
| **Weighted total** | **1.00** | **89** | **Grade: A−** |
|
||
|
||
**Adversarial review:**
|
||
- **Plan critic:** REVISE on the first draft — 3 blockers (FakeChatClient usage contradiction, missing `mcp` dep, unspecified two-layer HITL capture), 7 major, 4 minor. All addressed in Revisions below.
|
||
- **Scope guardian:** ALIGNED — 0 creep, all 9 criteria mapped, all 6 Non-Goals honored; 1 minor (two-layer HITL naming) addressed.
|
||
|
||
## Revisions
|
||
|
||
| # | Finding | Severity | Resolution |
|
||
|---|---------|----------|------------|
|
||
| 1 | FakeChatClient e2e can't satisfy usage-populated vs `None`-hard-fail | blocker | Step 13 test double emits synthetic `UsageDetails`; `strict_usage` flag (Step 4); real-usage assertion moved to gated Step 14 |
|
||
| 2 | `mcp`/FastMCP dependency never added | blocker | Step 1 conditional `mcp` add (GA only); Step 7 MVP uses in-process retriever-as-tool (no dep); MCP wrapper conditional/fast-follow |
|
||
| 3 | Two-layer HITL verdict-capture unspecified; verdict-id minting undefined | blocker | Step 3 adds `capture_verdict` + stable `Verdict.id` hash; Step 12 wires Layer-1 (in-run gate) + Layer-2 (out-of-band capture) + B11 notification stub, mapped explicitly |
|
||
| 4 | `extend_instructions` not retired by a real-session test | major | Step 3 test uses a REAL `agent_framework.SessionContext` (verified importable), not a fake |
|
||
| 5 | Session 2∥3 not independent (Step 7→6) + unowned `TextSpan` | major | `TextSpan` owned by `retrieval.py` (Step 5), imported by provenance (Step 6); Step 7 moved to Wave 3; Execution Strategy reworked |
|
||
| 6 | env vars / `.env.template` / no-silent-egress unverified | major | Step 8 adds `.env.template` + no-egress notes; Verification adds loopback-only + telemetry-unset checks (Step 14) |
|
||
| 7 | No circuit-breaker/halt language | major | Global halt rule + per-step "HALT" appended to every On-failure |
|
||
| 8 | Step 3 JSON persistence = scope creep | major | Removed; VerdictStore in-memory for MVP; durable persistence deferred to Fase 3 |
|
||
| 9 | self_repair token-budget edits frozen validator.py | major | Step 10 enforces the token bound in the generate loop; `validator.py` in Step 10 `forbidden_paths` |
|
||
| 10 | Step 1 manifest `min_file_count:3` may misfire on no-op lock | major | Manifest gates only `pyproject.toml` + test (min_file_count 2) |
|
||
| 11 | Estimated Scope counts inconsistent | minor | Corrected |
|
||
| 12 | model-map config artifact undefined | minor | `data/model_map.json` defined in Step 8, validated by Step 11 |
|
||
| 13 | Round-cap assertion doesn't catch off-by-one | minor | Step 9 pins the EXACT observed round count |
|
||
| 14 | Weak `realpath` manifest predicate | minor | Path-security moved into `retrieval.py` (Step 5) with `realpath` + behavioural EscapeRoute tests |
|
||
|
||
## Adversarial Pass 2 (gemini-bridge, v5.1.1 high-effort)
|
||
|
||
**Status: UNAVAILABLE.** The high-effort plan signal mandates an independent
|
||
gemini-bridge review of the post-revision plan. The `gemini-mcp` server has
|
||
returned an identical `400 BadRequestError` on every call this project (its
|
||
client SDK predates Google's May-2026 Interactions API change). A fifth
|
||
invocation would fail deterministically, so it was not attempted (cost
|
||
discipline D6). Independent second-opinion triangulation on the plan is
|
||
therefore absent — the plan rests on plan-critic + scope-guardian (Phase 9) +
|
||
installed-source-grounded exploration. To restore this pass, upgrade the
|
||
gemini-mcp client SDK to ≥ 2.0.0 and re-run `/trekplan` (or a standalone
|
||
gemini review) on `plan.md`.
|