chore(repo): planning artifacts become local-only; fixture builders become code
Operator ruling 2026-08-05, which settles decision (g): planning documents are generally never public, and what OUR OWN sessions generate does not go out on the forge at all. The example itself stays public so others can run the process. `.claude/projects/` is the Voyage session workbench -- 25 briefs/plans/reviews this project's own sessions produced. Untracked and gitignored, exactly as STATE.md already is, and for the same stated reason: this repo has a public mirror, so that class of material is local-only rather than tracked. The line is drawn at who wrote the document, and it is drawn deliberately: `docs/plan/`, `docs/research/` and `docs/rapport/` stay tracked. Those are curated, dated documents written for the repo's readers, three of them linked from the README as the decision record. Move that line if it was meant wider. Two files were NOT process artifacts and are not deleted. Both `build_fixture.py` scripts are cited by tracked tests (`test_ingest_golden_sql.py`, `test_ingest_golden_http.py`) as the documented rebuild path for byte-exact goldens -- reproduction code that had landed in the wrong directory. Moved next to the goldens they build; both docstrings updated, so no tracked file is left pointing into an untracked tree (verified: the only remaining `.claude/projects` string in a tracked file is the .gitignore rule itself). One prose reference in the dated Foundry auth recipe was dropped for the same reason. 652 tests still pass. Does NOT address the 27 of these already readable on open/ since the S12 release -- untracking stops future publication only. That retraction is a separate operator decision and is deliberately not taken here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GWsexbQjPo9rsV3aUE54ZS
This commit is contained in:
parent
93608d008d
commit
392f8493da
31 changed files with 11 additions and 7366 deletions
|
|
@ -1,122 +0,0 @@
|
|||
---
|
||||
type: trekbrief
|
||||
brief_version: "2.2"
|
||||
created: 2026-06-23
|
||||
task: "Fase 1 de-risk spikes (A–D): empirically validate the framework's most dangerous assumptions before the MVP vertical slice"
|
||||
slug: fase1-derisk-spikes
|
||||
project_dir: .claude/projects/2026-06-23-fase1-derisk-spikes/
|
||||
research_topics: 0
|
||||
research_status: skipped
|
||||
auto_research: false
|
||||
interview_turns: 3
|
||||
source: interview
|
||||
framing: refine
|
||||
phase_signals:
|
||||
- phase: research
|
||||
effort: standard
|
||||
- phase: plan
|
||||
effort: standard
|
||||
- phase: execute
|
||||
effort: standard
|
||||
- phase: review
|
||||
effort: standard
|
||||
---
|
||||
|
||||
# Task: Fase 1 — De-risk spikes (A–D)
|
||||
|
||||
> Generated by `/trekbrief` on 2026-06-23.
|
||||
> This brief is the contract between requirements and planning. `/trekplan`
|
||||
> reads it to produce the implementation plan. Every decision in the plan must
|
||||
> trace back to content in this brief.
|
||||
|
||||
## TL;DR
|
||||
|
||||
**Refine** of the locked plan ([§Fase 1](../../../docs/plan/2026-06-23-incremental-plan.md)) — same intent, narrowed to executable spike scope. Build four **throwaway** de-risk spikes (A–D) that convert the framework's most dangerous *documented-but-unverified* assumptions into *measured facts* before the Fase 2 vertical slice. Each spike has a falsifiable pass/fail from the plan's verification block. No new external research — reuse the research report's §15 implementation register. Runtime agent calls default to the **LOCAL profile** with hard token/round caps per D6.
|
||||
|
||||
## Intent
|
||||
|
||||
Fase 0 delivered the skeleton, locked decisions (D1–D6), and a synthetic reference domain (D4). Before committing to the full architecture in Fase 2 (vertical slice), we must empirically de-risk the four assumptions that — if wrong — would force a redesign. Specifically: (1) that a Group Chat maker-checker debate beats a single-agent baseline by enough to justify its multiplicative token cost (U3 / G7); (2) that the known MAF footguns behave as the research predicts and our guards hold — Magentic's unbounded termination when `limits=None` (G1/B4) and shared-`WorkflowBuilder` state corruption in fan-out (G2/B7); (3) that a blocking deterministic hybrid-validator (B1) can *structurally* block an out-of-range proposal from ever reaching the expert; and (4) that ExpeL retrieval (B2) actually surfaces a relevant prior verdict for a similar new proposal. These are throwaway spikes — code we expect to discard — whose only job is to turn §15 register assumptions into evidence. Getting each wrong *cheaply now* is vastly cheaper than discovering it mid-Fase-2.
|
||||
|
||||
## Goal
|
||||
|
||||
Four runnable spike modules (isolated so they are trivial to discard), each producing a **measurable pass/fail** against the plan §Fase 1 verification criteria, each running on the LOCAL profile by default with hard token and round/iteration caps, and each emitting a short findings note with the measured numbers (convergence rounds, stall frequency, token use, state-bleed observations, validator block/pass behavior, retrieval hit/miss). At the end, the framework's four most dangerous assumptions are each **confirmed or refuted with evidence**, and the findings are written down so they directly inform the Fase 2 design. The repo's quality gate (ruff + mypy(src) + pytest) stays green.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Production code for Fase 2 (the vertical slice) — these spikes are throwaway and may be deleted after their findings are recorded.
|
||||
- Compliance functions (D3 — the deployer owns DPIA/ROS/behandlingsformål; we build only technical preconditions).
|
||||
- Chasing the last 10% (D5) — spikes prove the generic core, not edge cases or polish.
|
||||
- Heavy Foundry/Azure runs (D6) — LOCAL profile is default; Foundry is used only for targeted, minimal verification if at all in Fase 1.
|
||||
- A production sandbox for user-supplied skill scripts (B8 / G3) — out of Fase 1 scope.
|
||||
- A full production VerdictStore — Spike D is a minimal 10–20 synthetic-verdict retrieval test only, not the durable store design.
|
||||
- Resolving the second-rank open questions (plan §Risiko: review-latency, project topology, schema ownership, Foundry-memory Preview) — these do not block Fase 1.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **D6 cost-discipline:** LOCAL profile default (OpenAI-compatible endpoint); cheapest models; tiny synthetic data; hard token + round caps; no heavy test runs. Foundry/Azure only for targeted, minimal verification.
|
||||
- **Deterministic validator is obligatory and blocking** (never an optional plugin) — Spike C must demonstrate *structural* blocking, not advisory warning.
|
||||
- **Stop-criteria + budget caps required at startup** — fail-fast if missing; never an unbounded loop.
|
||||
- **Group Chat maker-checker is the debate default**, NOT Magentic (which is experimental, G8/A2).
|
||||
- **Stack:** Python ≥3.10, MAF (`agent-framework`), `uv`, `ruff`, `mypy`, `pytest`. Type hints throughout; Pydantic for IR/validation.
|
||||
- Spikes reuse the existing synthetic domain (D4: `data/reference_projects.json`, `reference_domain.py`) and the backend-profile seam (D2: `backends.py`).
|
||||
|
||||
## Preferences
|
||||
|
||||
- **90%-principle (D5):** generic core + clear extension points; do not over-fit.
|
||||
- Per-agent model selection via chat-client (`FoundryChatClient` / `OpenAIChatClient`, common base `BaseChatClient`).
|
||||
- Keep spikes physically isolated (e.g. a `spikes/` package or `tests/spikes/`) so discarding them leaves the core untouched.
|
||||
- Findings recorded as short markdown notes inside the project dir (`.claude/projects/2026-06-23-fase1-derisk-spikes/`), not scattered.
|
||||
- Prefer reusing `mcp-solver` / OR-Tools / PuLP / Z3 (R1/R2) for Spike C's solver step rather than hand-rolling.
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
- Each spike enforces a **hard token-budget cap** and a **round/iteration cap**; exceeding either yields a structured stop event, never a silent hang.
|
||||
- Each spike **reports measured token usage** (per the plan's "every spike measures and reports token consumption").
|
||||
- Runtime defaults to the LOCAL profile; **no silent egress** to Azure/Foundry without explicit profile selection.
|
||||
- Spikes must not require real/sensitive data — synthetic only.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
*Falsifiable, mapped 1:1 to the plan §Fase 1 verification block.*
|
||||
|
||||
- **Spike A (U3 / G7):** Maker-checker (proposer · critic · validator) converges in ≤ N rounds (N fixed at spike-design time) AND the hard cap is respected; a findings note documents convergence rate, stall frequency, and token use for BOTH maker-checker and single-agent baseline, with an explicit cheaper/better verdict. Verify: the spike command/test exits 0 and prints the comparison table.
|
||||
- **Spike B (G1 / G2):** Unbounded Magentic (`limits=None`) does NOT self-terminate → confirms an explicit limit is required (B4); `ConcurrentBuilder` fan-out from a shared builder shows **zero state-bleed** when the fresh-instance helper (B7) is used. Verify: the spike asserts both observations and exits 0.
|
||||
- **Spike C (B1):** An out-of-range proposal is **structurally blocked** (never reaches the expert/output); a valid proposal passes and yields P10/P50/P90 from the Monte Carlo step; self-repair retries are capped at N. Verify: a test asserts the blocked case raises/returns a structured rejection and the valid case returns the percentiles.
|
||||
- **Spike D (B2):** Top-K retrieval fetches the relevant historical verdict for a similar new proposal from a store of 10–20 synthetic verdicts. Verify: a test asserts the expected verdict id is in the top-K for a crafted similar proposal.
|
||||
- **Quality gate:** `uv run ruff check .` exits 0, `uv run ruff format --check .` clean, `uv run mypy src` exits 0, `uv run pytest` exits 0.
|
||||
|
||||
## Research Plan
|
||||
|
||||
No external research needed — the codebase and the existing research report ([docs/research/2026-06-23-prior-art-platform.md](../../../docs/research/2026-06-23-prior-art-platform.md) §13 architecture, §15 implementation register, §15.3 footguns) plus this brief contain sufficient context for planning. The spikes themselves ARE the empirical de-risking. Any MAF API specifics (e.g. the exact current-version Python Group Chat termination API, U3) are confirmed inline via the `microsoft-learn` MCP at coding time rather than as a separate research pass.
|
||||
|
||||
## Open Questions / Assumptions
|
||||
|
||||
- **[ASSUMPTION]** The LOCAL profile resolves to an OpenAI-compatible endpoint reachable during spikes (per D6 and the `local` profile in `backends.py`). If no local endpoint is available at run time, Spike A/C/D live agent calls fall back to a minimal, capped Foundry run.
|
||||
- **[ASSUMPTION — verify before Spike A]** The installed `agent-framework` version's Python Group Chat termination API matches research §15 (U3: `termination_condition` lambda). Note a version ambiguity to resolve first: project `CLAUDE.md` states `agent-framework` 1.8.0 while `STATE.md` references `agent-framework-core` 1.9.0 — confirm the actual installed version (`uv pip show agent-framework-core`) and the current termination API via the `microsoft-learn` MCP before coding Spike A.
|
||||
- **[ASSUMPTION]** Spike A's convergence target "N rounds" will be fixed at spike-design time (candidate ≤ 3–5 rounds); the plan deliberately leaves N open.
|
||||
- Magentic is experimental (G8); Spike B exercises it ONLY to confirm the footgun (G1), never to build core flow on it (A2).
|
||||
|
||||
## Prior Attempts
|
||||
|
||||
None for the spikes themselves — fresh. Substrate from Fase 0 is complete and committed: repo scaffold, locked decisions D1–D6, GA-slimmed dependencies (`uv.lock`), the D4 synthetic "anleggskostnad" domain (`reference_domain.py` + `data/reference_projects.json`, 3 fictional projects), and the D2 backend-profile skeleton (`backends.py`: `Profile` azure|local + `ChatBackend` protocol + stubs raising `NotImplementedError` until Fase 1). Quality gate was green at Fase 0 close (ruff + mypy(src) + 12 pytest passed).
|
||||
|
||||
## Metadata
|
||||
|
||||
- **Created:** 2026-06-23
|
||||
- **Interview turns:** 3
|
||||
- **Auto-research opted in:** no
|
||||
- **Source:** trekbrief interview
|
||||
|
||||
---
|
||||
|
||||
## How to continue
|
||||
|
||||
Manual (default):
|
||||
|
||||
```bash
|
||||
# No research topics — go straight to planning:
|
||||
/trekplan --project .claude/projects/2026-06-23-fase1-derisk-spikes
|
||||
|
||||
# Then execute:
|
||||
/trekexecute --project .claude/projects/2026-06-23-fase1-derisk-spikes
|
||||
```
|
||||
|
|
@ -1,473 +0,0 @@
|
|||
# Fase 1 — De-risk spikes (A–D)
|
||||
|
||||
> **Plan quality: A-** (post adversarial review + revision) — APPROVE_WITH_NOTES
|
||||
>
|
||||
> Generated by trekplan v2.0 on 2026-06-23 — `plan_version: 1.7`
|
||||
|
||||
## Context
|
||||
|
||||
Fase 0 delivered the skeleton, locked decisions (D1–D6), and a synthetic reference domain (D4). Before committing to the full architecture in Fase 2, we empirically de-risk the four assumptions that — if wrong — force a redesign (brief Intent): (A) Group Chat maker-checker beats a single-agent baseline by enough to justify its multiplicative token cost (U3 / G7); (B) the known MAF footguns behave as predicted and our guards hold — Magentic unbounded termination when `limits=None` (G1/B4) and shared-builder/fan-out state corruption (G2/B7); (C) a blocking deterministic hybrid-validator (B1) can *structurally* block an out-of-range proposal; (D) ExpeL retrieval (B2) surfaces a relevant prior verdict. These are **throwaway** spikes whose only job is to turn §15-register assumptions into measured facts, cheaply, under cost-discipline (D6: LOCAL profile default, hard token/round caps).
|
||||
|
||||
**Decisive planning finding (premise-verification):** the GA-slimmed Fase 0 install deliberately dropped the `agent-framework[all]` meta, so the orchestration builders (`GroupChatBuilder`, `ConcurrentBuilder`, `MagenticBuilder`) are **not importable** — they live in the separate package `agent-framework-orchestrations`, which has a clean **GA release `1.0.0`** depending on `agent-framework-core<2,>=1.9.0` (exactly our installed core). Because the spikes are **throwaway**, this package is added to the **`dev`** dependency group (not core) — exactly like the PuLP solver — deferring any core-runtime commitment to Fase 2 (revised after scope review). Also resolved: the installed `agent-framework-core` is **1.9.0** (STATE.md correct; project `CLAUDE.md` says 1.8.0 and is corrected in Step 1; the `project-core` user-memory also says 1.8.0 — that lives outside the repo and is a session-end follow-up).
|
||||
|
||||
**Gate-green contract (the central design choice).** Each spike is split into a **logic layer** (pure functions/classes we author — always exercised by the quality gate, no live endpoint and no full MAF workflow run) and an **integration/live layer** (drives the actual MAF builders and/or a real LOCAL LLM — runs when available, otherwise `skip`s, except Spike B where driving the builders IS the de-risk; see Step 2/Step 4). The headline empirical claims (Spike A's better/cheaper verdict; live token numbers) live in the integration/live layer and are honestly reported as endpoint-dependent. The gate stays green from the logic layer alone.
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Reused (Fase 0, unchanged — src/ never touched)"
|
||||
RD["reference_domain.py<br/>(D4 synthetic projects)"]
|
||||
BC["agent_framework.BaseChatClient<br/>(subclassed by FakeChatClient)"]
|
||||
end
|
||||
subgraph "New — throwaway spikes/ package (dev-only)"
|
||||
H["_harness.py<br/>Budget+RoundCap (B4)<br/>FakeChatClient + minimal Agent<br/>live_local_client_or_skip() -> OpenAIChatClient"]
|
||||
SM["builder smoke (Step 2)<br/>proves FakeChatClient drives the GA builders"]
|
||||
A["a_groupchat.py — Spike A (U3/G7)"]
|
||||
B["b_footguns.py — Spike B (G1, G2/B7)"]
|
||||
C["c_validator.py — Spike C (B1)"]
|
||||
D["d_verdictstore.py — Spike D (B2)"]
|
||||
end
|
||||
subgraph "Deliverables (tracked)"
|
||||
F["docs/fase1-spikes/<br/>README + findings-{a,b,c,d}.md"]
|
||||
end
|
||||
H --> SM --> A & B & C & D
|
||||
RD --> A & C & D
|
||||
BC --> H
|
||||
A --> F
|
||||
B --> F
|
||||
C --> F
|
||||
D --> F
|
||||
```
|
||||
|
||||
## Codebase Analysis
|
||||
|
||||
- **Tech stack:** Python ≥3.10 (venv on 3.12), MAF GA (`agent-framework-core` 1.9.0, `-foundry` 1.8.2, `-openai` 1.8.2), `pydantic` 2.13.4 (stable), `uv`, `ruff` 0.15.18, `mypy` 2.1.0, `pytest` 9.1.1. Build: hatchling (wheel packages `src/portfolio_optimiser` only — top-level `spikes/` is never shipped).
|
||||
- **Key patterns:** small typed package; `from __future__ import annotations`; frozen dataclasses for the domain; `Protocol` + `runtime_checkable` for the D2 seam; fail-fast (`ValueError`/`NotImplementedError`); tests are plain pytest functions with `-> None`, `parametrize`, `pytest.raises`, module-scope fixtures, importing from `portfolio_optimiser`.
|
||||
- **Relevant files:** `src/portfolio_optimiser/backends.py` (D2 seam — `Profile`, `ChatBackend`, `get_backend`, `LocalBackend`; live-wiring of the seam is a **Fase 2** concern, NOT touched here), `src/portfolio_optimiser/reference_domain.py` (D4 — `Project`, `CostItem`, `load_reference_projects`, `.total_cost`), `src/portfolio_optimiser/data/reference_projects.json` (3 synthetic projects), `tests/test_*.py` (test style), `pyproject.toml`.
|
||||
- **Reusable code:** `load_reference_projects()` → spike fixtures (A/C/D); `Project.total_cost` / `CostItem.total_cost` → savings-IR constraint source (C); `agent_framework.BaseChatClient` → subclassed by the harness `FakeChatClient`; `agent_framework.openai.OpenAIChatClient` → built **directly** by the harness for the live arm (the D2 `LocalBackend` seam is intentionally left un-wired until Fase 2, so `src/` stays untouched and the throwaway spikes add no core surface).
|
||||
- **External tech (verified by introspection + PyPI, not docs):** orchestration builders in `agent_framework.orchestrations` (need `agent-framework-orchestrations`); chat clients in `agent_framework.openai` (`OpenAIChatClient`); `ContextProvider`/`MemoryContextProvider` top-level. **MAF workflows/orchestrations are async** (`await workflow.run(...)`), so the spike tests use `pytest-asyncio` (added in Step 1). Exact builder method signatures are confirmed by introspection at coding time (brief-sanctioned) and de-risked once up-front by the Step 2 builder smoke.
|
||||
- **Recent git activity:** `b57aa83` Fase 0 (D4+D2+tests), `491a746` deps GA-fix + lockfile. Clean tree on `main`, synced to origin.
|
||||
|
||||
## Research Sources
|
||||
|
||||
*Ground-truth API/dependency verification — introspection + PyPI metadata are authoritative here (no web docs needed).*
|
||||
|
||||
| Topic | Source | Key Findings | Confidence |
|
||||
|-------|--------|--------------|------------|
|
||||
| Orchestration builders' real import path | `import agent_framework.orchestrations` (introspection) | `GroupChatBuilder`, `ConcurrentBuilder`, `MagenticBuilder`, `StandardMagenticManager`, `SequentialBuilder`, `TerminationCondition`, `MagenticPlanReviewRequest` live there; not top-level | high |
|
||||
| Orchestrations package availability/stability | PyPI JSON | GA `1.0.0` (uploaded 2026-06-18, not yanked); `requires_dist: agent-framework-core<2,>=1.9.0`; rest b/rc | high |
|
||||
| Installed core version (resolves brief ambiguity) | `uv pip show agent-framework-core` | **1.9.0** (STATE correct; CLAUDE.md 1.8.0 → corrected Step 1) | high |
|
||||
| MAF orchestrations are async | introspection (`WorkflowBuilder`/orchestration `.run`) | tests need `pytest-asyncio`; integration arm is async | high |
|
||||
| Solver lib for Spike C | PyPI / project knowledge | `pulp` (PuLP) MIT, pure-Python, **bundles a CBC binary in its wheel** → faithful solver-in-the-loop (R2) | high |
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
Each step targets one focused change and follows the repo's TDD-ish style. Per the **gate-green contract** above, the deterministic logic layer is always exercised; integration/live layers are gated.
|
||||
|
||||
### Step 1: Scaffold throwaway spikes package and pin dev-only orchestration + solver + async deps
|
||||
|
||||
- **Files:** `pyproject.toml`, `CLAUDE.md`, `spikes/__init__.py`, `tests/spikes/__init__.py`, `tests/spikes/test_imports.py`, `docs/fase1-spikes/README.md`, `uv.lock`
|
||||
- **Changes:** In `pyproject.toml` add to `[project.optional-dependencies].dev` (NOT core — spikes are throwaway): `agent-framework-orchestrations>=1.0.0` (GA; resolves with core 1.9.0), `pulp>=2.8` (CBC solver), `pytest-asyncio>=0.24`. Set `[tool.pytest.ini_options] pythonpath = ["src", "."]` (repo-root added so `import spikes` resolves; `spikes/` has no name-collision with any installed dist, so it cannot shadow them) and `asyncio_mode = "auto"`. Set `[tool.ruff] src = ["src", "tests", "spikes"]`. Create `spikes/__init__.py` and `tests/spikes/__init__.py` (docstring: "Throwaway Fase 1 de-risk spikes — dev-only, never shipped in the wheel; safe to delete after findings recorded"). Create `docs/fase1-spikes/README.md` (the four spikes, the resolved version gate, a **Disposal** subsection: `rm -rf spikes tests/spikes docs/fase1-spikes` + revert the three `dev`/`tool` pyproject edits). In `CLAUDE.md` correct the stale stack line `MAF (`agent-framework` 1.8.0)` → `MAF (`agent-framework-core` 1.9.0)` (honors the brief's Spike-A version precondition). Run `uv sync --extra dev`, then re-confirm `uv pip show agent-framework-orchestrations` resolves cleanly (premise re-verification). (new files: all except `pyproject.toml`, `CLAUDE.md`)
|
||||
- **Reuses:** existing `pyproject.toml` / `CLAUDE.md` structure (Fase 0).
|
||||
- **Test first:**
|
||||
- File: `tests/spikes/test_imports.py` (new)
|
||||
- Verifies: `import spikes`; `from agent_framework.orchestrations import GroupChatBuilder, ConcurrentBuilder, MagenticBuilder, StandardMagenticManager, TerminationCondition`; `import pulp`; `import pytest_asyncio`.
|
||||
- Pattern: `tests/test_smoke.py`
|
||||
- **Verify:** `uv run --extra dev pytest tests/spikes/test_imports.py -q` → expected: passed; `uv pip show agent-framework-orchestrations` → `Version: 1.0.0`; `uv run ruff check .` → exit 0
|
||||
- **On failure:** retry — if the orchestrations import fails after sync, confirm the lock resolved `agent-framework-orchestrations==1.0.0`; if anything pulls a pre-release, pin tighter. Revert with `git checkout -- pyproject.toml CLAUDE.md uv.lock` if unrecoverable.
|
||||
- **Checkpoint:** `git commit -m "build(fase1): add dev orchestration + solver + async deps, scaffold spikes"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- pyproject.toml
|
||||
- CLAUDE.md
|
||||
- spikes/__init__.py
|
||||
- tests/spikes/__init__.py
|
||||
- tests/spikes/test_imports.py
|
||||
- docs/fase1-spikes/README.md
|
||||
- uv.lock
|
||||
min_file_count: 7
|
||||
commit_message_pattern: "^build\\(fase1\\): add dev orchestration \\+ solver \\+ async deps"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/backends.py
|
||||
- src/portfolio_optimiser/reference_domain.py
|
||||
must_contain:
|
||||
- path: pyproject.toml
|
||||
pattern: "agent-framework-orchestrations"
|
||||
- path: pyproject.toml
|
||||
pattern: "pytest-asyncio"
|
||||
- path: CLAUDE.md
|
||||
pattern: "agent-framework-core` 1.9.0"
|
||||
```
|
||||
|
||||
### Step 2: Shared spike harness + builder smoke (budget caps, fake client, live gate, async builder de-risk)
|
||||
|
||||
- **Files:** `spikes/_harness.py`, `tests/spikes/test_harness.py`
|
||||
- **Changes:** Implement the cost/stop invariant (B4) and the test seams shared by all spikes. (1) `@dataclass Budget(max_tokens: int, max_rounds: int)` — constructor raises `ValueError` for non-positive caps (refuse to start without caps, A4); a `TokenMeter` plus `charge(tokens)`/`tick_round()` raise `BudgetExceeded(kind, limit, observed)` the moment a cap is crossed. (Two exception types is intentional and documented in the module: `ValueError` = bad construction argument; `BudgetExceeded` = runtime breach.) (2) `FakeChatClient(BaseChatClient)` returning scripted deterministic responses and counting tokens by word-count; plus `fake_agent(client, name, instructions)` building a minimal real `agent_framework.Agent` backed by the fake client so the orchestration builders get genuine participants. (3) `live_local_client_or_skip()` → builds `agent_framework.openai.OpenAIChatClient(base_url=env PORTFOLIO_LOCAL_BASE_URL, model=env PORTFOLIO_LOCAL_MODEL)` **directly**, or `pytest.skip(...)` when env is unset (the D2 `LocalBackend` seam is deliberately not used — its live-wiring is Fase 2). (4) A **builder smoke** that front-loads the biggest risk: construct a `GroupChatBuilder` and a `ConcurrentBuilder` with two `fake_agent`s and `await` a trivial run, asserting they execute — this is the one place that proves `FakeChatClient` can drive the GA builders. If it cannot, that is itself a recorded de-risk finding (escalate), surfaced before any spike is built on the assumption. (new file)
|
||||
- **Reuses:** `agent_framework.BaseChatClient`, `agent_framework.Agent`; `agent_framework.openai.OpenAIChatClient`; `agent_framework.orchestrations.{GroupChatBuilder, ConcurrentBuilder}`; the D6 cost contract.
|
||||
- **Test first:**
|
||||
- File: `tests/spikes/test_harness.py` (new)
|
||||
- Verifies: `Budget(0, 5)` and `Budget(5, 0)` raise `ValueError`; `charge`/`tick_round` raise `BudgetExceeded` past their caps; `FakeChatClient` returns scripted replies in order and accumulates tokens; `live_local_client_or_skip()` skips when env unset; **builder smoke** (async test) constructs and runs `GroupChatBuilder`+`ConcurrentBuilder` with fake agents and asserts completion.
|
||||
- Pattern: `tests/test_backends.py` (`parametrize` + `pytest.raises`)
|
||||
- **Verify:** `uv run --extra dev pytest tests/spikes/test_harness.py -q` → expected: passed; `uv run ruff check spikes tests/spikes` → exit 0
|
||||
- **On failure:** escalate — if the builder smoke proves `FakeChatClient` cannot drive the GA builders, STOP and record it in `docs/fase1-spikes/README.md` as a primary de-risk finding (it changes Spikes A/B); decide with the operator whether to back the builders with a tiny canned-LLM stub or restructure. Otherwise revert with `git checkout -- spikes/_harness.py tests/spikes/test_harness.py`.
|
||||
- **Checkpoint:** `git commit -m "feat(fase1): shared spike harness + builder smoke (budget caps, fake client, live gate)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- spikes/_harness.py
|
||||
- tests/spikes/test_harness.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase1\\): shared spike harness \\+ builder smoke"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/backends.py
|
||||
must_contain:
|
||||
- path: spikes/_harness.py
|
||||
pattern: "class BudgetExceeded"
|
||||
- path: spikes/_harness.py
|
||||
pattern: "def live_local_client_or_skip"
|
||||
- path: tests/spikes/test_harness.py
|
||||
pattern: "GroupChatBuilder"
|
||||
```
|
||||
|
||||
### Step 3: Spike A — Group Chat maker-checker vs single-agent (U3 / G7)
|
||||
|
||||
- **Files:** `spikes/a_groupchat.py`, `tests/spikes/test_a_groupchat.py`, `docs/fase1-spikes/findings-a.md`
|
||||
- **Changes:** **Logic layer (always green):** `Arm` result dataclass (rounds, stalls, tokens, caught_flaw); `make_termination(n_rounds=3)` (N fixed at 3 per reviewer refinement #1); `verdict(mc: Arm, single: Arm) -> dict` computing `better = mc.caught_flaw and not single.caught_flaw` and `affordable = mc.tokens <= 3 * single.tokens` and `passed = better and affordable`, plus a comparison table renderer. **Integration/live layer (gated):** `run_live(project)` builds the 3-role maker-checker (proposer · critic · validator) via `GroupChatBuilder` + `make_termination()` and a single-agent baseline, runs both against `load_reference_projects()[0]` carrying a **planted flaw** (a savings candidate violating an obvious constraint, e.g. negative residual quantity), measures real numbers, writes them + the computed verdict to `findings-a.md`. (new file)
|
||||
- **Reuses:** `spikes._harness` (Budget, FakeChatClient, fake_agent, live gate); `reference_domain.load_reference_projects`; `agent_framework.orchestrations.GroupChatBuilder`/`TerminationCondition`.
|
||||
- **Test first:**
|
||||
- File: `tests/spikes/test_a_groupchat.py` (new)
|
||||
- Verifies (logic, non-tautological — inputs are **varied**, not a single scripted constant): `verdict()` returns `passed=True` when mc caught & single missed & tokens within 3×; `passed=False` when mc tokens = 5× single (affordable=False) even if better; `passed=False` when both caught the flaw (better=False); `make_termination(3)` stops at 3 rounds. A `@pytest.mark.skipif(no endpoint)` live test runs `run_live` and asserts ≤3 rounds, cap respected, a findings table emitted.
|
||||
- Pattern: `tests/test_reference_domain.py` + `tests/test_backends.py`
|
||||
- **Verify:** `uv run --extra dev pytest tests/spikes/test_a_groupchat.py -q` → expected: logic tests passed (live test skipped without endpoint)
|
||||
- **On failure:** retry — confirm `GroupChatBuilder` method names by introspection and adjust `run_live`; the logic layer is independent of the MAF surface. Revert with `git checkout -- spikes/a_groupchat.py tests/spikes/test_a_groupchat.py docs/fase1-spikes/findings-a.md` if unrecoverable.
|
||||
- **Checkpoint:** `git commit -m "feat(fase1): spike A - group chat maker-checker vs single-agent"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- spikes/a_groupchat.py
|
||||
- tests/spikes/test_a_groupchat.py
|
||||
- docs/fase1-spikes/findings-a.md
|
||||
min_file_count: 3
|
||||
commit_message_pattern: "^feat\\(fase1\\): spike A"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/reference_domain.py
|
||||
must_contain:
|
||||
- path: spikes/a_groupchat.py
|
||||
pattern: "GroupChatBuilder"
|
||||
- path: tests/spikes/test_a_groupchat.py
|
||||
pattern: "skipif"
|
||||
- path: docs/fase1-spikes/findings-a.md
|
||||
pattern: "token"
|
||||
```
|
||||
|
||||
### Step 4: Spike B — Magentic limits=None footgun + ConcurrentBuilder state isolation (G1 / G2)
|
||||
|
||||
- **Files:** `spikes/b_footguns.py`, `tests/spikes/test_b_footguns.py`, `docs/fase1-spikes/findings-b.md`
|
||||
- **Changes:** Driving the builders **is** the de-risk here, so these tests run in the gate (no live LLM — `FakeChatClient`/`fake_agent`), having been de-risked by the Step 2 builder smoke. (a) **Magentic unbounded (G1):** `MagenticBuilder` + `StandardMagenticManager` with `max_round_count=None` over a fake agent that never finalizes; drive it under the harness **round/iteration guard** (the deterministic assertion — wall-clock is only a secondary safety net, not asserted) and assert it does NOT self-terminate before the guard fires → confirms an explicit limit is required (B4). With explicit limits → terminates cleanly. (b) **Fan-out state isolation (G2/B7):** `fresh_workflow()` factory + a stateful executor; assert a SHARED instance bleeds state across the 3 projects and a FRESH instance per run shows zero bleed (B7). Writes observations + a token-use line ("0 — no live LLM") to `findings-b.md`. (new file)
|
||||
- **Reuses:** `spikes._harness` (round guard, FakeChatClient, fake_agent); `reference_domain.load_reference_projects`; `agent_framework.orchestrations.{MagenticBuilder, StandardMagenticManager, ConcurrentBuilder}`.
|
||||
- **Test first:**
|
||||
- File: `tests/spikes/test_b_footguns.py` (new)
|
||||
- Verifies: unbounded Magentic hits the round guard without self-terminating (asserts guard fired, not a natural stop); bounded Magentic stops within its limit; shared-instance fan-out leaks state (assertion of bleed); fresh-instance fan-out asserts no bleed across all 3 projects.
|
||||
- Pattern: `tests/test_backends.py`
|
||||
- **Verify:** `uv run --extra dev pytest tests/spikes/test_b_footguns.py -q` → expected: passed
|
||||
- **On failure:** escalate — if `MagenticBuilder`/`ConcurrentBuilder` cannot be driven by a fake agent despite the Step 2 smoke, record it as a genuine de-risk finding in `findings-b.md` (it answers the footgun question by other means) before reverting `git checkout -- spikes/b_footguns.py tests/spikes/test_b_footguns.py docs/fase1-spikes/findings-b.md`.
|
||||
- **Checkpoint:** `git commit -m "feat(fase1): spike B - magentic unbounded + concurrent state isolation"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- spikes/b_footguns.py
|
||||
- tests/spikes/test_b_footguns.py
|
||||
- docs/fase1-spikes/findings-b.md
|
||||
min_file_count: 3
|
||||
commit_message_pattern: "^feat\\(fase1\\): spike B"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: spikes/b_footguns.py
|
||||
pattern: "ConcurrentBuilder"
|
||||
- path: spikes/b_footguns.py
|
||||
pattern: "fresh_workflow"
|
||||
- path: docs/fase1-spikes/findings-b.md
|
||||
pattern: "token"
|
||||
```
|
||||
|
||||
### Step 5: Spike C — blocking hybrid validator (IR→PuLP→Monte Carlo, self-repair) (B1)
|
||||
|
||||
- **Files:** `spikes/c_validator.py`, `tests/spikes/test_c_validator.py`, `docs/fase1-spikes/findings-c.md`
|
||||
- **Changes:** Fully deterministic — no MAF builders. Pydantic IR `SavingsProposal` (project_id, measure, affected cost codes, claimed saving NOK, assumptions) with field validators + a cross-field `@model_validator` (claimed saving ≤ affected items' total; quantities ≥ 0). `validate_proposal(proposal) -> ValidatedProposal | Rejection`: (1) Pydantic schema validation; (2) **PuLP** feasibility/optimization over the project's cost items (uses PuLP's **bundled CBC** binary — if CBC is genuinely absent the step **escalates**; no silent LP-relaxation fallback); (3) **Monte Carlo** over uncertain unit-costs → P10/P50/P90 (stdlib `random` seeded + `statistics.quantiles`); (4) **structural block**: an out-of-range/infeasible proposal returns a `Rejection` that can never be consumed as a `ValidatedProposal`. `self_repair(generate, max_attempts=N)` capped, then hard-stops. The LLM-generation entry is a thin wrapper gated by `live_local_client_or_skip()`; all validator/solver/MC logic is tested with crafted IR directly. Writes a token-use line ("0 — validator deterministic; live generation gated") + percentiles to `findings-c.md`. (new file)
|
||||
- **Reuses:** `pydantic.BaseModel`/`model_validator`; `pulp`; `reference_domain.Project`/`CostItem`/`load_reference_projects`; `spikes._harness` (Budget for the self-repair cap, live gate).
|
||||
- **Test first:**
|
||||
- File: `tests/spikes/test_c_validator.py` (new)
|
||||
- Verifies: an out-of-range proposal is structurally blocked — `validate_proposal` returns `Rejection` and raises if treated as validated; a valid proposal returns `ValidatedProposal` with P10 ≤ P50 ≤ P90; Monte Carlo reproducible under the fixed seed; `self_repair` stops at `max_attempts`.
|
||||
- Pattern: `tests/test_reference_domain.py` + `pytest.raises`
|
||||
- **Verify:** `uv run --extra dev pytest tests/spikes/test_c_validator.py -q` → expected: passed
|
||||
- **On failure:** escalate (if CBC absent — record platform finding) / otherwise revert `git checkout -- spikes/c_validator.py tests/spikes/test_c_validator.py docs/fase1-spikes/findings-c.md`.
|
||||
- **Checkpoint:** `git commit -m "feat(fase1): spike C - blocking hybrid validator (IR/solver/monte-carlo)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- spikes/c_validator.py
|
||||
- tests/spikes/test_c_validator.py
|
||||
- docs/fase1-spikes/findings-c.md
|
||||
min_file_count: 3
|
||||
commit_message_pattern: "^feat\\(fase1\\): spike C"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: spikes/c_validator.py
|
||||
pattern: "model_validator"
|
||||
- path: spikes/c_validator.py
|
||||
pattern: "class Rejection"
|
||||
- path: tests/spikes/test_c_validator.py
|
||||
pattern: "P10"
|
||||
```
|
||||
|
||||
### Step 6: Spike D — VerdictStore + ExpeL retrieval (B2)
|
||||
|
||||
- **Files:** `spikes/d_verdictstore.py`, `tests/spikes/test_d_verdictstore.py`, `docs/fase1-spikes/findings-d.md`
|
||||
- **Changes:** Minimal in-memory VerdictStore: a list of `Verdict(id, proposal_features, decision, rationale)` seeded with 10–20 synthetic verdicts from the reference domain. **Similarity defined operationally (reviewer refinement #2):** a weighted score over *structured* fields — Jaccard on the affected cost-code set + match on `savings_measure_type` + a magnitude-bucket match on claimed saving — **not** raw description text. `retrieve(proposal, k) -> list[Verdict]` ranks by that score (this is the always-tested unit that satisfies SC-D). A thin custom `ContextProvider` subclass wraps `retrieve()` for ExpeL few-shot injection; its injection is asserted only against the introspected interface (if the abstract surface differs at coding time, the `retrieve()` ranking remains the deliverable and the SC-D assertion). The embedding-based similarity path is **removed** as out of scope for a throwaway spike (noted as a Fase-2 option in `findings-d.md`). Writes a token-use line ("0 — deterministic retrieval") + a top-K example to `findings-d.md`. (new file)
|
||||
- **Reuses:** `agent_framework.ContextProvider` (top-level); `reference_domain.load_reference_projects`; `spikes._harness`.
|
||||
- **Test first:**
|
||||
- File: `tests/spikes/test_d_verdictstore.py` (new)
|
||||
- Verifies (non-tautological by construction): seed a store where the true match shares the *structured* similarity fields with the query but uses **different description text**, while 2–3 **decoys** share surface description text but differ in the structured fields; assert the true match is top-1 and within top-K; assert deterministic ordering. (ContextProvider-injection assertion included only if the interface is confirmed; `retrieve()` ranking is the guaranteed SC-D check.)
|
||||
- Pattern: `tests/test_reference_domain.py`
|
||||
- **Verify:** `uv run --extra dev pytest tests/spikes/test_d_verdictstore.py -q` → expected: passed
|
||||
- **On failure:** retry — confirm `ContextProvider` abstract methods by introspection; keep `retrieve()` ranking separable. Revert `git checkout -- spikes/d_verdictstore.py tests/spikes/test_d_verdictstore.py docs/fase1-spikes/findings-d.md` if unrecoverable.
|
||||
- **Checkpoint:** `git commit -m "feat(fase1): spike D - verdictstore + expel retrieval"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- spikes/d_verdictstore.py
|
||||
- tests/spikes/test_d_verdictstore.py
|
||||
- docs/fase1-spikes/findings-d.md
|
||||
min_file_count: 3
|
||||
commit_message_pattern: "^feat\\(fase1\\): spike D"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: spikes/d_verdictstore.py
|
||||
pattern: "def retrieve"
|
||||
- path: tests/spikes/test_d_verdictstore.py
|
||||
pattern: "decoy"
|
||||
```
|
||||
|
||||
### Step 7: Consolidate findings, document disposal, confirm the green quality gate
|
||||
|
||||
- **Files:** `docs/fase1-spikes/README.md`
|
||||
- **Changes:** Update `docs/fase1-spikes/README.md` to a consolidated summary table (spike → assumption → result → confirmed/refuted → implication for Fase 2), pulling each per-spike note's verdict and token-use line (each `findings-{a,b,c,d}.md` already exists from Steps 3–6). Confirm the **Disposal** subsection (from Step 1) is complete. Run the full quality gate across the whole repo. (no new findings files here — they are owned by their spike steps)
|
||||
- **Reuses:** the per-spike findings written in Steps 3–6.
|
||||
- **Test first:** n/a (documentation + gate; the gate IS the verification).
|
||||
- **Verify:** `uv run ruff check .` → exit 0; `uv run ruff format --check .` → exit 0; `uv run mypy src` → exit 0; `uv run --extra dev pytest -q` → expected: all passed (originals + spike logic layers; live arms skipped without endpoint)
|
||||
- **On failure:** escalate — a red gate means a prior step regressed; bisect by step and fix before consolidating.
|
||||
- **Checkpoint:** `git commit -m "docs(fase1): consolidate spike findings + confirm green quality gate"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- docs/fase1-spikes/README.md
|
||||
- docs/fase1-spikes/findings-a.md
|
||||
- docs/fase1-spikes/findings-b.md
|
||||
- docs/fase1-spikes/findings-c.md
|
||||
- docs/fase1-spikes/findings-d.md
|
||||
min_file_count: 5
|
||||
commit_message_pattern: "^docs\\(fase1\\): consolidate spike findings"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/backends.py
|
||||
must_contain:
|
||||
- path: docs/fase1-spikes/README.md
|
||||
pattern: "Fase 2"
|
||||
- path: docs/fase1-spikes/README.md
|
||||
pattern: "Disposal"
|
||||
```
|
||||
|
||||
### Failure recovery rules
|
||||
|
||||
- **revert** — undo this step's changes (`git checkout -- {files}`), do not proceed.
|
||||
- **retry** — confirm the real MAF method signature by introspection, adjust, then revert if still failing.
|
||||
- **escalate** — stop entirely; needs human judgment (Step 2 builder smoke fails / Step 4 builder un-fakeable / Step 5 CBC absent / Step 7 gate regression). For Steps 2 and 4, an escalate that *records the MAF limitation* is itself a valid de-risk outcome.
|
||||
- **Checkpoint** — commit after each green step.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
| Approach | Pros | Cons | Why rejected |
|
||||
|----------|------|------|--------------|
|
||||
| Spikes under `src/portfolio_optimiser/spikes/` | Single import root | Ships throwaway code in the wheel | Top-level `spikes/` keeps the wheel clean (hatchling packages only `src/portfolio_optimiser`) |
|
||||
| `agent-framework-orchestrations` in **core** deps | One home for the locked debate default | Permanent Fase-2 commitment made under a throwaway task (scope creep, flagged in review) | Put it in `dev` like PuLP; promote to core in Fase 2 when the real flow is built |
|
||||
| Wire `LocalBackend.create_chat_client()` in Step 2 for the live arm | Fills the D2 seam | Touches `src/` under a throwaway task; forces editing `test_backends.py`; non-throwaway commitment | Build `OpenAIChatClient` directly in the throwaway harness; defer seam-wiring to Fase 2; `src/` stays untouched |
|
||||
| Require a live LOCAL endpoint for all spike tests | Tests the real thing | Gate red without an endpoint; violates SC-gate + D6 | Logic layer always green; integration/live arm gated |
|
||||
| numpy for Monte Carlo | Familiar | New dep for trivial sampling | stdlib `random` + `statistics.quantiles` — zero new dep |
|
||||
| `agent-framework[all]` to get orchestrations | One line | Re-introduces the beta/`--prerelease`/alpha-pydantic problem Fase 0 removed (`491a746`) | Pin the GA `agent-framework-orchestrations>=1.0.0` only |
|
||||
| Embedding similarity (Spike D) | "Real" RAG | Needs live endpoint; non-deterministic; serves no SC; over-builds throwaway | Deterministic structured-field similarity; embeddings = Fase-2 note only |
|
||||
| Hand-rolled feasibility instead of PuLP | No dep | Not a faithful solver-in-the-loop (B1/R2) | PuLP (bundled CBC) keeps the spike honest |
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- **Framework:** pytest 9.1.1 + `pytest-asyncio` (`asyncio_mode = "auto"`); plain functions with `-> None`, `parametrize`, `pytest.raises`; fixtures from `load_reference_projects()`. Async tests for the MAF-builder integration.
|
||||
- **Existing patterns:** `tests/test_backends.py`, `tests/test_reference_domain.py`, `tests/test_smoke.py`.
|
||||
- **New tests in this plan:** 6 modules (imports, harness+builder-smoke, A, B, C, D). Determinism via `FakeChatClient` + fixed seeds; live arms isolated behind `@pytest.mark.skipif(no endpoint)`. Spike B drives the builders in-gate (no endpoint) — de-risked by the Step 2 smoke.
|
||||
|
||||
### Tests to write
|
||||
|
||||
| Type | File | Verifies | Model test |
|
||||
|------|------|----------|------------|
|
||||
| Import | `tests/spikes/test_imports.py` | orchestrations + pulp + pytest-asyncio + `spikes` import | `tests/test_smoke.py` |
|
||||
| Unit+async | `tests/spikes/test_harness.py` | Budget caps; FakeChatClient; skip gate; builder smoke | `tests/test_backends.py` |
|
||||
| Unit | `tests/spikes/test_a_groupchat.py` | varied-input verdict logic; ≤3-round termination; gated live arm | `tests/test_reference_domain.py` |
|
||||
| Unit+async | `tests/spikes/test_b_footguns.py` | unbounded Magentic needs guard; zero fan-out bleed | `tests/test_backends.py` |
|
||||
| Unit | `tests/spikes/test_c_validator.py` | structural block; P10≤P50≤P90; self-repair cap | `tests/test_reference_domain.py` |
|
||||
| Unit | `tests/spikes/test_d_verdictstore.py` | non-tautological top-K retrieval | `tests/test_reference_domain.py` |
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Priority | Risk | Location | Impact | Mitigation |
|
||||
|----------|------|----------|--------|------------|
|
||||
| High | `FakeChatClient` cannot drive the async GA builders | `spikes/_harness.py` (smoke), A/B | Spikes A/B integration blocked | Step 2 builder smoke front-loads this; failure is a recorded de-risk finding (escalate), not a late surprise; logic layers (A verdict, B isolation) stay green regardless |
|
||||
| High | Real builder method signatures differ from planned wiring | A/B integration | Integration code won't run as written | Confirm by introspection at coding time (brief-sanctioned); per-step retry; logic layers independent of the MAF surface |
|
||||
| Medium | No LOCAL endpoint during the session | A/C live arms | Live numbers not captured | Logic layers prove the machinery; live arms `skip`; findings note records "live arm not run" honestly |
|
||||
| Medium | PuLP CBC binary missing on Intel mac | `spikes/c_validator.py` | Solver call fails | PuLP wheels bundle CBC; if genuinely absent → **escalate** (recorded), no silent fallback |
|
||||
| Low | Async-mode misconfig breaks existing sync tests | `pyproject.toml` | Original 12 tests affected | `asyncio_mode="auto"` only auto-marks coroutine tests; sync tests unaffected — Step 7 gate catches any regression |
|
||||
| Low | 2 `ExperimentalWarning`s (SKILLS/HARNESS) at import (G8) | all imports | Noise only | Harmless; do not enable `filterwarnings=error` |
|
||||
| Low | `.claude/projects/**` committed to a to-be-public repo | repo | Planning scratch in history | Out of scope here; flag at Fase 4 release prep |
|
||||
|
||||
## Assumptions
|
||||
|
||||
| # | Assumption | Why unverifiable now | Impact if wrong |
|
||||
|---|-----------|-----------------|-----------------|
|
||||
| 1 | LOCAL profile = reachable OpenAI-compatible endpoint via `PORTFOLIO_LOCAL_*` env | Depends on operator's running Ollama/LM Studio | Live arms skip; logic layers still pass — low impact |
|
||||
| 2 | `FakeChatClient` can drive the GA builders | Not yet installed/introspected to method level | Step 2 smoke decides early; failure is a documented finding, not silent breakage |
|
||||
| 3 | PuLP's bundled CBC works on the Intel-mac venv | Not installed yet | Step 5 escalates with a recorded platform finding |
|
||||
| 4 | N = 3 rounds is a fair maker-checker bound | Judgment from G7 token-discipline | If too tight to converge live, bump to ≤5 and re-record (documented range) |
|
||||
|
||||
## Verification
|
||||
|
||||
*Per-step manifests are checked automatically by trekexecute. These map to the brief Success Criteria.*
|
||||
|
||||
- [ ] Spike A: `uv run --extra dev pytest tests/spikes/test_a_groupchat.py -q` → varied-input verdict logic passes; gated live arm (when run) converges ≤3 rounds, cap respected, comparison table emitted (SC-A; empirical better/cheaper verdict is endpoint-dependent and so reported)
|
||||
- [ ] Spike B: `uv run --extra dev pytest tests/spikes/test_b_footguns.py -q` → unbounded Magentic needs an external guard; fresh-instance fan-out zero state-bleed (SC-B)
|
||||
- [ ] Spike C: `uv run --extra dev pytest tests/spikes/test_c_validator.py -q` → out-of-range blocked structurally; valid proposal yields P10/P50/P90; self-repair capped (SC-C)
|
||||
- [ ] Spike D: `uv run --extra dev pytest tests/spikes/test_d_verdictstore.py -q` → top-K retrieval returns the structurally-similar verdict over surface-text decoys (SC-D)
|
||||
- [ ] Gate: `uv run ruff check .` exit 0; `uv run ruff format --check .` clean; `uv run mypy src` exit 0; `uv run --extra dev pytest -q` all passed (SC-gate)
|
||||
- [ ] Findings: `docs/fase1-spikes/` has README (summary table + Disposal) + findings-{a,b,c,d}.md, each with a confirmed/refuted verdict and a token-use line (brief Goal + NFR)
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Files to modify:** 2 (`pyproject.toml`, `CLAUDE.md`; `uv.lock` regenerated)
|
||||
- **Files to create:** ~19 (4 spike modules + `_harness.py` + 6 test modules + 2 `__init__.py` + 5 docs notes + README owned by Step 1/7)
|
||||
- **Complexity:** medium (logic is modest; the real unknown — MAF builder ergonomics with a fake client — is de-risked up-front by the Step 2 smoke)
|
||||
|
||||
## Execution Strategy
|
||||
|
||||
### Session 1: Foundation
|
||||
- **Steps:** 1, 2
|
||||
- **Wave:** 1
|
||||
- **Depends on:** none
|
||||
- **Scope fence:**
|
||||
- Touch: `pyproject.toml`, `CLAUDE.md`, `spikes/__init__.py`, `spikes/_harness.py`, `tests/spikes/{__init__,test_imports,test_harness}.py`, `docs/fase1-spikes/README.md`, `uv.lock`
|
||||
- Never touch: `src/**`, the four spike modules
|
||||
|
||||
### Session 2: Spike A
|
||||
- **Steps:** 3
|
||||
- **Wave:** 2
|
||||
- **Depends on:** Session 1
|
||||
- **Scope fence:**
|
||||
- Touch: `spikes/a_groupchat.py`, `tests/spikes/test_a_groupchat.py`, `docs/fase1-spikes/findings-a.md`
|
||||
- Never touch: other spike modules, `src/**`, `pyproject.toml`, `CLAUDE.md`
|
||||
|
||||
### Session 3: Spike B
|
||||
- **Steps:** 4
|
||||
- **Wave:** 2
|
||||
- **Depends on:** Session 1
|
||||
- **Scope fence:**
|
||||
- Touch: `spikes/b_footguns.py`, `tests/spikes/test_b_footguns.py`, `docs/fase1-spikes/findings-b.md`
|
||||
- Never touch: other spike modules, `src/**`, `pyproject.toml`, `CLAUDE.md`
|
||||
|
||||
### Session 4: Spike C
|
||||
- **Steps:** 5
|
||||
- **Wave:** 2
|
||||
- **Depends on:** Session 1
|
||||
- **Scope fence:**
|
||||
- Touch: `spikes/c_validator.py`, `tests/spikes/test_c_validator.py`, `docs/fase1-spikes/findings-c.md`
|
||||
- Never touch: other spike modules, `src/**`, `pyproject.toml`, `CLAUDE.md`
|
||||
|
||||
### Session 5: Spike D
|
||||
- **Steps:** 6
|
||||
- **Wave:** 2
|
||||
- **Depends on:** Session 1
|
||||
- **Scope fence:**
|
||||
- Touch: `spikes/d_verdictstore.py`, `tests/spikes/test_d_verdictstore.py`, `docs/fase1-spikes/findings-d.md`
|
||||
- Never touch: other spike modules, `src/**`, `pyproject.toml`, `CLAUDE.md`
|
||||
|
||||
### Session 6: Consolidation
|
||||
- **Steps:** 7
|
||||
- **Wave:** 3
|
||||
- **Depends on:** Sessions 2–5
|
||||
- **Scope fence:**
|
||||
- Touch: `docs/fase1-spikes/README.md`
|
||||
- Never touch: `spikes/**`, `src/**`, `pyproject.toml`, `CLAUDE.md`
|
||||
|
||||
### Execution Order
|
||||
|
||||
- **Wave 1:** Session 1 (foundation — includes the builder smoke that de-risks all spikes)
|
||||
- **Wave 2:** Sessions 2, 3, 4, 5 (parallel — independent spike modules sharing only the read-only harness)
|
||||
- **Wave 3:** Session 6 (after Wave 2)
|
||||
|
||||
### Grouping rules applied
|
||||
|
||||
- Steps sharing files → same session (1+2 = foundation).
|
||||
- Independent spike modules → separate sessions (parallelizable in Wave 2).
|
||||
- Consolidation depends on all spikes → Wave 3.
|
||||
|
||||
## Plan Quality Score
|
||||
|
||||
| Dimension | Weight | Score | Notes |
|
||||
|-----------|--------|-------|-------|
|
||||
| Structural integrity | 0.15 | 90 | Foundation→spikes→consolidation; src/ untouched so scope fences are accurate |
|
||||
| Step quality | 0.20 | 88 | TDD-first; false LocalBackend reuse removed; findings owned per step |
|
||||
| Coverage completeness | 0.20 | 90 | SC-A/B/C/D + gate + findings mapped; A's empirical verdict honestly endpoint-dependent |
|
||||
| Specification quality | 0.15 | 86 | Real import paths; builder ergonomics de-risked by Step 2 smoke; CBC fallback decided |
|
||||
| Risk & pre-mortem | 0.15 | 90 | Biggest risk (fake-drives-builders) front-loaded with a green-keeping logic layer |
|
||||
| Headless readiness | 0.10 | 92 | On-failure + Checkpoint + Manifest per step; expected_paths ⊆ Files |
|
||||
| Manifest quality | 0.05 | 90 | All manifests checkable; orphan paths fixed |
|
||||
| **Weighted total** | **1.00** | **89** | **Grade: A-** |
|
||||
|
||||
**Adversarial review:**
|
||||
- **Plan critic:** REVISE → addressed (2 blockers + 8 major + 4 minor); see Revisions.
|
||||
- **Scope guardian:** MIXED → addressed (orchestrations moved to dev; embedding path dropped; token-reporting + findings-location ratified); see Revisions.
|
||||
|
||||
## Revisions
|
||||
|
||||
*Added after Phase 9 adversarial review (plan-critic + scope-guardian, deduped: 19 findings, 0 overlap).*
|
||||
|
||||
| # | Finding (rule_key) | Severity | Resolution |
|
||||
|---|--------------------|----------|------------|
|
||||
| 1 | live-arm/impossible-localbackend-wiring | blocker | Harness builds `OpenAIChatClient` directly; dropped the false `LocalBackend` reuse; D2 seam-wiring deferred to Fase 2; `src/` untouched |
|
||||
| 2 | fragile-assumption/fakeclient-drives-async-builders | blocker | Added Step 2 builder smoke (front-loads the risk); added `pytest-asyncio` + `asyncio_mode=auto`; split each spike into always-green logic layer + gated integration/live layer; failure is a recorded escalate |
|
||||
| 3 | structural/scope-fence-vs-live-arm | major | Resolved by #1 — `src/**` is now genuinely never touched; fences accurate |
|
||||
| 4 | missing-setup/async-test-runner | major | `pytest-asyncio>=0.24` added (Step 1); async integration tests |
|
||||
| 5 | verification/spike-a-verdict-tautological | major | Logic test now varies inputs to prove the verdict function; empirical U3/G7 verdict explicitly endpoint-dependent (live arm) |
|
||||
| 6 | underspecified/contextprovider-inject-no-target | major | `retrieve()` ranking is the guaranteed SC-D unit; ContextProvider injection asserted only against the introspected interface |
|
||||
| 7 | manifest/expected-paths-not-subset-of-files | major | All `Files:` lines now include test/findings/`uv.lock`/`CLAUDE.md` paths |
|
||||
| 8 | ordering/findings-files-creation-ambiguous | major | Each spike step (3–6) owns and creates its `findings-X.md`; Step 7 only consolidates the README |
|
||||
| 9 | missing-error-handling/cbc-fallback-deferred | major | Committed to PuLP's bundled CBC; absence → escalate (recorded), no silent LP-relaxation |
|
||||
| 10 | coverage/spike-a-precondition-downgraded | major | Step 1 corrects `CLAUDE.md` version; installed version confirmed 1.9.0; memory fix noted as session follow-up |
|
||||
| 11 | creep — orchestrations in core deps | major | Moved `agent-framework-orchestrations` to `dev` (throwaway scope), like PuLP |
|
||||
| 12 | creep — Spike D embedding path | minor | Removed; deterministic similarity only; embeddings = Fase-2 note |
|
||||
| 13 | gap — token reporting B/C/D | minor | Each findings note carries a token-use line (incl. "0 — no live LLM") |
|
||||
| 14 | gap — findings location vs brief | minor | Kept `docs/fase1-spikes/` (tracked, survives, consistent with research/plan) — ratified as a deliberate refinement; operator can object |
|
||||
| 15 | dependency — re-confirm orchestrations resolves | minor | Step 1 runs `uv pip show agent-framework-orchestrations` post-sync |
|
||||
| 16 | minor/pythonpath-root-shadowing | minor | Documented: repo-root added for `import spikes`; no name collision with installed dists |
|
||||
| 17 | minor/budget-two-exception-types | minor | Documented intent: `ValueError` = bad ctor arg, `BudgetExceeded` = runtime breach |
|
||||
| 18 | minor/wall-clock-flaky-assertion | minor | Spike B asserts the round/iteration guard; wall-clock is a secondary safety net only |
|
||||
| 19 | minor/no-teardown-for-throwaway-spikes | minor | Step 1/7 README **Disposal** subsection: `rm -rf` + revert the pyproject dev/tool edits |
|
||||
|
|
@ -1,125 +0,0 @@
|
|||
---
|
||||
type: trekreview
|
||||
review_version: "1.0"
|
||||
created: 2026-06-24
|
||||
task: "Fase 1 de-risk spikes (A–D): empirically validate the framework's most dangerous assumptions before the MVP vertical slice"
|
||||
slug: fase1-derisk-spikes
|
||||
project_dir: .claude/projects/2026-06-23-fase1-derisk-spikes/
|
||||
brief_path: .claude/projects/2026-06-23-fase1-derisk-spikes/brief.md
|
||||
scope_sha_start: ffd3ad4
|
||||
scope_sha_end: a2dff21
|
||||
reviewed_files_count: 20
|
||||
verdict: ALLOW
|
||||
findings: []
|
||||
---
|
||||
|
||||
# Review: Fase 1 de-risk spikes (A–D) — round 2 (post-remediation)
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Verdict: **ALLOW**. This is the round-2 re-review after remediation commit
|
||||
`a2dff21`. Both independent reviewers return **0 findings**.
|
||||
|
||||
Round 1 (`ffd3ad4..b81e22b`) returned **BLOCK** on one BLOCKER: Spike B(b)'s
|
||||
fan-out experiment asserted a tautological per-client `call_count` (3 vs 1)
|
||||
rather than the real G2/B7 shared-`Workflow` state-corruption footgun, making
|
||||
`findings-b.md`'s "CONFIRMED" a false-confirm. A MINOR flagged an unguarded
|
||||
findings-file write.
|
||||
|
||||
Both are now genuinely resolved, and the code-correctness reviewer verified the
|
||||
fix **against MAF internals**, not just the prose:
|
||||
|
||||
- **BLOCKER resolved.** `spikes/b_footguns.py` was rebuilt to observe the real
|
||||
mechanism — the conversation history each participant *receives* (new
|
||||
`FakeChatClient.received_texts` seam in `spikes/_harness.py`).
|
||||
`ConcurrentBuilder.build()` wraps each participant in an `AgentExecutor` whose
|
||||
`_session` is created once and reused across `Workflow.run()` calls; with no
|
||||
context provider, `Agent.run` auto-injects an `InMemoryHistoryProvider` and
|
||||
`SessionContext.get_messages(include_input=True)` feeds the accumulated
|
||||
prior-run messages to the chat client. So a reused workflow genuinely delivers
|
||||
run N the prompts/replies of runs 0..N-1 — measured as monotonic growth
|
||||
`[[p0], [p0,p1], [p0,p1,p2]]` — while a fresh instance per run gives each a
|
||||
clean thread (`[[p0], [p1], [p2]]`). The tests assert this by message content,
|
||||
non-vacuously (an empty extractor would make them fail, not pass).
|
||||
- **MINOR resolved.** `spikes/a_groupchat.py` now `mkdir(parents=True,
|
||||
exist_ok=True)`s the findings dir before `write_text`.
|
||||
- The `_harness.py` seam change (shared by all four spikes) introduced no new
|
||||
correctness defect (verified).
|
||||
|
||||
One transparent limitation, recorded as **prose context (not a finding)**, carried
|
||||
over from round 1: SC-A's *empirical* deliverable is partial — the codified
|
||||
`verdict()` logic and `render_comparison()` exist and are tested, but the measured
|
||||
two-arm comparison table was not produced because `run_live` is `skipif`-gated and
|
||||
no LOCAL endpoint ran. The brief's Open Questions/Assumptions explicitly sanctions
|
||||
the skip-without-endpoint fallback and the plan ratified the endpoint-dependent
|
||||
split, so it is not a `BROKEN_SUCCESS_CRITERION`. SC-A's empirical half remains open
|
||||
until a LOCAL endpoint run fills `findings-a.md`.
|
||||
|
||||
Spikes A, C, D, the quality gate, and all seven Non-Goals trace cleanly to delivered
|
||||
code (see Coverage). No scope creep; no plan/execute drift; `src/**` untouched.
|
||||
|
||||
## Coverage
|
||||
|
||||
| File | Treatment | Reason |
|
||||
|------|-----------|--------|
|
||||
| uv.lock | skip | lockfile (`*.lock` pattern) |
|
||||
| CLAUDE.md | summary-only | reviewed (plan-sanctioned 1.8.0→1.9.0 version fix) |
|
||||
| pyproject.toml | summary-only | reviewed |
|
||||
| docs/fase1-spikes/README.md | summary-only | reviewed (Spike B rows updated) |
|
||||
| docs/fase1-spikes/findings-a.md | summary-only | reviewed |
|
||||
| docs/fase1-spikes/findings-b.md | summary-only | reviewed (§(b) + Method note rewritten) |
|
||||
| docs/fase1-spikes/findings-c.md | summary-only | reviewed |
|
||||
| docs/fase1-spikes/findings-d.md | summary-only | reviewed |
|
||||
| spikes/__init__.py | summary-only | reviewed |
|
||||
| spikes/_harness.py | summary-only | reviewed (received_texts + message_texts seam added) |
|
||||
| spikes/a_groupchat.py | summary-only | reviewed (MINOR fix: mkdir before write) |
|
||||
| spikes/b_footguns.py | summary-only | reviewed (fan-out rebuilt — BLOCKER fix) |
|
||||
| spikes/c_validator.py | summary-only | reviewed |
|
||||
| spikes/d_verdictstore.py | summary-only | reviewed |
|
||||
| tests/spikes/test_a_groupchat.py | summary-only | reviewed |
|
||||
| tests/spikes/test_b_footguns.py | summary-only | reviewed (fan-out tests rebuilt) |
|
||||
| tests/spikes/test_c_validator.py | summary-only | reviewed |
|
||||
| tests/spikes/test_d_verdictstore.py | summary-only | reviewed |
|
||||
| tests/spikes/test_harness.py | summary-only | reviewed |
|
||||
| tests/spikes/test_imports.py | summary-only | reviewed |
|
||||
|
||||
## Findings (BLOCKER)
|
||||
|
||||
None.
|
||||
|
||||
## Findings (MAJOR)
|
||||
|
||||
None.
|
||||
|
||||
## Findings (MINOR)
|
||||
|
||||
None.
|
||||
|
||||
## Findings (SUGGESTION)
|
||||
|
||||
None.
|
||||
|
||||
## Suppressed Findings
|
||||
|
||||
None.
|
||||
|
||||
## Remediation Summary
|
||||
|
||||
- **BLOCKER:** 0
|
||||
- **MAJOR:** 0
|
||||
- **MINOR:** 0
|
||||
- **SUGGESTION:** 0
|
||||
|
||||
Round-1 BLOCKER and MINOR are both genuinely resolved by commit `a2dff21`,
|
||||
verified independently against MAF internals. **Fase 1 is clear to close.** The
|
||||
one open item (SC-A empirical comparison table) is brief-sanctioned and is carried
|
||||
into Fase 2 as a known LOCAL-endpoint task, not a blocker.
|
||||
|
||||
```json
|
||||
{
|
||||
"reviewer": "review-coordinator",
|
||||
"verdict": "ALLOW",
|
||||
"counts": { "BLOCKER": 0, "MAJOR": 0, "MINOR": 0, "SUGGESTION": 0 },
|
||||
"findings": []
|
||||
}
|
||||
```
|
||||
|
|
@ -1,339 +0,0 @@
|
|||
---
|
||||
type: trekbrief
|
||||
brief_version: "2.2"
|
||||
created: 2026-06-24
|
||||
task: "Fase 2 MVP vertical slice — one synthetic project end-to-end on MAF 1.9.0: debate → block-validate → HITL → learn"
|
||||
slug: fase2-mvp-vertical-slice
|
||||
project_dir: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/
|
||||
research_topics: 3
|
||||
research_status: complete
|
||||
auto_research: false
|
||||
interview_turns: 0
|
||||
source: interview
|
||||
framing: preserve
|
||||
phase_signals:
|
||||
- phase: research
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: plan
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: execute
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: review
|
||||
effort: high
|
||||
model: opus
|
||||
---
|
||||
|
||||
# Task: Fase 2 MVP vertical slice — one synthetic project end-to-end on MAF
|
||||
|
||||
> Generated by `/trekbrief` on 2026-06-24.
|
||||
> This brief is the contract between requirements and planning. `/trekplan`
|
||||
> reads it to produce the implementation plan. Every decision in the plan must
|
||||
> trace back to content in this brief.
|
||||
>
|
||||
> **Interview note (project driftsmodell):** this brief was driven from
|
||||
> documented operator intent — `docs/plan/2026-06-23-incremental-plan.md` §Fase 2,
|
||||
> `docs/research/2026-06-24-maf-capability-map.md` §1+§5, Fase 1 findings, and
|
||||
> `CLAUDE.md` invariants — not live Q&A, per the project's standing operating
|
||||
> model (operator's only input is the trigger phrase; the session executes the
|
||||
> next backbone step autonomously). `interview_turns: 0` reflects that. The
|
||||
> operator can correct any derived decision via the annotation HTML.
|
||||
|
||||
## TL;DR
|
||||
|
||||
Fase 2 **preserves** the locked plan, composing the verified Fase 1 spikes into one
|
||||
MVP vertical slice: ONE synthetic project end-to-end on MAF 1.9.0 — debate → blocking
|
||||
hybrid validator → two-layer HITL + provenance → ExpeL learning, in both profiles. MAF
|
||||
GA features wired per the capability-map (adopt tokens/budget-middleware/round-caps/@tool/MCP; keep hand-rolled VerdictStore + validator; promote `fresh_workflow()`). Fan-out over N projects is Fase 3.
|
||||
|
||||
## Intent
|
||||
|
||||
Fase 1 proved the four riskiest assumptions in isolation with throwaway spikes
|
||||
(maker-checker convergence, fan-out state-bleed, the hybrid validator, ExpeL
|
||||
retrieval) and produced a verified MAF 1.9.0 capability map that says, per need,
|
||||
whether to adopt a MAF feature or keep our own. We now need the first thing that
|
||||
is actually a *system*: a single project flowing through the whole method —
|
||||
candidate measures debated, a deterministic validator deciding the value, a domain
|
||||
expert judging via HITL, and the judgment fed back so the next run is smarter. This
|
||||
slice is the MVP backbone every later phase (generalisation in Fase 3, open-source
|
||||
in Fase 4, the D7 Claude-SDK sibling) builds on, so getting the seams right — token
|
||||
budget as middleware, fresh-workflow isolation factory, the inline validator gate,
|
||||
the ExpeL injection seam — matters more than breadth. The danger we are buying down
|
||||
is integration risk: each spike worked alone; the open question is whether they
|
||||
compose into one fail-fast, budget-bounded, provenance-stamped, learning loop on
|
||||
real (not faked) chat clients in both profiles.
|
||||
|
||||
## Goal
|
||||
|
||||
A runnable vertical slice in which one synthetic "anleggskostnad" project is taken
|
||||
end-to-end: load its data via a local-folder MCP source with citations, run the
|
||||
debate (single-agent baseline, escalating to Group Chat maker-checker under a
|
||||
deterministic termination contract with native round caps and a shared
|
||||
middleware-driven token budget), pass every candidate through the obligatory
|
||||
blocking hybrid validator (LLM → Pydantic IR → solver → Monte-Carlo P10/P50/P90)
|
||||
that returns either a `ValidatedProposal` or a structural `Rejection`, emit exactly
|
||||
one provenance-stamped saving proposal, capture a domain expert's verdict through a
|
||||
two-layer HITL, persist it to the hand-rolled VerdictStore, and have the *next* run
|
||||
retrieve that verdict via the ExpeL `ContextProvider` seam (with the corrected
|
||||
two-arg `extend_instructions(source_id, instructions)`). Nothing runs without
|
||||
valid, fail-fast-validated contracts (data-source schema, model-map, termination
|
||||
contract, feedback schema). The slice runs on the local profile (primary) and is
|
||||
verified minimally on the Azure/Foundry profile. The hand-rolled `Budget`/
|
||||
`TokenMeter` shrinks to a thin shared meter that a `ChatMiddleware` drives off real
|
||||
`UsageDetails` token counts; word-count proxies are gone.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **Fan-out over N projects (Fase 3).** Fase 2 *builds* `fresh_workflow()` as the
|
||||
core isolation factory (the documented MS State-Isolation pattern) but runs ONE
|
||||
project through it. Orchestrated fan-out and adding a second project via config
|
||||
only are Fase 3.
|
||||
- **Compliance functions (D3).** No DPIA/ROS/behandlingsformål features — the
|
||||
deployer owns those. We ship only the technical preconditions (local-only by
|
||||
default, provenance, no silent egress) plus a prominent disclaimer.
|
||||
- **Preview MAF features on the critical path (D2/D6).** mem0/redis memory,
|
||||
Cosmos/Redis history, and declarative YAML agent factory are Preview — kept OFF
|
||||
the MVP critical path. GA core only (sessions, middleware, observability, tools,
|
||||
MCP, checkpointing).
|
||||
- **Magentic orchestration.** Experimental; Group Chat maker-checker is the debate
|
||||
default (CLAUDE.md invariant).
|
||||
- **Replacing the hand-rolled VerdictStore or validator with MAF memory/eval.**
|
||||
Both MAF features are the wrong shape (bag-of-words / offline quality score) —
|
||||
capability-map verdict is KEEP ours.
|
||||
- **Chasing the last 10% (D5).** Build ~90% generic core + clear extension points;
|
||||
competent integrators configure the last mile.
|
||||
- **D7 Claude Agent SDK sibling.** Starts only after this slice works end-to-end.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Stack:** Python ≥3.10, `uv`, `ruff` (lint+format), `mypy`, `pytest`, Pydantic.
|
||||
- **MAF:** `agent-framework-core` 1.9.0 with GA packages pinned explicitly (NOT the
|
||||
`agent-framework[all]` meta). Promote `agent-framework-orchestrations` 1.0.0 from
|
||||
dev → core (it was throwaway-dev in Fase 1).
|
||||
- **API truth:** verified against *installed* 1.9.0 source via introspection; docs
|
||||
via the `microsoft-learn` MCP (Learn pages are NOT version-pinned — where source
|
||||
and docs disagree, installed source wins).
|
||||
- **Platform:** Intel mac (no Apple Silicon). CBC solver confirmed working in Fase 1
|
||||
(Spike C); PuLP via `pulp[cbc]` / `COIN_CMD`.
|
||||
- **Two profiles, one core API (D2):** local (fallback, primary for dev) +
|
||||
Azure/Foundry (full). Pluggable backend; same core API.
|
||||
- **Cost discipline (D6):** develop on the local profile (free); Foundry/Azure only
|
||||
for targeted, minimal verification; cheapest models, tiny synthetic data, hard
|
||||
token/round caps. No heavy test runs.
|
||||
- **Determinism is blocking:** the validator is obligatory and blocking — never an
|
||||
optional plugin; it cannot be bypassed.
|
||||
- **Fail-fast at startup:** no run begins without all contracts validated.
|
||||
- **STATE.md is local-only** (gitignored); the repo's private Forgejo remote is the
|
||||
only push target until Fase 4.
|
||||
|
||||
## Preferences
|
||||
|
||||
- **Adopt MAF GA features instead of reinventing (capability-map §1/§5):** real
|
||||
`UsageDetails` token counts (delete `len(text.split())`); budget cap as a shared
|
||||
`ChatMiddleware` short-circuit; native `GroupChatBuilder.with_max_rounds`; GA
|
||||
`@tool`/`FunctionTool` with explicit `schema=`; `MCPStdioTool` (local) /
|
||||
`MCPStreamableHTTPTool` (remote); observability via `gen_ai.client.token.usage`
|
||||
(derive cost = tokens × per-model pricing — there is no native cost metric).
|
||||
- **Promote `fresh_workflow()`** (Spike B) to a core fan-out factory — it IS the
|
||||
documented Microsoft isolation pattern; never reuse a built workflow across tasks.
|
||||
- **Keep hand-rolled** VerdictStore (structural Jaccard over typed cost-codes +
|
||||
measure-type + magnitude bucket) and validator (inline gate returning a domain
|
||||
object the system branches on). Promote the `ExpeLContextProvider` seam to core
|
||||
with the two-arg fix `extend_instructions(source_id, instructions)`.
|
||||
- **Debate ladder:** single-agent baseline first; escalate to Group Chat
|
||||
maker-checker only under documented need, under a deterministic termination
|
||||
contract (token-disiplin).
|
||||
- **Method as Agent Skill:** one `SKILL.md` + deterministic `scripts/` validator;
|
||||
MAF natively consumes `SKILL.md` via `SkillsProvider.from_paths(...)` — pin the
|
||||
experimental Skills surface and watch for breaks.
|
||||
- **Roll a tiny role→deployment map** (a dict/YAML → chat-client ctor); MAF
|
||||
declarative is Preview + not installed.
|
||||
- **Solver:** reuse Spike C's PuLP/CBC path in the skill's `scripts/` validator
|
||||
(proven on Intel mac) unless research surfaces a clearly better fit.
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
- **Hard caps enforced, fail-closed:** a token budget overrun short-circuits via the
|
||||
`ChatMiddleware`; a round overrun halts via `with_max_rounds`. No unbounded loop
|
||||
can start (Fase 1 B4: `max_round_count=None` does NOT self-terminate).
|
||||
- **No silent egress (D3 technical precondition):** on the local profile no project
|
||||
data leaves the machine; any egress is explicit and configured.
|
||||
- **Provenance on every emitted proposal (B6):** source citations + model/role +
|
||||
validator decision + token usage traceable on the single proposal.
|
||||
- **State isolation:** every project run gets a fresh workflow/session — zero
|
||||
cross-run conversation bleed (Fase 1 B7, verified on received-message content).
|
||||
- **Reproducible determinism:** the validator's accept/reject and the Monte-Carlo
|
||||
percentiles are deterministic for a fixed seed and input.
|
||||
- **Repo hygiene:** `uv run mypy src`, `uv run ruff check .` clean; tests pass.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- **Suite green:** `uv run pytest` exits 0 (including new Fase 2 tests);
|
||||
`uv run ruff check .` and `uv run mypy src` exit 0.
|
||||
- **End-to-end, one valid proposal:** a single-command run of the synthetic project
|
||||
on the local profile produces exactly one `ValidatedProposal` (not `Rejection`)
|
||||
carrying a populated provenance stamp (assert provenance fields: ≥1 citation,
|
||||
model/role, validator decision, token usage).
|
||||
- **Structural block works:** an out-of-range / infeasible candidate is blocked by
|
||||
the validator and returns a `Rejection` (assert type is `Rejection`, with reason).
|
||||
- **Verdict captured + persisted:** an expert verdict entered via the two-layer HITL
|
||||
is written to the VerdictStore (assert a verdict record exists for the proposal).
|
||||
- **Learning closes the loop:** a second run on a structurally similar proposal
|
||||
retrieves the prior verdict via the ExpeL `ContextProvider` (assert the retrieved
|
||||
verdict id == the one persisted), exercising `extend_instructions(source_id, …)`.
|
||||
- **Fail-fast contracts:** running with a malformed data-source / model-map /
|
||||
termination / feedback contract raises a validation error at startup BEFORE any
|
||||
agent call (assert it raises; assert no chat-client call was made).
|
||||
- **Hard-stop respected:** with a tiny budget/round cap, the run terminates via the
|
||||
middleware/round-cap path (assert `BudgetExceeded`-equivalent or round-cap halt;
|
||||
assert the loop did not exceed the cap).
|
||||
- **Real token accounting:** the core token meter reads token usage from
|
||||
`UsageDetails` (`response.usage_details["total_token_count"]`, `None`-safe) —
|
||||
assert the meter is populated from `UsageDetails` on a real run (positive,
|
||||
load-bearing assertion), and assert no `len(... .split())` word-count token proxy
|
||||
is introduced into `src/` (the Fase 1 proxy lives in `spikes/_harness.py`; `src/`
|
||||
must stay at 0 such proxies).
|
||||
- **Both profiles exercised:** the slice runs on the local profile (full run) and a
|
||||
trivial agent responds on the Azure/Foundry profile (targeted minimal check) —
|
||||
assert a successful response from each chat-client path.
|
||||
|
||||
## Research Plan
|
||||
|
||||
Three load-bearing topics the Fase 1 capability-map did NOT resolve. Each is
|
||||
answerable and gates a specific set of plan steps. (`research_topics: 3` in
|
||||
frontmatter == the three `### Topic` headings below; `research_status: pending` is
|
||||
correct — these run via the manual `/trekresearch` calls in *How to continue*
|
||||
BEFORE `/trekplan`; the orchestrator must not auto-advance to planning first.)
|
||||
|
||||
### Topic 1: Native human-in-the-loop in MAF workflows
|
||||
|
||||
- **Why this matters:** the "expert verdict captured via HITL" success criterion
|
||||
needs a concrete mechanism. If MAF has a native workflow pause/resume for human
|
||||
input (e.g. a `RequestInfoExecutor` / request-response executor or a
|
||||
checkpoint-and-resume seam), the two-layer HITL should use it; if not, we capture
|
||||
the verdict out-of-band post-run and feed it forward via the VerdictStore. The
|
||||
choice changes the workflow graph and the HITL design substantially.
|
||||
- **Research question:** "Does `agent-framework` 1.9.0 (core + orchestrations 1.0.0)
|
||||
provide a native human-in-the-loop primitive to pause a workflow for external
|
||||
input and resume it, and how does it interact with checkpointing and session
|
||||
state?"
|
||||
- **Suggested invocation:** `/trekresearch --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ --external "Does Microsoft Agent Framework 1.9.0 support native human-in-the-loop pause/resume in workflows (RequestInfoExecutor / request-response), and how does it interact with checkpointing and session state?"`
|
||||
- **Required for plan steps:** HITL design; per-project workflow graph; checkpointing
|
||||
integration; the verdict-capture step.
|
||||
- **Confidence needed:** high
|
||||
- **Estimated cost:** standard
|
||||
- **Scope hint:** both
|
||||
|
||||
### Topic 2: Local-folder data access via MCP with citation provenance
|
||||
|
||||
- **Why this matters:** the slice needs a `local_folder` data source surfaced as an
|
||||
MCP tool whose retrieved chunks carry enough metadata for a citation-aware
|
||||
`AIContextProvider` and the provenance stamp. We must know whether to use an
|
||||
existing filesystem MCP server (e.g. the official `@modelcontextprotocol/
|
||||
server-filesystem`) via `MCPStdioTool`, or build a thin custom local-folder MCP
|
||||
server that returns citation-able chunks. This decides a build-vs-reuse step and
|
||||
the provenance data model.
|
||||
- **Research question:** "What is the best way to expose a local document folder to
|
||||
a MAF agent via `MCPStdioTool` such that retrieved content carries citation
|
||||
metadata (file + locator) — is there a suitable existing filesystem MCP server, or
|
||||
is a thin custom server required, and what citation shape does an AIContextProvider
|
||||
expect?"
|
||||
- **Suggested invocation:** `/trekresearch --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ --external "Best way to expose a local document folder to a Microsoft Agent Framework agent via MCPStdioTool with citation metadata — existing filesystem MCP server vs thin custom server, and the citation shape an AIContextProvider expects?"`
|
||||
- **Required for plan steps:** data-access step; citation-aware context provider;
|
||||
provenance-stamping; build-vs-reuse decision for the local-folder server.
|
||||
- **Confidence needed:** high
|
||||
- **Estimated cost:** standard
|
||||
- **Scope hint:** both
|
||||
|
||||
### Topic 3: Real local-profile chat client for agent-framework 1.9.0
|
||||
|
||||
- **Why this matters:** every Fase 1 spike used `FakeChatClient` — no real LLM. The
|
||||
"both profiles" criterion needs a REAL local model on the primary (free) profile.
|
||||
We must know how `agent-framework` 1.9.0 talks to a local model (an
|
||||
OpenAI-compatible local endpoint via `OpenAIChatClient`, an Ollama integration, or
|
||||
another local `BaseChatClient`) and how `UsageDetails` is populated there (some
|
||||
providers return `None` — which our middleware must handle).
|
||||
- **Research question:** "How does `agent-framework-core` 1.9.0 run agents against a
|
||||
local model on the free/local profile (OpenAI-compatible endpoint, Ollama, or
|
||||
other local chat client), and does that path populate `UsageDetails` token counts
|
||||
or return `None`?"
|
||||
- **Suggested invocation:** `/trekresearch --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ --external "How does Microsoft agent-framework-core 1.9.0 run agents against a local model (OpenAI-compatible endpoint, Ollama, or local chat client) on a free local profile, and does it populate UsageDetails token counts?"`
|
||||
- **Required for plan steps:** backend-profile abstraction (local path); the budget
|
||||
middleware's `None`-handling; the end-to-end run; cost-discipline verification.
|
||||
- **Confidence needed:** high
|
||||
- **Estimated cost:** standard
|
||||
- **Scope hint:** both
|
||||
|
||||
## Open Questions / Assumptions
|
||||
|
||||
- **[ASSUMPTION]** Solver stays PuLP/CBC in the skill's `scripts/` validator (Spike
|
||||
C proved it on Intel mac); `mcp-solver` is not pursued unless Topic research shows
|
||||
a clear win.
|
||||
- **[ASSUMPTION]** The role→deployment model-map is a small hand-rolled dict/YAML →
|
||||
chat-client constructor; the available Foundry deployment names are
|
||||
tenant-specific (operator-supplied), not externally researchable.
|
||||
- **[ASSUMPTION]** "1 turn = 1 round" matches our intent for `with_max_rounds` on the
|
||||
Group Chat (capability-map flagged this to confirm during implementation).
|
||||
- **[ASSUMPTION]** The synthetic reference domain from Fase 0 (small fictional
|
||||
"anleggskostnad" projects) is reused as the single project for this slice; no new
|
||||
domain data is invented.
|
||||
- **[OPEN]** Whether checkpointing is on the Fase 2 critical path or deferred —
|
||||
capability-map said "ADOPT (later)"; default is defer past MVP unless Topic 1
|
||||
shows HITL needs checkpoint-resume.
|
||||
- **[OPEN]** Exact two-layer HITL semantics (synchronous review vs async +
|
||||
notification stub) — annenrangs risk in the plan; the notification stub (B11) is a
|
||||
stub only in Fase 2.
|
||||
- **[OPEN]** Which concrete local model runs on the Intel mac for the local profile.
|
||||
Topic 3 settles the *mechanism* (how 1.9.0 talks to a local model); the concrete
|
||||
installed model/endpoint is machine-specific and operator-supplied. The whole
|
||||
free-local-profile run hinges on a working local endpoint existing.
|
||||
- **[OPEN]** Which Foundry deployment names back the role→deployment map for the
|
||||
Azure/Foundry-profile check (tenant-specific, operator-supplied). The "both
|
||||
profiles exercised" success criterion depends on this operator input.
|
||||
|
||||
## Prior Attempts
|
||||
|
||||
Fase 0 (skeleton + synthetic domain + pluggable-backend shell) and Fase 1 (four
|
||||
throwaway de-risk spikes A–D) are complete and pushed. Fase 1 closed via
|
||||
`/trekreview` round 2 = ALLOW after a real BLOCKER was found and fixed: Spike B(b)'s
|
||||
fan-out "state isolation" had been confirmed with a tautological call-counter; the
|
||||
rebuilt experiment measured real conversation bleed on received-message content and
|
||||
confirmed the footgun, validated against Microsoft's own Workflows "State Isolation"
|
||||
doc. The MAF 1.9.0 capability map (the front-end input to this brief) was then
|
||||
produced from two grounded research passes (installed-source introspection + MS
|
||||
Learn MCP). Key Fase 1 facts carried forward: `BaseChatClient` drives the GA
|
||||
builders; `max_round_count=None` does NOT self-terminate (external guard required);
|
||||
reused workflow accumulates the conversation thread (use a fresh instance per task);
|
||||
`ContextProvider` requires `source_id`; `extend_instructions` is two-arg; real token
|
||||
usage lives on `UsageDetails`, not word-counts.
|
||||
|
||||
## Metadata
|
||||
|
||||
- **Created:** 2026-06-24
|
||||
- **Interview turns:** 0 (driven from documented operator intent per project driftsmodell)
|
||||
- **Auto-research opted in:** no
|
||||
- **Source:** trekbrief interview
|
||||
|
||||
---
|
||||
|
||||
## How to continue
|
||||
|
||||
Manual (default):
|
||||
|
||||
```bash
|
||||
# Run each research topic (order does not matter):
|
||||
/trekresearch --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ --external "Does Microsoft Agent Framework 1.9.0 support native human-in-the-loop pause/resume in workflows (RequestInfoExecutor / request-response), and how does it interact with checkpointing and session state?"
|
||||
/trekresearch --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ --external "Best way to expose a local document folder to a Microsoft Agent Framework agent via MCPStdioTool with citation metadata — existing filesystem MCP server vs thin custom server, and the citation shape an AIContextProvider expects?"
|
||||
/trekresearch --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ --external "How does Microsoft agent-framework-core 1.9.0 run agents against a local model (OpenAI-compatible endpoint, Ollama, or local chat client) on a free local profile, and does it populate UsageDetails token counts?"
|
||||
|
||||
# Then plan:
|
||||
/trekplan --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/
|
||||
|
||||
# Then execute:
|
||||
/trekexecute --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/
|
||||
```
|
||||
|
||||
Auto (opt-in during `/trekbrief`): research and planning run
|
||||
automatically; only execution is manual.
|
||||
|
|
@ -1,458 +0,0 @@
|
|||
<!--
|
||||
Remediation plan generated from a type:trekreview input (Handover 6). Lists the
|
||||
40-char hex IDs of the BLOCKER + MAJOR findings consumed from review.md.
|
||||
|
||||
---
|
||||
source_findings:
|
||||
- 4d8f1a2c9b6e3057f4a1d8c2b9e0537a6c1f4e92
|
||||
- 7b3e9c1d8a4f2056e9b0c3d7f1a85249c6e3b801
|
||||
- 9f2a4d7c1e8b3056a4f9d2c7b1e08345f6c29d1b
|
||||
- 2c7e1a9f4d8b3650c1e7a4f2d9b08537e6c14a90
|
||||
- 5a9d2f7c4e1b8360a9f4d7c2e1b95048f3c6a72d
|
||||
- 8e1c4a7f2d9b5063c8a1e4f7d2b9043507c6e1a8
|
||||
- 3f7a1c9e4d2b8560f7a3c1e9d4b25081c6e7a34f
|
||||
---
|
||||
-->
|
||||
|
||||
# Fase 2 MVP Vertical Slice — Remediation (connect the three load-bearing seams)
|
||||
|
||||
> **Plan quality: A** (91/100) — APPROVE_WITH_NOTES
|
||||
>
|
||||
> Generated by trekplan v5.6.1 on 2026-06-26 — `plan_version: 1.7`
|
||||
>
|
||||
> **Input:** `review.md` (type:trekreview, verdict BLOCK). Consumes the 2 BLOCKER +
|
||||
> 5 MAJOR findings as the actionable remediation set (MINOR F9 noted optional).
|
||||
> **Destination note (deviation from default):** the skill's default `--brief`+`project_dir`
|
||||
> destination is `{project_dir}/plan.md`, which is the ORIGINAL Fase 2 plan (the audit
|
||||
> trail referenced by `progress.json` + `review.md`). This remediation is written to
|
||||
> `plan-remediation.md` to avoid clobbering that record.
|
||||
|
||||
## Context
|
||||
|
||||
`/trekreview` returned **BLOCK**: the slice's three load-bearing seams — the
|
||||
debate→validator dataflow, the `ChatMiddleware` token-budget enforcement, and the
|
||||
MCP/retrieval tool path — are each built and unit-tested in isolation but are **never
|
||||
connected by the orchestrator** (`run.py`). The e2e suite is green because it tests the
|
||||
pieces against stand-ins and asserts proposal *shape* only. This plan wires the seams
|
||||
together so the brief's core contract — "candidate measures debated → a deterministic
|
||||
validator decides the value → enforced via a shared `ChatMiddleware` token budget,
|
||||
provenance-stamped" (brief Goal + NFR) — is actually delivered, not decorative.
|
||||
|
||||
The danger the review surfaced is that the green suite *masks* the defect. So a hard
|
||||
constraint on this remediation: **the tests must exercise the real mechanism, not a
|
||||
stand-in.** The single most important discovery during planning is that the existing
|
||||
test double (`SyntheticUsageChatClient`) extends the **minimal `BaseChatClient`**, which
|
||||
has **no `ChatMiddlewareLayer`** — so attaching the budget middleware to the debate
|
||||
agents would silently no-op in every test, reproducing the exact "green tests, dead
|
||||
mechanism" failure the review caught. The remediation therefore fixes the test double
|
||||
(Step 1) before wiring the mechanism (Steps 2–4), and proves the mechanism fires on a
|
||||
real layered client (Step 5).
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "run_project orchestrator (run.py) — the seams this plan connects"
|
||||
meter["TokenMeter(Budget)"]
|
||||
mw["BudgetMiddleware(meter) (NEW wiring)"]
|
||||
tool["make_retrieval_tool(docs_dir) (NEW wiring)"]
|
||||
fw["fresh_workflow(factory, tools=[tool], middleware=[mw])"]
|
||||
debate["debate = GroupChat (proposer/checker)"]
|
||||
run["result = await debate.run(...) → get_outputs() (NEW: captured)"]
|
||||
gen["generate_via_llm(proposer, debate_output+context, meter)"]
|
||||
val["validate_proposal → ValidatedProposal | Rejection"]
|
||||
stamp["ProvenanceStamp(token_usage=meter.tokens)"]
|
||||
|
||||
meter --> mw
|
||||
mw --> fw
|
||||
tool --> fw
|
||||
fw --> debate
|
||||
debate --> run
|
||||
run -->|debate_output| gen
|
||||
gen --> val
|
||||
val --> stamp
|
||||
end
|
||||
subgraph "MAF 1.9.0 (installed-source-verified)"
|
||||
layer["Agent(client, tools=, middleware=) — middleware fires only if client has ChatMiddlewareLayer"]
|
||||
mw -.attaches via.-> layer
|
||||
tool -.attaches via.-> layer
|
||||
end
|
||||
```
|
||||
|
||||
## Codebase Analysis
|
||||
|
||||
- **Tech stack:** Python ≥3.10, `uv`, `ruff`, `mypy`, `pytest` (asyncio_mode=auto),
|
||||
Pydantic 2.x. MAF `agent-framework-core` 1.9.0 + `-openai`/`-foundry`/`-orchestrations`.
|
||||
- **Key patterns:** fail-fast contracts before any chat client; first-class
|
||||
`ProvenanceStamp`; validator-as-retry as the reliability mechanism; `fresh_workflow`
|
||||
isolation factory; injected `client_factory` test seam returning synthetic-usage clients.
|
||||
- **Relevant files (all real, verified):**
|
||||
- `src/portfolio_optimiser/run.py` — orchestrator; carries F1, F2-wiring (F5), F7-wiring, F9.
|
||||
- `src/portfolio_optimiser/workflow.py` — `fresh_workflow`/`maker_checker_agents`; no `tools`/`middleware` params today.
|
||||
- `src/portfolio_optimiser/budget.py` — `BudgetMiddleware(ChatMiddleware)` (defined, unwired).
|
||||
- `src/portfolio_optimiser/datasource.py` — `make_retrieval_tool` → GA `FunctionTool` (defined, unexposed).
|
||||
- `src/portfolio_optimiser/generate.py` — `generate_via_llm` (async parse→validate→retry, meter-bounded).
|
||||
- `src/portfolio_optimiser/backends.py` — `OpenAIChatCompletionClient` (LOCAL) / `FoundryChatClient` (AZURE).
|
||||
- `tests/conftest.py` — `SyntheticUsageChatClient(BaseChatClient)` ← **minimal base, no middleware layer**.
|
||||
- `tests/test_vertical_slice_e2e.py`, `tests/test_budget.py`, `tests/test_workflow.py`.
|
||||
- `pyproject.toml` — `[dependency-groups]` (PEP 735) dev group.
|
||||
- **Reusable code:** `BudgetMiddleware`, `make_retrieval_tool`, `fresh_workflow`,
|
||||
`generate_via_llm`, `SyntheticUsageChatClient` (to be re-based), all e2e fixtures.
|
||||
- **Recent git activity:** `308b553` closed `/trekexecute` (14/14); `6ef4efc` recorded the
|
||||
BLOCK review. Branch `main`, clean.
|
||||
|
||||
## Research Sources
|
||||
|
||||
Installed-source introspection (MAF 1.9.0) performed during planning is the authority
|
||||
("installed source wins" — brief Constraints). Each mechanism below was verified
|
||||
empirically, not assumed.
|
||||
|
||||
| Mechanism | Source | Key finding | Confidence |
|
||||
|-----------|--------|-------------|------------|
|
||||
| `Agent(tools=, middleware=)` | `agent_framework._agents.Agent.__init__` (introspected) | Both kwargs exist; `ChatMiddleware ∈ MiddlewareTypes` | high |
|
||||
| Middleware application | `OpenAIChatCompletionClient.__mro__` = `…→ChatMiddlewareLayer→…→BaseChatClient` | Chat middleware fires only if the client carries `ChatMiddlewareLayer`; minimal `BaseChatClient` does NOT | high |
|
||||
| Middleware fires + propagates + override + strict_usage | empirical run (offline: layered synthetic client subclassing `OpenAIChatCompletionClient`, tiny cap) | `BudgetExceeded` raised by middleware **propagates out of `debate.run()`**; the `_inner_get_response(*, messages, stream, options, **kwargs)` override **intercepts offline (no network call)**; the synthetic `UsageDetails` **reaches `context.result`** inside the real middleware layer — it raised `BudgetExceeded`, NOT `UsageUnavailable`, so `strict_usage=True` propagation holds | high |
|
||||
| `debate.run()` output capture | empirical run (FakeChatClient debate to completion) | `get_outputs()` returns `AgentResponse` (has `.text`/`.messages`); **by DEFAULT it surfaces only the orchestrator's "reached max rounds" notice** (author `group_chat_orchestrator`) — the proposer's converged text is surfaced ONLY when the builder sets `output_from=[proposer]`. `get_intermediate_outputs()` is empty; the event list carries no participant `.text` | high |
|
||||
| Local client / UsageDetails | research 03 (`research/03-local-profile-chat-client.md`) | `OpenAIChatCompletionClient` non-streaming populates `UsageDetails` None-safely | high |
|
||||
| Pathguard on `.env*` | `~/.claude/hooks/pre-write-pathguard.sh:90` regex `\.env(\.[a-zA-Z]+)?$` | `env.template` (no leading dot) is permitted; `.env.template` is blocked | high |
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
Ordered by dependency. Steps 1–5 keep the suite green at every checkpoint; the test
|
||||
double is re-based (Step 1) *before* the mechanism is wired (Steps 2–4) so the mechanism
|
||||
is provable, not theater.
|
||||
|
||||
### Step 1: Re-base the synthetic test client on the layered client so chat middleware fires
|
||||
|
||||
- **Files:** `tests/conftest.py`
|
||||
- **Changes:** Change `SyntheticUsageChatClient` to subclass `agent_framework_openai.OpenAIChatCompletionClient` instead of the minimal `agent_framework.BaseChatClient`, so it inherits the `ChatMiddlewareLayer` (verified empirically: the minimal base has no middleware layer, so a `ChatMiddleware` attached to an agent silently no-ops; with the layered client the middleware fired offline and raised `BudgetExceeded`). Construct via `super().__init__(model="synthetic", api_key="synthetic", base_url="http://127.0.0.1:9/v1")` (offline — verified no network call until a request is actually sent). Keep the existing behavior verbatim: scripted/default replies, `call_count`, the **`tokens_per_reply` constructor kwarg name (unchanged)**, and a synthetic `UsageDetails(total_token_count=tokens_per_reply)` returned from the `_inner_get_response(*, messages, stream, options, **kwargs)` override (verified to be the slot the layered client calls for the raw response — it intercepts offline). Preserve the `make_client_factory`, `fresh_store`, `seeded_store`, `docs_dir` fixtures unchanged. No production code changes in this step.
|
||||
- **Reuses:** existing `SyntheticUsageChatClient` body (`tests/conftest.py:26-73`); `OpenAIChatCompletionClient` construction pattern from `src/portfolio_optimiser/backends.py:89-94`.
|
||||
- **Test first:**
|
||||
- File: `tests/conftest.py` is test infra; the regression gate is the existing suite.
|
||||
- Verifies: the re-based double is a drop-in — every existing test still passes (the double still emits synthetic `UsageDetails`, so strict-usage accounting does not hard-fail and `token_usage > 0` still holds). `conftest.py` is shared by the whole suite, so the gate MUST be the full suite, not a subset.
|
||||
- Pattern: `tests/conftest.py:26-73` (current double).
|
||||
- **Verify:** `uv run pytest -q` → expected: `97 passed, 3 skipped` (unchanged from pre-remediation — this proves the re-base is a true drop-in across every fixture consumer).
|
||||
- **On failure:** retry once adjusting the `_inner_get_response` override signature to match the layered client's call contract; if the layered base demands network on construction, escalate (it does not — verified offline). Then revert — `git checkout -- tests/conftest.py`.
|
||||
- **Checkpoint:** `git commit -m "test(fase2): re-base synthetic client on layered client so chat middleware fires"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- tests/conftest.py
|
||||
min_file_count: 1
|
||||
commit_message_pattern: "^test\\(fase2\\): re-base synthetic client on layered client so chat middleware fires$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/run.py
|
||||
- src/portfolio_optimiser/budget.py
|
||||
must_contain:
|
||||
- path: tests/conftest.py
|
||||
pattern: "OpenAIChatCompletionClient"
|
||||
```
|
||||
|
||||
### Step 2: Thread tools + middleware through fresh_workflow onto each debate agent
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/workflow.py`, `tests/test_workflow.py`
|
||||
- **Changes:** Add keyword-only params `tools: Sequence[Any] | None = None` and `middleware: Sequence[Any] | None = None` to both `maker_checker_agents(...)` and `fresh_workflow(...)`. In `maker_checker_agents`, pass them to each agent: `Agent(client_factory(role), _INSTRUCTIONS[role], name=role, tools=tools, middleware=middleware)`. `fresh_workflow` forwards both to `maker_checker_agents`. **Also set `output_from=[agents[0]]` (the proposer) on the `GroupChatBuilder`** — this is a verified prerequisite for F1: empirically, `get_outputs()` by default surfaces ONLY the orchestrator's "reached max rounds" notice; `output_from=[proposer]` is what surfaces the proposer's converged text (Step 4 consumes it). Defaults `None` preserve current behavior. Update the module docstring line that claims the budget middleware is "wired by the orchestrator" to point at the new param. (Verified: `Agent.__init__` accepts both `tools=`/`middleware=`; `ChatMiddleware ∈ MiddlewareTypes`; `GroupChatBuilder(output_from=[proposer])` surfaces the proposer output.)
|
||||
- **Reuses:** `maker_checker_agents` / `fresh_workflow` (`workflow.py:45-86`); `Agent` signature + `GroupChatBuilder(output_from=)` (introspected).
|
||||
- **Test first:**
|
||||
- File: `tests/test_workflow.py` (existing — add)
|
||||
- Verifies: TWO assertions. (a) Construction-spy: monkeypatch `portfolio_optimiser.workflow.Agent` with a recorder, call `maker_checker_agents(factory, tools=[sentinel_tool], middleware=[sentinel_mw])`, assert every recorded construction received `tools` containing `sentinel_tool` and `middleware` containing `sentinel_mw` (`.tools` is not a public attr on a built `Agent`, so assert at the construction boundary). (b) Behavioral: run a real `fresh_workflow(scripted_factory).run(...)` where the proposer's FakeChatClient reply is a distinctive marker, and assert that marker appears in `[getattr(o,"text","") for o in result.get_outputs()]` — proving `output_from=[proposer]` actually surfaces the proposer's output (not just the orchestrator notice).
|
||||
- Pattern: `tests/test_workflow.py:14-40` (factory style + `monkeypatch.setattr`).
|
||||
- **Verify:** `uv run pytest tests/test_workflow.py -q` → expected: `all passed`
|
||||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/workflow.py tests/test_workflow.py`
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): thread tools + budget middleware through fresh_workflow onto agents"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/workflow.py
|
||||
- tests/test_workflow.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase2\\): thread tools \\+ budget middleware through fresh_workflow onto agents$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/budget.py
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/workflow.py
|
||||
pattern: "middleware=middleware"
|
||||
- path: src/portfolio_optimiser/workflow.py
|
||||
pattern: "tools=tools"
|
||||
- path: src/portfolio_optimiser/workflow.py
|
||||
pattern: "output_from"
|
||||
```
|
||||
|
||||
### Step 3: Construct and attach BudgetMiddleware + retrieval tool in the orchestrator
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/run.py`, `tests/test_vertical_slice_e2e.py`
|
||||
- **Changes:** In `run_project`, after building `meter` (`run.py:121`), construct `mw = BudgetMiddleware(meter)` and `retrieval_tool = make_retrieval_tool(docs_dir, top_k=top_k)`, then pass both into the factory call: `debate = fresh_workflow(factory, max_rounds=max_rounds, enable_layer1_hitl=enable_layer1_hitl, tools=[retrieval_tool], middleware=[mw])`. Add the imports `from portfolio_optimiser.budget import Budget, TokenMeter, BudgetMiddleware` and `from portfolio_optimiser.datasource import ..., make_retrieval_tool`. This makes the debate's per-call token usage flow through the shared meter via the `ChatMiddleware` (the brief's named mechanism) and gives the agents the citation-bearing data-source tool. (F2 definition-site budget.py:81 + F5 wiring-site run.py:121 + F7 tool exposure all resolved here.)
|
||||
- **Reuses:** `BudgetMiddleware` (`budget.py:81`), `make_retrieval_tool` (`datasource.py:49`), `fresh_workflow` new params (Step 2).
|
||||
- **Test first:**
|
||||
- File: `tests/test_vertical_slice_e2e.py` (existing — add)
|
||||
- Verifies: a spy test that monkeypatches `portfolio_optimiser.run.fresh_workflow` to capture its kwargs, runs `run_project` with the synthetic factory, and asserts the captured `middleware` is a list containing a `BudgetMiddleware` and `tools` is a list containing a `FunctionTool` (proves the orchestrator wires both, independent of MAF internals).
|
||||
- Pattern: `tests/test_vertical_slice_e2e.py:85-100` (`spy` factory style).
|
||||
- **Verify:** `uv run pytest tests/test_vertical_slice_e2e.py -q` → expected: `all passed`
|
||||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/run.py tests/test_vertical_slice_e2e.py`
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): wire BudgetMiddleware + retrieval tool onto the debate in run_project"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/run.py
|
||||
- tests/test_vertical_slice_e2e.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase2\\): wire BudgetMiddleware \\+ retrieval tool onto the debate in run_project$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/budget.py
|
||||
- src/portfolio_optimiser/validator.py
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/run.py
|
||||
pattern: "BudgetMiddleware\\(meter\\)"
|
||||
- path: src/portfolio_optimiser/run.py
|
||||
pattern: "make_retrieval_tool\\("
|
||||
```
|
||||
|
||||
### Step 4: Feed the debate's converged output into candidate generation
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/run.py`, `tests/test_vertical_slice_e2e.py`
|
||||
- **Changes:** Capture the debate result: `result = await debate.run(...)` (currently a bare statement at `run.py:124`). Add a helper `_debate_text(result) -> str` with the **verified** extraction: iterate `result.get_outputs()`, read each element's `.text`, **exclude the orchestrator's termination notice** (an element whose `author_name == "group_chat_orchestrator"`, or whose text contains `"reached the maximum number of rounds"`), and return the last remaining non-empty text (the proposer's converged output surfaced by `output_from=[proposer]` from Step 2); return `""` if none. Build the generation context from BOTH the debate output and the citations: `gen_context = _debate_text(result) or context` — passed to `generate_via_llm(factory("proposer"), project, gen_context, meter)`, so the validated proposal provably derives from the debate (retrieval `context` only as a last-resort fallback; note: the fallback path is a *test failure* per `test_g`, not a silent degradation — the test asserts the debate marker reaches generation). Add `debate_output: str` to `RunResult` and populate it (the minimal observable surface F1's mandated assertion needs). **F9 (MINOR, the `fake-model` provenance label) is OUT OF SCOPE for this remediation** — it is advisory only (Handover 6 excludes MINOR); deferred as an optional follow-on, NOT addressed here, to keep this step focused on the BLOCKER.
|
||||
- **Reuses:** `generate_via_llm` (`generate.py:92`), `WorkflowRunResult.get_outputs()` returning `AgentResponse` with `.text` (verified), `RunResult` dataclass (`run.py:52`).
|
||||
- **Test first:**
|
||||
- File: `tests/test_vertical_slice_e2e.py` (existing — add)
|
||||
- Verifies: `test_g_validated_proposal_derives_from_debate` — a spy captures the `context` argument passed to `generate_via_llm` (monkeypatch `portfolio_optimiser.run.generate_via_llm`), scripts the debate agents (via the proposer's FakeChatClient/synthetic reply) to emit a distinctive marker string, runs `run_project`, and asserts (a) `result.debate_output` is non-empty and contains the scripted marker, and (b) the captured generation context contains that marker — i.e. deleting the debate would change the generation input (the precise gap the review named). The marker is chosen so it could only come from the debate, never from the retrieval `docs_dir` fixture.
|
||||
- Pattern: `tests/test_vertical_slice_e2e.py:85-100` (spy) + `:28-37` (assertion style).
|
||||
- **Verify:** `uv run pytest tests/test_vertical_slice_e2e.py -q` → expected: `all passed`
|
||||
- **On failure:** the extraction shape is verified (`AgentResponse.text`, orchestrator notice filtered by author); if `test_g` fails, the orchestrator-notice filter or the `output_from` wiring (Step 2) is the cause — fix the filter predicate, do not loosen the assertion. Else revert — `git checkout -- src/portfolio_optimiser/run.py tests/test_vertical_slice_e2e.py`
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): feed debate converged output into candidate generation"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/run.py
|
||||
- tests/test_vertical_slice_e2e.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase2\\): feed debate converged output into candidate generation$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/generate.py
|
||||
- src/portfolio_optimiser/validator.py
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/run.py
|
||||
pattern: "get_outputs\\("
|
||||
- path: src/portfolio_optimiser/run.py
|
||||
pattern: "debate_output"
|
||||
```
|
||||
|
||||
### Step 5: Prove the middleware short-circuits a real chat call and the debate path
|
||||
|
||||
- **Files:** `tests/test_budget.py`, `tests/test_vertical_slice_e2e.py`
|
||||
- **Changes:** Add the integration test the review demands (F8). In `tests/test_budget.py`, add `test_budget_middleware_fires_on_real_agent_chat`: build an `Agent(SyntheticUsageChatClient(tokens_per_reply=8), "x", name="proposer", middleware=[BudgetMiddleware(TokenMeter(Budget(max_tokens=5, max_rounds=10)))])`, `await agent.run("hi")`, assert `BudgetExceeded` is raised from the middleware path with `kind == "tokens"` (verified empirically: meter charged to 8 > cap 5). In `tests/test_vertical_slice_e2e.py`, add `test_h_tiny_budget_halts_via_debate_middleware` — and make it genuinely load-bearing (the critic's catch): **monkeypatch `portfolio_optimiser.run.generate_via_llm` to raise `AssertionError` if it is ever called**, then run `run_project` with a tiny `max_tokens` and per-reply tokens above it, asserting `BudgetExceeded` is raised. Because `generate_via_llm` independently raises `BudgetExceeded` (`generate.py:111`), a plain `pytest.raises` would pass even if the debate middleware no-ops (the exact "green-but-dead" trap) — failing the test the moment generation is reached forces the `BudgetExceeded` to originate in the **debate** `ChatMiddleware` (verified it propagates out of `debate.run()`), so detaching the debate middleware fails this test. Keep the existing `test_e` as-is (still green; its mechanism shifts from the generate loop to the debate middleware — note this in its docstring).
|
||||
- **Reuses:** `BudgetMiddleware` (`budget.py:81`), the re-based `SyntheticUsageChatClient` (Step 1), `run_project` wiring (Steps 2-4).
|
||||
- **Test first:**
|
||||
- File: `tests/test_budget.py` (existing — add) and `tests/test_vertical_slice_e2e.py` (existing — add)
|
||||
- Verifies: detaching `BudgetMiddleware` from the agent now fails a test (the middleware↔real-client integration is no longer untested — the exact gap from F8).
|
||||
- Pattern: `tests/test_budget.py:41-56` (middleware assertion) + `tests/test_vertical_slice_e2e.py:74-82` (tiny-budget e2e).
|
||||
- **Verify:** `uv run pytest tests/test_budget.py tests/test_vertical_slice_e2e.py -q` → expected: `all passed`
|
||||
- **On failure:** revert — `git checkout -- tests/test_budget.py tests/test_vertical_slice_e2e.py`
|
||||
- **Checkpoint:** `git commit -m "test(fase2): assert BudgetMiddleware short-circuits a real chat call + the debate"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- tests/test_budget.py
|
||||
- tests/test_vertical_slice_e2e.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^test\\(fase2\\): assert BudgetMiddleware short-circuits a real chat call \\+ the debate$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/budget.py
|
||||
must_contain:
|
||||
- path: tests/test_budget.py
|
||||
pattern: "agent.run"
|
||||
```
|
||||
|
||||
### Step 6: Commit the env contract artifact under a pathguard-permitted name
|
||||
|
||||
- **Files:** `env.template` (new)
|
||||
- **Changes:** Create `env.template` at the repo root with the staged content documenting `PORTFOLIO_LOCAL_BASE_URL` / `PORTFOLIO_LOCAL_API_KEY`, the Foundry `PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT`, and the research-03 no-egress notes (loopback only, Ollama bind `127.0.0.1`, pin ≥ 0.17.1, `OLLAMA_DEBUG` unset, model-pull = explicit egress). The leading-dot name `.env.template` is blocked by `pre-write-pathguard.sh:90` (`\.env(\.[a-zA-Z]+)?$`); `env.template` is permitted (verified). **The staged file is the source of truth** for the no-egress notes (`…/eed58028-…/scratchpad/env.template.staged`, content confirmed during planning) — the env-var *names* are in `backends.py` but the no-egress notes are NOT, so if the staged file is GC'd, reconstruct the notes from `research/03-local-profile-chat-client.md` §Dim 6, do not just emit bare var names. Grep `README*`/`docs/` for any `\.env\.template` reference and reconcile it to `env.template`.
|
||||
- **Reuses:** staged template content (source of truth); env-var names from `src/portfolio_optimiser/backends.py:74,90-91`; no-egress notes from `research/03-local-profile-chat-client.md` §Dim 6.
|
||||
- **Test first:**
|
||||
- File: none (it is a config artifact, not code) — the manifest is the predicate.
|
||||
- Verifies: `env.template` exists, documents BOTH profiles' env vars, AND carries the no-egress notes (so the commit message's "+ no-egress notes" is actually true).
|
||||
- **Verify:** `test -f env.template && grep -q PORTFOLIO_LOCAL_BASE_URL env.template && grep -q PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT env.template && grep -q 127.0.0.1 env.template && grep -qi egress env.template && echo OK` → expected: `OK`
|
||||
- **On failure:** if the pathguard hook blocks the write, the name is wrong — confirm no leading dot; if the no-egress grep fails, the notes are missing — reconstruct from research/03. Then escalate.
|
||||
- **Checkpoint:** `git commit -m "docs(fase2): add env.template documenting both profiles + no-egress notes"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- env.template
|
||||
min_file_count: 1
|
||||
commit_message_pattern: "^docs\\(fase2\\): add env.template documenting both profiles \\+ no-egress notes$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: env.template
|
||||
pattern: "PORTFOLIO_LOCAL_BASE_URL"
|
||||
- path: env.template
|
||||
pattern: "127.0.0.1"
|
||||
```
|
||||
|
||||
### Step 7: Record the PEP 735 dependency-groups migration as an intentional deviation
|
||||
|
||||
- **Files:** `pyproject.toml`
|
||||
- **Changes:** The execute-time migration of the dev group from `[project.optional-dependencies]` to PEP 735 `[dependency-groups]` was beyond the original plan Step 1 scope but is the correct structure — it is what makes the brief's SC1 bare `uv sync` + `uv run pytest` install dev tooling without `--extra` (reverting would break SC1). Keep the structure and resolve the drift by flagging it: extend the existing comment block immediately above the `[dependency-groups]` table (confirm the anchor at execute time; the review cites the table at `pyproject.toml:25`) to state explicitly that this is an **intentional, recorded deviation from the original plan Step 1** (which authorized only moving `orchestrations` + `pulp` to core), justified by SC1, and cross-reference this remediation. (No dependency change; documentation only.)
|
||||
- **Reuses:** existing comment block `pyproject.toml:23-24`.
|
||||
- **Test first:**
|
||||
- File: none (config comment) — the manifest is the predicate.
|
||||
- Verifies: the deviation is now explicitly recorded in-tree (no longer silent drift).
|
||||
- **Verify:** `grep -qi "intentional" pyproject.toml && grep -qi "Step 1" pyproject.toml && echo OK` → expected: `OK`
|
||||
- **On failure:** revert — `git checkout -- pyproject.toml`
|
||||
- **Checkpoint:** `git commit -m "docs(fase2): record PEP 735 dependency-groups as intentional Step 1 deviation"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- pyproject.toml
|
||||
min_file_count: 1
|
||||
commit_message_pattern: "^docs\\(fase2\\): record PEP 735 dependency-groups as intentional Step 1 deviation$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/backends.py
|
||||
must_contain:
|
||||
- path: pyproject.toml
|
||||
pattern: "[Ii]ntentional"
|
||||
```
|
||||
|
||||
### Failure recovery rules
|
||||
|
||||
- **revert** — `git checkout -- {files}`, do not proceed.
|
||||
- **retry** — one alternative attempt, then revert.
|
||||
- **escalate** — stop; the issue needs human judgment.
|
||||
- **Checkpoint** — commit after each green step so later failures cannot corrupt completed work.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
| Approach | Pros | Cons | Why rejected |
|
||||
|----------|------|------|--------------|
|
||||
| **F1: Parse the proposer's final debate message directly into the IR and validate that** (skip `generate_via_llm`) | Most literal "candidate from debate" | Small local models leak text / emit non-JSON (research 03 Dim 3); needs the parse-retry loop anyway; fragile | Rejected — `generate_via_llm` IS the validator-as-retry reliability mechanism the research mandates; feed the debate output into it instead |
|
||||
| **F2: attach `BudgetMiddleware` to the chat client (not the agent)** | Conceptually "per-client" | `BaseChatClient.__init__` takes no `middleware`; the layered client applies it via `client_kwargs`/agent — the GA-canonical seam is `Agent(middleware=…)` | Rejected — `Agent(middleware=…)` is the documented + verified seam |
|
||||
| **F2: keep the minimal `BaseChatClient` test double, assert the meter via the generate loop only** | No conftest change | The middleware would never fire in tests — reproduces the review's "green-but-dead" defect | Rejected — the whole point is to test the real mechanism |
|
||||
| **F4: revert dev deps to `[project.optional-dependencies]`** | Matches the literal Step 1 plan | Breaks SC1 (`uv sync` would no longer install dev tooling without `--extra`) | Rejected — keep PEP 735, record the deviation instead |
|
||||
| **Write to `{project_dir}/plan.md` (skill default)** | Default path | Clobbers the original Fase 2 plan (audit trail) | Rejected — write `plan-remediation.md` |
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- **Framework:** `pytest`, `asyncio_mode=auto`; synthetic-usage clients (no LLM, deterministic).
|
||||
- **Existing patterns:** spy/recording factories; assertions on `RunResult`; synthetic `UsageDetails`.
|
||||
- **New/changed tests:** re-based double (Step 1); construction-spy for tool+middleware threading (Step 2); orchestrator-wiring spy (Step 3); debate-derivation test (Step 4); real-client middleware + debate-halt tests (Step 5). Net: the three seams each gain a test that fails if the seam is detached.
|
||||
|
||||
### Tests to write
|
||||
|
||||
| Type | File | Verifies | Model test |
|
||||
|------|------|----------|------------|
|
||||
| Unit | `tests/test_workflow.py` | tool+middleware threaded to each `Agent` | `tests/test_workflow.py:14-40` |
|
||||
| Integration | `tests/test_vertical_slice_e2e.py` `test_g_…` | validated proposal derives from the debate | `tests/test_vertical_slice_e2e.py:85-100` |
|
||||
| Integration | `tests/test_budget.py` `test_…fires_on_real_agent_chat` | middleware short-circuits a real `agent.run()` | `tests/test_budget.py:41-56` |
|
||||
| Integration | `tests/test_vertical_slice_e2e.py` `test_h_…` | tiny budget halts via the debate middleware | `tests/test_vertical_slice_e2e.py:74-82` |
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Priority | Risk | Location | Impact | Mitigation |
|
||||
|----------|------|----------|--------|------------|
|
||||
| Low (was Medium — now verified) | `get_outputs()` surfaces only the orchestrator notice unless `output_from=[proposer]` is set | `workflow.py` Step 2 / `run.py` Step 4 | `_debate_text` extracts boilerplate → F1 hollow | RESOLVED in planning: Step 2 sets `output_from=[proposer]` (empirically surfaces proposer text); Step 4 filters the orchestrator notice by author; `test_g` fails loudly if the marker doesn't reach generation |
|
||||
| Low (was Medium — now verified) | Re-basing the test double on the layered client changes call semantics | `tests/conftest.py` Step 1 | existing tests regress | Verified offline: layered client constructs with no network, `_inner_get_response` override intercepts, synthetic `UsageDetails` reaches the middleware (raised `BudgetExceeded`, not `UsageUnavailable`); Step 1 gate is the FULL suite |
|
||||
| Low | Middleware on debate agents shifts where `test_e` raises (generate loop → debate) | `test_vertical_slice_e2e.py` | semantic surprise | Keep `test_e` green; document the shift in its docstring; `test_h` (generate-fenced) makes the debate-path halt explicit |
|
||||
| Low | `make_retrieval_tool` attached but the local model is weak at tool-calling | `run.py` Step 3 | tool not invoked on a weak model | Out of scope for wiring; research 03 covers it; validator-as-retry absorbs weak tool use; exposure (not invocation reliability) is the F7 fix |
|
||||
|
||||
## Assumptions
|
||||
|
||||
| # | Assumption | Why unverifiable now | Impact if wrong |
|
||||
|---|-----------|---------------------|-----------------|
|
||||
| 1 | Re-based double is a drop-in for ALL existing tests (the 3 sampled categories passed offline during planning; the full 97/3 not re-run) | Full suite not re-run during planning | Step 1's verify gate is the full `uv run pytest` → caught immediately, before any mechanism wiring |
|
||||
| 2 | The staged `env.template` content is still at the prior-session scratchpad | Cross-session scratchpad may be GC'd | Reconstruct no-egress notes from `research/03` §Dim 6 (Step 6 fallback) — NOT from `backends.py` (which lacks them) |
|
||||
|
||||
*Note: the three planning-time unknowns that mattered most — the `get_outputs()` output shape (needs `output_from=[proposer]`), the `_inner_get_response` override + `strict_usage` propagation, and `BudgetExceeded` propagation out of `debate.run()` — were all empirically resolved during planning and are no longer assumptions.*
|
||||
|
||||
## Verification
|
||||
|
||||
End-to-end checks that cross step boundaries (per-step manifests are auto-checked by trekexecute):
|
||||
|
||||
- [ ] `uv sync` (fresh resolve) → expected: exit 0 and the dev tooling (pytest/ruff/mypy) installed WITHOUT `--extra` — this is the SC1 premise that motivates keeping PEP 735 (Step 7).
|
||||
- [ ] `uv run pytest` → expected: exit 0, strictly MORE tests than the pre-remediation `97 passed / 3 skipped` (the new seam tests), 0 failed.
|
||||
- [ ] `uv run ruff check . && uv run ruff format --check .` → expected: clean.
|
||||
- [ ] `uv run mypy src` → expected: clean.
|
||||
- [ ] `grep -n "get_outputs" src/portfolio_optimiser/run.py` → expected: a match (debate output captured, not discarded).
|
||||
- [ ] `grep -n "BudgetMiddleware(meter)" src/portfolio_optimiser/run.py` → expected: a match (middleware constructed + wired).
|
||||
- [ ] `grep -n "make_retrieval_tool(" src/portfolio_optimiser/run.py` → expected: a match (tool exposed to agents).
|
||||
- [ ] `test -f env.template` → expected: present.
|
||||
- [ ] **Detach assertion (the anti-theater check):** temporarily removing `middleware=[mw]` from the Step 3 wiring MUST make `test_h` fail (it is generate-fenced, so the `BudgetExceeded` can only come from the debate); and temporarily reverting Step 4's debate capture MUST make `test_g` fail. If either still passes detached, the seam is still theater — do not close Fase 2.
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Files to modify:** 6 (`run.py`, `workflow.py`, `pyproject.toml`, `conftest.py`, `test_workflow.py`, `test_budget.py`, `test_vertical_slice_e2e.py` — 7 distinct files across steps; production code is 2: `run.py`, `workflow.py`).
|
||||
- **Files to create:** 1 (`env.template`).
|
||||
- **Complexity:** medium (the mechanisms are verified; the work is precise wiring + test-double rebasing, not new design).
|
||||
|
||||
## Execution Strategy
|
||||
|
||||
7 steps; sequential within a single session (the code seams share `run.py`/`workflow.py`). Two independent artifact steps can run in parallel if desired.
|
||||
|
||||
### Session 1: Wire and prove the three seams
|
||||
- **Steps:** 1, 2, 3, 4, 5
|
||||
- **Wave:** 1
|
||||
- **Depends on:** none
|
||||
- **Scope fence:**
|
||||
- Touch: `tests/conftest.py`, `src/portfolio_optimiser/workflow.py`, `src/portfolio_optimiser/run.py`, `tests/test_workflow.py`, `tests/test_vertical_slice_e2e.py`, `tests/test_budget.py`
|
||||
- Never touch: `validator.py`, `budget.py` (definitions are correct), `generate.py`, `ir.py`
|
||||
|
||||
### Session 2: Artifact + record reconciliation
|
||||
- **Steps:** 6, 7
|
||||
- **Wave:** 1 (independent of Session 1 — can run in parallel)
|
||||
- **Depends on:** none
|
||||
- **Scope fence:**
|
||||
- Touch: `env.template`, `pyproject.toml`
|
||||
- Never touch: `src/`, `tests/`
|
||||
|
||||
### Execution Order
|
||||
- **Wave 1:** Session 1 + Session 2 (independent; run Session 1 sequentially step-by-step, Session 2 anytime)
|
||||
|
||||
### Grouping rules applied
|
||||
- Steps sharing `run.py` (3, 4) ordered and same session.
|
||||
- Artifact steps (6, 7) touch no code → separate parallelizable session.
|
||||
- Step 1 (test double) precedes the mechanism steps so the mechanism is provable.
|
||||
|
||||
## Plan Quality Score
|
||||
|
||||
| Dimension | Weight | Score | Notes |
|
||||
|-----------|--------|-------|-------|
|
||||
| Structural integrity | 0.15 | 94 | Dependency-ordered; test double before mechanism; Step 1 gates full suite |
|
||||
| Step quality | 0.20 | 92 | Each step 1–2 files, TDD; `test_h` generate-fenced; F9 removed from prose |
|
||||
| Coverage completeness | 0.20 | 95 | All 2 BLOCKER + 5 MAJOR mapped; F8 debate-path now genuinely closed |
|
||||
| Specification quality | 0.15 | 93 | All three load-bearing mechanics empirically verified vs installed source |
|
||||
| Risk & pre-mortem | 0.15 | 92 | Test-double no-op trap + `get_outputs()` boilerplate trap both caught up front |
|
||||
| Headless readiness | 0.10 | 91 | On-failure + Checkpoint per step; deferred-discovery language removed |
|
||||
| Manifest quality | 0.05 | 90 | All steps have checkable manifests + must_contain (incl. `output_from`, no-egress) |
|
||||
| **Weighted total** | **1.00** | **93** | **Grade: A** |
|
||||
|
||||
**Adversarial review:**
|
||||
- **Plan critic:** REVISE — 0 blocker, 5 major, 5 minor (score 74/C on the pre-revision draft). All 10 findings addressed below (3 of them were already empirically verified during planning and were under-documented, not wrong).
|
||||
- **Scope guardian:** ALIGNED — 0 scope creep, 0 finding-gaps; all 7 `source_findings` map to concrete steps; F9 correctly optional. 2 minor verification gaps addressed below.
|
||||
|
||||
## Revisions
|
||||
|
||||
| # | Finding (source) | Severity | Resolution |
|
||||
|---|---------|----------|------------|
|
||||
| 1 | `test_h` is theater — passes even if debate middleware no-ops, because `generate_via_llm` raises `BudgetExceeded` independently (plan-critic) | major | Step 5: `test_h` now monkeypatches `generate_via_llm` to fail if reached → `BudgetExceeded` must originate in the debate; detaching the debate middleware fails it |
|
||||
| 2 | F1 rests on an unverified `get_outputs()` shape; `or context` fallback could re-introduce F1 (plan-critic) | major | Verified empirically: default `get_outputs()` surfaces only the orchestrator notice; Step 2 sets `output_from=[proposer]`; Step 4 filters the notice; `test_g` fails (not degrades) if the debate marker doesn't reach generation |
|
||||
| 3 | `_inner_get_response` override slot unverified — could hit the network (plan-critic) | major | Verified empirically offline: the override intercepts, no network call, middleware fires; recorded in Research Sources |
|
||||
| 4 | Step 1 verify gate too narrow (conftest is shared) (plan-critic) | major | Step 1 now gates on full `uv run pytest` (97/3) |
|
||||
| 5 | `_debate_text` shape deferred to execute-time discovery (plan-critic) | major | Extraction now concrete (verified `AgentResponse.text` + orchestrator-notice filter); deferred-discovery language removed |
|
||||
| 6 | F9 optional embedded inside Step 4 prose (plan-critic) | minor | F9 removed from Step 4; explicitly out-of-scope optional follow-on |
|
||||
| 7 | env.template no-egress notes unverified; fallback can't reproduce them (plan-critic) | minor | Step 6 verify greps `127.0.0.1` + `egress`; manifest asserts `127.0.0.1`; fallback sources notes from research/03 |
|
||||
| 8 | `strict_usage` propagation on the debate path unverified (plan-critic) | minor | Verified empirically (middleware raised `BudgetExceeded`, not `UsageUnavailable`); recorded in Research Sources |
|
||||
| 9 | `debate_output` adds a field to frozen `RunResult` — confirm scope (plan-critic) | minor | Kept — the minimal observable surface F1's mandated assertion requires (scope-guardian concurred it is justified, not creep) |
|
||||
| 10 | Step 7 line-ref drift (`23-24` vs `25`) (plan-critic) | minor | Anchor de-brittled — references the comment block above `[dependency-groups]`, confirm at execute |
|
||||
| 11 | SC1 bare `uv sync` (Step 7's motivating premise) not verified (scope-guardian) | minor | Added `uv sync` fresh-resolve check to Verification |
|
||||
| 12 | Step 5 example used `tokens=8`; client kwarg is `tokens_per_reply` (scope-guardian) | minor | Corrected to `tokens_per_reply=8` |
|
||||
|
|
@ -1,686 +0,0 @@
|
|||
# Fase 2 — MVP Vertical Slice: one synthetic project end-to-end on MAF 1.9.0
|
||||
|
||||
> **Plan quality: A−** (88/100) — APPROVE_WITH_NOTES (revised after adversarial review)
|
||||
>
|
||||
> Generated by trekplan v2.0 on 2026-06-24 — `plan_version: 1.7`
|
||||
|
||||
## Context
|
||||
|
||||
Fase 1 proved the four riskiest assumptions in isolation with throwaway spikes
|
||||
(maker-checker convergence, fan-out state-bleed, the hybrid blocking validator,
|
||||
ExpeL retrieval) and produced a verified MAF 1.9.0 capability map. Fase 2 builds
|
||||
the first thing that is actually a *system*: ONE synthetic "anleggskostnad"
|
||||
project flowing through the whole method — candidate measures debated (Group Chat
|
||||
maker-checker), a deterministic validator deciding the value, a domain expert
|
||||
judging via HITL, and the judgment fed back so the next run is smarter — on
|
||||
**real** (not faked) chat clients, in both profiles. The danger being bought down
|
||||
is **integration risk**: each spike worked alone; the open question is whether they
|
||||
compose into one fail-fast, budget-bounded, provenance-stamped, learning loop. This
|
||||
slice is the MVP backbone every later phase builds on (Fase 3 fan-out, Fase 4
|
||||
open-source, the D7 Claude-SDK sibling), so getting the seams right matters more
|
||||
than breadth (brief §Intent).
|
||||
|
||||
Three research briefs (installed-1.9.0-source-verified) correct the brief's earlier
|
||||
leanings and are binding on this plan:
|
||||
- **HITL (research 01):** capture the durable learned verdict **out-of-band** in the
|
||||
VerdictStore; use the GA in-run gate (`GroupChatBuilder.with_request_info`) only
|
||||
for optional synchronous Layer-1 review; **defer checkpointing off the MVP path**.
|
||||
- **Data access (research 02):** the official `@modelcontextprotocol/server-filesystem`
|
||||
cannot carry citations → the data source is a **framework-agnostic in-process
|
||||
retriever** returning `{file, locator, snippet, score}` (the D7-portable seam),
|
||||
exposed to the agents as a GA tool/ContextProvider; provenance is first-class
|
||||
Pydantic data, not MAF's `Annotation` (open streaming-drop bug #4316).
|
||||
- **Local client (research 03):** use **`OpenAIChatCompletionClient(base_url=…)`
|
||||
non-streaming** (NOT `OpenAIChatClient`/Responses); `UsageDetails` is populated
|
||||
None-safely non-streaming; the blocking validator becomes the **retry/repair**
|
||||
reliability mechanism for weak local models.
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "src/portfolio_optimiser/ — Fase 2 vertical slice"
|
||||
RUN[run.py orchestrator<br/>single-command entry] --> CONTRACTS[contracts.py<br/>fail-fast loaders]
|
||||
RUN --> WF[workflow.py<br/>fresh_workflow + GroupChat maker-checker]
|
||||
RUN --> CAP[verdicts.py<br/>capture_verdict Layer-2 + VerdictStore + ExpeL]
|
||||
WF --> BK[backends.py<br/>LocalBackend OpenAIChatCompletionClient / FoundryChatClient + model-map]
|
||||
WF --> BUD[budget.py<br/>TokenMeter + ChatMiddleware off UsageDetails]
|
||||
WF --> TOOL[retrieval.py exposed as GA tool/ContextProvider]
|
||||
TOOL --> RET[retrieval.py<br/>in-process retriever core + path-security - D7 seam]
|
||||
WF --> GEN[generate.py<br/>LLM->IR + validator-as-retry]
|
||||
GEN --> VAL[validator.py + ir.py<br/>blocking: IR->CBC->Monte-Carlo]
|
||||
VAL --> PROV[provenance.py<br/>first-class Pydantic stamp]
|
||||
PROV --> CAP
|
||||
CAP -.next run.-> WF
|
||||
end
|
||||
REF[reference_domain.py + data/*.json<br/>Fase 0, settled] --> RUN
|
||||
```
|
||||
|
||||
## Codebase Analysis
|
||||
|
||||
- **Tech stack:** Python ≥3.10, `uv`, hatchling, `ruff` (line-length 100), `mypy`,
|
||||
`pytest`+`pytest-asyncio` (`asyncio_mode="auto"`). Installed (ground-truth-verified):
|
||||
`agent-framework-core` 1.9.0, `-openai` 1.8.2, `-foundry` 1.8.2, `-orchestrations`
|
||||
1.0.0; underlying `openai` SDK **2.43.0**; `pulp` **3.3.2**; `pydantic` 2.13.4. CBC
|
||||
binary present (`.venv/.../pulp/solverdir/cbc/osx/i64/cbc`). `mcp` SDK is **NOT
|
||||
installed** (governs Step 7's build-vs-defer decision).
|
||||
- **Key patterns:** backend profiles = Strategy+Factory (`backends.py:26-84`,
|
||||
`ChatBackend` `@runtime_checkable` Protocol + `Profile` enum + `get_backend()`,
|
||||
fail-fast `ValueError`); validator = staged pipeline returning a type-discriminated
|
||||
`ValidatedProposal | Rejection` (frozen dataclasses); debate = `GroupChatBuilder`
|
||||
round-robin with deterministic termination; learning = retrieval + `ContextProvider`
|
||||
injection. Tests: `tests/` (production) + `tests/spikes/` (mirror); gated live arms
|
||||
via `@pytest.mark.skipif(_NO_ENDPOINT)`; determinism asserted over fixed inputs.
|
||||
- **Relevant files (verified):** `src/portfolio_optimiser/backends.py` (D2 seam, both
|
||||
`create_chat_client` raise `NotImplementedError` `:50-61,64-76`),
|
||||
`reference_domain.py` (`Project`/`CostItem` frozen + `load_reference_projects()`),
|
||||
`data/reference_projects.json` (3 projects, project[0]=`FV42-GSV-E1`). Real
|
||||
injection seam for tests: `SessionContext` (exported, `_sessions.py:154`;
|
||||
`extend_instructions(source_id, instructions)` two-arg `:253`).
|
||||
- **Reusable code (promote from spikes — REUSE near-verbatim unless noted):**
|
||||
`spikes/c_validator.py:58-205` (Pydantic IR + validate_proposal + CBC + Monte-Carlo
|
||||
seed `_MC_SEED=20260624` + `self_repair` attempts-bounded + `proposal_for`;
|
||||
`generate_via_llm` `:207` is the stub); `spikes/d_verdictstore.py:29-227`
|
||||
(ProposalFeatures/Verdict/similarity/VerdictStore/`ExpeLContextProvider`/seed_store —
|
||||
**bug `:115`** one-arg `extend_instructions`, masked by single-arg fake
|
||||
`test_d_verdictstore.py:99`); `spikes/b_footguns.py:100-106` (`fresh_workflow()`);
|
||||
`spikes/a_groupchat.py:48-63,130-163` (termination + maker-checker builder);
|
||||
`spikes/_harness.py:41-94` (Budget/TokenMeter — drop `_word_tokens` `:97-99`).
|
||||
- **External tech (researched):** see Research Sources.
|
||||
- **Recent git activity:** `src/` written in Fase 0 (`b57aa83`), cold/settled. Spike
|
||||
cluster most-recently churned by the round-2 fan-out rebuild (`a2dff21`).
|
||||
Conventions: Conventional Commits, `feat(fase2):`, `[skip-docs]` on doc commits.
|
||||
|
||||
## Research Sources
|
||||
|
||||
| Technology | Source | Key Findings | Confidence |
|
||||
|-----------|--------|--------------|------------|
|
||||
| MAF native HITL | research/01 | `ctx.request_info`/`@response_handler`/`run(responses=)`; `GroupChatBuilder.with_request_info(agents=[checker])`; durable checkpoint-resume fragile → out-of-band verdict, defer checkpointing | high |
|
||||
| Local-folder MCP + citations | research/02 | official filesystem server can't cite → in-process retriever (D7 seam) + provenance as own Pydantic (route around #4316); `SessionContext.extend_instructions(source_id,…)` confirmed | high |
|
||||
| Local chat client + UsageDetails | research/03 | `OpenAIChatCompletionClient(base_url)` non-streaming; `UsageDetails` None-safe (`_chat_completion_client.py:705,757`); `add_usage_details()`; qwen3:4b; Intel-CPU = plumbing only | high |
|
||||
| Solver | Spike C + installed | `pulp` 3.3.2, bundled CBC works Intel mac; PULP_CBC_CMD deprecation; PuLP 4.0 → `pulp[cbc]`/`COIN_CMD` | high |
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
Deterministic-core-first ordering (MAF-free modules before MAF-coupled wiring). TDD
|
||||
throughout. **Dependency note:** `TextSpan` is owned by `retrieval.py` (Step 5) and
|
||||
imported by `provenance.py` (Step 6) — so Step 6 follows Step 5. **Headless halt
|
||||
rule (applies to every step): on any failure, perform the step's On-failure action
|
||||
then HALT the session — never run a dependent step on a failed predecessor.**
|
||||
|
||||
### Step 1: Promote orchestrations + pulp to core; resolve the mcp dependency decision
|
||||
|
||||
- **Files:** `pyproject.toml`
|
||||
- **Changes:** Move `agent-framework-orchestrations>=1.0.0` and `pulp>=2.8` from `dev` into `[project.dependencies]` (now MVP-runtime). Update the pulp comment to note installed 3.3.2 + the PuLP-4.0 `pulp[cbc]`/`COIN_CMD` migration. **mcp decision:** attempt `uv add mcp` — if it resolves a GA (non-pre) release, add it to core deps (enables the optional thin MCP-server wrapper in Step 7); if it resolves only as a pre-release, DO NOT add it (GA-pin discipline) and record that the MVP data source is the in-process retriever-as-tool (research 02 fallback), with the MCP-server wrapper deferred as a documented fast-follow. Re-lock. (Per brief Constraints.)
|
||||
- **Reuses:** existing dependency block `pyproject.toml:7-26`.
|
||||
- **Test first:**
|
||||
- File: `tests/test_imports_core.py` (new)
|
||||
- Verifies: `agent_framework_orchestrations` + `pulp` import; `PULP_CBC_CMD().available()` is True.
|
||||
- Pattern: `tests/spikes/test_imports.py`
|
||||
- **Verify:** `uv lock && uv sync && uv run pytest tests/test_imports_core.py` → expected: `1 passed`
|
||||
- **On failure:** retry once pinning exact installed versions; if still failing, escalate — then HALT.
|
||||
- **Checkpoint:** `git commit -m "build(fase2): promote orchestrations + pulp to core deps"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- pyproject.toml
|
||||
- tests/test_imports_core.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^build\\(fase2\\): promote orchestrations \\+ pulp to core deps$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/backends.py
|
||||
must_contain:
|
||||
- path: pyproject.toml
|
||||
pattern: "agent-framework-orchestrations"
|
||||
```
|
||||
|
||||
### Step 2: Promote the Pydantic IR + blocking validator to src/
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/ir.py` (new), `src/portfolio_optimiser/validator.py` (new)
|
||||
- **Changes:** Move `AffectedItem`/`SavingsProposal` (+ validators) into `ir.py`. Move `validate_proposal`, `ValidatedProposal`, `Rejection`, `_solve_max_feasible`, `_monte_carlo` (keep `_MC_SEED=20260624`), `self_repair` (attempts-bounded — promoted verbatim; token-budget bounding is added by the generate loop in Step 10, NOT here), `proposal_for`, `CbcUnavailable`, `_quiet_pulp` into `validator.py`. Pure modules — NO `agent_framework` import (D7-portable core). Import `Project` from `reference_domain`. (new files)
|
||||
- **Reuses:** `spikes/c_validator.py:44-205`; `reference_domain.Project`.
|
||||
- **Test first:**
|
||||
- File: `tests/test_validator.py` (new)
|
||||
- Verifies: valid IR → `ValidatedProposal` (`p10<=p50<=p90`); out-of-range → `Rejection` (reason, no percentiles); same fixed input twice → identical `(p10,p50,p90)`; Pydantic blocks negative qty / claim>affected-total.
|
||||
- Pattern: `tests/spikes/test_c_validator.py:24-78` (determinism over FIXED inputs, never an LLM)
|
||||
- **Verify:** `uv run pytest tests/test_validator.py` → expected: `5 passed`
|
||||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/ir.py src/portfolio_optimiser/validator.py tests/test_validator.py`; HALT.
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): promote blocking validator + IR to src"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/ir.py
|
||||
- src/portfolio_optimiser/validator.py
|
||||
- tests/test_validator.py
|
||||
min_file_count: 3
|
||||
commit_message_pattern: "^feat\\(fase2\\): promote blocking validator \\+ IR to src$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/validator.py
|
||||
pattern: "class Rejection"
|
||||
- path: src/portfolio_optimiser/validator.py
|
||||
pattern: "_MC_SEED"
|
||||
```
|
||||
|
||||
### Step 3: Promote VerdictStore + ExpeL + verdict capture (two-arg fix; real-SessionContext test)
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/verdicts.py` (new)
|
||||
- **Changes:** Move `ProposalFeatures`, `Verdict`, `similarity`, `VerdictStore` (in-memory, deterministic top-k retrieve), `ExpeLContextProvider(ContextProvider)`, `seed_store()`. **Fix the Fase 1 bug:** `before_run` must call `context.extend_instructions(self.source_id, [self.format_fewshot()])` — TWO args (`_sessions.py:253`). Add **`Verdict.id` minting** = stable content hash `sha256(canonical(ProposalFeatures))[:16]` (so a structurally identical proposal maps to a stable id) and **`capture_verdict(features, decision, rationale) -> Verdict`** (the Layer-2 out-of-band constructor). VerdictStore is **in-memory only for the MVP** — no JSON/file persistence (durable persistence deferred to Fase 3; the brief's learning-loop criterion is satisfied within a session via `seed_store()` + in-memory state). (new file)
|
||||
- **Reuses:** `spikes/d_verdictstore.py:29-227` (with the `:115` two-arg correction); `agent_framework.ContextProvider`, `agent_framework.SessionContext`.
|
||||
- **Test first:**
|
||||
- File: `tests/test_verdicts.py` (new)
|
||||
- Verifies: structural match beats text decoys (`hits[0].id=="TRUE"`); retrieval deterministic; `k<=0` raises; `capture_verdict` mints a stable id (same features → same id); **`before_run` populates a REAL `SessionContext` via two-arg `extend_instructions(source_id, instructions)`** — construct `from agent_framework import SessionContext`, call `await provider.before_run(...)` with it, assert the retrieved verdict text landed in `ctx.instructions` (NOT a single-arg fake — this exercises the genuine GA signature and retires the Critical risk).
|
||||
- Pattern: `tests/spikes/test_d_verdictstore.py:67-89` (retrieval) + real `SessionContext`
|
||||
- **Verify:** `uv run pytest tests/test_verdicts.py` → expected: `5 passed`
|
||||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/verdicts.py tests/test_verdicts.py`; HALT.
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): VerdictStore + ExpeL + capture_verdict (two-arg, real SessionContext test)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/verdicts.py
|
||||
- tests/test_verdicts.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase2\\): VerdictStore \\+ ExpeL \\+ capture_verdict \\(two-arg, real SessionContext test\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/verdicts.py
|
||||
pattern: "extend_instructions\\(self.source_id"
|
||||
- path: src/portfolio_optimiser/verdicts.py
|
||||
pattern: "def capture_verdict"
|
||||
```
|
||||
|
||||
### Step 4: Token meter + budget ChatMiddleware off real UsageDetails
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/budget.py` (new)
|
||||
- **Changes:** Promote `Budget`/`BudgetExceeded`/`TokenMeter` (fail-fast positive caps, structured `BudgetExceeded.kind/limit/observed`) fed from real `UsageDetails`: a `ChatMiddleware` reads `response.usage_details["total_token_count"]` (None-safe via `add_usage_details`) and short-circuits when the cap is crossed. **`usage_details is None` handling:** a `strict_usage: bool` flag (default True in dev/prod) makes a `None` usage a HARD FAIL (a usage regression must never silently disable the cap, research 03 Rec 3); test doubles that legitimately supply synthetic usage do not trip it. NO `len(...split())` proxy. (new file)
|
||||
- **Reuses:** `spikes/_harness.py:41-94` (drop `_word_tokens`); `agent_framework.UsageDetails`/`add_usage_details` (`_types.py:417`), `agent_framework.ChatMiddleware`.
|
||||
- **Test first:**
|
||||
- File: `tests/test_budget.py` (new)
|
||||
- Verifies (hand-built `UsageDetails`, no LLM): meter reads `total_token_count` + accumulates; cap crossed → `BudgetExceeded`; non-positive cap rejected; with `strict_usage=True`, `usage_details=None` raises. Meta-guard: `grep -rE "\.split\(\)" src/` finds zero token proxies.
|
||||
- Pattern: `tests/spikes/test_harness.py:27-60`
|
||||
- **Verify:** `uv run pytest tests/test_budget.py` → expected: `5 passed`
|
||||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/budget.py tests/test_budget.py`; HALT.
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): real-UsageDetails token meter + budget middleware"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/budget.py
|
||||
- tests/test_budget.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase2\\): real-UsageDetails token meter \\+ budget middleware$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/budget.py
|
||||
pattern: "usage_details"
|
||||
```
|
||||
|
||||
### Step 5: Framework-agnostic local-folder retriever core (owns TextSpan + path-security)
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/retrieval.py` (new)
|
||||
- **Changes:** Define `TextSpan` (`start_index`,`end_index` — the single owner; imported by provenance in Step 6), `RetrievedChunk` (`file`, `locator: TextSpan`, `snippet`, `score`), and `retrieve(query, docs_dir, top_k) -> list[RetrievedChunk]`. Chunk-at-ingest with EXACT char-span locators (citations exact by construction). Keyword/substring scoring (no heavy vector deps — D5/D6). **Path-security (this module reads the docs folder):** canonicalise + `os.path.realpath` + boundary-check every accessed path against `docs_dir`, fail closed on prefix-collision and symlink-escape (the EscapeRoute CVE class, research 02 Dim 5); native file APIs only. NO `agent_framework`/`mcp` imports — D7-portable seam. (new file)
|
||||
- **Reuses:** stdlib only; `TextSpan` aligns with `provenance.Citation.locator` (Step 6).
|
||||
- **Test first:**
|
||||
- File: `tests/test_retrieval.py` (new)
|
||||
- Verifies: `retrieve` returns chunks whose `locator` char-span exactly slices `snippet`; deterministic for fixed query+dir; top_k respected; **a `../` traversal and a symlink pointing outside `docs_dir` are both rejected (fail-closed); a sibling dir sharing a name prefix is rejected**.
|
||||
- Pattern: `tests/test_reference_domain.py:8-38` + `tests/test_backends.py:30-32` (raises)
|
||||
- **Verify:** `uv run pytest tests/test_retrieval.py` → expected: `7 passed`
|
||||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/retrieval.py tests/test_retrieval.py`; HALT.
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): local-folder retriever core with path-security"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/retrieval.py
|
||||
- tests/test_retrieval.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase2\\): local-folder retriever core with path-security$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/retrieval.py
|
||||
pattern: "class RetrievedChunk"
|
||||
- path: src/portfolio_optimiser/retrieval.py
|
||||
pattern: "realpath"
|
||||
```
|
||||
|
||||
### Step 6: First-class Pydantic provenance stamp
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/provenance.py` (new)
|
||||
- **Changes:** Define `Citation` (`file`, `locator: TextSpan` imported from `retrieval`, `snippet`) and `ProvenanceStamp` (`citations: list[Citation]` `min_length=1`, `model`, `role`, `validator_decision: Literal["validated","rejected"]`, `token_usage: int`). Authoritative provenance — independent of MAF `Annotation` (#4316). Provide `to_annotations()` mapping to MAF `Annotation(type="citation", …)` for display only. (new file; imports `TextSpan` from Step 5)
|
||||
- **Reuses:** `retrieval.TextSpan` (Step 5); `agent_framework._types.Annotation`/`TextSpanRegion` (display adapter); `pydantic.BaseModel`.
|
||||
- **Test first:**
|
||||
- File: `tests/test_provenance.py` (new)
|
||||
- Verifies: 0 citations → `ValidationError`; valid stamp exposes ≥1 citation + model/role + decision + token usage; `to_annotations()` yields `type=="citation"` dicts.
|
||||
- Pattern: `tests/spikes/test_c_validator.py:66-78`
|
||||
- **Verify:** `uv run pytest tests/test_provenance.py` → expected: `3 passed`
|
||||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/provenance.py tests/test_provenance.py`; HALT.
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): first-class Pydantic provenance stamp"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/provenance.py
|
||||
- tests/test_provenance.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase2\\): first-class Pydantic provenance stamp$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/provenance.py
|
||||
pattern: "class ProvenanceStamp"
|
||||
```
|
||||
|
||||
### Step 7: Expose the retriever to the agents as a citation-bearing data source
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/datasource.py` (new)
|
||||
- **Changes:** Wire `retrieval.retrieve()` into the debate as a citation-bearing data source. **MVP (GA-safe) path:** a GA `@tool`/`FunctionTool` (or a `ContextProvider`) that calls `retrieve()` and returns chunks whose `{file, locator, snippet}` the orchestrator maps into `provenance.Citation` — no new dependency, D7-portable. **Conditional MCP wrapper:** IF Step 1 added `mcp` as a GA dependency, ALSO expose a thin custom stdio MCP server (FastMCP) returning the same chunks as `structuredContent`, consumed via `MCPStdioTool(parse_tool_results=…)`, to honor the CLAUDE.md "data access via MCP" convention; if `mcp` was not GA-addable, the MCP wrapper is a documented fast-follow and the MVP uses the in-process tool. Either way the path-security lives in `retrieval.py` (Step 5). (new file)
|
||||
- **Reuses:** `retrieval.retrieve` (Step 5); `provenance.Citation` (Step 6); `agent_framework` `@tool`/`FunctionTool`; (conditional) `MCPStdioTool` + `mcp`.
|
||||
- **Test first:**
|
||||
- File: `tests/test_datasource.py` (new)
|
||||
- Verifies: the tool/ContextProvider returns citation-ready chunks for a query over the synthetic docs dir; chunks carry exact locators usable to build a `Citation`; (if `mcp` present) the MCP wrapper returns the same `structuredContent` shape.
|
||||
- Pattern: `tests/test_retrieval.py` (Step 5) + `tests/test_backends.py`
|
||||
- **Verify:** `uv run pytest tests/test_datasource.py` → expected: `all passed`
|
||||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/datasource.py tests/test_datasource.py`; HALT.
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): expose retriever as citation-bearing data source"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/datasource.py
|
||||
- tests/test_datasource.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase2\\): expose retriever as citation-bearing data source$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/datasource.py
|
||||
pattern: "retrieve"
|
||||
```
|
||||
|
||||
### Step 8: Wire the two backend profiles + role→model map + env template
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/backends.py`, `src/portfolio_optimiser/data/model_map.json` (new), `.env.template` (new), `tests/test_backends.py`
|
||||
- **Changes:** Implement `LocalBackend.create_chat_client` → `OpenAIChatCompletionClient(base_url=<PORTFOLIO_LOCAL_BASE_URL>, api_key=<PORTFOLIO_LOCAL_API_KEY default "ollama">, model_id=<resolved>)` configured **non-streaming** (research 03 — NOT `OpenAIChatClient`). Implement `AzureFoundryBackend.create_chat_client` → `FoundryChatClient` (deployment names from env). Add a `data/model_map.json` (the role→model/deployment map artifact, the SAME file Step 11's contract validates) loaded into a small resolver feeding `create_chat_client(model=…)`. Add `.env.template` documenting `PORTFOLIO_LOCAL_BASE_URL`, `PORTFOLIO_LOCAL_API_KEY`, Foundry `project_endpoint`/credential + the research-03 no-egress notes (Ollama bind `127.0.0.1`, pin ≥0.17.1, `OLLAMA_DEBUG` unset, model-pull = explicit egress). Keep `get_backend()` fail-fast. Invert the skeleton test.
|
||||
- **Reuses:** `backends.py:26-84`; `spikes/_harness.py:191-215` (env pattern, switch client class); `agent_framework_openai.OpenAIChatCompletionClient`, `agent_framework_foundry.FoundryChatClient`.
|
||||
- **Test first:**
|
||||
- File: `tests/test_backends.py` (existing — modify)
|
||||
- Verifies: both backends satisfy `ChatBackend`; unknown profile raises `ValueError`; the model_map resolves a role→model id; `create_chat_client` returns a client object (no network) — replace `test_create_chat_client_is_skeleton` (`:36`).
|
||||
- Pattern: `tests/test_backends.py:14-39`
|
||||
- **Verify:** `uv run pytest tests/test_backends.py` → expected: `all passed`
|
||||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/backends.py src/portfolio_optimiser/data/model_map.json .env.template tests/test_backends.py`; HALT.
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): wire local + Foundry profiles + model-map + env template"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/backends.py
|
||||
- src/portfolio_optimiser/data/model_map.json
|
||||
- .env.template
|
||||
- tests/test_backends.py
|
||||
min_file_count: 4
|
||||
commit_message_pattern: "^feat\\(fase2\\): wire local \\+ Foundry profiles \\+ model-map \\+ env template$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/backends.py
|
||||
pattern: "OpenAIChatCompletionClient"
|
||||
```
|
||||
|
||||
### Step 9: fresh_workflow isolation factory + maker-checker GroupChat (Layer-1 HITL gate)
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/workflow.py` (new)
|
||||
- **Changes:** Promote `fresh_workflow()` as the core factory — FRESH `GroupChatBuilder` + FRESH chat clients per project run (zero cross-run bleed, B7). Maker-checker debate: proposer + checker, deterministic termination (`make_termination`), `with_max_rounds(n)` AND the external budget-middleware guard (B4: `max_round_count=None` does NOT self-terminate). **Layer-1 HITL** = optionally enable `with_request_info(agents=[checker])` (in-process synchronous review, no checkpoint). Non-streaming. (new file)
|
||||
- **Reuses:** `spikes/b_footguns.py:100-106`; `spikes/a_groupchat.py:48-63,130-163`; `budget.ChatMiddleware` (Step 4); `backends` (Step 8); `agent_framework_orchestrations.GroupChatBuilder`.
|
||||
- **Test first:**
|
||||
- File: `tests/test_workflow.py` (new)
|
||||
- Verifies (FakeChatClient): fresh workflow per run → each run a clean thread; a participant sees ONLY its own project (assert on received-message CONTENT, not a call counter — the Fase 1 round-2 fix); a never-converging debate with `with_max_rounds(3)` halts at **exactly 3 rounds** (pin the observed count, retiring the 1-turn=1-round off-by-one assumption — not just `<=`).
|
||||
- Pattern: `tests/spikes/test_b_footguns.py:40-45` + `tests/spikes/test_harness.py:103-126`
|
||||
- **Verify:** `uv run pytest tests/test_workflow.py` → expected: `all passed`
|
||||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/workflow.py tests/test_workflow.py`; HALT.
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): fresh_workflow factory + maker-checker GroupChat"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/workflow.py
|
||||
- tests/test_workflow.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase2\\): fresh_workflow factory \\+ maker-checker GroupChat$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/workflow.py
|
||||
pattern: "with_max_rounds"
|
||||
```
|
||||
|
||||
### Step 10: LLM→IR generation wired to validator-as-retry (token budget in the generate loop)
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/generate.py` (new)
|
||||
- **Changes:** Implement `generate_via_llm(chat_client, project, context, meter) -> SavingsProposal` (replaces the stub `c_validator.py:207`): a NON-STREAMING chat call requesting structured output, parsed into the IR. On `ValidationError` (small models emit text-leaked/wrong-typed tool calls — research 03 Dim 3), invoke `validator.self_repair` — bounded by `self_repair`'s own `max_attempts` AND by the **budget `meter` checked here in the generate loop between attempts** (NO edit to `validator.py` — the validator stays the verbatim-promoted module from Step 2; the token bound lives in this loop). (new file)
|
||||
- **Reuses:** `validator.self_repair`/`ir.SavingsProposal` (Step 2); `budget` meter (Step 4); a chat client from `backends` (Step 8).
|
||||
- **Test first:**
|
||||
- File: `tests/test_generate.py` (new)
|
||||
- Verifies (FakeChatClient scripted): well-formed reply → `SavingsProposal`; malformed/text-leaked reply → bounded `self_repair` (retried, not silently accepted); exhausting attempts OR crossing the meter cap → typed failure (never a malformed proposal); validator.py is unmodified (no diff).
|
||||
- Pattern: `tests/spikes/test_harness.py:66-76` + `test_c_validator.py`
|
||||
- **Verify:** `uv run pytest tests/test_generate.py` → expected: `3 passed`
|
||||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/generate.py tests/test_generate.py`; HALT.
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): LLM->IR generation with validator-as-retry"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/generate.py
|
||||
- tests/test_generate.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase2\\): LLM->IR generation with validator-as-retry$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/validator.py
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/generate.py
|
||||
pattern: "self_repair"
|
||||
```
|
||||
|
||||
### Step 11: Fail-fast contract loaders
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/contracts.py` (new)
|
||||
- **Changes:** Pydantic models + `load_contracts(...)` for the four contracts: data-source config, **model-map (validates `data/model_map.json` from Step 8)**, termination contract, feedback schema. Validate ALL at startup and raise before any chat-client is constructed or called (brief NFR fail-fast). Pydantic gives JSON-Schema-grade validation (CLAUDE.md convention). (new file)
|
||||
- **Reuses:** `pydantic.BaseModel`; `data/model_map.json` (Step 8); `backends.Profile`.
|
||||
- **Test first:**
|
||||
- File: `tests/test_contracts.py` (new)
|
||||
- Verifies: malformed data-source / model-map / termination / feedback → `ValidationError` at load; AND **no chat-client call was made** (inject a FakeChatClient spy, assert `call_count==0`).
|
||||
- Pattern: `tests/test_backends.py:30-32` + FakeChatClient `call_count` (`spikes/_harness.py:138`)
|
||||
- **Verify:** `uv run pytest tests/test_contracts.py` → expected: `all passed`
|
||||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/contracts.py tests/test_contracts.py`; HALT.
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): fail-fast contract loaders"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/contracts.py
|
||||
- tests/test_contracts.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase2\\): fail-fast contract loaders$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/contracts.py
|
||||
pattern: "def load_contracts"
|
||||
```
|
||||
|
||||
### Step 12: Orchestrator + single-command entry (two-layer HITL wiring)
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/run.py` (new), `src/portfolio_optimiser/__init__.py`
|
||||
- **Changes:** `run_project(project_id, profile, verdict_input=None)` composes the slice: `load_contracts` (fail-fast) → load project + retrieve cited chunks via the Step-7 data source → `fresh_workflow` maker-checker debate (budget middleware + round cap; **Layer-1** = the optional in-run `with_request_info` gate) → `generate_via_llm` candidate → blocking `validate_proposal` → `ValidatedProposal | Rejection` → attach `ProvenanceStamp` → **Layer-2 (out-of-band)**: `capture_verdict(features, decision, rationale)` where `decision` comes from `verdict_input` (function arg / CLI prompt / fixture in tests; the **B11 notification is a stub** in Fase 2) → `VerdictStore` persist → (next run) `ExpeLContextProvider` retrieval. Document the two layers explicitly in a module docstring. Console entry for the single-command run. Export public API from `__init__.py`. (new file + modify `__init__.py`)
|
||||
- **Reuses:** every Step 2–11 module.
|
||||
- **Test first:**
|
||||
- File: `tests/test_run_smoke.py` (new)
|
||||
- Verifies: `run_project` importable; runs end-to-end on an injected FakeChatClient (with a fixture `verdict_input`) and returns a `ValidatedProposal` with a `ProvenanceStamp`.
|
||||
- Pattern: `tests/test_smoke.py`
|
||||
- **Verify:** `uv run python -c "from portfolio_optimiser.run import run_project"` → no error; `uv run pytest tests/test_run_smoke.py` → `1 passed`
|
||||
- **On failure:** revert `git checkout -- src/portfolio_optimiser/run.py src/portfolio_optimiser/__init__.py tests/test_run_smoke.py`; HALT.
|
||||
- **Checkpoint:** `git commit -m "feat(fase2): vertical-slice orchestrator + two-layer HITL wiring"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/run.py
|
||||
- tests/test_run_smoke.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase2\\): vertical-slice orchestrator \\+ two-layer HITL wiring$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/run.py
|
||||
pattern: "def run_project"
|
||||
```
|
||||
|
||||
### Step 13: End-to-end integration test (success criteria, on FakeChatClient with synthetic usage)
|
||||
|
||||
- **Files:** `tests/test_vertical_slice_e2e.py` (new), `tests/conftest.py` (new)
|
||||
- **Changes:** A deterministic CI integration test (scripted FakeChatClient — no real LLM). **The test double emits a synthetic `UsageDetails`** (e.g. `{"total_token_count": N}`) so the budget middleware (Step 4, `strict_usage=True`) does NOT hard-fail and the provenance stamp's `token_usage` is a positive number sourced from `UsageDetails` (this is the in-CI accounting check; the REAL-provider populated-usage assertion lives in Step 14). Add a shared `tests/conftest.py` providing the scripted FakeChatClient + synthetic-usage factory + `seed_store` fixtures. Criteria covered: (a) one valid proposal e2e → exactly one `ValidatedProposal` with populated provenance (≥1 citation, model/role, validator decision, token usage>0 from synthetic UsageDetails); (b) out-of-range → `Rejection` with reason; (c) Layer-2 verdict captured via `capture_verdict` → written to VerdictStore (record exists); (d) second run on a structurally similar proposal retrieves the prior verdict via `ExpeLContextProvider` (assert `retrieved.id == persisted.id`, exercising two-arg `extend_instructions`); (e) tiny budget/round cap → halts via middleware/round-cap (assert cap not exceeded); (f) malformed contract → raises before any chat call. (new files)
|
||||
- **Reuses:** `run.run_project`; `tests/conftest.py` FakeChatClient + synthetic-usage; `seed_store()`.
|
||||
- **Test first:** this IS the test step.
|
||||
- File: `tests/test_vertical_slice_e2e.py` (new)
|
||||
- Verifies: all six assertions above.
|
||||
- Pattern: `tests/spikes/test_harness.py:103-126` + `test_c_validator.py` + `test_d_verdictstore.py`
|
||||
- **Verify:** `uv run pytest && uv run ruff check . && uv run mypy src` → expected: all pass, ruff clean, mypy clean
|
||||
- **On failure:** escalate — an e2e failure means a composition seam is wrong; diagnose against the failing criterion; HALT.
|
||||
- **Checkpoint:** `git commit -m "test(fase2): end-to-end vertical-slice integration suite"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- tests/test_vertical_slice_e2e.py
|
||||
- tests/conftest.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^test\\(fase2\\): end-to-end vertical-slice integration suite$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: tests/test_vertical_slice_e2e.py
|
||||
pattern: "ValidatedProposal"
|
||||
```
|
||||
|
||||
### Step 14: Gated live-profile checks (real UsageDetails + no-egress)
|
||||
|
||||
- **Files:** `tests/test_local_profile_live.py` (new), `tests/test_foundry_profile_live.py` (new)
|
||||
- **Changes:** Separately-gated tests (`@pytest.mark.skipif`, NOT default CI). **Local:** a REAL `OpenAIChatCompletionClient(base_url)` non-streaming call returns a response AND the meter is populated from real `UsageDetails` — positive, load-bearing `total_token_count > 0` (this is the criterion FakeChatClient cannot prove). Assert the call goes only to the configured loopback `base_url` (no-egress: no outbound to `api.openai.com`). **Foundry:** a trivial agent responds on `FoundryChatClient` (cheapest model, hard cap — D6). (new files)
|
||||
- **Reuses:** `backends` (Step 8); `live_local_client_or_skip` skip pattern (`spikes/_harness.py:191-215`).
|
||||
- **Test first:** these ARE gated tests.
|
||||
- File: `tests/test_local_profile_live.py`, `tests/test_foundry_profile_live.py` (new)
|
||||
- Verifies: skip cleanly without env; when env present, a successful response + `total_token_count > 0` from each path; local call targets only the loopback endpoint.
|
||||
- Pattern: `tests/spikes/test_a_groupchat.py:14-16,65`
|
||||
- **Verify:** `uv run pytest tests/test_local_profile_live.py tests/test_foundry_profile_live.py` → expected: `2 skipped` (no endpoint in CI) — or `passed` with env set
|
||||
- **On failure:** skip — no endpoint is the expected CI outcome; a non-skip failure with env set escalates; HALT.
|
||||
- **Checkpoint:** `git commit -m "test(fase2): gated live local + Foundry profile checks"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- tests/test_local_profile_live.py
|
||||
- tests/test_foundry_profile_live.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^test\\(fase2\\): gated live local \\+ Foundry profile checks$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: tests/test_local_profile_live.py
|
||||
pattern: "skipif"
|
||||
```
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
| Approach | Pros | Cons | Why rejected |
|
||||
|----------|------|------|--------------|
|
||||
| Reuse `@modelcontextprotocol/server-filesystem` | No custom code; official | Cannot produce citations; npx/Node dep; CVE/supply-chain surface (research 02) | Citation NFR is load-bearing; in-process retriever produces exact locators |
|
||||
| Standalone thin MCP server on the MVP critical path | Honors "data access via MCP" convention fully | Requires `mcp` (not installed; possibly `--pre` → violates GA-pin); extra process | In-process retriever-as-tool for MVP (GA-safe); MCP wrapper conditional on `mcp` GA (Step 7) |
|
||||
| Native `OllamaChatClient` for local | MS-recommended; sidesteps `/v1` edges | `--pre` beta + new `ollama` dep — violates GA-pin | `OpenAIChatCompletionClient` non-streaming; native is spike-gated fallback |
|
||||
| MAF checkpoint-resume for durable verdict | Native durable pause | Fragile (open bugs into 1.9.0; pickle vs Pydantic IR) + D7 lock-in | Out-of-band VerdictStore; defer checkpointing |
|
||||
| MAF `Annotation` propagation for provenance | Native | Python streaming-drop bug #4316 → silent empty citations | First-class own Pydantic provenance |
|
||||
| JSON-persistent VerdictStore in MVP | Cross-process durability | Not required by any Fase 2 criterion (in-session learning suffices) | In-memory for MVP; durable persistence deferred to Fase 3 |
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- **Framework:** `pytest` + `pytest-asyncio` (`asyncio_mode="auto"`). `pythonpath=["src","."]`, `testpaths=["tests"]`. New shared `tests/conftest.py` (Step 13) provides the scripted FakeChatClient + synthetic-`UsageDetails` factory + `seed_store` fixtures.
|
||||
- **Existing patterns:** deterministic validator/Monte-Carlo over FIXED inputs (`test_c_validator.py:59-63`, seed `20260624`); FakeChatClient driving real GA builders with round caps (`test_harness.py:103-126`); gated live arm via `skipif` (`test_a_groupchat.py:14-16`); real-`SessionContext` exercise for the two-arg `extend_instructions` (Step 3).
|
||||
- **Determinism discipline (research 03 Dim 5):** determinism asserted ONLY over the validator + Monte-Carlo with fixed inputs; NO test asserts LLM output. The propose (non-deterministic) → validate (deterministic) boundary is explicit.
|
||||
- **Usage-accounting split:** in-CI accounting uses the FakeChatClient's synthetic `UsageDetails` (Step 13); the REAL-provider `total_token_count > 0` assertion is gated/live (Step 14) — this is the only place "real UsageDetails" is provable.
|
||||
|
||||
### Tests to write
|
||||
|
||||
| Type | File | Verifies | Model test |
|
||||
|------|------|----------|------------|
|
||||
| Unit | `tests/test_validator.py` | validate/reject/determinism/Pydantic-block | `tests/spikes/test_c_validator.py` |
|
||||
| Unit | `tests/test_verdicts.py` | retrieval + REAL-SessionContext two-arg extend_instructions + id minting | `tests/spikes/test_d_verdictstore.py` |
|
||||
| Unit | `tests/test_budget.py` | UsageDetails meter, strict None hard-fail, cap, no-proxy guard | `tests/spikes/test_harness.py` |
|
||||
| Unit | `tests/test_retrieval.py` | exact-locator chunks + EscapeRoute path rejection | `tests/test_reference_domain.py` |
|
||||
| Unit | `tests/test_provenance.py` | ≥1 citation, fields, annotation adapter | `tests/spikes/test_c_validator.py` |
|
||||
| Unit | `tests/test_datasource.py` | citation-ready chunks via tool/(opt) MCP | `tests/test_retrieval.py` |
|
||||
| Unit | `tests/test_backends.py` | profiles wired, model-map, no-NotImplementedError | `tests/test_backends.py` (existing) |
|
||||
| Integration | `tests/test_workflow.py` | fresh-instance isolation (content), exact round cap | `tests/spikes/test_b_footguns.py` |
|
||||
| Unit | `tests/test_generate.py` | LLM→IR + bounded self_repair (attempts+budget); validator.py unmodified | `tests/spikes/test_harness.py` |
|
||||
| Unit | `tests/test_contracts.py` | fail-fast before any chat call | `tests/test_backends.py` |
|
||||
| Integration | `tests/test_vertical_slice_e2e.py` | all 6 e2e criteria (synthetic usage) | `tests/spikes/test_harness.py` |
|
||||
| Live (gated) | `tests/test_local_profile_live.py` | real client + real UsageDetails>0 + loopback-only | `tests/spikes/test_a_groupchat.py` |
|
||||
| Live (gated) | `tests/test_foundry_profile_live.py` | trivial Foundry response | `tests/spikes/test_a_groupchat.py` |
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Priority | Risk | Location | Impact | Mitigation |
|
||||
|----------|------|----------|--------|------------|
|
||||
| Critical | `extend_instructions` single-arg bug masked by a tautological fake | `spikes/d_verdictstore.py:115`, `test_d_verdictstore.py:99` | Learning-loop close crashes against a real `SessionContext` | Step 3: two-arg call + a test using a REAL `SessionContext` (not a fake), retiring the risk in CI |
|
||||
| Critical | Budget enforcement word-count theatre / `usage=None` silently disables cap | `spikes/_harness.py:97-99` | Hard-cap NFR fake; unbounded run | Step 4: middleware off `UsageDetails`; `strict_usage` None=hard fail; non-streaming; src grep-guard |
|
||||
| Critical | Round cap not self-terminating | Fase 1 B4 | Unbounded debate on a real client | Step 9: explicit `with_max_rounds` + external budget guard; test pins EXACT round count |
|
||||
| High | `usage=None` on FakeChatClient breaks e2e under strict middleware | Steps 4/13 | e2e cannot run | Step 13: test double emits synthetic `UsageDetails`; real-usage assertion moved to gated Step 14 |
|
||||
| High | `mcp` not installed / possibly `--pre` | Steps 1/7 | Step 7 import fails; GA-pin violated | Step 1 conditional add; Step 7 MVP uses in-process tool (no dep); MCP wrapper only if `mcp` GA |
|
||||
| High | Wrong OpenAI client (Responses vs Chat Completions) | `spikes/_harness.py:210`, `backends.py:65` | Local profile fails / UsageDetails not populated | Step 8: `OpenAIChatCompletionClient` non-streaming |
|
||||
| High | Small-model tool-calling unreliability | research 03 Dim 3 | Malformed candidates crash/pass | Step 10: validator-as-retry (attempts+budget); qwen3:4b; num_ctx≥32k |
|
||||
| High | Provenance via MAF annotation (#4316) | research 02 Dim 2 | Citations silently drop on streaming | Step 6: own Pydantic provenance; non-streaming |
|
||||
| High | Retriever owns path/symlink security (EscapeRoute) | Step 5 | Out-of-folder read → silent exfil | Step 5: canonicalise+realpath+boundary fail-closed; TDD bypass patterns |
|
||||
| Medium | self_repair token-budget vs frozen validator | Steps 2/10 | Scope-fence violation | Step 10: budget checked in generate loop; `validator.py` in `forbidden_paths` |
|
||||
| Medium | HITL verdict on checkpoint-resume | research 01 | Silent loss / pickle rejection | Out-of-band `capture_verdict`; checkpointing deferred |
|
||||
| Medium | Determinism asserted through the LLM | research 03 Dim 5 | Flaky tests | Pin validator+MC inputs only |
|
||||
| Medium | Intel-CPU latency makes a real debate minutes-long | research 03 Dim 4 | Live test hangs | Tiny runs (1 maker+1 checker, ≤3 rounds, hard caps); gate live; representative runs on Foundry |
|
||||
| Medium | `fresh_workflow` reuse → B7 bleed returns | `spikes/b_footguns.py:100` | Silent cross-project contamination | Step 9: fresh workflow + clients per run; B7 content-level test |
|
||||
| Medium | Headless cascade on dependency-chained failure | all steps | Downstream steps run on a failed predecessor | Global halt rule + per-step "HALT" in On-failure |
|
||||
| Low | `#1772` double system prompt (OpenAIChatClient+Ollama+middleware) | research 03 Dim 3 | Wasted tokens | Verify vs 1.9.0 in client spike-gate; native client avoids it |
|
||||
| Low | no-op `uv.lock` diff on re-lock | Step 1 | Manifest min_file_count misfire | Step 1 manifest gates only pyproject.toml + test (min_file_count 2) |
|
||||
|
||||
## Assumptions
|
||||
|
||||
| # | Assumption | Why unverifiable | Impact if wrong |
|
||||
|---|-----------|-----------------|-----------------|
|
||||
| 1 | Research is COMPLETE (3 valid briefs on disk) | — (verified) | Would re-run research |
|
||||
| 2 | Follow research over brief prose (HITL split, in-process retriever, OpenAIChatCompletionClient non-streaming) | research post-dates brief prose | Planning against superseded premises |
|
||||
| 3 | Operator supplies a working local OpenAI-compatible endpoint + model (Ollama qwen3:4b on `127.0.0.1:11434/v1/`) | machine-specific (brief [OPEN]) | Local full-run + "both profiles" criterion can't execute; local-only run still satisfies the other 8 |
|
||||
| 4 | Operator supplies Foundry deployment names | tenant-specific (brief [OPEN]) | Foundry minimal check skips |
|
||||
| 5 | `mcp` GA availability decides Step 7's MCP wrapper (else in-process tool only) | not installed; resolved in Step 1 | If `--pre`-only, the "data access via MCP" convention is met by a fast-follow, not the MVP |
|
||||
| 6 | Synthetic Fase 0 domain reused as the single project | brief [ASSUMPTION] | — |
|
||||
| 7 | Solver stays PuLP/CBC (installed 3.3.2, CBC present); PuLP 4.0 → `pulp[cbc]`/`COIN_CMD` | — (verified) | Future PuLP bump breaks bundled CBC |
|
||||
|
||||
## Verification
|
||||
|
||||
End-to-end integration checks (per-step manifests verify each step):
|
||||
|
||||
- [ ] `uv run pytest` → all pass (live arms skip without env)
|
||||
- [ ] `uv run ruff check .` → clean
|
||||
- [ ] `uv run mypy src` → clean
|
||||
- [ ] `uv run python -c "from portfolio_optimiser.run import run_project"` then a FakeChatClient e2e → exactly one `ValidatedProposal` with a populated `ProvenanceStamp` (≥1 citation, model/role, validator decision, token usage>0)
|
||||
- [ ] Out-of-range candidate → `Rejection` (type-asserted, reason)
|
||||
- [ ] Second run on a structurally similar proposal → `retrieved.id == persisted.id`
|
||||
- [ ] Malformed contract → raises at startup; no chat-client call made
|
||||
- [ ] Tiny budget/round cap → halts; loop did not exceed cap (exact round count pinned)
|
||||
- [ ] `grep -rE "\.split\(\)" src/` → no word-count token proxy
|
||||
- [ ] No silent egress: gated live local call targets only the loopback `base_url` (Step 14); telemetry env vars unset
|
||||
- [ ] Gated live (when env set): real `UsageDetails.total_token_count > 0` on the local profile; trivial Foundry response
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Files to modify:** 2 existing (`pyproject.toml`, `backends.py`, `__init__.py`, `tests/test_backends.py` — 4 touch points)
|
||||
- **Files to create:** ~25 (11 `src/` modules incl. `datasource.py` + `data/model_map.json` + `.env.template` + ~12 test files incl. `conftest.py`)
|
||||
- **Complexity:** high (integration of four subsystems on real clients, two profiles)
|
||||
|
||||
## Execution Strategy
|
||||
|
||||
14 steps > 5 → grouped into sessions/waves. **Dependencies are explicit; sessions in
|
||||
the same wave touch disjoint files AND have no cross-session import edge.** Global
|
||||
headless rule: on any step failure, perform the On-failure action then HALT — never
|
||||
run a dependent step on a failed predecessor.
|
||||
|
||||
### Session 1: Dependencies
|
||||
- **Steps:** 1
|
||||
- **Wave:** 1
|
||||
- **Depends on:** none
|
||||
- **Scope fence:** Touch `pyproject.toml`, `uv.lock`, `tests/test_imports_core.py`; Never touch `src/**`
|
||||
|
||||
### Session 2: Deterministic core (no MAF), dependency-ordered
|
||||
- **Steps:** 2, 5, 6, 3, 4 (note: 5 retrieval before 6 provenance — TextSpan ownership; all internally independent except 6→5)
|
||||
- **Wave:** 2
|
||||
- **Depends on:** Session 1
|
||||
- **Scope fence:** Touch `src/portfolio_optimiser/{ir,validator,retrieval,provenance,verdicts,budget}.py` + tests; Never touch `backends.py`, `workflow.py`, `datasource.py`
|
||||
|
||||
### Session 3: Data source + backend wiring
|
||||
- **Steps:** 7, 8
|
||||
- **Wave:** 3 (Step 7 depends on Steps 5+6; Step 8 independent of Session 2 internals)
|
||||
- **Depends on:** Session 2 (Step 7 → retrieval/provenance)
|
||||
- **Scope fence:** Touch `src/portfolio_optimiser/{datasource,backends}.py`, `data/model_map.json`, `.env.template` + tests; Never touch the core modules' internals (import only)
|
||||
|
||||
### Session 4: Composition
|
||||
- **Steps:** 9, 10, 11, 12
|
||||
- **Wave:** 4
|
||||
- **Depends on:** Sessions 2, 3
|
||||
- **Scope fence:** Touch `src/portfolio_optimiser/{workflow,generate,contracts,run,__init__}.py` + tests; Never touch core/datasource/backends internals (incl. `validator.py` — Step 10 forbidden_path)
|
||||
|
||||
### Session 5: Integration + live
|
||||
- **Steps:** 13, 14
|
||||
- **Wave:** 5
|
||||
- **Depends on:** Session 4
|
||||
- **Scope fence:** Touch `tests/test_vertical_slice_e2e.py`, `tests/conftest.py`, `tests/test_local_profile_live.py`, `tests/test_foundry_profile_live.py`; Never touch `src/**`
|
||||
|
||||
### Execution Order
|
||||
- **Wave 1:** Session 1
|
||||
- **Wave 2:** Session 2 (internal order 2 → 5 → 6 → 3 → 4)
|
||||
- **Wave 3:** Session 3 (after Wave 2)
|
||||
- **Wave 4:** Session 4 (after Wave 3)
|
||||
- **Wave 5:** Session 5 (after Wave 4)
|
||||
|
||||
### Grouping rules applied
|
||||
- Steps sharing files / import edges → same session, sequenced; 3–5 steps/session; sessions ordered by dependency. No two sessions in the same wave share an import edge (the Step 7→6 edge moved Step 7 to Wave 3).
|
||||
|
||||
## Plan Quality Score
|
||||
|
||||
| Dimension | Weight | Score | Notes |
|
||||
|-----------|--------|-------|-------|
|
||||
| Structural integrity | 0.15 | 90 | dependency-ordered; TextSpan ownership fixed; Step 7 moved to its own wave; halt rule added |
|
||||
| Step quality | 0.20 | 88 | each step 1–2 files, TDD, concrete reuse; verdict capture + id minting now explicit |
|
||||
| Coverage completeness | 0.20 | 92 | all 9 criteria → steps + verification; two-layer HITL mapped; no-egress verified |
|
||||
| Specification quality | 0.15 | 86 | model-map artifact + env template defined; some new-module internals left to impl |
|
||||
| Risk & pre-mortem | 0.15 | 92 | 17 risks incl. the 3 premise corrections; extend_instructions retired by a real-session test |
|
||||
| Headless readiness | 0.10 | 88 | On-failure + HALT + Checkpoint + Manifest per step |
|
||||
| Manifest quality | 0.05 | 86 | checkable manifests; strengthened retrieval/datasource predicates; lock-step gating fixed |
|
||||
| **Weighted total** | **1.00** | **89** | **Grade: A−** |
|
||||
|
||||
**Adversarial review:**
|
||||
- **Plan critic:** REVISE on the first draft — 3 blockers (FakeChatClient usage contradiction, missing `mcp` dep, unspecified two-layer HITL capture), 7 major, 4 minor. All addressed in Revisions below.
|
||||
- **Scope guardian:** ALIGNED — 0 creep, all 9 criteria mapped, all 6 Non-Goals honored; 1 minor (two-layer HITL naming) addressed.
|
||||
|
||||
## Revisions
|
||||
|
||||
| # | Finding | Severity | Resolution |
|
||||
|---|---------|----------|------------|
|
||||
| 1 | FakeChatClient e2e can't satisfy usage-populated vs `None`-hard-fail | blocker | Step 13 test double emits synthetic `UsageDetails`; `strict_usage` flag (Step 4); real-usage assertion moved to gated Step 14 |
|
||||
| 2 | `mcp`/FastMCP dependency never added | blocker | Step 1 conditional `mcp` add (GA only); Step 7 MVP uses in-process retriever-as-tool (no dep); MCP wrapper conditional/fast-follow |
|
||||
| 3 | Two-layer HITL verdict-capture unspecified; verdict-id minting undefined | blocker | Step 3 adds `capture_verdict` + stable `Verdict.id` hash; Step 12 wires Layer-1 (in-run gate) + Layer-2 (out-of-band capture) + B11 notification stub, mapped explicitly |
|
||||
| 4 | `extend_instructions` not retired by a real-session test | major | Step 3 test uses a REAL `agent_framework.SessionContext` (verified importable), not a fake |
|
||||
| 5 | Session 2∥3 not independent (Step 7→6) + unowned `TextSpan` | major | `TextSpan` owned by `retrieval.py` (Step 5), imported by provenance (Step 6); Step 7 moved to Wave 3; Execution Strategy reworked |
|
||||
| 6 | env vars / `.env.template` / no-silent-egress unverified | major | Step 8 adds `.env.template` + no-egress notes; Verification adds loopback-only + telemetry-unset checks (Step 14) |
|
||||
| 7 | No circuit-breaker/halt language | major | Global halt rule + per-step "HALT" appended to every On-failure |
|
||||
| 8 | Step 3 JSON persistence = scope creep | major | Removed; VerdictStore in-memory for MVP; durable persistence deferred to Fase 3 |
|
||||
| 9 | self_repair token-budget edits frozen validator.py | major | Step 10 enforces the token bound in the generate loop; `validator.py` in Step 10 `forbidden_paths` |
|
||||
| 10 | Step 1 manifest `min_file_count:3` may misfire on no-op lock | major | Manifest gates only `pyproject.toml` + test (min_file_count 2) |
|
||||
| 11 | Estimated Scope counts inconsistent | minor | Corrected |
|
||||
| 12 | model-map config artifact undefined | minor | `data/model_map.json` defined in Step 8, validated by Step 11 |
|
||||
| 13 | Round-cap assertion doesn't catch off-by-one | minor | Step 9 pins the EXACT observed round count |
|
||||
| 14 | Weak `realpath` manifest predicate | minor | Path-security moved into `retrieval.py` (Step 5) with `realpath` + behavioural EscapeRoute tests |
|
||||
|
||||
## Adversarial Pass 2 (gemini-bridge, v5.1.1 high-effort)
|
||||
|
||||
**Status: UNAVAILABLE.** The high-effort plan signal mandates an independent
|
||||
gemini-bridge review of the post-revision plan. The `gemini-mcp` server has
|
||||
returned an identical `400 BadRequestError` on every call this project (its
|
||||
client SDK predates Google's May-2026 Interactions API change). A fifth
|
||||
invocation would fail deterministically, so it was not attempted (cost
|
||||
discipline D6). Independent second-opinion triangulation on the plan is
|
||||
therefore absent — the plan rests on plan-critic + scope-guardian (Phase 9) +
|
||||
installed-source-grounded exploration. To restore this pass, upgrade the
|
||||
gemini-mcp client SDK to ≥ 2.0.0 and re-run `/trekplan` (or a standalone
|
||||
gemini review) on `plan.md`.
|
||||
|
|
@ -1,273 +0,0 @@
|
|||
---
|
||||
type: trekresearch-brief
|
||||
created: 2026-06-24
|
||||
question: "Does Microsoft Agent Framework 1.9.0 (core + orchestrations 1.0.0) provide a native human-in-the-loop primitive to pause a workflow for external input and resume it, and how does it interact with checkpointing and session state?"
|
||||
confidence: 0.85
|
||||
dimensions: 6
|
||||
mcp_servers_used: [microsoft-learn, tavily]
|
||||
local_agents_used: [installed-source-introspection (orchestrator, main context)]
|
||||
external_agents_used: [docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (unavailable)]
|
||||
topic: 1
|
||||
brief: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md
|
||||
---
|
||||
|
||||
# Native human-in-the-loop in MAF 1.9.0 workflows
|
||||
|
||||
> Generated by trekresearch (Voyage 5.6.0) on 2026-06-24. Topic 1 of 3 for the
|
||||
> Fase 2 MVP-vertical-slice brief. Scope: external swarm + installed-source
|
||||
> introspection (API truth from the installed 1.9.0 package wins over Learn docs,
|
||||
> per CLAUDE.md). Gemini second opinion was unavailable (MCP SDK broke on
|
||||
> Google's May-2026 Interactions API change) — triangulation rests on docs +
|
||||
> community + contrarian + installed source.
|
||||
|
||||
## Research Question
|
||||
|
||||
Does `agent-framework` 1.9.0 (`agent-framework-core` + `agent-framework-orchestrations` 1.0.0)
|
||||
provide a native human-in-the-loop (HITL) mechanism to pause a workflow for
|
||||
external/human input and resume it, and how does it interact with checkpointing
|
||||
and session/conversation state? Decision it feeds: the Fase 2 "expert verdict
|
||||
captured via HITL" success criterion — design the two-layer HITL + verdict-capture
|
||||
+ feed-forward seam, and the per-project workflow graph.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Yes — native HITL exists and is GA in 1.9.0**, in three distinct surfaces:
|
||||
(1) a workflow-level request/response primitive (`ctx.request_info()` + `@response_handler`,
|
||||
resume via `run(responses=...)`), (2) durable checkpoint-and-resume that persists
|
||||
pending requests, and (3) an orchestration-level human-approval gate that — verified
|
||||
in installed source — **works on our Group Chat maker-checker default** via
|
||||
`GroupChatBuilder.with_request_info(agents=[...])`. **Confidence on the mechanism is
|
||||
HIGH (installed-source-verified); confidence on durable checkpoint-resume RELIABILITY
|
||||
is LOW/contradictory** — official docs promise lossless durable resume, but the bug
|
||||
tracker shows the *pause* half is solid while the *persist-and-resume* half has
|
||||
multiple open/silent-failure bugs surviving into the 1.9.0 line (#5818, #6127, #6372,
|
||||
#5621) plus a pickle type-allowlist that rejects custom Pydantic types on restore (#5810).
|
||||
**Key caveat:** our learned-verdict path is an *indefinite, out-of-band, never-resume-the-
|
||||
original-run* decision whose durable artifact is the hand-rolled VerdictStore — so the
|
||||
recommendation is to use the GA *in-run* approval gate for synchronous Layer-1 review,
|
||||
keep the durable learned verdict OUT-OF-BAND in the VerdictStore, and **defer
|
||||
checkpointing off the MVP critical path**.
|
||||
|
||||
## Dimensions
|
||||
|
||||
### 1. Native HITL primitive — Confidence: high
|
||||
|
||||
**Installed-source findings (ground truth, 1.9.0):**
|
||||
- `WorkflowContext.request_info(request_data: object, response_type: type, *, request_id: str | None = None)` — any executor calls this to suspend the workflow and request external input (`agent_framework/_workflows/_workflow_context.py:393`).
|
||||
- Every `Executor` inherits `RequestInfoMixin` (`_workflows/_executor.py:30`); a method decorated `@response_handler` receives the typed response on resume; handlers are matched by the request/response **type annotations** (`_workflows/_request_info_mixin.py`).
|
||||
- `RequestInfoExecutor` is **NOT present** in installed 1.9.0 (only `RequestInfoMixin` + `response_handler` + `ctx.request_info`). `send_responses`/`send_responses_streaming` are **NOT public** on `Workflow` — internal `_send_responses_internal` only.
|
||||
|
||||
**External findings:**
|
||||
- Docs + AutoGen→MAF migration guide confirm "workflows can pause execution and wait for external input before continuing" — a capability AutoGen's `Team` lacked. <https://learn.microsoft.com/agent-framework/migration-guide/from-autogen/>
|
||||
- The versioned upgrade guides explain why the installed surface looks as it does: `RequestInfoExecutor` removed (python-1.0.0b251104) in favour of `ctx.request_info()` + `@response_handler`; `send_responses*` removed (python-1.0.0b260210, PR #3720) in favour of `run(responses=...)`. Both predate 1.9.0 (2026-06-18). <https://learn.microsoft.com/agent-framework/support/upgrade/requests-and-responses-upgrade-guide-python>, <https://learn.microsoft.com/agent-framework/support/upgrade/python-2026-significant-changes>
|
||||
|
||||
**Contradictions:**
|
||||
- The auto-generated **API-reference** page (`agent-framework-python-latest`) still lists `send_responses*` and prose-mentions `RequestInfoExecutor`. This **conflicts** with both the dated changes-guide and the installed source. Resolution: **installed source wins** (CLAUDE.md invariant) — those names are doc-lag / the C# (`RequestPort`) idiom. Do not write them into Python 1.9.0 code.
|
||||
|
||||
### 2. Pause/resume mechanics — Confidence: high
|
||||
|
||||
**Installed-source findings:**
|
||||
- `Workflow.run(message=None, *, responses=None, checkpoint_id=None, checkpoint_storage=None)` — three intents, exactly one of `message` / `responses` / `checkpoint_id` per call (`_workflows/_workflow.py:675+`). `responses` is `Mapping[str, Any]` keyed by `request_id`; mutually exclusive with `message`; **can be combined with `checkpoint_id`** ("restore then send responses in a single call" per the docstring).
|
||||
- Run-state enum (`_workflows/_events.py:58-65`): `IDLE`, `IDLE_WITH_PENDING_REQUESTS` (paused awaiting input — non-terminal), `IN_PROGRESS_PENDING_REQUESTS`. `result.get_final_state()` returns it.
|
||||
- `RunnerContext.send_request_info_response(request_id, response)` validates the response **type** against the original request and raises `ValueError` on unknown `request_id` or type mismatch (`_workflows/_runner_context.py:457-471`).
|
||||
- Events collapsed to a generic `WorkflowEvent[DataT]` carrying `.request_id` / `.data` / `.type` (no `RequestInfoEvent` subclass).
|
||||
|
||||
**External findings:**
|
||||
- Canonical loop (official sample `guessing_game_with_human_input.py`): `run(..., stream=True)` → collect `request_info` events → `run(stream=True, responses=pending)` → repeat until no pending requests. **In-process resume needs NO checkpoint** — "state is preserved across multiple calls to run." <https://learn.microsoft.com/agent-framework/workflows/human-in-the-loop>
|
||||
- Request IDs are caller-supplied or auto-UUID; responses strongly typed via `response_type`.
|
||||
|
||||
### 3. Checkpointing interaction — Confidence: high (mechanism) / contradictory (reliability)
|
||||
|
||||
**Installed-source + docs findings (mechanism — high):**
|
||||
- `WorkflowCheckpoint.pending_request_info_events: dict[str, ...]` — pending HITL requests ARE serialized into the checkpoint (`_workflows/_checkpoint.py:81`). On restore the runner rehydrates them (`_runner_context.py:424-425`) and **re-emits** them as `request_info` events; you then answer with a *separate* `run(responses=...)` call (you cannot inject responses during the restore call itself).
|
||||
- Storage backends (all implement the `CheckpointStorage` protocol, swap without code change): `InMemoryCheckpointStorage` (ephemeral), `FileCheckpointStorage` (local disk, explicit `storage_path`, **pickle** + restricted unpickler), `CosmosCheckpointStorage` (Azure, preview — egress). Checkpoints fire at superstep boundaries, so a HITL pause lands on one cleanly. <https://learn.microsoft.com/agent-framework/workflows/checkpoints>
|
||||
|
||||
**Contradictions (reliability — LOW):**
|
||||
- **Docs** present durable "checkpoint → exit process → restore → respond" as a supported happy path.
|
||||
- **Community / bug tracker** shows persist-and-resume is the fragile half:
|
||||
- #5818 (OPEN, Magentic + `AgentSession` resume): same `request_id` sent back, workflow does not resume. <https://github.com/microsoft/agent-framework/issues/5818>
|
||||
- #6127 (unanswered, Sequential): resume re-prompts the same approval **and** re-executes the function; duplicate `CallId` in streaming chunks. <https://github.com/microsoft/agent-framework/discussions/6127>
|
||||
- #6372 (fixed PR #6491, but affected 1.1.0–**1.9.0**): fan-in barrier silently loses buffered messages across checkpoint/resume — **no error raised**. <https://github.com/microsoft/agent-framework/issues/6372>
|
||||
- #5621 (OPEN, Handoff restore): `Expected exactly one update for key 'SharedState'`, unmatched ToolApproval. <https://github.com/microsoft/agent-framework/issues/5621>
|
||||
- #5810 (fixed PR #6049): restore **type-allowlist** blocks non-safe types (`Checkpoint deserialization blocked for type … MessageRole`) — custom/Pydantic payloads must be registered via `allowed_checkpoint_types`. <https://github.com/microsoft/agent-framework/issues/5810>
|
||||
- #3255 (fixed PR #3689): sub-workflow restore re-sent already-answered requests → `Response provided for unknown request ID`. <https://github.com/microsoft/agent-framework/issues/3255>
|
||||
- **Version-skew (documented):** PR #3744 — "**Existing checkpoints cannot be resumed between versions.**" A MAF bump invalidates stored checkpoints.
|
||||
|
||||
### 4. Session / conversation-state interaction — Confidence: high
|
||||
|
||||
**External findings:**
|
||||
- The built-in `AgentExecutor` (wraps an agent inside a workflow) serializes on checkpoint: internal message cache, **full conversation history**, agent session state, and pending requests/responses — and restores them. So a HITL-paused agent-bearing workflow does NOT lose chat history. <https://learn.microsoft.com/agent-framework/workflows/advanced/agent-executor>
|
||||
- **Material gap for the Azure/Foundry profile:** "Checkpointing with agents that use **server-side sessions** (e.g. `FoundryAgent`) has limitations. Server-side session state is **not captured in checkpoints**." A durable HITL pause with a Foundry-hosted agent will not have its conversation reliably restored from the checkpoint alone. (Same doc.)
|
||||
- For pure custom-executor workflows (no `AgentExecutor`): only shared `state`, in-transit messages, and pending requests are captured; executor-local fields persist only if you override `on_checkpoint_save()` / `on_checkpoint_restore()`.
|
||||
- Pre-1.9.0 rename relevant to code around a pause: `SharedState`→`State`, `ctx.shared_state`→`ctx.state`, state getters/setters now **synchronous** (PR #3667).
|
||||
|
||||
**Cross-link:** this is the Fase 1 B7 bleed vector — cross-run conversation state lives in `AgentSession.state` + `InMemoryHistoryProvider`. The HITL/checkpoint state is workflow-level and distinct, but `fresh_workflow()` isolation still governs whether a restored conversation contaminates the next project run. (See `docs/research/2026-06-24-maf-capability-map.md`, Fase 1 B7.)
|
||||
|
||||
### 5. Group-chat / orchestration-level approval gate — Confidence: high
|
||||
|
||||
**Installed-source findings (decisive for our debate default):**
|
||||
- `GroupChatBuilder.with_request_info(*, agents: Sequence[str | SupportsAgentRun] | None = None)` **EXISTS** (`agent_framework_orchestrations/_group_chat.py:882`). It pauses after the named agent(s) respond and emits a `request_info` event (`type='request_info'`) "that allows the caller to review the conversation and optionally [approve/edit] … the standard response_handler/request_info pattern." Same method exists on `SequentialBuilder` (`_sequential.py:154`).
|
||||
- Participants matching the filter are wrapped as `AgentApprovalExecutor(WorkflowExecutor)` (`_orchestration_request_info.py:168`), constructed `allow_direct_output=True` so the user-approved final response surfaces as workflow output.
|
||||
- The human reply object is `AgentRequestInfoResponse` (public export) with `.approve()` (accept as-is), `.from_strings([text])`, `.from_messages([...])` (`_orchestration_request_info.py:44-85`). Supplied via `run(responses={request_id: AgentRequestInfoResponse...})`.
|
||||
- `resolve_request_info_filter(agents)` selects which agents pause for approval — e.g. pause only before the **checker** in maker-checker.
|
||||
- A separate, lighter gate also exists: tool-approval via `@tool(approval_mode="always_require")` → `function_approval_request` content → approve/deny (in-run, no graph).
|
||||
|
||||
**External findings:**
|
||||
- Docs/maintainer confirm there is **no single universal "approve/edit/reject" object**; the documented routes are (a) `with_request_info` + `AgentRequestInfoResponse`, (b) tool-approval, (c) Magentic plan-review (`enable_plan_review`) — Magentic is experimental and OFF our path. For arbitrary maker-checker over agent output, the intended route is `with_request_info` (now installed-source-confirmed for GroupChat) or a custom executor calling `ctx.request_info()`. <https://github.com/microsoft/agent-framework/discussions/1287>
|
||||
- **Caveat:** community shows the orchestration approval *resume* path is where bugs cluster (#5818, #6127, #6006) — the gate fires reliably; persisting/resuming the approval across serialization is fragile.
|
||||
|
||||
### 6. Design implication for our two-layer HITL + verdict capture — Confidence: high (recommendation)
|
||||
|
||||
This is decision-relevant — see **Recommendation** below. Short form: the brief's
|
||||
fallback assumption ("if MAF lacks native HITL, capture out-of-band") is **partly
|
||||
inverted**: MAF *has* native HITL, and the right split is to use the GA *in-run*
|
||||
pieces for synchronous review but keep the *durable learned verdict* out-of-band.
|
||||
|
||||
## External Knowledge
|
||||
|
||||
### Best Practice
|
||||
- Canonical 1.9.0 HITL = `ctx.request_info()` + `@response_handler` + `run(responses=)`; detect pause via `IDLE_WITH_PENDING_REQUESTS` / `get_request_info_events()`. Official samples: `guessing_game_with_human_input.py`, `sequential_request_info.py`, `checkpoint/checkpoint_with_human_in_the_loop.py`, `magentic_human_plan_review.py` (all under `microsoft/agent-framework` `python/samples/03-workflows/`).
|
||||
|
||||
### Security (relevant to D3 no-silent-egress + local-only)
|
||||
- `FileCheckpointStorage` writes **unencrypted pickle** blobs containing conversation history + pending HITL payloads + shared state. Encryption-at-rest / permissions / ACLs are **the developer's responsibility** (no built-in encryption). Lock the `storage_path` down (dedicated dir, `0700`/`0600`, encrypted volume; macOS FileVault helps).
|
||||
- Restore is hardened: restricted unpickler **ON by default** since 1.0.1 (we're on 1.9.0); non-safe types throw `WorkflowCheckpointException` unless registered in `allowed_checkpoint_types`. Treat it as a safety net, not the control — "never load checkpoints from untrusted sources."
|
||||
- **No CVEs** against `agent-framework*` (GitHub Security Advisories empty; OSV empty). Two transitive Starlette advisories on deps.dev — confirm via `uv run pip-audit`. Semantic Kernel CVEs (CVE-2026-26030/-25592) are a *different package* and not in the checkpoint path; only relevant if `semantic-kernel` is pulled in (it is not, per our pinned tree).
|
||||
- **Telemetry OFF by default** — no exporter ships; `ENABLE_INSTRUMENTATION`/`ENABLE_SENSITIVE_DATA`/`ENABLE_CONSOLE_EXPORTERS` all default false. No-silent-egress holds out of the box if we (a) don't set those, (b) set no `OTEL_EXPORTER_OTLP_*`, (c) use File/InMemory (never Cosmos). One subtlety: MAF auto-propagates OTel trace context into MCP `tools/call` `_meta` when a span is active — inert with instrumentation off; keep MCP servers local (stdio) regardless.
|
||||
|
||||
### Known Issues
|
||||
- See Dimension 3 contradictions. Plus: `with_request_info()` naming is opaque and there is no event-type filter for which events trigger a HITL pause (#3534, open). Local-model + HITL is **unverified by anyone** — zero community signal; several worst resume bugs are Azure-server-side-persistence-specific and may simply not apply locally, but then we own conversation-history persistence ourselves. Spike locally; do not rely on precedent.
|
||||
|
||||
## Gemini Second Opinion
|
||||
|
||||
Unavailable. The `gemini-mcp` server's client SDK predates Google's May-2026
|
||||
Interactions API breaking change and returned `400 BadRequestError` before any
|
||||
research ran. No independent Gemini triangulation was obtained for this topic;
|
||||
treat the second opinion as absent (not negative). To restore: upgrade the
|
||||
gemini-mcp server's client SDK to ≥ 2.0.0.
|
||||
|
||||
## Synthesis
|
||||
|
||||
The triangulation surfaces an insight no single source states: **MAF 1.9.0 has
|
||||
three different HITL surfaces, and the strongest one for *our* need is the
|
||||
lightest one — while the heaviest one (durable checkpoint-resume) is both the
|
||||
shakiest in practice and a poor fit for the problem.**
|
||||
|
||||
1. **In-run synchronous review** (`GroupChatBuilder.with_request_info` →
|
||||
`AgentApprovalExecutor` → `run(responses={id: AgentRequestInfoResponse...})`)
|
||||
is GA, installed-source-confirmed for our Group Chat maker-checker default, and
|
||||
needs **no checkpointing** (state persists across `run()` calls in-process).
|
||||
This is the solid, happy-path piece.
|
||||
|
||||
2. **Durable cross-process pause** (checkpoint + restore + responses) is where the
|
||||
docs promise and the bug tracker diverge hardest: open/silent-failure resume
|
||||
bugs into the 1.9.0 line, a pickle type-allowlist that fights our Pydantic IR,
|
||||
and "checkpoints cannot be resumed between versions." The contrarian pass is
|
||||
right that coupling our **highest-value data path** (the verdict the system
|
||||
*learns* from) to MAF's **most-churned, least-durable, Azure-favoring** surface
|
||||
is a self-inflicted risk — and a lock-in against the D7 Claude-SDK sibling,
|
||||
which has no executor/checkpoint model and can only share a *framework-agnostic*
|
||||
verdict seam.
|
||||
|
||||
3. The actual shape of "fagekspert enters a verdict the next run learns from" is
|
||||
an **indefinite, out-of-band, never-resume-the-original-run** decision. Its
|
||||
durable artifact is the hand-rolled **VerdictStore**, not an in-flight workflow
|
||||
checkpoint. A workflow checkpoint is engineered for "pause seconds-to-minutes,
|
||||
resume the same process" — the wrong tool for "pause indefinitely, decide
|
||||
elsewhere."
|
||||
|
||||
So the brief's binary ("native HITL → use it; else out-of-band") resolves to a
|
||||
**split**: adopt the GA in-run approval gate for the *synchronous Layer-1 review*
|
||||
where it fits; keep the *durable, learning-loop verdict* out-of-band in the
|
||||
VerdictStore; and **defer checkpointing off the MVP critical path** (matches the
|
||||
brief's `[OPEN]` default — and Topic 1 confirms the default rather than overturning
|
||||
it, because durable checkpoint-resume is the fragile part).
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **Layer-1 review: native gate vs. simplest possible?** `with_request_info` is
|
||||
GA and fits, but a custom executor calling `ctx.request_info()` gives full
|
||||
control over the request payload (the `ValidatedProposal` + provenance). Decide
|
||||
in /trekplan: native `with_request_info(agents=[checker])` vs custom request-info
|
||||
executor. Either way: in-process, no checkpoint.
|
||||
- **Is Layer-1 even in the MVP, or is it async-only?** The brief marks two-layer
|
||||
HITL semantics (sync review vs async+notification stub) as `[OPEN]`. If Fase 2
|
||||
ships only the async/out-of-band verdict + notification *stub* (B11), the native
|
||||
in-run gate may be deferred too — fewer moving parts. Resolve in /trekplan.
|
||||
- **Local-profile HITL behaviour** — unverified by anyone. Needs a self-spike:
|
||||
run the approval gate / `request_info` loop against the local OpenAI-compatible
|
||||
endpoint (Topic 3) and confirm it fires and resumes. Tie to Topic 3's outcome.
|
||||
- **If durable pause is ever needed:** the acceptance gate must be a
|
||||
resume-integrity test (pause → checkpoint → restore → assert pending requests
|
||||
match + conversation intact), given #6372/#5621-class silent failures. Out of
|
||||
MVP scope but record the condition.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**For the Fase 2 MVP, capture the expert verdict OUT-OF-BAND and keep the durable
|
||||
artifact in the hand-rolled VerdictStore; do NOT couple the learned-verdict path
|
||||
to native checkpoint-resume.** Concretely:
|
||||
|
||||
1. **Layer 2 (durable, learning loop) = out-of-band VerdictStore.** The emitted
|
||||
`ValidatedProposal` + provenance is presented to the fagekspert out-of-band; the
|
||||
verdict is written to the VerdictStore; the next run retrieves it via the ExpeL
|
||||
`ContextProvider` seam (`extend_instructions(source_id, instructions)`). This is
|
||||
framework-agnostic, D7-portable, and the success-criterion ("second run
|
||||
retrieves the prior verdict") is satisfied without any MAF checkpoint.
|
||||
2. **Layer 1 (optional in-run synchronous review) = GA native gate IF included.**
|
||||
If Fase 2 ships a synchronous review, use `GroupChatBuilder.with_request_info(
|
||||
agents=[checker])` (or a custom `ctx.request_info()` executor for a richer typed
|
||||
payload) — in-process, resume via `run(responses=...)`, **no checkpointing**.
|
||||
Register any custom Pydantic response type's expectations now so it survives if
|
||||
checkpointing is ever added.
|
||||
3. **Defer checkpointing off the MVP critical path** (brief `[OPEN]` default
|
||||
upheld). `InMemoryCheckpointStorage` is fine for tests; do not put
|
||||
`FileCheckpointStorage` durable HITL on the critical path. If added later:
|
||||
File/InMemory only (never Cosmos — egress), locked-down `storage_path`,
|
||||
`allowed_checkpoint_types` for our IR, a resume-integrity acceptance test, and
|
||||
awareness of cross-version checkpoint invalidation.
|
||||
4. **Provenance, not checkpoint, carries the audit trail** — the single emitted
|
||||
proposal's provenance stamp (citations + model/role + validator decision + token
|
||||
usage) is the durable record, consistent with the no-silent-egress + provenance
|
||||
NFRs.
|
||||
|
||||
Risks to carry into the plan: (a) MAF HITL/checkpoint API churn → any MAF-native
|
||||
HITL code carries upgrade cost; pin the surface and watch the changes guide.
|
||||
(b) Local-profile HITL is unverified → spike with Topic 3's local client.
|
||||
(c) If a synchronous in-run gate is used, the orchestration *resume* path is where
|
||||
community bugs cluster — keep it in-process (no serialization boundary) to dodge
|
||||
that whole class.
|
||||
|
||||
## Sources
|
||||
|
||||
| # | Source | Type | Quality | Used in |
|
||||
|---|--------|------|---------|---------|
|
||||
| 1 | `.venv/.../agent_framework/_workflows/_workflow_context.py:393` (`request_info`) | codebase | high | Dim 1, 2 |
|
||||
| 2 | `.venv/.../agent_framework/_workflows/_request_info_mixin.py` (`response_handler`, `RequestInfoMixin`) | codebase | high | Dim 1 |
|
||||
| 3 | `.venv/.../agent_framework/_workflows/_workflow.py:675+` (`run(responses=, checkpoint_id=)`) | codebase | high | Dim 2, 3 |
|
||||
| 4 | `.venv/.../agent_framework/_workflows/_events.py:58-65` (`WorkflowRunState`) | codebase | high | Dim 2 |
|
||||
| 5 | `.venv/.../agent_framework/_workflows/_runner_context.py:424,457-471` (rehydrate + `send_request_info_response`) | codebase | high | Dim 2, 3 |
|
||||
| 6 | `.venv/.../agent_framework/_workflows/_checkpoint.py:81` (`pending_request_info_events`) | codebase | high | Dim 3 |
|
||||
| 7 | `.venv/.../agent_framework_orchestrations/_group_chat.py:882` (`GroupChatBuilder.with_request_info`) | codebase | high | Dim 5 |
|
||||
| 8 | `.venv/.../agent_framework_orchestrations/_orchestration_request_info.py:44-85,168` (`AgentRequestInfoResponse`, `AgentApprovalExecutor`) | codebase | high | Dim 5 |
|
||||
| 9 | <https://learn.microsoft.com/agent-framework/workflows/human-in-the-loop> | official | high | Dim 1,2,3,5 |
|
||||
| 10 | <https://learn.microsoft.com/agent-framework/workflows/checkpoints> | official | high | Dim 3, Security |
|
||||
| 11 | <https://learn.microsoft.com/agent-framework/workflows/advanced/agent-executor> | official | high | Dim 4 |
|
||||
| 12 | <https://learn.microsoft.com/agent-framework/support/upgrade/python-2026-significant-changes> | official | high | Dim 1,2,3 |
|
||||
| 13 | <https://learn.microsoft.com/agent-framework/support/upgrade/requests-and-responses-upgrade-guide-python> | official | high | Dim 1 |
|
||||
| 14 | <https://learn.microsoft.com/agent-framework/migration-guide/from-autogen/> | official | high | Dim 1 |
|
||||
| 15 | <https://github.com/microsoft/agent-framework/issues/5818> (Magentic AgentSession resume, OPEN) | community | high | Dim 3,5 |
|
||||
| 16 | <https://github.com/microsoft/agent-framework/discussions/6127> (Sequential re-prompt+re-exec) | community | medium | Dim 3,5 |
|
||||
| 17 | <https://github.com/microsoft/agent-framework/issues/6372> (fan-in barrier silent loss, 1.1.0–1.9.0) | community | high | Dim 3 |
|
||||
| 18 | <https://github.com/microsoft/agent-framework/issues/5621> (Handoff restore fails, OPEN) | community | high | Dim 3 |
|
||||
| 19 | <https://github.com/microsoft/agent-framework/issues/5810> (checkpoint type-allowlist) | community | high | Dim 3, Security |
|
||||
| 20 | <https://github.com/microsoft/agent-framework/issues/3255> (sub-workflow dup request, fixed) | community | high | Dim 3 |
|
||||
| 21 | <https://github.com/microsoft/agent-framework/issues/3534> (with_request_info naming/filter, OPEN) | community | medium | Known Issues |
|
||||
| 22 | <https://github.com/microsoft/agent-framework/discussions/1287> (no universal approval object; custom executor) | community | high | Dim 5 |
|
||||
| 23 | <https://www.diagrid.io/blog/still-not-durable-how-microsoft-agent-framework-and-strands-agents-repeat-the-same-mistake> | community | medium | Synthesis |
|
||||
| 24 | <https://github.com/microsoft/agent-framework/discussions/2305> (checkpoint limitations) | community | medium | Synthesis |
|
||||
| 25 | <https://github.com/microsoft/agent-framework/issues/4078> (stability/production timeline, unanswered) | community | medium | Synthesis |
|
||||
| 26 | <https://github.com/microsoft/agent-framework/security/advisories> (no advisories) | official | high | Security |
|
||||
| 27 | <https://osv.dev/list?ecosystem=PyPI&q=agent-framework> (no results) | official | high | Security |
|
||||
| 28 | <https://learn.microsoft.com/agent-framework/agents/observability> (telemetry off by default) | official | high | Security |
|
||||
| 29 | <https://learn.microsoft.com/agent-framework/agents/providers/ollama> (local provider exists) | official | medium | Known Issues |
|
||||
| 30 | <https://pypi.org/project/agent-framework-core/> (1.9.0 = 2026-06-18) | official | high | Exec summary |
|
||||
|
|
@ -1,245 +0,0 @@
|
|||
---
|
||||
type: trekresearch-brief
|
||||
created: 2026-06-24
|
||||
question: "What is the best way to expose a local document folder to a MAF 1.9.0 agent via MCPStdioTool such that retrieved content carries citation metadata (file + locator) — existing filesystem MCP server vs thin custom server, and the citation shape a context provider expects?"
|
||||
confidence: 0.82
|
||||
dimensions: 6
|
||||
mcp_servers_used: [microsoft-learn, tavily]
|
||||
local_agents_used: [installed-source-introspection (orchestrator, main context)]
|
||||
external_agents_used: [docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (unavailable)]
|
||||
topic: 2
|
||||
brief: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md
|
||||
---
|
||||
|
||||
# Local-folder data access via MCP with citation provenance (MAF 1.9.0)
|
||||
|
||||
> Generated by trekresearch (Voyage 5.6.0) on 2026-06-24. Topic 2 of 3 for the
|
||||
> Fase 2 MVP-vertical-slice brief. Scope: external swarm + installed-source
|
||||
> introspection (installed 1.9.0 API truth wins over Learn docs). Gemini
|
||||
> unavailable (MCP SDK broke on Google's May-2026 API change).
|
||||
|
||||
## Research Question
|
||||
|
||||
What is the best way to expose a LOCAL document folder to a MAF 1.9.0 agent via
|
||||
`MCPStdioTool` so retrieved content carries citation metadata (file + locator) —
|
||||
reuse an existing filesystem MCP server, or build a thin custom server — and what
|
||||
citation shape does MAF's context/annotation model expect? Decision it feeds: the
|
||||
data-access step, the citation-aware context provider, provenance-stamping, and the
|
||||
build-vs-reuse decision for the local-folder server.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**The brief's lean ("reuse the official `@modelcontextprotocol/server-filesystem`
|
||||
+ attach citations via `parse_tool_results`") does not survive contact with the
|
||||
requirement and should be dropped.** The official filesystem server returns raw
|
||||
text and bare path strings only — **no line/char offsets, no chunk IDs; its
|
||||
`search_files` is filename-glob, not content search** — so it physically cannot
|
||||
yield a `(file + locator + snippet)` citation; you would have to re-read, re-chunk
|
||||
and re-locate inside the callback (i.e. build the whole citation engine anyway, in
|
||||
the worst place, behind a Node subprocess). MAF also does **not** auto-create
|
||||
citation annotations from MCP results. **Recommendation (honouring the CLAUDE.md
|
||||
"data access via MCP, JSON-Schema-validated, fail-fast" convention): build a THIN
|
||||
CUSTOM local-folder MCP server that returns citation-ready structured chunks
|
||||
(`{file, locator, snippet, score}`), wrapping a framework-agnostic in-process
|
||||
retriever core that is the D7-portable seam.** Carry provenance as first-class
|
||||
Pydantic data on the emitted proposal, independent of MAF's annotation propagation
|
||||
(which has an open Python streaming-drop bug, #4316). Confidence 0.82: the
|
||||
capability facts are HIGH (installed-source + official-README verified); the
|
||||
exact MVP retrieval depth (keyword vs local embeddings) is a deferrable design
|
||||
choice.
|
||||
|
||||
## Dimensions
|
||||
|
||||
### 1. Can the official filesystem server carry citations? — Confidence: high (NO)
|
||||
|
||||
**External findings (docs + community + contrarian agree):**
|
||||
- `@modelcontextprotocol/server-filesystem` tools: `read_text_file` (whole file, or `head`/`tail` by line *count* only), `read_multiple_files`, `list_directory`, `directory_tree`, `get_file_info`, `search_files`, `write_file`, `edit_file`. <https://github.com/modelcontextprotocol/servers/blob/main/src/filesystem/README.md>
|
||||
- Read results = **raw UTF-8 text, no line numbers, no byte/char offsets, no chunk IDs.** `search_files` returns **matching file PATHS only** (filename glob — NOT content grep). `get_file_info` = size/timestamps, not a locator.
|
||||
- Net: the only locator it gives is the file path you already passed in. Enhanced community forks advertise `read_file_lines` / grep-with-line-numbers / chunking precisely because the official server lacks them.
|
||||
|
||||
**Conclusion:** the official server cannot produce `(file + locator + snippet)` citations. It is fine only for "let the agent open named files," not citation-backed retrieval.
|
||||
|
||||
### 2. Does MAF auto-create citations from MCP results? — Confidence: high (NO)
|
||||
|
||||
**Installed-source findings (1.9.0):**
|
||||
- Default parser `_parse_contents_from_mcp_tool_result` / `_parse_tool_result_from_mcp` maps MCP content → `Content.from_text/from_data/from_uri`; **attaches no annotations** (`agent_framework/_mcp.py:507-517`).
|
||||
- `structuredContent` IS now serialized to text Content (`_mcp.py:586-587`) — so the #3313 "structuredContent dropped → None" bug **is fixed in installed 1.9.0**. But it lands as JSON *text*, not as typed/annotated content — you still parse it yourself for provenance.
|
||||
- The seam to attach citations is `MCPTool.__init__(parse_tool_results: Callable[[CallToolResult], str | list[Content]])` — your callback fully replaces the default parse.
|
||||
|
||||
**External findings:**
|
||||
- Confirmed: docs/RAG paths use **prompt-level** "cite your sources" (a `string_mapper`), not structured annotations. The one citation-aware RAG provider, `TextSearchProvider`, is **.NET-only — no Python equivalent**. <https://learn.microsoft.com/agent-framework/agents/rag>
|
||||
- **Open bug #4316:** file-citation annotations are **silently dropped in the Python streaming path** (`annotations=None`); .NET fixed, Python lagging. → do NOT rely on MAF's `Annotation` propagation for load-bearing provenance. <https://github.com/microsoft/agent-framework/issues/4316>
|
||||
|
||||
### 3. Citation shape MAF expects — Confidence: high
|
||||
|
||||
**Installed-source findings (1.9.0, ground truth):**
|
||||
- `Annotation` is a **TypedDict** (`agent_framework/_types.py:374`): `type: Literal["citation"]`, `title`, `url`, `file_id`, `tool_name`, `snippet`, `annotated_regions: Sequence[TextSpanRegion]`, `additional_properties`, `raw_representation`.
|
||||
- `TextSpanRegion` TypedDict (`_types.py:366`): `type: Literal["text_span"]`, `start_index`, `end_index`.
|
||||
- The class form `CitationAnnotation` was replaced by this TypedDict in 1.0.0b260123 (PR #3252) — installed 1.9.0 = **TypedDict** form. Build citations as dicts: `{"type":"citation","file_id":...,"title":...,"snippet":...,"annotated_regions":[{"start_index":..,"end_index":..}]}`.
|
||||
- Carrier: `FunctionResultContent.annotations: list[... ] | None` (accepts dict/TypedDict).
|
||||
|
||||
**Use:** our provenance stamp's "≥1 citation" maps cleanly onto this `Annotation`. But because of #4316, treat `Annotation` as a *display/serialization* surface and keep the authoritative provenance as our own Pydantic object.
|
||||
|
||||
### 4. Context provider injection seam — Confidence: high (corrects an external error)
|
||||
|
||||
**Installed-source findings (1.9.0) — and a flagged docs-vs-source conflict:**
|
||||
- `class ContextProvider` (`agent_framework/_sessions.py:351`), `__init__(self, source_id: str)` — **source_id REQUIRED** (confirms Fase 1).
|
||||
- Hooks: `async def before_run(...)` (`:370`) and `async def after_run(...)` (`:391`) — add messages/instructions/tools in `before_run`, process/store in `after_run`.
|
||||
- `Context.extend_instructions(self, source_id: str, instructions: str | Sequence[str])` (`:253`) and `extend_tools(source_id, tools)` (`:266`) — **the two-arg seam the brief and Fase 1 specified, confirmed verbatim.**
|
||||
- `MemoryContextProvider` / `InMemoryHistoryProvider` exist but are conversation memory, **not** retrievers.
|
||||
|
||||
> **Premiss correction (important for /trekplan):** the docs-researcher reported, with high confidence from the Learn API-ref page, that Python uses `invoking`/`invoked` hooks and that `source_id` / `extend_instructions` are "C#-only, not in Python." **That is wrong for installed 1.9.0** — the installed source has `before_run`/`after_run`, `ContextProvider(source_id)`, and `extend_instructions(source_id, …)`. Per CLAUDE.md, installed source wins. Do NOT let the planning phase adopt the `invoking`/`invoked` / no-source_id surface.
|
||||
|
||||
**Design fit:** a custom `ExpeLContextProvider(ContextProvider)` (Fase 1 seam, promote to core) injects retrieved+cited content + prior verdicts in `before_run` via `extend_instructions(source_id, …)`, and the same provider family is where the ExpeL learning loop reads VerdictStore. No built-in local-folder retrieval provider exists in Python — retrieval is ours to build.
|
||||
|
||||
### 5. Build vs reuse (the decision) — Confidence: high (recommendation)
|
||||
|
||||
**Triangulated picture:**
|
||||
- **Reuse official server:** cannot cite (Dim 1); adds a Node runtime + `npx` cold-start + version-pin burden; npx PATH/nvm handshake failure is THE most-reported stdio blocker on macOS (B5); supply-chain surface (chalk/debug + Shai-Hulud hit MCP-SDK transitives). Security: usable only if pinned ≥ `2025.7.1` (EscapeRoute CVE-2025-53109/53110, fixed there; current `2026.1.14` clean), vendored (never `npx @latest`), allowlist scoped to the doc folder (never `$HOME`).
|
||||
- **Thin custom MCP server:** removes the npm supply-chain surface, emits provenance natively (`{file, line_range/char_span, snippet, score}` via `structuredContent`), honours the CLAUDE.md "data access via MCP, JSON-Schema-validated, fail-fast" convention, and is portable to the D7 Claude-SDK sibling's in-process-MCP idiom. Cost: **we own** path-canonicalisation + symlink-realpath validation (the exact bug class the official server had fixed twice — TDD against the EscapeRoute scenarios) and input validation (no shelling out; use native file APIs).
|
||||
- **In-process retriever, no MCP** (contrarian's first choice): simplest + lowest latency + best D7 fit, but conflicts with the project's "data access via MCP" convention. Keep as the fallback if that convention is relaxed.
|
||||
- Reference design: `shinpr/mcp-local-rag` (LanceDB file-based + local embeddings; returns `path + chunk index + title + score + chunk text` + `read_chunk_neighbors` context expansion) — the closest local-only, citation-grade pattern. <https://github.com/shinpr/mcp-local-rag>
|
||||
|
||||
### 6. Retrieval depth for the MVP — Confidence: medium
|
||||
|
||||
- The MVP uses a **tiny synthetic "anleggskostnad" domain** (Fase 0). A full vector stack is not required for D5 (90%) / D6 (cost). A simple in-process retriever (keyword/substring + chunk-at-ingest with exact locators) likely suffices and keeps deps minimal.
|
||||
- The Python "real RAG" path (SK `VectorStore.create_search_function().as_agent_framework_tool()`) **requires pulling in `semantic-kernel ≥ 1.38`** — a different package with its own RCE CVE (CVE-2026-26030, eval in in-memory vector-store filter, < 1.39.4). Adding it conflicts with the "GA-pakker pinnet eksplisitt, ikke metaen" dependency discipline and adds CVE surface. **Defer / avoid for MVP** unless semantic retrieval proves necessary; if adopted, pin ≥ 1.39.4.
|
||||
|
||||
## External Knowledge
|
||||
|
||||
### Best Practice
|
||||
- Canonical `MCPStdioTool` wiring (official docstring + Learn): `MCPStdioTool(name="filesystem", command="npx", args=["-y","@modelcontextprotocol/server-filesystem", <dir>])`, used as `async with`; requires `pip install mcp --pre`. <https://learn.microsoft.com/agent-framework/agents/tools/local-mcp-tools>
|
||||
- MAF can also expose an *agent* as an MCP server (`agent.as_mcp_server()`) — not relevant to local-folder retrieval. Generic server-building is the MCP Python SDK / FastMCP domain. <https://github.com/modelcontextprotocol/python-sdk>
|
||||
|
||||
### Security (D3 no-silent-egress + local-only)
|
||||
- Stdio MCP server makes **no network calls**; only inherent egress in the path is the `npx` fetch at launch (removed by vendoring/pinning). OTel `_meta` trace-context injection into `tools/call` is inert with instrumentation off and carries only random W3C IDs (no project data).
|
||||
- EscapeRoute CVEs (CVE-2025-53110 prefix-collision, CVE-2025-53109 symlink-escape) — both **fixed in 2025.7.1**; exploit trigger is prompt injection of the agent (relevant: our maker-checker + blocking validator are compensating controls). Never run the server with elevated privileges. <https://cymulate.com/blog/cve-2025-53109-53110-escaperoute-anthropic/>
|
||||
- npx supply chain: pin exact version + lockfile, verify tarball for `postinstall`, quarantine fresh versions, consider containerization. A thin custom dependency-light server removes this surface entirely.
|
||||
|
||||
### Known Issues
|
||||
- MAF MCP bugs — **verified against installed 1.9.0**: #3313 structuredContent parsing **FIXED** (`_mcp.py:586`); #2884 stdio session-invalidation handled (`is_connected` reset + `ClosedResourceError` catch, `_mcp.py:935/1355/1544`). Still: #2284 `_meta` dropped (don't rely on `_meta` for provenance — use `structuredContent`/`content`); #4316 Python streaming citation-annotation drop (open).
|
||||
- npx/nvm PATH handshake failure on macOS (servers#64) — use absolute `command` path + explicit `env` PATH/NODE_PATH, prefer pre-installed/`uvx`-pinned over cold `npx -y`. Filesystem server fails to start if any allowed dir is unavailable (servers#3232).
|
||||
- Don't split a citation across `content` vs `structuredContent` vs `_meta` — clients (MAF included) forward them inconsistently; put it in one reliably-forwarded channel.
|
||||
|
||||
## Gemini Second Opinion
|
||||
|
||||
Unavailable (same MCP-SDK/Interactions-API failure as Topic 1). No independent
|
||||
Gemini triangulation for this topic; treat as absent, not negative.
|
||||
|
||||
## Synthesis
|
||||
|
||||
Three sources converge on a conclusion the brief did not anticipate: **the
|
||||
citation requirement, not the data-access requirement, drives the design — and it
|
||||
rules out the off-the-shelf filesystem server.** Citations of the form
|
||||
`(file + locator + snippet)` must be *exact by construction*, which means
|
||||
chunk-and-locate **at ingest**, in code we own. Bolting them on afterward — over a
|
||||
third-party server's lossy whole-file text, through a `parse_tool_results` callback,
|
||||
relying on a MAF annotation path with an open streaming-drop bug — stacks three
|
||||
fragilities to recreate something we could have produced cleanly up front.
|
||||
|
||||
The right shape is a **two-layer seam**: (1) a framework-agnostic in-process
|
||||
retriever core — `retrieve(query) -> list[RetrievedChunk]`, where `RetrievedChunk`
|
||||
is a Pydantic type carrying `file`, `locator` (line range / char span), `snippet`,
|
||||
`score` — which is the **D7-portable contract** (MAF and Claude Agent SDK both
|
||||
satisfy it); and (2) a **thin custom MCP server** wrapping that core, returning the
|
||||
chunks as structured `structuredContent`, which honours the project's "data access
|
||||
via MCP, JSON-Schema-validated, fail-fast" convention and ports to the Claude SDK's
|
||||
in-process-MCP idiom. Provenance lives as first-class Pydantic data on the emitted
|
||||
proposal; the MAF `Annotation` TypedDict is only a display/serialisation view of it,
|
||||
so #4316 never sits on the load-bearing path. The official filesystem server is
|
||||
demoted to a possible *future* "browse named files" convenience, pinned and scoped.
|
||||
|
||||
This also threads the CLAUDE.md needle: the contrarian's "skip MCP, in-process
|
||||
only" is the simplest and the best D7 fit, but it breaks the MCP-data-access
|
||||
convention; the thin-custom-MCP-server-over-an-in-process-core keeps the convention
|
||||
*and* the D7 portability *and* citation fidelity — at the cost of owning sandbox
|
||||
correctness (testable, bounded).
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **MVP retrieval depth:** keyword/substring + ingest-time chunking (no extra deps,
|
||||
D5/D6-aligned) vs local embeddings (LanceDB/SK VectorStore, heavier + CVE surface).
|
||||
Default to keyword for the tiny synthetic domain; revisit only if recall is poor.
|
||||
Resolve in /trekplan.
|
||||
- **Locator format:** line range vs char span. It is the shared D7 contract — define
|
||||
and test it in the golden-suite up front. Default: char span (`start_index`/
|
||||
`end_index`) to map straight onto `TextSpanRegion`; carry line numbers in
|
||||
`additional_properties` for human display.
|
||||
- **Does the MVP even need MCP, or is the in-process retriever enough?** The CLAUDE.md
|
||||
convention says data access via MCP. The thin-custom-server honours it; confirm the
|
||||
operator wants the MCP boundary in Fase 2 vs deferring it (the in-process core is
|
||||
needed either way). Surface in /trekplan.
|
||||
- **Chunk-at-ingest vs at-query:** ingest-time chunking gives exact, cacheable
|
||||
locators; decide whether the MVP ingests once or reads live. Tie to determinism NFR.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Build a thin custom local-folder MCP server that returns citation-ready
|
||||
structured chunks, wrapping a framework-agnostic in-process retriever core. Do NOT
|
||||
reuse `@modelcontextprotocol/server-filesystem` for citation-backed retrieval.**
|
||||
|
||||
1. **Retriever core (D7 seam):** `retrieve(query, top_k) -> list[RetrievedChunk]`;
|
||||
`RetrievedChunk(file: str, locator: TextSpan, snippet: str, score: float)`.
|
||||
Chunk + assign locators at ingest so citations are exact by construction.
|
||||
Start with keyword/substring retrieval (no heavy deps; D5/D6). Pure Python,
|
||||
no MAF/MCP imports — portable to the Claude SDK sibling.
|
||||
2. **Thin custom MCP server (MAF-facing, honours the convention):** a minimal
|
||||
`mcp`/FastMCP stdio server exposing a `search(query)` tool that returns the
|
||||
chunks via `structuredContent` (JSON-Schema-validated, fail-fast on bad config).
|
||||
Path handling is security-critical: canonicalise + boundary-check (no
|
||||
`startsWith`), resolve symlink realpath and fail closed, native file APIs (no
|
||||
shelling out) — TDD against the EscapeRoute bypass patterns.
|
||||
3. **Wire into MAF:** `MCPStdioTool(name="docfolder", command=<abs python>, args=[...])`
|
||||
with a `parse_tool_results` callback that maps `structuredContent` chunks into
|
||||
`FunctionResultContent` + `Annotation(type="citation", file_id, snippet,
|
||||
annotated_regions=[TextSpanRegion])`. On Intel mac, pass absolute `command` +
|
||||
explicit `env` PATH; pre-install the server (no cold `npx`/uvx fetch on the path).
|
||||
4. **Provenance = first-class Pydantic data** on the emitted `ValidatedProposal`
|
||||
(≥1 citation: file + locator + snippet, plus model/role + validator decision +
|
||||
token usage). The MAF `Annotation` TypedDict is only its display view — route
|
||||
around #4316 by never depending on MAF annotation propagation for the assertion.
|
||||
5. **Citation-aware context injection:** a custom `ExpeLContextProvider(ContextProvider)`
|
||||
(`source_id` required) uses `before_run` + `extend_instructions(source_id, …)` to
|
||||
inject cited content (and prior verdicts) — the confirmed two-arg seam.
|
||||
6. **If the official filesystem server is ever used** (future "browse named files"):
|
||||
pin ≥ `2025.7.1` vendored (never `npx @latest`), allowlist the doc folder only
|
||||
(never `$HOME`), absolute node path + explicit env.
|
||||
|
||||
Fallback (only if the MCP-data-access convention is relaxed for the MVP): skip the
|
||||
MCP server and call the in-process retriever directly via a `@tool`/`ContextProvider`
|
||||
— simplest, lowest-latency, best D7 fit.
|
||||
|
||||
Risks to carry into the plan: (a) we own MCP-server sandbox correctness — treat
|
||||
path/symlink validation as security-critical, TDD it; (b) MAF Python citation
|
||||
annotation propagation is buggy in streaming (#4316) — provenance must be our own
|
||||
data; (c) adding `semantic-kernel` for vector RAG pulls a CVE-bearing package — defer.
|
||||
|
||||
## Sources
|
||||
|
||||
| # | Source | Type | Quality | Used in |
|
||||
|---|--------|------|---------|---------|
|
||||
| 1 | `.venv/.../agent_framework/_mcp.py:507-517,586-587,655` (default parser, structuredContent) | codebase | high | Dim 2, Known Issues |
|
||||
| 2 | `.venv/.../agent_framework/_mcp.py:911,935,1330,1355,1544` (session lifecycle, ClosedResourceError) | codebase | high | Known Issues |
|
||||
| 3 | `.venv/.../agent_framework/_types.py:366,374` (`TextSpanRegion`, `Annotation` TypedDict) | codebase | high | Dim 3 |
|
||||
| 4 | `.venv/.../agent_framework/_sessions.py:351,370,391,253,266` (`ContextProvider`, `before_run`/`after_run`, `extend_instructions`) | codebase | high | Dim 4 |
|
||||
| 5 | <https://github.com/modelcontextprotocol/servers/blob/main/src/filesystem/README.md> (tool surface, no locators) | official | high | Dim 1 |
|
||||
| 6 | <https://www.npmjs.com/package/@modelcontextprotocol/server-filesystem> | official | high | Dim 1,5 |
|
||||
| 7 | <https://learn.microsoft.com/python/api/agent-framework-core/agent_framework.mcpstdiotool?view=agent-framework-python-latest> | official | high | Dim 5, Best Practice |
|
||||
| 8 | <https://learn.microsoft.com/agent-framework/agents/tools/local-mcp-tools> | official | high | Best Practice |
|
||||
| 9 | <https://learn.microsoft.com/agent-framework/agents/rag> (TextSearchProvider .NET-only; SK VectorStore path) | official | high | Dim 2, 6 |
|
||||
| 10 | <https://learn.microsoft.com/agent-framework/support/upgrade/python-2026-significant-changes> (Annotation TypedDict PR #3252) | official | high | Dim 3 |
|
||||
| 11 | <https://github.com/microsoft/agent-framework/issues/4316> (Python streaming citation drop, OPEN) | community | high | Dim 2, Synthesis |
|
||||
| 12 | <https://github.com/microsoft/agent-framework/issues/3313> (structuredContent dropped, fixed) | community | high | Dim 2, Known Issues |
|
||||
| 13 | <https://github.com/microsoft/agent-framework/issues/2884> (stdio session not invalidated, fixed #3154) | community | high | Known Issues |
|
||||
| 14 | <https://github.com/microsoft/agent-framework/issues/2284> (`_meta` discarded) | community | medium | Known Issues |
|
||||
| 15 | <https://github.com/modelcontextprotocol/servers/issues/64> (npx/nvm PATH handshake) | community | high | Dim 5, Known Issues |
|
||||
| 16 | <https://github.com/modelcontextprotocol/servers/issues/3232> (startup fails on unavailable dir) | community | medium | Known Issues |
|
||||
| 17 | <https://cymulate.com/blog/cve-2025-53109-53110-escaperoute-anthropic/> (EscapeRoute CVEs) | community | high | Dim 5, Security |
|
||||
| 18 | <https://github.com/advisories/GHSA-q66q-fx2p-7w4m> (CVE-2025-53109) | official | high | Security |
|
||||
| 19 | <https://github.com/advisories/GHSA-hc55-p739-j48w> (CVE-2025-53110) | official | high | Security |
|
||||
| 20 | <https://security.snyk.io/package/npm/@modelcontextprotocol%2Fserver-filesystem> (no advisories > 2025.7.1) | community | high | Security |
|
||||
| 21 | <https://stacklok.com/blog/examining-the-impact-of-npm-supply-chain-attacks-on-mcp> (chalk/debug, Shai-Hulud) | community | high | Dim 5, Security |
|
||||
| 22 | <https://github.com/shinpr/mcp-local-rag> (local-only citation-grade RAG reference) | community | medium | Dim 5 |
|
||||
| 23 | <https://docs.claude.com/en/api/agent-sdk/custom-tools> (Claude SDK in-process MCP idiom) | official | high | Dim 5, Synthesis |
|
||||
| 24 | <https://learn.microsoft.com/azure/foundry/agents/how-to/tools/file-search> (hosted FileSearch cites service-side) | official | medium | Dim 2 |
|
||||
| 25 | <https://www.anthropic.com/engineering/code-execution-with-mcp> (MCP token bloat) | community | medium | Known Issues |
|
||||
|
|
@ -1,237 +0,0 @@
|
|||
---
|
||||
type: trekresearch-brief
|
||||
created: 2026-06-24
|
||||
question: "How does agent-framework-core 1.9.0 run agents against a local model on the free/local profile (OpenAI-compatible endpoint, Ollama, or other local chat client), and does that path populate UsageDetails token counts or return None?"
|
||||
confidence: 0.80
|
||||
dimensions: 6
|
||||
mcp_servers_used: [microsoft-learn, tavily]
|
||||
local_agents_used: [installed-source-introspection (orchestrator, main context)]
|
||||
external_agents_used: [docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (unavailable)]
|
||||
topic: 3
|
||||
brief: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md
|
||||
---
|
||||
|
||||
# Real local-profile chat client for agent-framework 1.9.0 + UsageDetails
|
||||
|
||||
> Generated by trekresearch (Voyage 5.6.0) on 2026-06-24. Topic 3 of 3 for the
|
||||
> Fase 2 MVP-vertical-slice brief. Scope: external swarm + installed-source
|
||||
> introspection (installed API truth wins). Gemini unavailable (MCP SDK broke on
|
||||
> Google's May-2026 API change).
|
||||
|
||||
## Research Question
|
||||
|
||||
How does `agent-framework-core` 1.9.0 (+ `agent-framework-openai`) run agents
|
||||
against a LOCAL model on the free/local profile, and does that path populate
|
||||
`UsageDetails` token counts or return `None`? Decision it feeds: the backend-profile
|
||||
abstraction (local path), the budget middleware's `None`-handling, the end-to-end
|
||||
run, and cost-discipline verification.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Two local paths exist, and the brief's framing needs two corrections.** Path A —
|
||||
`OpenAIChatCompletionClient(base_url=<local /v1>)` — is **installed (1.8.2), needs
|
||||
zero new dependencies, and populates `UsageDetails` None-safely from the local
|
||||
server's `usage` object**. Path B — the native `OllamaChatClient` — is a **`--pre`
|
||||
beta package (`agent-framework-ollama 1.0.0b260521`) that is NOT installed** and
|
||||
adds the `ollama` client dependency. **Correction 1 (client):** use
|
||||
`OpenAIChatCompletionClient` (Chat Completions API — the surface Ollama/LM Studio
|
||||
`/v1` exposes), NOT `OpenAIChatClient` (Responses API). **Correction 2 (streaming):**
|
||||
run the debate/validator path **NON-STREAMING** — that simultaneously dodges the
|
||||
well-documented `/v1` streaming+tool-calling drop bugs AND the streaming
|
||||
`usage=None` problem (`UsageDetails` is reliably populated only non-streaming).
|
||||
**Token accounting works** (confirmed in installed source), so the "real token
|
||||
accounting" success criterion is satisfiable on the local profile. The real limits
|
||||
are model capability and Intel-Mac CPU speed: small CPU-runnable models are weak at
|
||||
tool-calling, and an Intel Mac (no Metal/MLX, ~4–6 tok/s on 7B) makes a multi-round
|
||||
debate take minutes — so local is for plumbing/smoke verification, with
|
||||
representative-quality runs pushed to Foundry. Confidence 0.80.
|
||||
|
||||
## Dimensions
|
||||
|
||||
### 1. Local chat-client paths — Confidence: high
|
||||
|
||||
**Installed-source findings (ground truth):**
|
||||
- `agent-framework-openai` 1.8.2 exports **both** `OpenAIChatClient` (Responses API) and `OpenAIChatCompletionClient` (Chat Completions API) — both import OK in the venv. `RawOpenAIChatClient`/`RawOpenAIChatCompletionClient` are the lower-level variants.
|
||||
- Both accept `base_url` (→ env `OPENAI_BASE_URL`) and `api_key` (required even for local — pass a dummy like `"ollama"`/`"not-needed"`). `model_id` selects the model.
|
||||
- The native Ollama path: `agent_framework/ollama/__init__.pyi` re-exports `OllamaChatClient`, `OllamaChatOptions`, `OllamaEmbeddingClient`, … from `agent_framework_ollama` — but **`agent_framework_ollama` is NOT installed** (`import agent_framework_ollama` → ModuleNotFoundError; only the lazy re-export stub exists). Installing it (`uv add agent-framework-ollama --prerelease=allow`) pulls `ollama >=0.5.3,<0.5.4`.
|
||||
|
||||
**External findings:**
|
||||
- **Correct client for local `/v1`:** Ollama/LM Studio/vLLM expose a Chat Completions surface (`/v1/chat/completions`), so `OpenAIChatCompletionClient` is the technically correct client; `OpenAIChatClient` targets the Responses API which local servers don't expose. Two official MS docs disagree on which to use for Ollama — **flagged conflict; the Chat-Completions client is correct for `/v1`.** <https://learn.microsoft.com/agent-framework/integrations/openai-endpoints>
|
||||
- Official local-endpoint table: Ollama `http://localhost:11434/v1/`, LM Studio `http://localhost:1234/v1/`, vLLM `http://localhost:8000/v1/`. The `/v1/` suffix matters; the native `OllamaChatClient` uses `:11434` WITHOUT `/v1`.
|
||||
- Native `OllamaChatClient` is **preview/beta** (`1.0.0b260521`, "4-Beta", `--pre`), wraps `ollama.AsyncClient` (native `/api/chat`), `OTEL_PROVIDER_NAME='ollama'`. MS docs describe it as having "full support for function tools and streaming." <https://learn.microsoft.com/agent-framework/agents/providers/ollama>
|
||||
|
||||
### 2. UsageDetails population — Confidence: high
|
||||
|
||||
**Installed-source findings (the load-bearing answer):**
|
||||
- `OpenAIChatCompletionClient`: `_chat_completion_client.py:705` — `usage_details=self._parse_usage_from_openai(response.usage) if response.usage else None`. **None-safe.** `_parse_usage_from_openai` (`:757-776`) maps `usage.prompt_tokens`→`input_token_count`, `usage.completion_tokens`→`output_token_count`, `usage.total_tokens`→`total_token_count` (+ reasoning/cached/audio details).
|
||||
- `UsageDetails` is the TypedDict in `agent_framework/_types.py` (`input/output/total_token_count: int | None`); `add_usage_details(u1, u2)` helper exists (`_types.py:417`) — use it to accumulate across calls in the shared budget meter. Lands on `response.usage_details` (`_types.py:547`); read `response.usage_details["total_token_count"]` (None-safe).
|
||||
- Streaming: `_chat_completion_client.py:724-726` — `if chunk.usage: Content.from_usage(...)` — usage only arrives if the provider emits a usage chunk.
|
||||
|
||||
**External findings:**
|
||||
- **Non-streaming:** Ollama/LM Studio/llama.cpp `/v1` all return a populated `usage` object → `UsageDetails` populated. Confirmed multi-source. <https://docs.ollama.com/api/openai-compatibility>
|
||||
- **Streaming:** OpenAI semantics require `stream_options={"include_usage": true}`; whether MAF sets it automatically is **unverified** — so streamed `UsageDetails` may be `None`. → run non-streaming for the accounting path.
|
||||
- **Native `OllamaChatClient`:** maps Ollama's `prompt_eval_count`→input, `eval_count`→output; **`total_token_count` is NOT set (derive it)**. Streaming updates carry no usage. <https://docs.ollama.com/api/usage>
|
||||
- MAF usage plumbing is young: "Fix OTel usage detail attributes" landed in python-1.8.0 (already in our 1.9.0).
|
||||
|
||||
### 3. Tool-calling / structured output on local models — Confidence: high (load-bearing risk)
|
||||
|
||||
**External findings (strong multi-source pattern):**
|
||||
- Small CPU-runnable models are markedly weak at tool-calling: 7–8B tool-selection F1 ≈ 0.48–0.57; failure modes = never calling the tool (answers from memory), emitting the tool call as **plain text/JSON** instead of a structured `tool_calls` object, and argument type drift (`"5"` vs `5`). <https://www.docker.com/blog/local-llm-tool-calling-a-practical-evaluation/>
|
||||
- Qwen3-class is the most reliable small family (F1 ≈ 0.93 in the same eval); MAF docs explicitly bless `qwen3:4b` / `llama3.2` for tools and warn "not all models support function calling." `num_ctx ≥ 32k` improves reliability.
|
||||
- Streaming + tools on `/v1` historically silently dropped/leaked tool calls (5+ repos; fixed 2025-05-28 but a 0.12.3 regression remains) — **another reason to run non-streaming.** <https://ollama.com/blog/streaming-tool>
|
||||
- Structured output: `response_format: type[BaseModel]` is available on both local-path clients, but reliability is model-dependent.
|
||||
- MAF bug #1772: `OpenAIChatClient` + Ollama + ChatContext middleware inserts the system prompt twice — verify against 1.9.0; the native client may avoid it. <https://github.com/microsoft/agent-framework/issues/1772>
|
||||
|
||||
**Design fit:** our **blocking Pydantic validator becomes the mitigation** — it must retry/repair on parse failure, turning model unreliability into a bounded retry loop rather than silent corruption.
|
||||
|
||||
### 4. Intel-Mac CPU performance reality — Confidence: medium-high
|
||||
|
||||
**External findings:**
|
||||
- Intel Macs run Ollama **CPU-only** (no Metal/MLX — Apple-Silicon-only). Reported ~4–6 tok/s on a 7B model (i9 2019); 3–8 tok/s CPU-only generally — "usable for testing, painful for actual work." <https://localaimaster.com/blog/ollama-system-requirements>
|
||||
- A maker-checker debate (N agents × M rounds × tool round-trips × validator retries) at single-digit tok/s runs into minutes per candidate.
|
||||
|
||||
**Conclusion:** Intel-Mac local is a **plumbing-verification + smoke-test** environment (does the slice wire up, do tool calls fire, does the validator parse, does usage come back?), not an iterative/batch run environment — exactly matching D6 ("develop on local, verify minimally on Foundry"). Keep local runs tiny: 1 maker + 1 checker, ≤3 rounds, tiny synthetic data, hard token caps, a 3–4B model.
|
||||
|
||||
### 5. Determinism boundary — Confidence: high
|
||||
|
||||
- Local LLM output is non-deterministic even at temperature 0 / fixed seed (documented edge cases). This does **not** conflict with the success criterion **iff** the determinism criterion is scoped to the **validator + Monte-Carlo** (fixed seed + fixed candidate input → identical accept/reject + identical P10/P50/P90), NOT to the LLM that produces the candidate.
|
||||
- **Trap to avoid:** never assert end-to-end determinism *through* the LLM. Golden tests pin the validator's *inputs*, not the LLM's *outputs*. Our architecture already separates "agents propose (non-deterministic) → validator decides (deterministic)", so the boundary is sound — make it explicit in the test design.
|
||||
|
||||
### 6. Security / no-egress posture — Confidence: high
|
||||
|
||||
**External findings:**
|
||||
- `OpenAIChatCompletionClient(base_url=localhost)` keeps inference on-box; the OpenAI SDK sends only to `base_url` and never contacts `api.openai.com`; a dummy `api_key` is safe. **No new dependency** (uses already-installed `agent-framework-openai`). This is the lower-supply-chain-cost path.
|
||||
- Native `agent-framework-ollama` is official MS but **beta** and adds the `ollama` client dep (pinned `<0.5.4`) — a supply-chain + GA-discipline cost.
|
||||
- Ollama daemon: bind **`127.0.0.1`** (never `0.0.0.0` — it has NO auth by design, CVE-2025-63389); pin Ollama **≥ 0.17.1** (clears CVE-2024-37032 "Probllama" RCE, the 39719-22 set, and CVE-2026-7482 "Bleeding Llama" GGUF heap-read); keep `OLLAMA_DEBUG` unset (else prompts are written to `server.log`).
|
||||
- **Model-pull is explicit egress** (downloads from a registry) — document it in provenance; not "silent" but real. Air-gap option: pre-stage model blobs offline. LM Studio persists chats/RAG locally by default (treat as data-at-rest).
|
||||
- No native telemetry/phone-home from MAF client (instrumentation off by default), Ollama (no inference telemetry), or LM Studio (off by default).
|
||||
|
||||
## External Knowledge
|
||||
|
||||
### Best Practice
|
||||
- Local wiring: `OpenAIChatCompletionClient(base_url="http://127.0.0.1:11434/v1/", api_key="ollama", model_id="qwen3:4b")` (non-streaming); or native `OllamaChatClient(host=..., model_id=...)` after `--pre` install. Cost: no native cost metric — derive `cost = tokens × per-model price` (local = 0); token usage on OTel `gen_ai.client.token.usage`.
|
||||
|
||||
### Known Issues
|
||||
- Streaming+tools `/v1` drop/leak (run non-streaming); `#1772` double-system-prompt with OpenAIChatClient+Ollama+middleware (verify vs 1.9.0); native client `total_token_count` not set (derive); `prompt_eval_count` historically flaky for cached/large prompts. **No community report exists of MAF *Python* against a *local* model on an *Intel Mac* with token usage — this scenario is under-documented; verify empirically.**
|
||||
|
||||
## Gemini Second Opinion
|
||||
|
||||
Unavailable (same MCP-SDK/Interactions-API failure as Topics 1–2). No independent
|
||||
Gemini triangulation; treat as absent, not negative.
|
||||
|
||||
## Synthesis
|
||||
|
||||
The brief's premise — "run locally via `OpenAIChatClient(base_url=...)`; `UsageDetails`
|
||||
works because the local server returns OpenAI-style `usage`" — is *directionally
|
||||
right but wrong in two specifics that would fail success criteria on the very
|
||||
profile we develop on*. First, the **Chat Completions** client
|
||||
(`OpenAIChatCompletionClient`), not the **Responses** client (`OpenAIChatClient`),
|
||||
is the one that talks to a local `/v1` endpoint and the one whose verified
|
||||
`_parse_usage_from_openai` populates `UsageDetails`. Second, **streaming is the
|
||||
trap**: it is exactly where local `/v1` silently drops/leaks tool calls AND where
|
||||
`usage` comes back `None` unless `include_usage` is set (uncertain in MAF). Running
|
||||
the debate/validator path **non-streaming** fixes both at once — and non-streaming
|
||||
is what we want anyway for deterministic accounting.
|
||||
|
||||
The contrarian's "use the native `OllamaChatClient` instead" is a real option (MS
|
||||
docs route you to it; it sidesteps `/v1` translation edges and gets
|
||||
`prompt_eval_count`/`eval_count` directly), but it conflicts with the project's
|
||||
**GA-pin dependency discipline** (it is a `--pre` beta + a new `ollama` dep). The
|
||||
resolution is a **spike-gate**: start with the zero-new-dep Chat-Completions
|
||||
`base_url` path non-streaming; if a quick spike shows tool-calls leaking as text, or
|
||||
`usage=None`, or the #1772 middleware bug biting, switch to the native client and
|
||||
accept the scoped `--pre` dependency. Either way the model-map → chat-client ctor
|
||||
abstraction (D2) hides the choice behind the profile.
|
||||
|
||||
The deeper truth the swarm surfaces: on Intel-Mac CPU, "local primary" means
|
||||
**local proves the slice RUNS** (plumbing, tool-calls fire, validator parses, usage
|
||||
populated) — it does **not** prove the agents produce *good* candidates. "Both
|
||||
profiles exercised" therefore means: local = it runs; one minimal Foundry pass
|
||||
(cheapest model, hard cap) = the candidates are substantively sound. And the model's
|
||||
tool-calling weakness is absorbed by our **blocking validator-as-retry** — the
|
||||
obligatory validator stops being just a gate and becomes the reliability mechanism
|
||||
that makes a weak local model usable.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **Concrete local model + endpoint** (brief `[OPEN]`, operator-supplied): default
|
||||
to `qwen3:4b` via Ollama on `127.0.0.1:11434` for dev/smoke; `qwen2.5:7b`/`qwen3:8b`
|
||||
for reliability checks. The whole free-local run hinges on a working local endpoint
|
||||
existing on the machine.
|
||||
- **Foundry deployment names** (brief `[OPEN]`, tenant-specific, operator-supplied) —
|
||||
back the role→deployment map for the minimal Azure/Foundry-profile check.
|
||||
- **base_url vs native client** — resolve via a one-call spike in /trekexecute:
|
||||
does `OpenAIChatCompletionClient(base_url)` non-streaming return populated
|
||||
`UsageDetails` and structured tool-calls on the chosen model? If yes, no new dep;
|
||||
if no, switch to native `OllamaChatClient`.
|
||||
- **Does MAF set `include_usage` on streaming?** Unverified — irrelevant if we run
|
||||
non-streaming, but confirm if streaming is ever needed for narration.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Run the local profile via `OpenAIChatCompletionClient(base_url=<local /v1>)`,
|
||||
non-streaming, behind the D2 model-map abstraction; keep the native `OllamaChatClient`
|
||||
as a spike-gated fallback.**
|
||||
|
||||
1. **Client:** `OpenAIChatCompletionClient(base_url="http://127.0.0.1:11434/v1/",
|
||||
api_key="ollama", model_id="qwen3:4b")` — installed, **zero new deps**, honours the
|
||||
GA-pin discipline. NOT `OpenAIChatClient` (Responses API).
|
||||
2. **Non-streaming for debate + validator** — dodges `/v1` streaming+tools drop AND
|
||||
streaming `usage=None` in one move. Streaming only for non-tool narration, if ever.
|
||||
3. **Budget middleware: treat `usage_details is None` as a HARD FAIL in dev**
|
||||
(assertion) so a usage-reporting regression can never silently disable the budget
|
||||
cap. Maps directly to the "real token accounting" success criterion (assert the
|
||||
meter is populated from `UsageDetails` on a real run). Accumulate via
|
||||
`add_usage_details(...)`. No `len(...split())` proxy in `src/`.
|
||||
4. **Validator-as-retry:** the blocking Pydantic validator must retry/repair on parse
|
||||
failure — small models emit unparseable/text-leaked tool calls; this turns the
|
||||
obligatory validator into the reliability mechanism.
|
||||
5. **Model:** `qwen3:4b` for dev/smoke (MAF-blessed, tool-capable, CPU-tolerable);
|
||||
`qwen2.5:7b`/`qwen3:8b` for reliability checks; `num_ctx=32k`.
|
||||
6. **Determinism golden tests:** pin validator + Monte-Carlo inputs (fixed seed →
|
||||
identical decision/percentiles); never assert determinism through the LLM.
|
||||
7. **"Both profiles exercised":** local (full run) proves the slice RUNS; one minimal
|
||||
Foundry pass (cheapest model, hard token cap, per D6) proves candidates are
|
||||
substantively sound. Keep local runs tiny (1 maker + 1 checker, ≤3 rounds).
|
||||
8. **Spike-gate the client choice** in /trekexecute: if `base_url` non-streaming shows
|
||||
tool-call text-leakage, `usage=None`, or #1772, switch to native `OllamaChatClient`
|
||||
(accept the `--pre` `agent-framework-ollama` dep + verify its `prompt_eval_count`/
|
||||
`eval_count` mapping; derive `total_token_count`).
|
||||
9. **Security hardening:** Ollama bind `127.0.0.1`, pin **≥ 0.17.1**, `OLLAMA_DEBUG`
|
||||
unset; document model-pull as explicit egress; dummy `api_key`.
|
||||
|
||||
Risks to carry into the plan: (a) Intel-CPU latency → local is plumbing/smoke, not
|
||||
fast iteration; (b) small-model tool-calling weakness → lean on validator-retry +
|
||||
Qwen3; (c) native-client fallback is `--pre` (GA-discipline tension) — adopt only if
|
||||
spike-gated.
|
||||
|
||||
## Sources
|
||||
|
||||
| # | Source | Type | Quality | Used in |
|
||||
|---|--------|------|---------|---------|
|
||||
| 1 | `.venv/.../agent_framework_openai/__init__.py:33-43` (`OpenAIChatClient`, `OpenAIChatCompletionClient`) | codebase | high | Dim 1 |
|
||||
| 2 | `.venv/.../agent_framework_openai/_chat_completion_client.py:705,724-726,757-776` (UsageDetails mapping, None-safe) | codebase | high | Dim 2 |
|
||||
| 3 | `.venv/.../agent_framework/_types.py:402-417,547` (`UsageDetails`, `add_usage_details`, `response.usage_details`) | codebase | high | Dim 2 |
|
||||
| 4 | `.venv/.../agent_framework_openai/_chat_client.py:376,382` (`base_url`, `api_key`) | codebase | high | Dim 1 |
|
||||
| 5 | `agent_framework/ollama/__init__.pyi` + `import agent_framework_ollama` fails (not installed) | codebase | high | Dim 1, 6 |
|
||||
| 6 | <https://learn.microsoft.com/agent-framework/integrations/openai-endpoints> (base_url table, ChatCompletion client) | official | high | Dim 1, 2 |
|
||||
| 7 | <https://learn.microsoft.com/agent-framework/agents/providers/ollama> (native client, tool caveat, qwen3:4b) | official | high | Dim 1,3 |
|
||||
| 8 | <https://learn.microsoft.com/python/api/agent-framework-core/agent_framework.usagedetails?view=agent-framework-python-latest> | official | high | Dim 2 |
|
||||
| 9 | <https://pypi.org/project/agent-framework-ollama/> (1.0.0b260521 beta, --pre) | official | high | Dim 1, 6 |
|
||||
| 10 | <https://github.com/microsoft/agent-framework/blob/main/python/packages/ollama/pyproject.toml> (deps: ollama <0.5.4) | official | high | Dim 1, 6 |
|
||||
| 11 | <https://docs.ollama.com/api/openai-compatibility> (usage; streaming needs include_usage) | official | high | Dim 2 |
|
||||
| 12 | <https://docs.ollama.com/api/usage> (prompt_eval_count/eval_count) | official | high | Dim 2 |
|
||||
| 13 | <https://github.com/microsoft/agent-framework/releases> (1.8.0 OTel usage fix; 1.9.0 ollama tools fix) | official | high | Dim 2 |
|
||||
| 14 | <https://github.com/microsoft/agent-framework/issues/1772> (double system prompt, Ollama+middleware) | community | medium | Dim 3 |
|
||||
| 15 | <https://ollama.com/blog/streaming-tool> (streaming+tools fixed 2025-05-28) | official | high | Dim 3 |
|
||||
| 16 | <https://github.com/ollama/ollama/issues/12557> (streaming tool-call regression 0.12.3) | community | medium | Dim 3 |
|
||||
| 17 | <https://www.docker.com/blog/local-llm-tool-calling-a-practical-evaluation/> (tool-call F1 by model) | community | high | Dim 3 |
|
||||
| 18 | <https://www.morphllm.com/best-ollama-models> (Qwen3 best tool-calling) | community | medium | Dim 3 |
|
||||
| 19 | <https://localaimaster.com/blog/ollama-system-requirements> (Intel i9 4-6 tok/s, no MLX) | community | medium | Dim 4 |
|
||||
| 20 | <https://github.com/ollama/ollama/issues/586> (seed+temp0 not deterministic) | community | medium | Dim 5 |
|
||||
| 21 | <https://www.wiz.io/blog/probllama-ollama-vulnerability-cve-2024-37032> (CVE-2024-37032) | community | high | Dim 6 |
|
||||
| 22 | <https://github.com/advisories/GHSA-f6mr-38g8-39rg> (CVE-2025-63389 no-auth by design) | official | high | Dim 6 |
|
||||
| 23 | <https://www.runzero.com/blog/ollama/> (CVE-2026-7482 Bleeding Llama, < 0.17.1) | community | high | Dim 6 |
|
||||
| 24 | <https://www.indusface.com/blog/exposed-ollama-servers-llm-security-risks/> (bind 127.0.0.1) | community | high | Dim 6 |
|
||||
| 25 | <https://learn.microsoft.com/agent-framework/agents/observability> (no native cost metric) | official | high | Best Practice |
|
||||
| 26 | <https://lmstudio.ai/app-privacy> (LM Studio local persistence, no telemetry default) | official | medium | Dim 6 |
|
||||
|
|
@ -1,86 +0,0 @@
|
|||
---
|
||||
type: trekreview
|
||||
review_version: "1.0"
|
||||
created: 2026-06-26
|
||||
task: "Fase 2 MVP vertical slice — one synthetic project end-to-end on MAF 1.9.0: debate → block-validate → HITL → learn"
|
||||
slug: fase2-mvp-vertical-slice
|
||||
project_dir: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/
|
||||
brief_path: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md
|
||||
scope_sha_start: 6ef4efc
|
||||
scope_sha_end: 01c3f0d
|
||||
reviewed_files_count: 9
|
||||
findings: []
|
||||
---
|
||||
|
||||
# Review: Fase 2 MVP vertical slice — one synthetic project end-to-end on MAF 1.9.0: debate → block-validate → HITL → learn
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Verdict: **ALLOW** — zero findings after dedup and the HubSpot Judge filters. This is a **remediation review**: the first review of this task (scope `9973d9f..308b553`) returned **BLOCK** (2 BLOCKER + 5 MAJOR + 1 MINOR) because the slice's three load-bearing seams — debate→validator dataflow, ChatMiddleware token-budget enforcement, and the MCP-tool retrieval path — were unit-tested in isolation but never connected by the orchestrator, so the e2e suite passed on proposal *shape* only (green-but-dead). The remediation (`plan-remediation.md`, 7 steps, commits `bf0bbff..01c3f0d`) wired all three seams; both independent reviewers verified the new tests are genuinely load-bearing by the detach criterion (each fails when its seam is removed) and explicitly refused to treat the green suite as evidence. Standing items remain (F9 deferred MINOR, SC9 real-profile coverage, retrieval-exposed-not-invoked, untested `strict_usage` fail-closed) — these are named below and are NOT findings; do not read ALLOW as "nothing left to do."
|
||||
|
||||
## Coverage
|
||||
|
||||
| File | Treatment | Reason |
|
||||
|------|-----------|--------|
|
||||
| `.claude/projects/2026-06-24-fase2-mvp-vertical-slice/plan-remediation.md` [uncommitted-tracked] | summary-only | Remediation plan of record (7 steps); documentation, no executable seam. |
|
||||
| `env.template` | summary-only | Resolves F3 — documents both profiles + `127.0.0.1` + no-egress notes; config artifact. |
|
||||
| `pyproject.toml` | summary-only | Resolves F4 — records intentional PEP 735 `[dependency-groups]` deviation (`:26-31`); `agent_framework_openai` declared as core dep. |
|
||||
| `src/portfolio_optimiser/run.py` | summary-only | Orchestrator — the file where all three seams were previously unwired. F1 (`:162-169,166`), F2/F5/F8 (`:153,160`), F7 (`:154,159`) now wired; F9 (`:173`) deferred residual. |
|
||||
| `src/portfolio_optimiser/workflow.py` | summary-only | Attaches middleware + tools to every Agent (`:62-63`); `output_from=[agents[0]]` (`:105`) surfaces proposer's converged output for F1. |
|
||||
| `tests/conftest.py` | summary-only | `:27` re-bases `SyntheticUsageChatClient` onto `OpenAIChatCompletionClient` — verified NECESSARY (the prior `BaseChatClient` base silently no-op'd middleware: the exact green-but-dead trap). |
|
||||
| `tests/test_budget.py` | summary-only | `:70` real `agent.run("hi")` asserts `BudgetExceeded` + `meter.tokens==8`; load-bearing for F2/F5/F8. |
|
||||
| `tests/test_vertical_slice_e2e.py` | summary-only | `test_g` (F1 marker in `debate_output` + generation context), `test_h` (F2 — `BudgetExceeded` can only originate in debate middleware); both fail on detach. |
|
||||
| `tests/test_workflow.py` | summary-only | Construction-spy + wiring-spy tests for retrieval-tool exposure (F7) and middleware attachment. |
|
||||
|
||||
Triage map: 9 files in scope, all `summary-only`, **0 skipped**. Independent ground-truth check at HEAD (`01c3f0d`): offline suite **103 passed / 3 skipped** — the 3 skips are operator-gated live-profile (Azure) tests, not silent drops.
|
||||
|
||||
Confirmed remediation outcomes (folded from both reviewers' verified conclusions):
|
||||
|
||||
- **F1 — debate output discarded → RESOLVED.** `run.py:162-169` extracts the proposer's converged output (surfaced via `output_from=[agents[0]]`, `workflow.py:105`) and feeds it via `gen_context = debate_output or context` (`run.py:166`). `test_g` asserts the `"Reduce scope"` marker (in `_VALID`, absent from the `docs_dir` fixture) appears in both `result.debate_output` and the captured generation context — fails if either seam reverts. `output_from` verified a real `GroupChatBuilder.__init__` param (`_group_chat.py:629`).
|
||||
- **F2/F5/F8 — BudgetMiddleware never wired / no real-client test → RESOLVED.** `run.py:153,160` constructs `BudgetMiddleware(meter)` and passes `middleware=[...]` into `fresh_workflow`; `workflow.py:62-63` attaches to every Agent. `conftest.py:27` re-base verified NECESSARY against `_clients.py:214-231`. `test_budget.py:70` and `test_h` both fail on detach.
|
||||
- **F7 — retrieval tool not exposed → RESOLVED.** `run.py:154,159` builds `make_retrieval_tool`, passed via `tools=[...]`; `workflow.py:62` attaches per agent; spy tests assert it. (Exposure, not invocation-reliability — per F7's recommendation + 90% principle.)
|
||||
- **F3 — env contract → RESOLVED** (`env.template`).
|
||||
- **F4 — PEP 735 deviation unrecorded → RESOLVED** (`pyproject.toml:26-31`).
|
||||
- **Security:** exposing the retrieval tool creates no injection vector — LLM-controlled `query` reaches only in-memory keyword scoring; `docs_dir` is boundary-checked, fail-closed against `..`/symlink/prefix-collision (`retrieval.py:48-70`). New keyword-only params default to `None`; all call sites updated or default-covered; no regression.
|
||||
|
||||
## Findings (BLOCKER)
|
||||
|
||||
None.
|
||||
|
||||
## Findings (MAJOR)
|
||||
|
||||
None.
|
||||
|
||||
## Findings (MINOR)
|
||||
|
||||
None — verdict ALLOW. See Standing items for the deferred F9 MINOR (not re-raised as a finding because it is unchanged by this diff and explicitly deferred by the remediation plan).
|
||||
|
||||
## Findings (SUGGESTION)
|
||||
|
||||
None.
|
||||
|
||||
## Standing items (NOT findings)
|
||||
|
||||
These are honest residuals carried forward, not defects introduced or left unaddressed by this remediation. They do not affect the mechanical verdict and are recorded so ALLOW is not mistaken for "complete":
|
||||
|
||||
- **F9 (MINOR, deferred)** — `model = "fake-model"` keyed off `client_factory is not None` (`run.py:173`) is still present, unchanged by this diff. Explicitly deferred by the remediation plan (Handover 6 consumes only BLOCKER + MAJOR). Real but latent; deserves the planned follow-on, not closure here.
|
||||
- **SC9 real-profile coverage** — exercised in CI only via the offline synthetic client (local) plus operator-gated/skipped live arms (Azure). Unchanged from round 1, which accepted it; the brief frames concrete endpoints as operator-supplied OPEN items. Not a remediation defect.
|
||||
- **Retrieval tool exposed but not proven invoked** by agents — the synthetic client ignores `tools=`. Matches F7's recommendation + the 90% principle.
|
||||
- **`strict_usage=True`** on the budget middleware — on the gated real backend, a debate response lacking `usage_details.total_token_count` raises `UsageUnavailable` (intentional fail-closed; untested offline).
|
||||
|
||||
## Suppressed Findings
|
||||
|
||||
None — both reviewers returned zero findings; nothing was dropped by Pass 1 (dedup) or Pass 2 (HubSpot Judge filters). Pass 3 (Cloudflare reasonableness) was bypassed under high-effort mode; with an empty findings set there were no rule_keys to normalize.
|
||||
|
||||
## Remediation Summary
|
||||
|
||||
- BLOCKER: 0
|
||||
- MAJOR: 0
|
||||
- MINOR: 0
|
||||
- SUGGESTION: 0
|
||||
|
||||
Clean ALLOW resolving a prior BLOCK: all 7 actionable original findings (2 BLOCKER + 5 MAJOR) are verified RESOLVED with genuinely load-bearing tests (detach criterion confirmed by both reviewers). `/trekplan --brief review.md` will consume an empty findings set — there is no remediation work to schedule from this review. The four named Standing items above remain operator-owned follow-ons (notably the deferred F9 MINOR) and are not actionable findings.
|
||||
|
||||
```json
|
||||
{"verdict":"ALLOW","counts":{"BLOCKER":0,"MAJOR":0,"MINOR":0,"SUGGESTION":0},"findings":[]}
|
||||
```
|
||||
|
|
@ -1,202 +0,0 @@
|
|||
---
|
||||
type: trekbrief
|
||||
brief_version: "2.2"
|
||||
created: 2026-06-26
|
||||
task: "Fase 3: orkestrator fan-out over N porteføljeprosjekter med state-isolasjon + delt læring"
|
||||
slug: fase3-portfolio-fanout
|
||||
project_dir: .claude/projects/2026-06-26-fase3-portfolio-fanout/
|
||||
research_topics: 0
|
||||
research_status: skipped
|
||||
auto_research: false
|
||||
interview_turns: 9
|
||||
source: interview
|
||||
framing: refine
|
||||
phase_signals:
|
||||
- phase: research
|
||||
effort: standard
|
||||
- phase: plan
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: execute
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: review
|
||||
effort: high
|
||||
model: opus
|
||||
---
|
||||
|
||||
# Task: Fase 3 — orkestrator fan-out over N porteføljeprosjekter
|
||||
|
||||
> Generated by `/trekbrief` on 2026-06-26.
|
||||
> This brief is the contract between requirements and planning. `/trekplan`
|
||||
> reads it to produce the implementation plan. Every decision in the plan must
|
||||
> trace back to content in this brief.
|
||||
|
||||
## TL;DR
|
||||
|
||||
Refine av den låste planen (Fase 3, §52-57): bygg en **sekvensiell fan-out-orkestrator**
|
||||
som kjører N porteføljeprosjekter gjennom eksisterende `run_project`, **isolerer
|
||||
eksekverings-state** per prosjekt men **deler én VerdictStore** (ExpeL akkumulerer over
|
||||
porteføljen). Bevis at et nytt syntetisk prosjekt kjører ende-til-ende **kun via konfig**
|
||||
(null kjerne-endring), med **load-bearing** isolasjons-test + extension-dok.
|
||||
|
||||
## Intent
|
||||
|
||||
Rammeverket beviser i dag metoden på ÉTT prosjekt (Fase 2 vertikal skive,
|
||||
`run_project`). Kjerneløftet — en *generisk* porteføljeoptimerer — er ubevist
|
||||
helt til rammeverket kjører over N uavhengige prosjekter uten kjerne-kodeendring
|
||||
per prosjekt. Fase 3 generaliserer: en orkestrator fan-out'er over en konfigurert
|
||||
portefølje, isolerer hvert prosjekts eksekverings-state (budsjett-meter, debatt,
|
||||
retrieval-kontekst) så det er null state-bleed, samtidig som den tråder én delt
|
||||
VerdictStore så læringsløkken (ExpeL) komponerer over porteføljen — en dom på ett
|
||||
prosjekt informerer like forslag på andre. Å bevise at et splitter nytt syntetisk
|
||||
prosjekt slippes inn kun via konfig er den konkrete falsifiseringen av
|
||||
genericitets-påstanden (D4/D5). Dette låser opp Fase 4 (open-source) og D7
|
||||
Claude-SDK-søsken, som begge avhenger av en genuint generisk, konfig-drevet kjerne.
|
||||
|
||||
## Goal
|
||||
|
||||
En `run_portfolio`-orkestrator (navn avgjøres i plan) eksisterer som tar en
|
||||
konfigurert liste prosjekt-id-er + én delt VerdictStore, kjører hvert gjennom
|
||||
`run_project` **sekvensielt** med isolert eksekverings-state, og returnerer en
|
||||
ordnet liste av per-prosjekt `RunResult` pluss et **tynt aggregat** (antall
|
||||
validert/avvist, sum claimed saving, sum token-forbruk). Et nytt syntetisk
|
||||
referanse-prosjekt lagt til **kun** i `reference_projects.json` (+ dets
|
||||
docs-fixture) kjører gjennom porteføljen uten endring i noen kjerne-modul.
|
||||
Referanse-domenet er pakket som dokumentert eksempel, med extension-point-dok for
|
||||
«legg til egen metode / egen datakilde / egen modell-map».
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **Concurrent/parallell fan-out** (`asyncio.gather` / `ConcurrentBuilder`) — kun
|
||||
sekvensiell; concurrency er et dokumentert extension point (90%-prinsipp).
|
||||
- **Dynamisk/runtime prosjekt-topologi** — kun fast konfig-liste.
|
||||
- **Rik porteflje-rapport** (HTML/markdown-dashboards) — kun tynt aggregat-struct.
|
||||
- **Per-prosjekt isolert VerdictStore** — vi deler bevisst én store (kryss-prosjekt-læring).
|
||||
- **Live Azure/Foundry-kjøring som gate** — offline syntetisk forblir default; ekte-profil-arm
|
||||
forblir gated/skipped (Standing item SC9, uendret).
|
||||
- **Compliance-funksjoner** (D3 — uendret invariant; deployer eier DPIA/ROS).
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Ingen kjerne-modul-kodeendring for å legge til et nytt prosjekt** (det bærende
|
||||
genericitets-kriteriet). Å legge til et prosjekt = JSON-konfig + docs-fixture, intet annet.
|
||||
- **Per-prosjekt `docs_dir` MÅ være konfig-drevet** (et felt i prosjekt-konfigen,
|
||||
resolvet av loaderen) — IKKE en hardkodet id→sti-mapping i orkestrator-kode. Ellers brytes
|
||||
SC1 for det NESTE prosjektet. (Selve mekanismen å støtte per-prosjekt-`docs_dir` er en
|
||||
engangs kjerne-endring under Fase 3-bygget; det er den marginale kostnaden ved å legge til
|
||||
prosjekt N+1 *etterpå* SC1 måler.)
|
||||
- **Kostnadsdisiplin (D6):** utvikle/verifiser på lokal/offline syntetisk profil; ingen tunge live-kjøringer.
|
||||
- **Deterministisk validator forblir obligatorisk + blokkerende** (uendret).
|
||||
- **MAF 1.9.0; `uv`; Python ≥3.10.** Eksisterende `run_project`-signatur er per-prosjekt-sømmen
|
||||
(gjenbruk — IKKE fork/dupliser logikken).
|
||||
- **STATE.md local-only** (uendret).
|
||||
|
||||
## Preferences
|
||||
|
||||
- **Gjenbruk `run_project` som-det-er** (det eksponerer allerede `store=` og
|
||||
`client_factory=`-sømmene) — orkestratoren *komponerer* det, dupliserer ikke logikken.
|
||||
- Match eksisterende modul-stil: `frozen` dataclasses for resultat-typer, type hints overalt,
|
||||
ruff/mypy-rene.
|
||||
- Tynt aggregat som frozen dataclass (f.eks. `PortfolioResult`) som speiler `RunResult`-stilen.
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
- Suite forblir grønn offline (syntetisk), ingen ny live-avhengighet.
|
||||
- **Isolasjons-testen MÅ være load-bearing:** den FALLER hvis eksekverings-state
|
||||
bleed-er mellom prosjekter (f.eks. delt meter) — ikke green-but-dead (Fase 2-lærdommen,
|
||||
[[load-bearing-tests-quality]]).
|
||||
- `uv run mypy src` ren; `uv run ruff check .` ren.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- **SC1 — Nytt prosjekt kun via konfig:** et nytt syntetisk prosjekt lagt til KUN i
|
||||
`reference_projects.json` (+ docs-fixture, inkl. dets konfig-drevne `docs_dir`) kjører
|
||||
gjennom orkestratoren og produserer et `RunResult` for det. Bevis: `git diff --name-only`
|
||||
for tilføyelsen viser KUN endringer i `*.json` + docs-fixture + tester — null
|
||||
`src/portfolio_optimiser/*.py`-kjernemodul.
|
||||
- **SC2 — Fan-out over ≥2 prosjekter:** orkestratoren returnerer ett `RunResult` per
|
||||
prosjekt + ett aggregat. Falsifiserbart: `uv run pytest tests/test_portfolio.py` exit 0,
|
||||
asserterer `len(result.runs) == N` og at aggregatet summerer korrekt.
|
||||
- **SC3 — State-isolasjon er load-bearing:** en test beviser at per-prosjekt
|
||||
eksekverings-state (budsjett-meter) IKKE bleed-er; testen FALLER hvis orkestratoren
|
||||
endres til å dele én meter over alle prosjekter (detach-bevis). Foretrukket: enkod
|
||||
detach-en som en automatisk guard (inject en delt meter i orkestratoren, assert at
|
||||
isolasjons-testen da blir rød) framfor kun en manuell reviewer-revert — manuelle
|
||||
detach-bevis hoppes over under tidspress (Fase 2-feilmodus).
|
||||
- **SC4 — Delt VerdictStore akkumulerer:** etter fan-out inneholder den delte storen
|
||||
én verdict per prosjekt, og et senere prosjekts ExpeL-retrieval kan overflate en
|
||||
tidligere prosjekt-dom for et likt forslag. Assert: `store.verdicts` har lengde N,
|
||||
og `retrieved` er ikke-tom for en matchende feature i et senere prosjekt. (Retrieval keyer
|
||||
på `_features_of(proposal)` (`run.py:185`) — se Open Questions for fixture-kravet.)
|
||||
- **SC5 — Full suite grønn offline, ingen regresjon:** `uv run pytest` → alle nye tester
|
||||
passerer OG dagens grønne baseline-antall holder (current + new), de 3 eksisterende skip
|
||||
bevart, null nye failures; `uv run mypy src` ren; `uv run ruff check .` ren. (Baseline-tallet
|
||||
hentes fra faktisk suite ved plan/execute-start — IKKE hardkodet gjetning.)
|
||||
- **SC6 — Extension-dok eksisterer:** en doc-seksjon «legg til eget prosjekt / egen
|
||||
datakilde / egen modell-map» som peker på konfig-sømmene; verifiserbart ved at fila finnes
|
||||
og refererer `reference_projects.json` + modell-map.
|
||||
- **SC7 — Begge profiler eksersert (offline):** en parametrisert test kjører orkestratoren
|
||||
under `profile=local` OG `profile=azure` med den syntetiske `client_factory`, og asserterer
|
||||
at begge produserer et `RunResult` (kontrakt-stien faktisk kjørt, ikke bare backend
|
||||
instansiert). Ekte Azure-arm forblir `@pytest.mark.skip` (SC9 uendret).
|
||||
|
||||
## Research Plan
|
||||
|
||||
No external research needed — the codebase and this brief contain sufficient
|
||||
context for planning. MAF-concurrency-mekanikk ble spiket i Fase 1 (Spike B) og
|
||||
concurrency er bevisst utenfor scope (sekvensiell valgt); `run_project` eksponerer
|
||||
allerede `store=` og `client_factory=`-sømmene planen trenger.
|
||||
|
||||
## Open Questions / Assumptions
|
||||
|
||||
- **[ASSUMPTION]** `run_project`s eksisterende `store=`-param tråder en delt VerdictStore
|
||||
korrekt over sekvensielle kall (den `add`-er per kall; retrieval leser tidligere). Verifiser
|
||||
i plan mot `run.py:184-193`.
|
||||
- **[ASSUMPTION]** Hvert `run_project`-kall bygger en fersk meter + fresh_workflow internt
|
||||
(`run.py:151-161`), så eksekverings-isolasjon er i stor grad iboende ved sekvensiell gjenbruk —
|
||||
orkestratoren må unngå å introdusere delt muterbar state. Verifiser at meteren konstrueres per
|
||||
kall (`run.py:151`).
|
||||
- **[OPEN] docs_dir per prosjekt:** `run_project` tar én `docs_dir`. En portefølje av N
|
||||
uavhengige prosjekter trenger naturlig hver sin docs-kilde. Løsningen er bundet av Constraints:
|
||||
per-prosjekt-`docs_dir` MÅ være et konfig-felt (resolvet av loaderen), ikke en kode-mapping —
|
||||
ellers brytes SC1. Plan ferdigstiller felt-navn + hvor det bor (i `reference_projects.json`
|
||||
per prosjekt vs en egen portefølje-konfig).
|
||||
- **[OPEN] Aggregat-felter:** eksakt sett (antall validert/avvist, sum claimed saving, sum
|
||||
tokens) ferdigstilles i plan.
|
||||
- **[OPEN] SC4 fixture-design:** for at et senere prosjekts retrieval skal overflate et
|
||||
tidligere prosjekts dom, må de to syntetiske prosjektene være laget så den syntetiske klienten
|
||||
gir OVERLAPPENDE features (retrieval keyer på `_features_of`: affected codes, measure_type,
|
||||
claimed saving). Plan må designe fixturene for et reelt feature-match, ikke oppdage gapet ved test-tid.
|
||||
- **[ASSUMPTION/PROVENANCE] Nytt syntetisk prosjekt er AI-forfattet domene-innhold.** SC1s nye
|
||||
prosjekt (kostnadskoder, enheter, enhetspriser) er AI-foreslått; operatøren er ikke domene-ekspert
|
||||
([[user-not-domain-expert]]) og kan ikke fange domene-feil. Planen/fixturen markerer dataen som
|
||||
AI-forslag / «ikke verifisert», konsistent med eksisterende `_note` i `reference_projects.json`.
|
||||
|
||||
## Prior Attempts
|
||||
|
||||
Ingen tidligere fan-out-forsøk. Fase 1 Spike B (`tests/spikes/test_b_footguns.py`, throwaway)
|
||||
testet `ConcurrentBuilder` state-isolasjon — informerer isolasjons-bekymringen, men brukte en
|
||||
ANNEN (concurrent) mekanisme enn det sekvensielle valget her. Lærdom fra Fase 2:
|
||||
load-bearing-tester må FALLE når sømmen detach-es ([[load-bearing-tests-quality]]).
|
||||
|
||||
## Metadata
|
||||
|
||||
- **Created:** 2026-06-26
|
||||
- **Interview turns:** 9
|
||||
- **Auto-research opted in:** no
|
||||
- **Source:** trekbrief interview
|
||||
|
||||
---
|
||||
|
||||
## How to continue
|
||||
|
||||
Manual (default) — 0 research-tema, så rett til plan:
|
||||
|
||||
```bash
|
||||
# Ingen /trekresearch (0 tema).
|
||||
/trekplan --project .claude/projects/2026-06-26-fase3-portfolio-fanout
|
||||
|
||||
# Deretter:
|
||||
/trekexecute --project .claude/projects/2026-06-26-fase3-portfolio-fanout
|
||||
```
|
||||
|
|
@ -1,260 +0,0 @@
|
|||
---
|
||||
source_findings:
|
||||
- b3abbdd7035b5463b3bb9e23f83a158d678cb4f1
|
||||
- a2376be90affdffea4c9b4ef83014d0a30243e23
|
||||
---
|
||||
|
||||
# Fase 3 remediation — provenance model leak + portfolio rejection-arm coverage
|
||||
|
||||
> **Plan quality: A** (93/100) — APPROVE
|
||||
>
|
||||
> Generated by trekplan v5.6.1 on 2026-06-26 — `plan_version: 1.7`
|
||||
>
|
||||
> Destination note: written as `plan-remediation.md` (not `plan.md`) to preserve
|
||||
> the existing Fase 3 delivery plan in this project dir.
|
||||
>
|
||||
> Revised once after adversarial review (see Revisions): F1 narrowed to the
|
||||
> injected branch only; minor specification + headless fixes folded in.
|
||||
|
||||
## Context
|
||||
|
||||
`/trekreview` closed Fase 3 on **WARN** with two MAJOR findings, both in
|
||||
`src/portfolio_optimiser/run.py`. This plan remediates exactly those two,
|
||||
nothing more:
|
||||
|
||||
- **F1 (`b3abbdd…`, PLACEHOLDER_IN_CODE, run.py:197)** — `model = "fake-model"
|
||||
if client_factory is not None else resolve_model(...)` stamps the literal
|
||||
`"fake-model"` into `ProvenanceStamp.model` for **any** deployer who injects a
|
||||
custom `client_factory` (a public seam threaded through `run_portfolio(...,
|
||||
client_factory=...)`). This falsifies a load-bearing provenance invariant —
|
||||
one of the few hard technical guarantees the framework actually owns — on the
|
||||
exact path an external MAF user hits. Confirmed against ground truth.
|
||||
- **F2 (`a2376be9…`, MISSING_TEST, run.py:240-241)** — `_aggregate` computes
|
||||
`rejected_count=len(rejected)` and a validated-only `sum_claimed_saving_nok`,
|
||||
but no `run_portfolio` test ever produces a `Rejection` (`test_portfolio.py:64`
|
||||
asserts `rejected_count==0`). The rejection partition and the validated-only
|
||||
filter ship green-but-unproven under SC2.
|
||||
|
||||
Intended outcome: provenance reflects the real proposer model on the injected
|
||||
seam (never a fabricated id), and the portfolio aggregate's rejection arm is
|
||||
exercised by a deterministic rejection. Each finding's `id` is carried in
|
||||
`source_findings` above for the audit trail back to `review.md`.
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Fase 3 remediation"
|
||||
F[injected client_factory] -->|"factory('proposer')"| PC[proposer_client]
|
||||
PC -->|generate_via_llm| OUT[outcome: Validated/Rejection]
|
||||
PC -->|"F1: injected branch -> getattr(.model) or 'unknown'"| PS[ProvenanceStamp.model]
|
||||
DEF[default path] -->|"unchanged: resolve_model(profile)"| PS
|
||||
OUT -->|_aggregate partition| AGG[PortfolioResult]
|
||||
AGG -->|"F2 test: 1 rejected"| RC[rejected_count / validated-only sum]
|
||||
end
|
||||
style PS fill:#cfe
|
||||
style RC fill:#fec
|
||||
```
|
||||
|
||||
## Codebase Analysis
|
||||
|
||||
- **Tech stack:** Python ≥3.10, Microsoft Agent Framework (`agent-framework-core`
|
||||
1.9.0), Pydantic IR, `uv` / `pytest` / `mypy` / `ruff`. Repo-wide `find` sweep =
|
||||
82 files; `mypy src` type-checks 14 source modules (the two counts measure
|
||||
different scopes — repo vs `src/`).
|
||||
- **Key patterns:** vertical-slice orchestrator (`run_project` → `run_portfolio`),
|
||||
frozen result dataclasses, `client_factory` test-injection seam, deterministic
|
||||
blocking validator, synthetic-usage fake clients in `conftest.py`.
|
||||
- **Relevant files:**
|
||||
- `src/portfolio_optimiser/run.py:176` — `factory = client_factory or _default_factory(profile)`
|
||||
- `src/portfolio_optimiser/run.py:193` — proposer client currently inlined into `generate_via_llm`
|
||||
- `src/portfolio_optimiser/run.py:197` — the `fake-model` leak (F1 site)
|
||||
- `src/portfolio_optimiser/run.py:231-243` — `_aggregate` (F2 target, correct as-is)
|
||||
- `tests/test_vertical_slice_e2e.py:28` — `run_project` call shape + `_VALID`/`_VI` fixtures
|
||||
- `tests/test_portfolio.py:28-45` — `REPLIES` + `_PORTFOLIO_IDS` (portfolio fixtures)
|
||||
- `tests/conftest.py:84-93,144-163` — `make_client_factory` / `make_portfolio_client_factory`
|
||||
- **Reusable code:**
|
||||
- `_default_factory` (run.py:126-130) — the default (non-injected) path is left exactly as-is on `resolve_model(profile, "proposer")`; only the injected branch changes.
|
||||
- `SyntheticUsageChatClient` (conftest.py:46) — injected fakes carry `.model == "synthetic"`, giving the F1 test a concrete non-fake value to assert.
|
||||
- `_OUT_OF_RANGE` magnitude `800000` (test_vertical_slice_e2e.py:23) — already proven to reject FV42 (claim 800000 > P90 444750, ≤ Σ 1,482,500), reused for the portfolio rejection.
|
||||
- **External tech (researched):** none — fully internal change.
|
||||
- **Recent git activity:** `4253dd6` (review.md committed), `497399e..207f057` Fase 3 delivery (10 commits). Suite green at 114 passed / 4 skipped, mypy + ruff clean (observed this session).
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Step 1: Stamp the real proposer model on the injected seam
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/run.py`, `tests/test_vertical_slice_e2e.py`
|
||||
- **Changes:** In `run_project`, capture the proposer client once instead of
|
||||
inlining it: replace `outcome = await generate_via_llm(factory("proposer"),
|
||||
project, gen_context, meter)` (run.py:193) with two lines — `proposer_client =
|
||||
factory("proposer")` then `outcome = await generate_via_llm(proposer_client,
|
||||
project, gen_context, meter)`. Then change ONLY the injected arm of run.py:197:
|
||||
keep the ternary on `client_factory is not None`, leave the default arm as
|
||||
`resolve_model(profile, "proposer")` untouched, and replace the injected arm's
|
||||
`"fake-model"` literal with `getattr(proposer_client, "model", None) or
|
||||
"unknown"` — i.e. `model = (getattr(proposer_client, "model", None) or
|
||||
"unknown") if client_factory is not None else resolve_model(profile,
|
||||
"proposer")`. Add a short inline comment: the injected branch stamps the
|
||||
injected client's real model (`"unknown"` is the neutral fallback for a client
|
||||
that does not surface one — never a fabricated name); the default path keeps
|
||||
the deterministic `resolve_model`. This is the narrowest fix — the leak lived
|
||||
only on the injected branch, and the proven production path is unchanged. No
|
||||
signature change; `resolve_model` stays imported.
|
||||
- **Reuses:** `factory` local (run.py:176), `ProvenanceStamp` (provenance.py:32-39); default path keeps `resolve_model` (backends.py:54-65) verbatim.
|
||||
- **Test first:**
|
||||
- File: `tests/test_vertical_slice_e2e.py` *(existing)* — add `test_a1_provenance_stamps_injected_client_model_not_sentinel`
|
||||
- Verifies: an injected `make_client_factory(_VALID)` (SyntheticUsageChatClient, `.model == "synthetic"`) yields `result.provenance.model == "synthetic"` and `result.provenance.model != "fake-model"`. FAILS on current code (stamps `"fake-model"`); PASSES after the fix. Load-bearing: reverting run.py:197 reddens it.
|
||||
- Pattern: `tests/test_vertical_slice_e2e.py:28` `test_a_valid_proposal_end_to_end` (same `run_project` call shape, `docs_dir`/`make_client_factory`/`fresh_store` fixtures, `_VALID`/`_VI`)
|
||||
- **Verify:** `uv run pytest tests/test_vertical_slice_e2e.py -q && uv run mypy src` → expected: all e2e tests pass (incl. the new one) and `Success: no issues found in 14 source files`
|
||||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/run.py tests/test_vertical_slice_e2e.py` — then stop; do not proceed to Step 2.
|
||||
- **Checkpoint:** `git commit -m "fix(fase3): stamp real proposer model into provenance, kill fake-model leak (F1)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/run.py
|
||||
- tests/test_vertical_slice_e2e.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^fix\\(fase3\\): stamp real proposer model into provenance, kill fake-model leak \\(F1\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- tests/test_portfolio.py
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/run.py
|
||||
pattern: "proposer_client"
|
||||
- path: tests/test_vertical_slice_e2e.py
|
||||
pattern: "test_a1_provenance_stamps_injected_client_model_not_sentinel"
|
||||
```
|
||||
|
||||
### Step 2: Exercise the portfolio rejection arm and the portfolio provenance seam
|
||||
|
||||
- **Files:** `tests/test_portfolio.py`
|
||||
- **Depends on:** Step 1 (the portfolio-seam model assertion below expects the F1 fix in place).
|
||||
- **Changes:** Add `from portfolio_optimiser.validator import Rejection` to the
|
||||
imports. Add `test_a3_rejected_proposal_excluded_from_aggregate`: build
|
||||
`replies = {**REPLIES, "FV42-GSV-E1": REPLIES["FV42-GSV-E1"].replace("200000",
|
||||
"800000")}` so FV42's claim (800000) exceeds its P90 (444750) but stays ≤ its
|
||||
affected-items total (1,482,500) — it constructs cleanly and the validator
|
||||
rejects it, while RV13 and BRU still validate. Drive `run_portfolio(_PORTFOLIO_IDS,
|
||||
"local", store=fresh_store, client_factory=make_portfolio_client_factory(replies))`
|
||||
and assert: `rejected_count == 1`, `validated_count == 2`,
|
||||
`sum_claimed_saving_nok == 340000` (130000 + 210000, EXCLUDING the rejected
|
||||
800000). Locate the rejection explicitly — `rejected = [r for r in result.runs
|
||||
if isinstance(r.outcome, Rejection)]; assert len(rejected) == 1` — then assert
|
||||
`rejected[0].outcome.proposal.claimed_saving_nok == 800000` (carried but not
|
||||
summed) and `rejected[0].provenance.validator_decision == "rejected"`. Add the
|
||||
portfolio-seam F1 cross-check on the same result: `assert all(r.provenance.model
|
||||
== "synthetic" for r in result.runs)` — proves the injected model is stamped
|
||||
across **all N** records via `run_portfolio` (the seam finding `b3abbdd` names),
|
||||
not just the single-project `run_project`. Finally, correct the now-stale
|
||||
docstring in `test_d_both_profiles` (the `run.py:152,173 model="fake-model"`
|
||||
parenthetical) to read that under an injected `client_factory` the stamp
|
||||
mirrors the injected client's own model (run.py:197), with `resolve_model`
|
||||
remaining the offline profile-dependent seam that test exercises.
|
||||
- **Reuses:** `make_portfolio_client_factory` (conftest.py:144-163), `fresh_store`
|
||||
(conftest.py:166-168), `REPLIES` / `_PORTFOLIO_IDS` (test_portfolio.py:28-45),
|
||||
the proven `800000` rejection magnitude (test_vertical_slice_e2e.py:23).
|
||||
Exercises `_aggregate` (run.py:235/240/241) as-is — no `src` change.
|
||||
- **Test first:**
|
||||
- File: `tests/test_portfolio.py` *(existing)* — this step IS the test
|
||||
- Verifies: the rejection partition + validated-only sum + portfolio-seam model. Load-bearing — reverting run.py:241's `for r in validated` filter to sum over all runs makes the sum 1,140,000 and reddens it; reverting the Step-1 fix reddens the `provenance.model == "synthetic"` assertion.
|
||||
- Pattern: `tests/test_portfolio.py:48` `test_a_fanout_returns_one_runresult_per_project` (same `run_portfolio` call shape + fixtures)
|
||||
- **Verify:** `uv run pytest tests/test_portfolio.py -q && uv run pytest -q` → expected: portfolio tests pass; full suite `116 passed, 4 skipped` (the observed 114 baseline + 2 new tests; equivalently: +2 passed, zero new failures/skips)
|
||||
- **On failure:** revert — `git checkout -- tests/test_portfolio.py` — then stop and report.
|
||||
- **Checkpoint:** `git commit -m "test(fase3): portfolio rejection arm + provenance-seam assertions (F2 + F1 follow-up)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- tests/test_portfolio.py
|
||||
min_file_count: 1
|
||||
commit_message_pattern: "^test\\(fase3\\): portfolio rejection arm \\+ provenance-seam assertions \\(F2 \\+ F1 follow-up\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/run.py
|
||||
must_contain:
|
||||
- path: tests/test_portfolio.py
|
||||
pattern: "test_a3_rejected_proposal_excluded_from_aggregate"
|
||||
```
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
| Approach | Pros | Cons | Why rejected |
|
||||
|----------|------|------|--------------|
|
||||
| F1: unify both branches onto `getattr(client,"model")` | One expression, no ternary | Changes the proven default (production) path to read a third-party `.model` attribute no test guards; silent degrade to `"unknown"` if `agent_framework` moves it | Violates smallest-change; flagged MAJOR by both reviewers — narrowed to the injected branch only |
|
||||
| F1: make `client_factory` surface its model (signature change) | Explicit model contract | Breaks every injection site (both conftest fixtures, ~12 e2e + ~7 portfolio call sites, workflow.py callers) for zero extra fidelity | Largest blast radius |
|
||||
| F1: neutral sentinel `"injected"` (no client read) | One-token change | Discards the real `.model` an injected client genuinely carries — lossy | `getattr(.model)` is honest AND minimal; sentinel kept only as the model-less fallback |
|
||||
| F2: unit-test `_aggregate` directly | Smaller | Doesn't exercise the real `run_portfolio` → validator → Rejection path the finding names | Portfolio-level test is the faithful proof |
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- **Framework:** `pytest` (async, `pytest-asyncio` auto mode), deterministic synthetic clients — no LLM, no egress.
|
||||
- **Existing patterns:** `run_project`/`run_portfolio` driven by `make_client_factory`/`make_portfolio_client_factory`; the synthetic reply JSON *is* the proposal; P90 = 0.30·Σ when `assumptions` omitted.
|
||||
- **New tests in this plan:** 2 (one per step).
|
||||
|
||||
### Tests to write
|
||||
|
||||
| Type | File | Verifies | Model test |
|
||||
|------|------|----------|------------|
|
||||
| Integration | `tests/test_vertical_slice_e2e.py` | injected client's real model stamped (`"synthetic"`, not `"fake-model"`) on `run_project` | `test_a_valid_proposal_end_to_end` |
|
||||
| Integration | `tests/test_portfolio.py` | rejection partition + validated-only sum + injected model across all N `run_portfolio` records | `test_a_fanout_returns_one_runresult_per_project` |
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Priority | Risk | Location | Impact | Mitigation |
|
||||
|----------|------|----------|--------|------------|
|
||||
| Low | `SyntheticUsageChatClient.model` is not exactly `"synthetic"` | `tests/conftest.py:46` | F1 test asserts wrong value | Runtime-verified `.model == "synthetic"`; the `!= "fake-model"` assertion is the load-bearing half regardless |
|
||||
| Low | FV42 `800000` no longer rejects if fixture magnitudes change | `tests/test_portfolio.py:28` | F2 test goes green vacuously | Test asserts `rejected_count == 1` explicitly (not `>= 0`); a non-rejection reddens it |
|
||||
| Low | A custom injected client exposes its model under a non-`.model` attribute | `src/portfolio_optimiser/run.py:197` | injected-path stamp falls to `"unknown"` | Honest fallback (not a fabricated id); the default production path is untouched and keeps `resolve_model` |
|
||||
|
||||
## Assumptions
|
||||
|
||||
| # | Assumption | Why unverifiable | Impact if wrong |
|
||||
|---|-----------|-----------------|-----------------|
|
||||
| 1 | `SyntheticUsageChatClient` exposes `.model == "synthetic"` at the stamp site | Confirmed by sub-agent runtime check, not re-run here | F1 test value adjusts to the actual `.model`; fix logic unaffected |
|
||||
|
||||
## Verification
|
||||
|
||||
- [ ] `uv run pytest -q` → expected: `116 passed, 4 skipped` (114 observed baseline + 2 new; equivalently +2 passed, no new failures)
|
||||
- [ ] `uv run mypy src` → expected: `Success: no issues found in 14 source files`
|
||||
- [ ] `uv run ruff check .` → expected: `All checks passed!`
|
||||
- [ ] `grep -rn "fake-model" src/` → expected: no matches (the leak literal is gone from `src`)
|
||||
- [ ] Load-bearing check (F1): revert the run.py:197 edit, run `uv run pytest tests/test_vertical_slice_e2e.py -q` → the new test FAILS; restore the edit → it passes
|
||||
- [ ] Load-bearing check (F2): change run.py:241 to sum over all runs, run `uv run pytest tests/test_portfolio.py -q` → `test_a3` FAILS (sum 1,140,000 ≠ 340,000); restore → passes
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Files to modify:** 3 (`run.py`, `test_vertical_slice_e2e.py`, `test_portfolio.py`)
|
||||
- **Files to create:** 0
|
||||
- **Complexity:** low
|
||||
|
||||
## Plan Quality Score
|
||||
|
||||
| Dimension | Weight | Score | Notes |
|
||||
|-----------|--------|-------|-------|
|
||||
| Structural integrity | 0.15 | 92 | 2 steps, F1→F2 order correct, Step-2 dependency on Step-1 made explicit |
|
||||
| Step quality | 0.20 | 92 | TDD, concrete diffs, exact fixtures + magnitudes, explicit rejection locator |
|
||||
| Coverage completeness | 0.20 | 95 | Both findings addressed; F1 now proven on both run_project and run_portfolio seams |
|
||||
| Specification quality | 0.15 | 92 | Exact line numbers, no placeholders, all arithmetic verified by review |
|
||||
| Risk & pre-mortem | 0.15 | 90 | Narrowed fix removed the default-path coupling risk; remaining risks surfaced |
|
||||
| Headless readiness | 0.10 | 92 | On-failure revert + halt + checkpoint per step; ASCII commit messages |
|
||||
| Manifest quality | 0.05 | 92 | Checkable manifests, forbidden_paths fences, ASCII-only patterns |
|
||||
| **Weighted total** | **1.00** | **93** | **Grade: A** |
|
||||
|
||||
**Adversarial review:**
|
||||
- **Plan critic:** APPROVE_WITH_NOTES — 0 blockers, 1 major, 8 minor. Major (F1 unnecessarily changed the proven default path) ADDRESSED by narrowing the fix to the injected branch. Minors on the rejection locator, portfolio-seam coverage, em-dash commit, docstring commit-scope, file-count wording, and circuit breaker all ADDRESSED.
|
||||
- **Scope guardian:** ALIGNED — 0 creep, 0 gaps. Its lone minor (default-path `.model` untested) is dissolved by the narrowed fix (default path unchanged).
|
||||
|
||||
## Revisions
|
||||
|
||||
| # | Finding | Severity | Resolution |
|
||||
|---|---------|----------|------------|
|
||||
| 1 | F1 unified both ternary branches, changing the proven default path to an untested `.model` read | major (plan-critic + scope-guardian) | Narrowed: `getattr` only on the injected branch; default path keeps `resolve_model` verbatim |
|
||||
| 2 | F1 tested at `run_project`, not the `run_portfolio` seam the finding names | minor | Added `assert all(r.provenance.model == "synthetic")` on the portfolio result in `test_a3` |
|
||||
| 3 | `test_a3` did not specify how to locate the single Rejection | minor | Added explicit `rejected = [r ... if isinstance(r.outcome, Rejection)]; assert len(rejected) == 1` |
|
||||
| 4 | F1 docstring fallout committed under an F2-only `test()` message | minor | Step-2 commit message now names "F2 + F1 follow-up"; docstring correction scoped into it |
|
||||
| 5 | Em-dash in Step-2 commit message + `commit_message_pattern` on an em-dash-fragile machine | minor | Replaced with ASCII hyphen |
|
||||
| 6 | Verify pinned absolute `116 passed` | minor | Kept (114 baseline now observed) + added relative "+2 passed, no new failures" framing |
|
||||
| 7 | `82 source files` vs `14 source files` inconsistency | minor | Clarified the two counts measure repo vs `src/` scope |
|
||||
| 8 | On-failure clauses did not state halt vs proceed | minor | Both steps now say revert then stop / do not proceed |
|
||||
|
|
@ -1,704 +0,0 @@
|
|||
# Fase 3 — Sequential fan-out orchestrator over N portfolio projects
|
||||
|
||||
> **Plan quality: B+** (88/100) — APPROVE_WITH_NOTES
|
||||
>
|
||||
> Generated by trekplan v5.6.1 on 2026-06-26 — `plan_version: 1.7`
|
||||
> Profile: premium · phase_signal plan = effort:high, model:opus
|
||||
> Brief: `.claude/projects/2026-06-26-fase3-portfolio-fanout/brief.md`
|
||||
|
||||
## Context
|
||||
|
||||
The framework today proves the method on ONE project (`run_project`, Fase 2 vertical
|
||||
slice). The core promise — a *generic* portfolio optimiser — is unproven until the
|
||||
framework runs over N independent projects with **zero core-code change per project**.
|
||||
Fase 3 generalises: a sequential `run_portfolio` orchestrator fans out over a configured
|
||||
portfolio, **isolates each project's execution state** (budget meter, debate, retrieval
|
||||
context) so there is no state-bleed, while threading **one shared `VerdictStore`** so the
|
||||
ExpeL learning loop composes across the portfolio (a verdict on project A informs a similar
|
||||
proposal on project C). Proving a brand-new synthetic project drops in **via config only**
|
||||
is the concrete falsification of the genericity claim (D4/D5) and unlocks Fase 4
|
||||
(open-source) + D7 (Claude-SDK sibling). Motivation traces to the brief's Intent verbatim.
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Fase 3 — fan-out (NEW + changed)"
|
||||
CFG["reference_projects.json<br/>+ docs_dir + verdict_input (NEW fields)"] --> LD["load_reference_projects<br/>(extended loader)"]
|
||||
LD --> RP["run_portfolio (NEW)<br/>sequential for-loop"]
|
||||
RP -->|"per project, await"| RJ["run_project (REUSE as-is<br/>+ additive meter= seam)"]
|
||||
RP -->|"ONE instance, threaded"| VS["VerdictStore<br/>(SHARED — ExpeL accumulates)"]
|
||||
RJ -->|"fresh per call"| TM["TokenMeter<br/>(ISOLATED per project)"]
|
||||
RJ -->|"retrieve before add"| VS
|
||||
RP --> PR["PortfolioResult (NEW)<br/>runs + store + thin aggregate"]
|
||||
end
|
||||
```
|
||||
|
||||
## Codebase Analysis
|
||||
|
||||
- **Tech stack:** Python ≥3.10, Microsoft Agent Framework (`agent-framework-core` 1.9.0),
|
||||
Pydantic IR, PuLP/CBC validator, `uv`, `pytest` (`asyncio_mode=auto`), `ruff`, `mypy`.
|
||||
- **Key patterns:** layered/hexagonal composition root; every seam explicit
|
||||
(`client_factory`, `store`, `ChatBackend` protocol). Result types are `@dataclass(frozen=True)`;
|
||||
the one mutable accumulator (`VerdictStore`) is a non-frozen `@dataclass` by design.
|
||||
`from __future__ import annotations` + full type hints + keyword-only (`*,`) options + absolute
|
||||
imports in every module. Tests mirror src module names; each opens with a docstring + `Pattern:`
|
||||
pointer; lettered SC names (`test_a_…`); guard-tests grep src for anti-patterns
|
||||
(`test_budget.py:88-99`). JSON config carries a `_note` provenance/disclaimer first key.
|
||||
- **The seam to compose — `run_project`** (`src/portfolio_optimiser/run.py:114-127`): signature
|
||||
`run_project(project_id, profile=LOCAL, *, docs_dir, verdict_input, store=None,
|
||||
client_factory=None, max_rounds=3, max_tokens=100_000, top_k=3, enable_layer1_hitl=False,
|
||||
notify=None) -> RunResult`. All execution state is built **fresh per call** — `TokenMeter`
|
||||
(`run.py:151`), `fresh_workflow` (`run.py:155`), retrieval context (`run.py:141`) — so
|
||||
sequential reuse is **inherently isolated**. The ONE shared seam is `store=`: retrieve
|
||||
(`run.py:189`) happens **before** `store.add` (`run.py:193`), so project k+1 sees project k's
|
||||
verdict. `RunResult` (`run.py:57-68`, frozen): `outcome` (`ValidatedProposal | Rejection`,
|
||||
both expose `.proposal`), `provenance` (`token_usage = meter.tokens`, `run.py:179`), `verdict`,
|
||||
`retrieved`, `store`, `debate_output`.
|
||||
- **Learning store** (`verdicts.py`): `VerdictStore.add` is **idempotent on a content-hash id**
|
||||
minted over `(affected_codes, measure_type, claimed_saving_nok)` — `description` excluded
|
||||
(`_mint_id`, `verdicts.py:78-90`). `retrieve` ranks by structural `similarity`
|
||||
(codes-Jaccard 0.60 + measure-match 0.25 + magnitude-bucket 0.15) with **no similarity floor**
|
||||
(`verdicts.py:110-119`). `_features_of` (`run.py:98-104`) maps proposal → features.
|
||||
- **Project loader** (`reference_domain.py:53-75`): reads bundled `data/reference_projects.json`
|
||||
via `importlib.resources.files`. `Project` (`reference_domain.py:38-51`, frozen) has
|
||||
`id/name/description/currency/cost_items` — **no `docs_dir`, no `verdict_input`**. `docs_dir`
|
||||
is supplied per-call to `run_project` today (CLI `--docs-dir`); there is **no project→docs
|
||||
mapping** anywhere. This is the gap Step 2 closes config-side.
|
||||
- **Reusable code:** `_project_by_id` (`run.py:91`), `load_reference_projects`,
|
||||
`SyntheticUsageChatClient` + `make_client_factory`/`fresh_store`/`docs_dir` fixtures
|
||||
(`tests/conftest.py:27-114`), the two-call accumulation precedent
|
||||
(`test_vertical_slice_e2e.py:69-90`), the decoy-vs-true-match precedent (`test_verdicts.py`),
|
||||
the load-bearing fence (`test_h`, `test_vertical_slice_e2e.py:172-199`), the gated-live skip
|
||||
(`test_foundry_profile_live.py:19-24`), the src-grep guard (`test_budget.py:88-99`).
|
||||
- **No external tech.** MAF concurrency was spiked in Fase 1 and deliberately out of scope
|
||||
(sequential chosen). No research-scout needed.
|
||||
- **Baseline (verified this session):** `uv run pytest -q` → **103 passed, 3 skipped** (the 3
|
||||
skips are env-gated live arms). `mypy src` + `ruff check .` clean.
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
Steps are TDD (test-first; the project has tests). Steps 1–2 are the authorized **one-time**
|
||||
core changes (the marginal per-project cost SC1 measures *afterwards*). Steps 3+ are additive.
|
||||
Forbidden across every step: `src/portfolio_optimiser/validator.py` and `ir.py` (the
|
||||
D7-portable pure core — never imported/edited by the orchestrator).
|
||||
|
||||
### Step 1: Add additive `meter=` injection seam to run_project
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/run.py`, `tests/test_vertical_slice_e2e.py`
|
||||
- **Changes:** Add a keyword-only param `meter: TokenMeter | None = None` to `run_project`
|
||||
(after `notify`, `run.py:126`). Change the meter construction at `run.py:151` from
|
||||
`meter = TokenMeter(Budget(...))` to `meter = meter if meter is not None else
|
||||
TokenMeter(Budget(max_tokens=max_tokens, max_rounds=max(max_rounds * 4, 4)))`. Behaviour is
|
||||
**preserved** when `meter is None` (the default) — every existing call is unaffected. This is
|
||||
the injection point SC3 needs to encode the isolation detach as an automatic guard (there is
|
||||
no meter seam today; without it the detach degrades to a manual reviewer-revert, the Fase 2
|
||||
failure mode the brief forbids). **Why a seam and not Step 5's seamless per-project-baseline
|
||||
fallback:** the baseline assertion proves isolation *holds now*, but provides no automatic
|
||||
guard that reddens if a future edit makes `run_portfolio` share a meter. SC3 explicitly prefers
|
||||
the encoded detach ("inject en delt meter i orkestratoren, assert at isolasjons-testen da blir
|
||||
rød") — that requires an injection point. The param is additive, optional, and
|
||||
behaviour-preserving (`None` = today's code), consistent with the brief's authorization of
|
||||
one-time core changes for the Fase 3 build (cf. `docs_dir`).
|
||||
- **Reuses:** `TokenMeter`/`Budget` (`budget.py:40,57`), already imported at `run.py:35`.
|
||||
- **Test first:**
|
||||
- File: `tests/test_vertical_slice_e2e.py` (existing) — add `test_i_injected_meter_is_used`
|
||||
- Verifies: a `TokenMeter` pre-charged to a known value and passed as `meter=` is the meter
|
||||
used by the run — assert `result.provenance.token_usage == pre_charged + run_delta` (i.e.
|
||||
strictly greater than the same run with a fresh meter), proving the injected instance flows
|
||||
to `provenance.token_usage` (`run.py:179`). Detaches cleanly: revert the `run.py:151` change
|
||||
and the injected meter is ignored → assertion fails.
|
||||
- Pattern: `tests/test_vertical_slice_e2e.py:28-41` (`test_a`, run_project call shape)
|
||||
- **Verify:** Run `test_i_injected_meter_is_used` BEFORE the `run.py:151` change → expected:
|
||||
**red** (injected meter ignored). After the change: `uv run pytest tests/test_vertical_slice_e2e.py -q`
|
||||
→ expected: all pass incl. the new test (Iron-Law red→green gate).
|
||||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/run.py tests/test_vertical_slice_e2e.py`
|
||||
- **Checkpoint:** `git commit -m "feat(fase3): additive meter= seam on run_project (SC3 detach hook)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/run.py
|
||||
- tests/test_vertical_slice_e2e.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(fase3\\): additive meter= seam on run_project \\(SC3 detach hook\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/validator.py
|
||||
- src/portfolio_optimiser/ir.py
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/run.py
|
||||
pattern: "meter: TokenMeter \\| None = None"
|
||||
- path: tests/test_vertical_slice_e2e.py
|
||||
pattern: "test_i_injected_meter_is_used"
|
||||
```
|
||||
|
||||
### Step 2: Add config-driven docs_dir + verdict_input to the project loader
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/reference_domain.py`,
|
||||
`src/portfolio_optimiser/data/reference_projects.json`,
|
||||
`src/portfolio_optimiser/data/docs/FV42-GSV-E1/notes.txt` (new),
|
||||
`src/portfolio_optimiser/data/docs/RV13-RAS-TP/notes.txt` (new),
|
||||
`src/portfolio_optimiser/data/docs/BRU-LAKS-REHAB/notes.txt` (new),
|
||||
`tests/test_reference_domain.py`
|
||||
- **Changes:**
|
||||
1. JSON: add to **each** of the 3 existing projects a `"docs_dir"` (relative, e.g.
|
||||
`"docs/FV42-GSV-E1"`) and a `"verdict_input"` object `{"decision","rationale"}`. Mark the
|
||||
`verdict_input` values SYNTHETIC/AI-authored in the file-level `_note` (operator is not a
|
||||
domain expert — `[[user-not-domain-expert]]`; production replaces the reference domain with
|
||||
a real data source + real HITL — see Step 8 doc).
|
||||
2. `Project` dataclass (`reference_domain.py:38-51`): add fields `docs_dir: str` and
|
||||
`verdict_input: dict[str, str]`.
|
||||
3. Loader (`reference_domain.py:53-75`): read `p["docs_dir"]` and resolve it to an absolute
|
||||
filesystem path against the package data root —
|
||||
`str(files("portfolio_optimiser").joinpath("data", p["docs_dir"]))` — and read
|
||||
`p["verdict_input"]` into the dataclass. Missing keys raise `KeyError` (fail-fast, matches
|
||||
existing loader contract).
|
||||
4. Create one small synthetic `notes.txt` per project (AI-authored, flagged) whose text
|
||||
contains the **cost-saving query terms** the orchestrator searches for
|
||||
(`retrieve_chunks("cost saving measure", …)`, `run.py:141`) — e.g. a line naming a
|
||||
cost-saving measure on a cost code — so `retrieve_chunks` returns ≥1 chunk and
|
||||
`run_project` does not raise `ValueError` (`run.py:143-144`). Mirror the existing
|
||||
`docs_dir` fixture content style (`conftest.py:110-113`).
|
||||
- **Reuses:** `importlib.resources.files` pattern already at `reference_domain.py:55`; the JSON
|
||||
`_note` provenance discipline (`reference_projects.json:2`, `model_map.json:2`).
|
||||
- **Test first:**
|
||||
- File: `tests/test_reference_domain.py` (existing)
|
||||
- Verifies: every loaded `Project` has a non-empty `docs_dir` that points to an existing
|
||||
directory, and a `verdict_input` dict containing exactly `decision` + `rationale` keys.
|
||||
- Pattern: `tests/test_reference_domain.py:8-38` (loader shape assertions)
|
||||
- **Verify:** Run the extended loader assertions BEFORE the `reference_domain.py` change →
|
||||
expected: **red** (`Project` has no `docs_dir`/`verdict_input`). After: `uv run pytest
|
||||
tests/test_reference_domain.py -q` → expected: all pass (Iron-Law red→green gate).
|
||||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/reference_domain.py src/portfolio_optimiser/data/reference_projects.json tests/test_reference_domain.py` and `git clean -fd src/portfolio_optimiser/data/docs/`
|
||||
- **Checkpoint:** `git commit -m "feat(fase3): config-driven docs_dir + verdict_input on Project loader"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/reference_domain.py
|
||||
- src/portfolio_optimiser/data/reference_projects.json
|
||||
- src/portfolio_optimiser/data/docs/FV42-GSV-E1/notes.txt
|
||||
- src/portfolio_optimiser/data/docs/RV13-RAS-TP/notes.txt
|
||||
- src/portfolio_optimiser/data/docs/BRU-LAKS-REHAB/notes.txt
|
||||
- tests/test_reference_domain.py
|
||||
min_file_count: 6
|
||||
commit_message_pattern: "^feat\\(fase3\\): config-driven docs_dir \\+ verdict_input on Project loader$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/validator.py
|
||||
- src/portfolio_optimiser/ir.py
|
||||
- src/portfolio_optimiser/run.py
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/reference_domain.py
|
||||
pattern: "docs_dir: str"
|
||||
- path: src/portfolio_optimiser/reference_domain.py
|
||||
pattern: "verdict_input: dict\\[str, str\\]"
|
||||
- path: src/portfolio_optimiser/data/reference_projects.json
|
||||
pattern: "verdict_input"
|
||||
```
|
||||
|
||||
### Step 3: Create PortfolioResult + run_portfolio orchestrator (SC2 fan-out)
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/run.py`, `src/portfolio_optimiser/__init__.py`,
|
||||
`tests/conftest.py`, `tests/test_portfolio.py` (new)
|
||||
- **Changes:**
|
||||
1. `run.py`: add `@dataclass(frozen=True) class PortfolioResult` mirroring `RunResult` style
|
||||
(`run.py:57-68`): fields `runs: tuple[RunResult, ...]`, `store: VerdictStore`,
|
||||
`validated_count: int`, `rejected_count: int`, `sum_claimed_saving_nok: float`,
|
||||
`sum_token_usage: int`. Class docstring describing each field.
|
||||
2. `run.py`: add `async def run_portfolio(project_ids: Sequence[str] | None = None,
|
||||
profile: Profile | str = Profile.LOCAL, *, store: VerdictStore | None = None,
|
||||
client_factory=None, max_rounds=3, max_tokens=100_000, top_k=3,
|
||||
meter_factory: Callable[[], TokenMeter] | None = None) -> PortfolioResult`. Body: build
|
||||
`{p.id: p for p in load_reference_projects()}`; resolve `ids` (None → all loaded); create
|
||||
ONE `store` if not given; sequential `for pid in ids: await run_project(pid, profile,
|
||||
docs_dir=project.docs_dir, verdict_input=project.verdict_input, store=store,
|
||||
client_factory=client_factory, max_rounds=..., max_tokens=..., top_k=...,
|
||||
meter=meter_factory() if meter_factory else None)`; collect `RunResult`s; return
|
||||
`_aggregate(runs, store)`. Unknown id → `ValueError` (reuse `_project_by_id` semantics).
|
||||
Import `Sequence` from `collections.abc`.
|
||||
3. `run.py`: add private `_aggregate(runs, store) -> PortfolioResult` —
|
||||
`validated = [r for r in runs if isinstance(r.outcome, ValidatedProposal)]`;
|
||||
`sum_claimed = sum(r.outcome.proposal.claimed_saving_nok for r in validated)`;
|
||||
`sum_tokens = sum(r.provenance.token_usage for r in runs)`.
|
||||
4. `__init__.py`: export `PortfolioResult`, `run_portfolio` (add to imports + `__all__`).
|
||||
5. `tests/conftest.py`: add `make_portfolio_client_factory(replies: dict[str, str])` fixture →
|
||||
a project-aware `SyntheticUsageChatClient` subclass that selects its reply by scanning the
|
||||
incoming prompt for a known `project_id` substring (the prompt embeds `project.id` at
|
||||
`run.py:162` and `generate.py:48`), falling back to a default. Keeps `run_portfolio`'s
|
||||
single `client_factory` production-shaped while letting tests vary the proposal per project.
|
||||
6. **Define the shared `REPLIES` fixture constant (used by Steps 3, 4, 7)** in
|
||||
`tests/test_portfolio.py`. The synthetic reply IS the proposal — `generate._parse_ir`
|
||||
(`generate.py:60-64`) builds `affected_items` (each with its own `quantity`/`unit_cost`)
|
||||
straight from this JSON, and the validator's P90 = `0.30 × Σ(quantity·unit_cost)`
|
||||
**only when `assumptions` is empty** (degenerate Monte Carlo, `validator.py:108-113`). All
|
||||
three replies therefore OMIT `assumptions` and carry explicit magnitudes so each
|
||||
`claimed_saving_nok ≤ P90`. Exact constants (verified):
|
||||
|
||||
| project_id | affected_items (code, qty, unit_cost) | Σ item totals | P90 (0.30·Σ) | claimed | validates? |
|
||||
|---|---|---|---|---|---|
|
||||
| FV42-GSV-E1 | (05.2, 4300, 215), (03.1, 1800, 310) | 1,482,500 | 444,750 | 200,000 | yes |
|
||||
| RV13-RAS-TP (decoy) | (88.2, 180, 4200) | 756,000 | 226,800 | 130,000 | yes |
|
||||
| BRU-LAKS-REHAB | (05.2, 4300, 215), (07.4, 2400, 690) | 2,580,500 | 774,150 | 210,000 | yes |
|
||||
|
||||
`measure` strings are **byte-identical** "Reduce scope" for FV42 + BRU (measure-match is
|
||||
exact string equality, `verdicts.py:68`) and "Material substitution" for the decoy. This
|
||||
makes `validated_count==3`, `rejected_count==0`, `sum_claimed_saving_nok==540000`
|
||||
derivable IN this step (no forward dependency on Step 4), and sets up the Step-4 overlap
|
||||
(FV42 and BRU share code 05.2 + measure + magnitude bucket → similarity 0.60).
|
||||
- **Reuses:** `run_project` (`run.py:114`, composed as-is), `RunResult`/`ValidatedProposal`/
|
||||
`Rejection`, `load_reference_projects`, `SyntheticUsageChatClient` (`conftest.py:27`), the
|
||||
`__init__.py:3,7` export pattern.
|
||||
- **Test first:**
|
||||
- File: `tests/test_portfolio.py` (new) — `test_a_fanout_returns_one_runresult_per_project`
|
||||
- Verifies: `run_portfolio(["FV42-GSV-E1","RV13-RAS-TP","BRU-LAKS-REHAB"], "local",
|
||||
store=fresh_store, client_factory=make_portfolio_client_factory(REPLIES))` → `len(runs)==3`,
|
||||
each `isinstance(r, RunResult)`; aggregate pinned to the Step-3 fixture constants:
|
||||
`validated_count==3`, `rejected_count==0`, `sum_claimed_saving_nok==540000`; and
|
||||
`sum_token_usage` equals the **explicit element-wise sum**
|
||||
`runs[0]+runs[1]+runs[2]` token_usage (a real wiring check, not the field's own definition)
|
||||
with each run's `token_usage > 0`. Add `test_a2_unknown_project_id_raises`:
|
||||
`with pytest.raises(ValueError): await run_portfolio(["NOPE-DOES-NOT-EXIST"], ...)` (the
|
||||
unknown-id error path, `run.py` `_project_by_id` semantics).
|
||||
- Pattern: `tests/test_vertical_slice_e2e.py:28-41` + module docstring/`Pattern:` convention
|
||||
- **Verify:** Run the new tests BEFORE the `run.py` change → expected: **red** (ImportError /
|
||||
`run_portfolio` undefined) — the Iron-Law failing-test-first gate. Then after the change:
|
||||
`uv run pytest tests/test_portfolio.py -q` → expected: pass
|
||||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/run.py src/portfolio_optimiser/__init__.py tests/conftest.py` and `git clean -fd tests/test_portfolio.py`
|
||||
- **Checkpoint:** `git commit -m "feat(fase3): run_portfolio sequential orchestrator + PortfolioResult aggregate"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/run.py
|
||||
- src/portfolio_optimiser/__init__.py
|
||||
- tests/conftest.py
|
||||
- tests/test_portfolio.py
|
||||
min_file_count: 4
|
||||
commit_message_pattern: "^feat\\(fase3\\): run_portfolio sequential orchestrator \\+ PortfolioResult aggregate$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/validator.py
|
||||
- src/portfolio_optimiser/ir.py
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/run.py
|
||||
pattern: "async def run_portfolio"
|
||||
- path: src/portfolio_optimiser/run.py
|
||||
pattern: "class PortfolioResult"
|
||||
- path: src/portfolio_optimiser/__init__.py
|
||||
pattern: "run_portfolio"
|
||||
- path: tests/test_portfolio.py
|
||||
pattern: "test_a_fanout_returns_one_runresult_per_project"
|
||||
```
|
||||
|
||||
### Step 4: Shared VerdictStore accumulates + load-bearing retrieval (SC4)
|
||||
|
||||
- **Files:** `tests/test_portfolio.py`
|
||||
- **Changes:** Add `test_b_shared_store_accumulates_and_surfaces_prior_verdict`. Fan out over
|
||||
`["FV42-GSV-E1","RV13-RAS-TP","BRU-LAKS-REHAB"]` reusing the **Step-3 `REPLIES`** (already
|
||||
specified with explicit magnitudes + empty `assumptions` — no re-specification here). The three
|
||||
proposals are deliberately distinct-but-overlapping and all validate, minting **3 distinct ids**.
|
||||
similarity(BRU, FV42) = 0.60, decomposed as: codes-Jaccard({05.2,07.4}, {03.1,05.2}) = 1/3 →
|
||||
`0.60 × 0.333 = 0.20`, measure-match ("Reduce scope" == "Reduce scope", exact equality) →
|
||||
`0.25`, magnitude-bucket (both in `[1e5,5e5)`) → `0.15`; **total 0.60**. similarity(BRU, decoy) =
|
||||
codes 0 + measure 0 + magnitude 0.15 = **0.15**. So BRU's retrieval ranks FV42's verdict above
|
||||
the decoy.
|
||||
- **Reuses:** the two-call precedent (`test_vertical_slice_e2e.py:69-90`); the decoy/true-match
|
||||
ranking idea (`test_verdicts.py`); `_mint_id`/`similarity` semantics (`verdicts.py:65-90`).
|
||||
- **Test first:**
|
||||
- File: `tests/test_portfolio.py`
|
||||
- Verifies (load-bearing, not mere non-emptiness): after fan-out
|
||||
`len(result.store.verdicts) == 3`; the 3 verdict ids are pairwise distinct;
|
||||
`result.runs[2].retrieved[0].id == result.runs[0].verdict.id` (BRU surfaces FV42, the
|
||||
structural match, ranked above the decoy). If retrieval ranking breaks or proposals collide
|
||||
to one id, this fails — it asserts the specific match, not store-non-empty.
|
||||
- Pattern: `tests/test_verdicts.py` (true-match-beats-decoy assertion style)
|
||||
- **Verify:** `uv run pytest tests/test_portfolio.py::test_b_shared_store_accumulates_and_surfaces_prior_verdict -q` → expected: pass
|
||||
- **On failure:** revert — `git checkout -- tests/test_portfolio.py`
|
||||
- **Checkpoint:** `git commit -m "test(fase3): SC4 shared-store accumulation + load-bearing cross-project retrieval"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- tests/test_portfolio.py
|
||||
min_file_count: 1
|
||||
commit_message_pattern: "^test\\(fase3\\): SC4 shared-store accumulation \\+ load-bearing cross-project retrieval$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/validator.py
|
||||
- src/portfolio_optimiser/verdicts.py
|
||||
must_contain:
|
||||
- path: tests/test_portfolio.py
|
||||
pattern: "test_b_shared_store_accumulates_and_surfaces_prior_verdict"
|
||||
```
|
||||
|
||||
### Step 5: Load-bearing state-isolation guard (SC3, meter detach)
|
||||
|
||||
- **Files:** `tests/test_portfolio.py`
|
||||
- **Changes:** Add `test_c_execution_state_isolation_is_load_bearing`, a cap-independent
|
||||
automatic detach guard built on the Step-1 `meter=` seam + Step-3 `meter_factory`. The factory
|
||||
`f` here emits a **uniform but valid JSON `SavingsProposal`** (reuse `REPLIES["FV42-GSV-E1"]`,
|
||||
the same string for both projects) — NOT a bare default like `"ok"`: `generate_via_llm` loops
|
||||
re-fetching until `_parse_ir` succeeds, charging the meter each pass (`generate.py:106-115`), so
|
||||
an unparseable reply would run to `BudgetExceeded` instead of completing. Observable is
|
||||
per-project `RunResult.provenance.token_usage` (`= meter.tokens`, `run.py:180`):
|
||||
1. **Baseline:** `standalone = await run_project("RV13-RAS-TP", ..., client_factory=f)`.
|
||||
2. **Isolated (default):** `iso = await run_portfolio(["FV42-GSV-E1","RV13-RAS-TP"], ...,
|
||||
client_factory=f)` (uniform reply) → assert `iso.runs[1].provenance.token_usage ==
|
||||
standalone.provenance.token_usage` — project 1's usage is its OWN only, independent of
|
||||
project 0. This `==` is the load-bearing assertion: if `run_portfolio` shared a meter by
|
||||
default, runs[1] would be cumulative and this breaks.
|
||||
3. **Shared (injected):** `shared = TokenMeter(Budget(...))`; `sh = await run_portfolio([...],
|
||||
..., client_factory=f, meter_factory=lambda: shared)` → assert
|
||||
`sh.runs[1].provenance.token_usage > sh.runs[0].provenance.token_usage` — sharing
|
||||
accumulates, proving the detach is real and the `==` arm would redden under sharing.
|
||||
Both arms run every CI → the detach is encoded automatically (no manual revert).
|
||||
- **Reuses:** the load-bearing fence technique (`test_h`, `test_vertical_slice_e2e.py:172-199`);
|
||||
`TokenMeter`/`Budget`.
|
||||
- **Test first:**
|
||||
- File: `tests/test_portfolio.py`
|
||||
- Verifies: the three assertions above (baseline equality under isolation; strict growth under
|
||||
sharing).
|
||||
- Pattern: `tests/test_vertical_slice_e2e.py:172-199` (load-bearing detach)
|
||||
- **Verify:** `uv run pytest tests/test_portfolio.py::test_c_execution_state_isolation_is_load_bearing -q` → expected: pass
|
||||
- **On failure:** retry — if the isolated `==` does not hold because the two projects issue
|
||||
unequal chat-call counts, switch the isolation observable to a per-project standalone baseline
|
||||
for BOTH projects (`iso.runs[i] == standalone_i`); if still flaky, fall back to the
|
||||
cap-based variant (size `max_tokens` so 1 project < cap < 2 shared → `pytest.raises(BudgetExceeded)`
|
||||
only under sharing). Then revert on persistent failure: `git checkout -- tests/test_portfolio.py`
|
||||
- **Checkpoint:** `git commit -m "test(fase3): SC3 load-bearing meter-isolation detach guard"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- tests/test_portfolio.py
|
||||
min_file_count: 1
|
||||
commit_message_pattern: "^test\\(fase3\\): SC3 load-bearing meter-isolation detach guard$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/validator.py
|
||||
must_contain:
|
||||
- path: tests/test_portfolio.py
|
||||
pattern: "test_c_execution_state_isolation_is_load_bearing"
|
||||
```
|
||||
|
||||
### Step 6: Both profiles exercised offline + gated Azure arm (SC7)
|
||||
|
||||
- **Files:** `tests/test_portfolio.py`, `tests/test_portfolio_live.py` (new)
|
||||
- **Changes:**
|
||||
1. `test_portfolio.py`: add `test_d_both_profiles_run_offline`, parametrized
|
||||
`@pytest.mark.parametrize("profile", ["local", "azure"])`, running `run_portfolio` with the
|
||||
synthetic `client_factory` and asserting a `PortfolioResult` with `len(runs)==N` of
|
||||
`RunResult` for each profile (the contract path is actually executed offline, not just the
|
||||
backend instantiated). **Plus a load-bearing teeth assertion** that the profile reaches a
|
||||
profile-dependent seam offline: `resolve_model(profile, "proposer")` resolves to that
|
||||
profile's configured model from the bundled `model_map.json` and `resolve_model("local",
|
||||
"proposer") != resolve_model("azure", "proposer")` (deleting the azure model-map entry
|
||||
reddens it). **Honest limitation documented:** under an injected `client_factory`, `profile`
|
||||
is otherwise inert inside `run_project` (`run.py:152,173` — `model="fake-model"`, the F9
|
||||
standing item); the orchestrator end-to-end run proves the path executes under both, and the
|
||||
`resolve_model` check is the only profile-dependent seam provable without egress.
|
||||
2. `test_portfolio_live.py` (new): a `@pytest.mark.skipif`-gated real-Azure portfolio run
|
||||
mirroring `test_foundry_profile_live.py:14-24` — stays skipped offline (SC9 unchanged).
|
||||
- **Reuses:** `resolve_model` (`backends.py:54`), the gated-live skip pattern
|
||||
(`test_foundry_profile_live.py:19-24`).
|
||||
- **Test first:**
|
||||
- File: `tests/test_portfolio.py` + `tests/test_portfolio_live.py`
|
||||
- Verifies: both profiles produce a `PortfolioResult` offline; `resolve_model` distinguishes
|
||||
the profiles; the live arm is collected-but-skipped.
|
||||
- Pattern: `tests/test_foundry_profile_live.py:14-24`
|
||||
- **Verify:** `uv run pytest tests/test_portfolio.py -k both_profiles tests/test_portfolio_live.py -q` → expected: offline params pass, live arm skipped
|
||||
- **On failure:** revert — `git checkout -- tests/test_portfolio.py` and `git clean -fd tests/test_portfolio_live.py`
|
||||
- **Checkpoint:** `git commit -m "test(fase3): SC7 both profiles offline + gated Azure portfolio arm"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- tests/test_portfolio.py
|
||||
- tests/test_portfolio_live.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^test\\(fase3\\): SC7 both profiles offline \\+ gated Azure portfolio arm$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/validator.py
|
||||
must_contain:
|
||||
- path: tests/test_portfolio.py
|
||||
pattern: "both_profiles"
|
||||
- path: tests/test_portfolio_live.py
|
||||
pattern: "skipif"
|
||||
```
|
||||
|
||||
### Step 7: New project via config only + no-code-mapping guard (SC1)
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/data/reference_projects.json`,
|
||||
`src/portfolio_optimiser/data/docs/SKOLE-VVS-OPPGR/notes.txt` (new),
|
||||
`tests/test_portfolio.py`
|
||||
- **Changes:**
|
||||
1. Append a 4th synthetic project (id e.g. `SKOLE-VVS-OPPGR`) to `reference_projects.json`
|
||||
with the full key set **including** `docs_dir` + `verdict_input` — AND its bundled docs
|
||||
dir. All domain data (codes/units/prices, the verdict decision/rationale) is AI-authored
|
||||
and flagged SYNTHETIC/ "ikke verifisert" in the `_note` discipline
|
||||
(`[[user-not-domain-expert]]`). **No `src/portfolio_optimiser/*.py` change** — this is the
|
||||
SC1 demonstration that adding a project is config + docs only.
|
||||
2. `test_portfolio.py`: `test_e_new_project_flows_through_via_config_only` — run
|
||||
`run_portfolio(["SKOLE-VVS-OPPGR"], ...)` and assert it produces a `RunResult` whose
|
||||
`outcome.proposal.project_id == "SKOLE-VVS-OPPGR"`, proving config-only onboarding.
|
||||
3. `test_portfolio.py`: `test_f_no_hardcoded_project_ids_in_src` — a src-grep guard asserting
|
||||
**no** reference-project id literal (`FV42-GSV-E1|RV13-RAS-TP|BRU-LAKS-REHAB|SKOLE-VVS-OPPGR`)
|
||||
appears anywhere under `src/portfolio_optimiser/` (ids live only in the data JSON). If a
|
||||
future project is added via a code-side id→path/verdict mapping, this guard reddens —
|
||||
load-bearing for SC1's "config-only" claim.
|
||||
- **Reuses:** the `_note` provenance discipline; the src-grep guard idiom (`test_budget.py:88-99`).
|
||||
- **Test first:**
|
||||
- File: `tests/test_portfolio.py`
|
||||
- Verifies: the new project runs end-to-end via config; no hardcoded ids in src.
|
||||
- Pattern: `tests/test_budget.py:88-99` (src anti-pattern grep guard)
|
||||
- **Verify:** `uv run pytest tests/test_portfolio.py -k "new_project or no_hardcoded" -q` → expected: pass; then `git diff --name-only HEAD~1` shows only `*.json`, `data/docs/**`, `tests/**` (no `src/portfolio_optimiser/*.py`)
|
||||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/data/reference_projects.json tests/test_portfolio.py` and `git clean -fd src/portfolio_optimiser/data/docs/SKOLE-VVS-OPPGR`
|
||||
- **Checkpoint:** `git commit -m "test(fase3): SC1 new project via config-only + no-hardcoded-id src guard"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/data/reference_projects.json
|
||||
- src/portfolio_optimiser/data/docs/SKOLE-VVS-OPPGR/notes.txt
|
||||
- tests/test_portfolio.py
|
||||
min_file_count: 3
|
||||
commit_message_pattern: "^test\\(fase3\\): SC1 new project via config-only \\+ no-hardcoded-id src guard$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/run.py
|
||||
- src/portfolio_optimiser/reference_domain.py
|
||||
- src/portfolio_optimiser/validator.py
|
||||
- src/portfolio_optimiser/ir.py
|
||||
must_contain:
|
||||
- path: tests/test_portfolio.py
|
||||
pattern: "test_f_no_hardcoded_project_ids_in_src"
|
||||
- path: src/portfolio_optimiser/data/reference_projects.json
|
||||
pattern: "SKOLE-VVS-OPPGR"
|
||||
```
|
||||
|
||||
### Step 8: Extension documentation (SC6)
|
||||
|
||||
- **Files:** `docs/extending.md` (new), `tests/test_portfolio.py`
|
||||
- **Changes:**
|
||||
1. `docs/extending.md` — sections "Legg til eget prosjekt", "Legg til egen datakilde",
|
||||
"Legg til egen modell-map", each pointing at the concrete config seams:
|
||||
`reference_projects.json` (the per-project `docs_dir` + `verdict_input` fields, the `_note`
|
||||
SYNTHETIC discipline), the bundled `data/docs/<id>/` docs source, and `model_map.json`
|
||||
(role→deployment). State plainly that the bundled reference domain + its `verdict_input`
|
||||
values are SYNTHETIC fixtures and that a real deployer replaces the data source and supplies
|
||||
Layer-2 verdicts via real HITL (not static config) — keeps the architecture honest.
|
||||
2. `test_portfolio.py`: `test_g_extension_doc_exists_and_references_seams` — assert
|
||||
`docs/extending.md` exists and contains `reference_projects.json`, `docs_dir`,
|
||||
`verdict_input`, and `model_map.json`.
|
||||
- **Reuses:** the file-existence + content guard idiom.
|
||||
- **Test first:**
|
||||
- File: `tests/test_portfolio.py`
|
||||
- Verifies: doc exists and names the config seams.
|
||||
- Pattern: `tests/test_budget.py:88-99` (file-content assertion)
|
||||
- **Verify:** `uv run pytest tests/test_portfolio.py -k extension_doc -q` → expected: pass
|
||||
- **On failure:** revert — `git checkout -- tests/test_portfolio.py` and `git clean -fd docs/extending.md`
|
||||
- **Checkpoint:** `git commit -m "docs(fase3): extension-point guide (add project / data source / model-map) + SC6 test"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- docs/extending.md
|
||||
- tests/test_portfolio.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^docs\\(fase3\\): extension-point guide \\(add project / data source / model-map\\) \\+ SC6 test$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/validator.py
|
||||
must_contain:
|
||||
- path: docs/extending.md
|
||||
pattern: "reference_projects.json"
|
||||
- path: tests/test_portfolio.py
|
||||
pattern: "test_g_extension_doc_exists_and_references_seams"
|
||||
```
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
| Approach | Pros | Cons | Why rejected |
|
||||
|----------|------|------|--------------|
|
||||
| Per-project `client_factory` arg on `run_portfolio` (test seam) | Explicit; no prompt-scanning | Adds test-only API surface; production always uses one factory; diverges from `run_project`'s single-factory seam | Project-aware synthetic client keeps `run_portfolio` production-shaped (one `client_factory`), pushing per-project reply logic into the test double only |
|
||||
| Per-project isolated `VerdictStore` | Simpler isolation story | Kills cross-project ExpeL learning — the whole point of Fase 3 | Explicit Non-Goal; the brief deliberately shares one store |
|
||||
| `asyncio.gather` concurrent fan-out | Faster | Races the shared store's retrieve-before-add ordering; interleaves debates; needs the meter/store made concurrency-safe | Explicit Non-Goal (sequential only; concurrency is a documented extension point, 90%-principle) |
|
||||
| Hardcoded `{project_id: docs_dir}` map in orchestrator code | No JSON change | Adding project N+1 edits `src/*.py` → **breaks SC1** | Config-driven `docs_dir` on the loader is the authorized one-time change that makes N+1 config-only |
|
||||
| Make `profile` load-bearing by changing `run.py:173` (`model="fake-model"` → `resolve_model`) | Would make SC7 fully load-bearing end-to-end | Touches a Fase 2 seam (F9) — scope creep beyond the brief | Out of scope; SC7 teeth come from a direct `resolve_model` assertion instead, F9 left as the standing item |
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- **Framework:** `pytest` + `pytest-asyncio` (`asyncio_mode=auto` — bare `async def test_*`).
|
||||
- **Existing patterns:** synthetic offline client (`conftest.py:27`), shared-store two-call
|
||||
accumulation (`test_vertical_slice_e2e.py:69-90`), load-bearing fence (`test_h`), gated-live
|
||||
skip (`test_foundry_profile_live.py`), src-grep guard (`test_budget.py:88-99`),
|
||||
decoy-vs-match ranking (`test_verdicts.py`).
|
||||
- **New tests in this plan:** 1 in `test_vertical_slice_e2e.py` (meter seam) + ~7 in
|
||||
`tests/test_portfolio.py` (SC2,4,3,7,1×2,6) + 1 in `test_portfolio_live.py` (gated) + loader
|
||||
assertions in `test_reference_domain.py`.
|
||||
|
||||
### Tests to write
|
||||
|
||||
| Type | File | Verifies | Model test |
|
||||
|------|------|----------|------------|
|
||||
| Unit | `tests/test_vertical_slice_e2e.py` | injected `meter=` is used (Step 1) | `test_a` (`:28`) |
|
||||
| Loader | `tests/test_reference_domain.py` | `docs_dir`+`verdict_input` resolved per project | `test_reference_domain.py:8` |
|
||||
| Integration | `tests/test_portfolio.py` | SC2 fan-out + aggregate sums | `test_vertical_slice_e2e.py:28` |
|
||||
| Integration | `tests/test_portfolio.py` | SC4 accumulation + load-bearing retrieval | `test_verdicts.py` |
|
||||
| Load-bearing | `tests/test_portfolio.py` | SC3 meter-isolation detach (auto guard) | `test_h` (`:172`) |
|
||||
| Parametrized | `tests/test_portfolio.py` | SC7 both profiles offline + `resolve_model` teeth | `test_foundry_profile_live.py` |
|
||||
| Integration+Guard | `tests/test_portfolio.py` | SC1 config-only project + no-hardcoded-id src grep | `test_budget.py:88` |
|
||||
| Guard | `tests/test_portfolio.py` | SC6 extension doc exists + references seams | `test_budget.py:88` |
|
||||
| Gated live | `tests/test_portfolio_live.py` | real-Azure portfolio arm (skipped offline) | `test_foundry_profile_live.py:19` |
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Priority | Risk | Location | Impact | Mitigation |
|
||||
|----------|------|----------|--------|------------|
|
||||
| Critical | SC4 collapses to `len==1` if projects mint identical ids | `verdicts.py:121-124,78-90` | green-but-dead | Project-aware factory yields 3 structurally **distinct** proposals; assert 3 distinct ids, not just `len==3` (Step 4) |
|
||||
| Critical | SC4 "surfaces prior" passes trivially — `retrieve` has no similarity floor | `verdicts.py:110-119` | zero-overlap projects pass | Assert `runs[2].retrieved[0].id == runs[0].verdict.id` (specific match beats the RV13 decoy), not non-emptiness (Step 4) |
|
||||
| High | SC3 has no meter-injection point today | `run.py:151` | detach degrades to manual revert | Add additive `meter=` seam (Step 1) + `meter_factory` (Step 3); both detach arms run every CI |
|
||||
| High | SC3 `==` arm flaky if the two projects issue unequal chat-call counts | `run.py:151-169` | false red/green | Cap-independent baseline: `iso.runs[1] == standalone(project_1)`; documented fallbacks in Step 5 On-failure (per-project baseline, or cap-based `BudgetExceeded`) |
|
||||
| High | SC7 profile inert under injected client (`model="fake-model"`) | `run.py:152,173` (F9) | parametrize proves nothing | Add direct `resolve_model(profile,…)` teeth + document the limitation honestly; do not silently claim full profile coverage (Step 6) |
|
||||
| High | `verdict_input` hidden non-config dependency breaks SC1 | `run.py:119` | adding N+1 edits core | Make `verdict_input` a per-project config field resolved by the loader (Step 2), same home as `docs_dir` |
|
||||
| Medium | Bundled-docs path resolution via `importlib.resources` is source/editable-install dependent | `reference_domain.py:55` | a future zip-wheel install can't `open()` a Traversable dir | Acceptable for the dev+test context (90%); note as Assumption; if packaged, switch to `importlib.resources.as_file` |
|
||||
| Medium | Partial-failure mid-fan-out aborts the sequential loop with a half-accumulated store | `run.py:130-131,193` | ambiguous portfolio result | v1 policy = **fail-fast** (a raising project aborts; consistent with the offline synthetic suite where all validate); collect-and-continue is a documented future extension |
|
||||
| Low | Magnitude-bucket boundary flips drop the magnitude similarity component | `verdicts.py:29,53-57` | fixture overlap not as intended | All three SC4 savings (130k/200k/210k) sit inside bucket `[1e5,5e5)` interior — verified numerically (Step 4) |
|
||||
|
||||
## Assumptions
|
||||
|
||||
| # | Assumption | Why unverifiable | Impact if wrong |
|
||||
|---|-----------|-----------------|-----------------|
|
||||
| 1 | The two/three offline projects issue **equal** synthetic chat-call counts, so the SC3 `==` isolation arm holds | Depends on `fresh_workflow` round dynamics under the synthetic client; not statically determinable | SC3 isolation arm needs the documented fallback (per-project baseline or cap-based variant) — Step 5 already carries it |
|
||||
| 2 | `importlib.resources.files(...).joinpath("data", docs_dir)` yields an `os.PathLike` real path under the `uv` editable/source install | `files()` returns a `Traversable`; concrete behavior differs for zip-imported packages | A zip-wheel install would need `as_file`; harmless in the current dev/test context |
|
||||
| 3 | Putting synthetic `verdict_input` (decision/rationale) in `reference_projects.json` is the right config home (vs a separate portfolio-config) | Brief leaves it open | If a separate portfolio-config is later preferred, the loader field moves; SC1 still holds as long as it stays config-driven |
|
||||
| 4 | Static config `verdict_input` faithfully stands in for Layer-2 HITL in the offline synthetic framework | It is a fixture convenience, not the production HITL path | Documented in Step 8 doc as synthetic — production supplies verdicts via real HITL, not config |
|
||||
|
||||
## Verification
|
||||
|
||||
*Per-step manifests are checked automatically by trekexecute. These are the end-to-end gates.*
|
||||
|
||||
- [ ] `uv run pytest -q` → expected: `(103 + new) passed, 3 skipped` — the 3 existing live
|
||||
skips preserved, zero new failures (SC5; baseline re-read at execute-start, not hardcoded).
|
||||
- [ ] `uv run mypy src` → expected: clean (SC5).
|
||||
- [ ] `uv run ruff check .` → expected: clean; `uv run ruff format .` → no diff (SC5).
|
||||
- [ ] `uv run pytest tests/test_portfolio.py -q` → all SC2/SC3/SC4/SC7/SC1/SC6 tests pass.
|
||||
- [ ] SC1 git-diff proof — for the Step-7 commit:
|
||||
`git show --name-only --format= HEAD` (the SC1 commit) lists ONLY `*.json`,
|
||||
`data/docs/**`, and `tests/**` — **no `src/portfolio_optimiser/*.py`**.
|
||||
- [ ] SC3 detach is genuinely load-bearing — temporarily make `run_portfolio` share one meter
|
||||
across projects; `test_c_execution_state_isolation_is_load_bearing` MUST go red; revert.
|
||||
- [ ] SC4 is genuinely load-bearing — temporarily give two projects identical proposals; the
|
||||
distinct-id assertion MUST go red; revert.
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Files to modify:** 7 — `run.py`, `reference_domain.py`, `reference_projects.json`,
|
||||
`__init__.py`, `conftest.py`, `test_reference_domain.py`, `test_vertical_slice_e2e.py`.
|
||||
- **Files to create:** ~9 — `tests/test_portfolio.py`, `tests/test_portfolio_live.py`,
|
||||
`docs/extending.md`, and 4 bundled `data/docs/<id>/notes.txt` (+ the SC1 new project's docs).
|
||||
- **Complexity:** medium — additive orchestration over a proven seam; the difficulty is in the
|
||||
load-bearing test design (SC3/SC4), which the exploration de-risked with concrete fixtures.
|
||||
|
||||
## Execution Strategy
|
||||
|
||||
8 steps. Dependencies make Steps 1–3 a foundation wave; Steps 4–8 are additive and largely
|
||||
independent (they extend `tests/test_portfolio.py` + sibling files). A single sequential
|
||||
`/trekexecute` session is appropriate (one author, clean tree); the grouping below is the
|
||||
dependency map, not a mandate to parallelise.
|
||||
|
||||
### Session 1: Core seams
|
||||
- **Steps:** 1, 2
|
||||
- **Wave:** 1
|
||||
- **Depends on:** none
|
||||
- **Scope fence:**
|
||||
- Touch: `run.py` (meter seam only), `reference_domain.py`, `reference_projects.json`,
|
||||
`data/docs/**`, `test_vertical_slice_e2e.py`, `test_reference_domain.py`
|
||||
- Never touch: `validator.py`, `ir.py`, `verdicts.py`
|
||||
|
||||
### Session 2: Orchestrator
|
||||
- **Steps:** 3
|
||||
- **Wave:** 2
|
||||
- **Depends on:** Session 1 (loader provides `docs_dir`/`verdict_input`)
|
||||
- **Scope fence:**
|
||||
- Touch: `run.py` (PortfolioResult + run_portfolio), `__init__.py`, `conftest.py`, `test_portfolio.py`
|
||||
- Never touch: `validator.py`, `ir.py`
|
||||
|
||||
### Session 3: Success-criteria tests + docs
|
||||
- **Steps:** 4, 5, 6, 7, 8
|
||||
- **Wave:** 3
|
||||
- **Depends on:** Session 2 (`run_portfolio` exists)
|
||||
- **Scope fence:**
|
||||
- Touch: `tests/test_portfolio.py`, `tests/test_portfolio_live.py`, `docs/extending.md`,
|
||||
`reference_projects.json` (SC1 new project), `data/docs/SKOLE-VVS-OPPGR/**`
|
||||
- Never touch: `src/portfolio_optimiser/*.py` (SC1 demands config-only) except none
|
||||
|
||||
### Execution Order
|
||||
|
||||
- **Wave 1:** Session 1
|
||||
- **Wave 2:** Session 2 (after Wave 1)
|
||||
- **Wave 3:** Session 3 (after Wave 2)
|
||||
|
||||
### Grouping rules applied
|
||||
|
||||
- Steps sharing files → same session (Steps 4–8 all extend `test_portfolio.py`).
|
||||
- Core changes isolated to Waves 1–2 so Wave 3 can assert config-only (SC1).
|
||||
- Sessions ordered strictly by dependency.
|
||||
|
||||
## Plan Quality Score
|
||||
|
||||
| Dimension | Weight | Score | Notes |
|
||||
|-----------|--------|-------|-------|
|
||||
| Structural integrity | 0.15 | 92 | Steps dependency-ordered; foundation seams before consumers |
|
||||
| Step quality | 0.20 | 90 | Each step TDD, single focused change, concrete fixtures + commands |
|
||||
| Coverage completeness | 0.20 | 92 | All 7 SCs mapped to steps; brief Open Questions (docs_dir, verdict_input, SC4 fixture, aggregate) resolved |
|
||||
| Specification quality | 0.15 | 88 | Exact file:line, computed P90/similarity math, no placeholders |
|
||||
| Risk & pre-mortem | 0.15 | 90 | The 5 green-but-dead traps surfaced by exploration each have a mitigation + a documented fallback |
|
||||
| Headless readiness | 0.10 | 85 | On-failure + Checkpoint per step; SC3 carries an explicit fallback path |
|
||||
| Manifest quality | 0.05 | 80 | All steps have checkable manifests with forbidden_paths fencing the pure core |
|
||||
| **Weighted total** | **1.00** | **88** | **Grade: B+** |
|
||||
|
||||
**Adversarial review:**
|
||||
- **Plan critic:** APPROVE_WITH_NOTES — 0 blockers, 2 major, 9 minor (score 81/B). Both majors
|
||||
(SC4 fixture magnitudes unspecified + Step 3↔4 forward coupling) addressed by moving the
|
||||
concrete `REPLIES` table into Step 3. Minors folded in (red-phase gates, JSON uniform reply,
|
||||
error-path test, similarity wording, notes.txt relevance, file count, tautology).
|
||||
- **Scope guardian:** ALIGNED — 0 creep, 0 gaps. All 7 SCs covered; the 3 flagged items
|
||||
(`meter=`, `resolve_model` teeth, config `verdict_input`) each judged in-scope; 4 over-reaches
|
||||
correctly rejected in Alternatives.
|
||||
|
||||
## Revisions
|
||||
|
||||
*Added by adversarial review (Phase 9).*
|
||||
|
||||
| # | Finding | Severity | Resolution |
|
||||
|---|---------|----------|------------|
|
||||
| 1 | SC4/SC2 proposal magnitudes (quantity·unit_cost → P90) never specified; identity assumes empty `assumptions` | major | Step 3 now carries the explicit `REPLIES` table (item magnitudes, Σ totals, P90, claimed) + states empty `assumptions`; all three validate by construction |
|
||||
| 2 | Step 3 `540000`/`validated_count==3` depended on a fixture deferred to Step 4 (forward coupling) | major | `REPLIES` constant defined in Step 3; Step 4 now reuses it (no re-specification) |
|
||||
| 3 | Step 5 "uniform reply" could be a bare non-JSON string → runs to `BudgetExceeded` | minor | Step 5 now reuses `REPLIES["FV42-GSV-E1"]` (valid JSON) for `f` |
|
||||
| 4 | Step 3 `sum_token_usage` assertion was tautological (field's own definition) | minor | Re-stated as an explicit element-wise sum wiring check + each run `>0` |
|
||||
| 5 | `meter=`/`meter_factory` production surface not justified vs the seamless fallback | minor | Step 1 now states why the encoded detach (SC3-preferred) needs the injection point |
|
||||
| 6 | Steps 1–3 lacked an explicit TDD red-phase gate (Iron Law) | minor | Steps 1–3 Verify now require the test to be **red before** the production change |
|
||||
| 7 | `notes.txt` relevance to `retrieve_chunks("cost saving measure")` not tied | minor | Step 2 now requires the notes text to contain the cost-saving query terms |
|
||||
| 8 | `run_portfolio` unknown-id / error paths untested | minor | Step 3 adds `test_a2_unknown_project_id_raises` (`pytest.raises(ValueError)`) |
|
||||
| 9 | similarity(BRU,FV42) parenthetical overstated the codes contribution; measure equality is exact | minor | Step 4 now shows the full 0.20+0.25+0.15 decomposition + byte-identical measure requirement |
|
||||
| 10 | Estimated Scope "6 files to modify" but listed 7 | minor | Corrected to 7 |
|
||||
| 11 | SC7 azure offline arm near-inert under injected client | minor (acknowledged) | Already documented honestly in Step 6 + Risks; `resolve_model` teeth retained; no change |
|
||||
|
||||
## Adversarial Pass 2 (gemini-bridge, v5.1.1 high-effort)
|
||||
|
||||
**Status: attempted, unavailable — no independent findings.** The high-effort plan phase
|
||||
(`phase_signal: plan effort=high`) prescribes an additional independent gemini-bridge review of
|
||||
the post-revision plan. The pass was launched but the Gemini Deep Research API returned a
|
||||
**deterministic** failure: its legacy Interactions API schema was deprecated (May 2026) and the
|
||||
local `gemini-mcp` client SDK predates the replacement. This is an operator-side tooling issue,
|
||||
not a plan defect, and not retryable as-is.
|
||||
|
||||
**Decision:** proceed without the third cross-check. The plan already cleared two independent
|
||||
adversarial reviewers (plan-critic APPROVE_WITH_NOTES, scope-guardian ALIGNED) with all
|
||||
substantive findings consumed; the gemini pass was additive triangulation, not a gate. *Operator
|
||||
action if desired:* upgrade `gemini-mcp` per the May-2026 breaking-change notice, then this pass
|
||||
can be re-run for an independent second opinion before `/trekexecute`.
|
||||
|
|
@ -1,121 +0,0 @@
|
|||
---
|
||||
type: trekreview
|
||||
review_version: "1.0"
|
||||
created: 2026-06-26
|
||||
task: "Fase 3: orkestrator fan-out over N porteføljeprosjekter med state-isolasjon + delt læring"
|
||||
slug: fase3-portfolio-fanout
|
||||
project_dir: .claude/projects/2026-06-26-fase3-portfolio-fanout/
|
||||
brief_path: .claude/projects/2026-06-26-fase3-portfolio-fanout/brief.md
|
||||
scope_sha_start: 497399e0aa523208b3188f0c8ca54627ec5863b7
|
||||
scope_sha_end: 207f0570755cc5f0d664cfb16f18a5a00aee1ef0
|
||||
reviewed_files_count: 15
|
||||
verdict: WARN
|
||||
findings:
|
||||
- b3abbdd7035b5463b3bb9e23f83a158d678cb4f1
|
||||
- a2376be90affdffea4c9b4ef83014d0a30243e23
|
||||
---
|
||||
|
||||
# Review: Fase 3 — orkestrator fan-out over N porteføljeprosjekter med state-isolasjon + delt læring
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Verdict: **WARN** — 0 BLOCKER, 2 MAJOR. Look first at the `fake-model` placeholder leak in `run.py:197`: it stamps a hardcoded sentinel into `ProvenanceStamp.model` for any deployer who supplies a custom `client_factory`, falsifying a load-bearing provenance invariant on every project in the fan-out. The second MAJOR is a coverage gap — the portfolio `rejected_count` / validated-only-sum branch (`run.py:240-241`) is never exercised by an actual rejection, so a regression there ships green under SC2. Two honesty caveats: SC5 (full-suite green / mypy / ruff) was a runtime gate the reviewers could not execute and declined to fabricate, so its pass is asserted from code shape only, not observed; and this is a HIGH-effort review, so Pass 3 (Cloudflare reasonableness filter) was skipped by design — borderline findings are left for the operator to weigh. Both incoming `rule_key`s (`PLACEHOLDER_IN_CODE`, `MISSING_TEST`) are valid catalogue keys, so no high-effort rule_key normalization was applied.
|
||||
|
||||
## Coverage
|
||||
|
||||
15 files reviewed, all `summary-only`. 0 files marked `skip` → 0 silent skips. Working tree clean → no `[uncommitted]` annotations.
|
||||
|
||||
| File | Treatment | Reason |
|
||||
|------|-----------|--------|
|
||||
| docs/extending.md | summary-only | SC6 extension-point doc; prose, no executable surface |
|
||||
| src/portfolio_optimiser/__init__.py | summary-only | Package exports; mechanical |
|
||||
| src/portfolio_optimiser/data/docs/BRU-LAKS-REHAB/notes.txt | summary-only | Synthetic project fixture data |
|
||||
| src/portfolio_optimiser/data/docs/FV42-GSV-E1/notes.txt | summary-only | Synthetic project fixture data |
|
||||
| src/portfolio_optimiser/data/docs/RV13-RAS-TP/notes.txt | summary-only | Synthetic project fixture data |
|
||||
| src/portfolio_optimiser/data/docs/SKOLE-VVS-OPPGR/notes.txt | summary-only | Synthetic project fixture data |
|
||||
| src/portfolio_optimiser/data/reference_projects.json | summary-only | Config-only project registry (SC1 seam) |
|
||||
| src/portfolio_optimiser/reference_domain.py | summary-only | Domain reference module; no diff-critical logic flagged |
|
||||
| src/portfolio_optimiser/run.py | summary-only | Orchestrator fan-out + aggregate; both MAJOR findings land here |
|
||||
| tests/conftest.py | summary-only | Shared fixtures |
|
||||
| tests/spikes/test_b_footguns.py | summary-only | SC1 no-hardcoded-id / 4th-project spike guard |
|
||||
| tests/test_portfolio.py | summary-only | SC2 aggregate; asserts rejected_count==0 (root of MISSING_TEST gap) |
|
||||
| tests/test_portfolio_live.py | summary-only | SC7 gated Azure portfolio arm (skip-gated) |
|
||||
| tests/test_reference_domain.py | summary-only | Domain reference tests |
|
||||
| tests/test_vertical_slice_e2e.py | summary-only | End-to-end slice |
|
||||
|
||||
## Findings (MAJOR)
|
||||
|
||||
### b3abbdd7035b5463b3bb9e23f83a158d678cb4f1
|
||||
- file: src/portfolio_optimiser/run.py
|
||||
- line: 197
|
||||
- rule_key: PLACEHOLDER_IN_CODE
|
||||
- brief_ref: NFR — code correctness
|
||||
- title: fake-model placeholder leaks into provenance via public client_factory seam
|
||||
- detail: `run.py:197` sets `model = 'fake-model' if client_factory is not None else resolve_model(profile, 'proposer')`, writing a hardcoded sentinel into `ProvenanceStamp.model` (run.py:197-204). The gate `client_factory is not None` is a public, documented seam threaded through `run_portfolio(..., client_factory=...)` (run.py:251,278), so a deployer with a custom backend factory stamps `'fake-model'` across all N provenance records, falsifying the load-bearing provenance invariant. Only the default (no injected factory) path is compensated by `resolve_model`.
|
||||
- recommended_action: Resolve the real model identifier from the injected `client_factory` (or require the factory to surface its model) instead of branching to the `'fake-model'` literal; reserve the sentinel for test-only fixtures, not the public seam.
|
||||
|
||||
### a2376be90affdffea4c9b4ef83014d0a30243e23
|
||||
- file: src/portfolio_optimiser/run.py
|
||||
- line: 241
|
||||
- rule_key: MISSING_TEST
|
||||
- brief_ref: NFR — code correctness
|
||||
- title: Portfolio rejected_count and validated-only sum never exercised by a rejection
|
||||
- detail: `_aggregate` computes `rejected_count=len(rejected)` (run.py:240) and a validated-only `sum_claimed_saving_nok` (run.py:241). Every portfolio test feeds proposals that validate — `test_portfolio.py:63` asserts `rejected_count==0` — so no `run_portfolio` test produces a Rejection. The rejected branch and the validated-only filter are unproven: a regression that sums rejected claims or miscounts rejections would pass green while SC2 still claims the aggregate is correct.
|
||||
- recommended_action: Add a `run_portfolio` test that feeds at least one proposal the deterministic validator rejects, then assert `rejected_count >= 1` and that `sum_claimed_saving_nok` excludes the rejected claim.
|
||||
|
||||
## Suppressed Findings
|
||||
|
||||
None. No finding was dropped by Pass 1 (dedup) or Pass 2 (HubSpot Judge). Pass 3 (reasonableness) was skipped under HIGH effort mode.
|
||||
|
||||
## Conformance trace
|
||||
|
||||
From `brief-conformance-reviewer` (`findings: []` — zero conformance/scope findings):
|
||||
|
||||
- **SC1** — config-only new project: PASS (guarded by no-hardcoded-id grep + 4th-project spike).
|
||||
- **SC2** — fan-out aggregate: PASS shape-wise, but see MAJOR `a2376…` — the rejected/validated-only branch is unexercised.
|
||||
- **SC3** — state-isolation (`==`): PASS, load-bearing (shared meter detach breaks the test).
|
||||
- **SC4** — shared VerdictStore accumulation: PASS, load-bearing.
|
||||
- **SC5** — full-suite green / mypy / ruff: NOT OBSERVED — runtime gate the reviewers could not execute; no fabricated finding emitted. Operator must confirm at the gate.
|
||||
- **SC6** — extension doc: PASS (`docs/extending.md`).
|
||||
- **SC7** — both profiles offline; live-Azure skip-gated: PASS.
|
||||
- **Non-Goals** — all honored: no concurrency, no dynamic topology, no rich report, one shared store, live-Azure skip-gated, no compliance code. Validator/IR have zero diff (out of scope by absence).
|
||||
|
||||
## Remediation Summary
|
||||
|
||||
- BLOCKER: 0
|
||||
- MAJOR: 2
|
||||
- MINOR: 0
|
||||
- SUGGESTION: 0
|
||||
|
||||
/trekplan (`--brief review.md`) will consume the two MAJOR findings from the trailing JSON block: fix the `fake-model` provenance leak on the public `client_factory` seam, and add a rejection-path test for the portfolio aggregate. Fase 3 closes on WARN per the gate (a BLOCK would require a BLOCKER).
|
||||
|
||||
```json
|
||||
{
|
||||
"verdict": "WARN",
|
||||
"counts": { "BLOCKER": 0, "MAJOR": 2, "MINOR": 0, "SUGGESTION": 0 },
|
||||
"findings": [
|
||||
{
|
||||
"id": "b3abbdd7035b5463b3bb9e23f83a158d678cb4f1",
|
||||
"severity": "MAJOR",
|
||||
"rule_key": "PLACEHOLDER_IN_CODE",
|
||||
"file": "src/portfolio_optimiser/run.py",
|
||||
"line": 197,
|
||||
"brief_ref": "NFR — code correctness",
|
||||
"title": "fake-model placeholder leaks into provenance via public client_factory seam",
|
||||
"detail": "run.py:197 sets model = 'fake-model' if client_factory is not None else resolve_model(profile, 'proposer'), writing a hardcoded sentinel into ProvenanceStamp.model (run.py:197-204). The gate client_factory is not None is a public, documented seam threaded through run_portfolio(..., client_factory=...) (run.py:251,278), so a deployer with a custom backend factory stamps 'fake-model' across all N provenance records, falsifying the load-bearing provenance invariant. Only the default (no injected factory) path is compensated by resolve_model.",
|
||||
"recommended_action": "Resolve the real model identifier from the injected client_factory (or require the factory to surface its model) instead of branching to the 'fake-model' literal; reserve the sentinel for test-only fixtures, not the public seam."
|
||||
},
|
||||
{
|
||||
"id": "a2376be90affdffea4c9b4ef83014d0a30243e23",
|
||||
"severity": "MAJOR",
|
||||
"rule_key": "MISSING_TEST",
|
||||
"file": "src/portfolio_optimiser/run.py",
|
||||
"line": 241,
|
||||
"brief_ref": "NFR — code correctness",
|
||||
"title": "Portfolio rejected_count and validated-only sum never exercised by a rejection",
|
||||
"detail": "_aggregate computes rejected_count=len(rejected) (run.py:240) and a validated-only sum_claimed_saving_nok (run.py:241). Every portfolio test feeds proposals that validate — test_portfolio.py:63 asserts rejected_count==0 — so no run_portfolio test produces a Rejection. The rejected branch and the validated-only filter are unproven: a regression that sums rejected claims or miscounts rejections would pass green while SC2 still claims the aggregate is correct.",
|
||||
"recommended_action": "Add a run_portfolio test that feeds at least one proposal the deterministic validator rejects, then assert rejected_count >= 1 and that sum_claimed_saving_nok excludes the rejected claim."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
|
@ -1,263 +0,0 @@
|
|||
---
|
||||
type: trekbrief
|
||||
brief_version: "2.2"
|
||||
created: 2026-07-03
|
||||
task: "I2 — MAF reference implementation of the ingest layer: file catalogue/CSV (offline)"
|
||||
slug: i2-ingest-csv-maf
|
||||
project_dir: .claude/projects/2026-07-03-i2-ingest-csv-maf/
|
||||
research_topics: 0
|
||||
research_status: skipped
|
||||
auto_research: false
|
||||
interview_turns: 0
|
||||
source: manual
|
||||
framing: preserve
|
||||
phase_signals:
|
||||
- phase: research
|
||||
effort: high
|
||||
model: fable
|
||||
- phase: plan
|
||||
effort: high
|
||||
model: fable
|
||||
- phase: execute
|
||||
effort: high
|
||||
model: fable
|
||||
- phase: review
|
||||
effort: high
|
||||
model: fable
|
||||
---
|
||||
|
||||
# Task: I2 — MAF reference implementation: ingest file catalogue/CSV (offline)
|
||||
|
||||
> Generated by `/trekbrief` on 2026-07-03.
|
||||
> This brief is the contract between requirements and planning. `/trekplan`
|
||||
> reads it to produce the implementation plan. Every decision in the plan must
|
||||
> trace back to content in this brief.
|
||||
>
|
||||
> **Provenance note (framing defense):** this brief was derived autonomously from
|
||||
> two FROZEN, operator-approved sources — the I2 section of
|
||||
> `docs/plan/2026-07-03-sesjonsplan-ingest.md` and the normative
|
||||
> `shared/ingest-spec.md` — under the STATE.md operating model ("Les STATE.md og
|
||||
> følg instruksjonene" = go-signal for the NEXT step; I2 is not gated). No
|
||||
> interview questions were asked because every answer already exists in written,
|
||||
> approved form; re-asking would violate the operator's recorded
|
||||
> plan-over-menu preference. `framing: preserve` is documented, not guessed:
|
||||
> I2 continues the frozen plan arc unchanged (STATE.md: "I2 [NESTE]").
|
||||
|
||||
## TL;DR
|
||||
|
||||
Implement the ingest layer's first source type on the MAF side, exactly as frozen in
|
||||
`shared/ingest-spec.md`: fail-fast pydantic manifest contract, CSV/file-catalogue
|
||||
connector, deterministic materialization into an OKF bundle with provenance
|
||||
frontmatter + index generation — offline, zero model calls, TDD. Framing: **preserve**
|
||||
— the "I2 [NESTE]" step of the operator-approved, frozen ingest session plan.
|
||||
|
||||
## Intent
|
||||
|
||||
The 8-step optimiser loop consumes hand-curated OKF bundles today; the frozen ingest
|
||||
målbilde closes the gap to real data sources while preserving the architecture-defining
|
||||
rule that **data reaches the model ONLY via OKF bundles** (no RAG, no query-time
|
||||
retrieval — method spec §3). I2 is the first implementation session of that program:
|
||||
it proves the frozen contract (`shared/ingest-spec.md`, authored and guarded in I1) is
|
||||
actually implementable from the spec alone, on the reference (MAF) side, using the
|
||||
offline `file` source type so the whole session stays at zero model calls and zero
|
||||
network (cost discipline D6). It also closes the plan's key assumption 2 — that an
|
||||
ingest manifest can be validated fail-fast without network — which is currently
|
||||
marked RISK with only pattern-level evidence (`contracts.py` is in-process). Finally,
|
||||
I2 turns the planning session's ephemeral ad-hoc proof (a synthetic generated bundle
|
||||
passing `navigate_bundle`/`bundle_context`) into a durable load-bearing test, so the
|
||||
seam can never silently detach.
|
||||
|
||||
## Goal
|
||||
|
||||
`uv run pytest` green (without network or credentials) on a repo where: a pydantic
|
||||
manifest contract validates ingest manifests fail-fast per spec §4 (polymorphic on
|
||||
`source.type`, verdict-layer reservation enforced before any source call); a
|
||||
file-catalogue/CSV connector executes extractions per §4 (boundary-checked path
|
||||
resolution, `max_rows` fail-fast); materialization per §5–§7 writes
|
||||
`ingest-{id}.md` concept files with the exact 7-key provenance frontmatter and
|
||||
generates/updates `index.md` per §6 (idempotent by target, curated/promoted links
|
||||
preserved); and a golden extraction case at `examples/ingest-golden-file/` (spec §11
|
||||
layout) reproduces byte for byte from fixture + fixed `ingested_at`. The existing
|
||||
loop consumes the generated bundle through UNCHANGED `okf.py` navigation code. Every
|
||||
seam in the I2 load-bearing set has a test proven RED at detach.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **SQL connector** — I4 (the pydantic manifest schema MAY validate the `sql`/`http`
|
||||
variants per §4, but no `sql` connector execution in I2; see Open Questions).
|
||||
- **HTTP/MCP connector and the network opt-in gate test** — I6 (gated: network
|
||||
capability). The `file` source type requires no network flag (§5, §8).
|
||||
- **Sibling repo (D7) work** — I3/I5 (gated: other repo). No files outside this repo.
|
||||
- **Commons/spec changes** — any needed spec change means STOP and a separate gated
|
||||
commons round (session-plan frame rule); never folded into I2. The golden case is
|
||||
authored LOCALLY in this repo in I2; sharing it via commons is a later gated round.
|
||||
- **Wiring ingest into `run_project`** — ingest is an explicit operator-run step
|
||||
BEFORE the loop (spec §2, §9); the run path is untouched.
|
||||
- **Extension points explicitly deferred by the spec:** per-row file splitting,
|
||||
multiple manifests per bundle, incremental re-ingest, approval registry /
|
||||
signed `approved_by` (spec §5, §9, §10).
|
||||
- **S12 release work** — independently gated (LICENSE is an operator-owned choice).
|
||||
|
||||
Note on spec conformance: full §1 conformance requires both `file` and `sql`
|
||||
connectors — I2 is deliberately a partial-conformance milestone; §1 conformance
|
||||
completes at I4. Reviewers must not read §1 as an I2 failure.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **The spec is normative and frozen.** Implement from `shared/ingest-spec.md` alone;
|
||||
MUST/MUST NOT clauses are binding. On any discovered spec defect or needed change:
|
||||
stop, report — separate gated commons round.
|
||||
- **Offline, deterministic, zero model calls** (D6 + spec §1, §11): tests run without
|
||||
credentials and without network; no live sources anywhere.
|
||||
- **TDD Iron Law:** no production code without a failing test first. Load-bearing
|
||||
tests must be proven RED at detach (detach → observe red → reattach), the method
|
||||
spec §11 regime.
|
||||
- **`okf.py` navigation code stays UNCHANGED** — the navigability criterion is
|
||||
explicitly "consumable via UNCHANGED `okf.py`". Reusing its existing primitives
|
||||
(e.g. the index-linking primitive, whose idempotent-by-target semantics §6
|
||||
references) is expected; modifying them is not in scope.
|
||||
- **Context-layer neutrality invariant:** `okf.py` and `shared/` remain pure stdlib,
|
||||
MAF-free (`test_okf_is_maf_free`). The new ingest module makes zero model calls and
|
||||
MUST NOT import `agent_framework`/`mcp` (it is a deterministic pre-loop step);
|
||||
pydantic is fine (already a core dependency, `contracts.py` pattern).
|
||||
- **Commons is PULL-ONLY**; nothing in I2 writes to commons or `shared/`.
|
||||
- **One session = one closed delivery:** green suite, STATE.md overwritten,
|
||||
commit + push to Forgejo; never a broken state across the session boundary.
|
||||
- **Surgical scope:** new ingest module + tests + golden example; no refactoring of
|
||||
adjacent code.
|
||||
|
||||
## Preferences
|
||||
|
||||
- Follow the existing `contracts.py` pydantic style for the manifest contract
|
||||
(fail-fast, explicit models, type hints everywhere).
|
||||
- Mirror the existing load-bearing test idiom (e.g.
|
||||
`tests/test_step8_promotion_loadbearing.py`): each test names its seam and fails
|
||||
when the seam is detached.
|
||||
- Reuse `okf.py`'s existing index-linking primitive for §6 index updates rather than
|
||||
reimplementing link semantics.
|
||||
- Golden case directory per spec §11 convention: `examples/ingest-golden-file/`
|
||||
with `manifest.json`, `fixture/`, `ingested-at.txt`, `expected-bundle/`.
|
||||
- Small synthetic fixture data (cost discipline — no large test payloads).
|
||||
- Code/comments/docs in English; Conventional Commits.
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
- **Bit-determinism:** same source content + manifest + `ingested_at` → byte-identical
|
||||
bundle; repeated runs idempotent (spec §10, §11).
|
||||
- **Zero new runtime dependencies** (pydantic and stdlib suffice; CSV via stdlib `csv`).
|
||||
- **Fail-fast, never silent:** malformed manifest, verdict-typed mapping, path escape,
|
||||
`max_rows` overflow, non-CSV-representable cell value, curated-file collision — all
|
||||
ERRORS, never warnings/truncation/coercion (spec §3, §4, §5, §8).
|
||||
- **File hygiene per spec §5:** LF-only, exactly one trailing newline, single-line
|
||||
frontmatter values, whitespace-collapsed `source_query`.
|
||||
- **Source calls are logged** (which source, when, row count — spec §8); test-visible
|
||||
via stdlib `logging`, no new dependency.
|
||||
- **Contract completeness:** the plan traces spec §12's cross-check table
|
||||
field-by-field (`manifest_version` … `generated`, golden-case entries) so no
|
||||
contract field is silently dropped — including `bundle_summary` (§4/§6) and the
|
||||
`ingest_manifest` stamp format `{stem}@{hash16}` (§5).
|
||||
|
||||
## Success Criteria
|
||||
|
||||
All commands run from the repo root; "without network/credentials" is the standing
|
||||
test condition (spec §11).
|
||||
|
||||
1. **Golden extraction, bit-deterministic:** a test materializes
|
||||
`examples/ingest-golden-file/` from `manifest.json` + `fixture/` +
|
||||
`ingested-at.txt` and compares against `expected-bundle/` file by file, byte for
|
||||
byte — `uv run pytest tests/ -k ingest` exits 0. Running materialization twice
|
||||
over the same target produces identical bytes (idempotence, §10).
|
||||
2. **Load-bearing: provenance detach** — with the §7 stamping detached (e.g.
|
||||
provenance keys not written), the suite goes RED. Proof recorded (detach → red →
|
||||
reattach) in the session log.
|
||||
3. **Load-bearing: navigability via UNCHANGED `okf.py`** — the generated bundle is
|
||||
consumed by the existing navigation code (`navigate_bundle`/`bundle_context`
|
||||
seam): generated files reachable via index cross-links, unknown provenance fields
|
||||
preserved. Test goes RED if index generation/linking is detached. `git diff` for
|
||||
the delivery shows no modification to `okf.py`.
|
||||
4. **Load-bearing: verdict reservation** — a manifest with `okf_type: verdict`
|
||||
(case-insensitive) or a generated-filename collision with `index.md`/the
|
||||
`promoted-verdict-*` namespace is REJECTED at validation, before any source call;
|
||||
test goes RED if the reservation is removed.
|
||||
5. **Load-bearing: re-ingest layer safety** — re-materialization over a bundle
|
||||
containing a promoted verdict (created via the real `promote_verdict`) preserves
|
||||
the verdict file AND its index link; test goes RED if replacement semantics stop
|
||||
honouring the ingest-stamp ownership rule (§3, §6).
|
||||
6. **Fail-fast manifest validation without network (closes key assumption 2):**
|
||||
malformed manifests (missing fields, bad `id` grammar, empty `extractions`,
|
||||
non-positive `max_rows`, unknown `source.type`) raise before any source access —
|
||||
asserted by tests that run with no network available.
|
||||
7. **Full suite + gates green:** `uv run pytest` exits 0 — the pre-existing suite
|
||||
passes unchanged (baseline captured at session start before any I2 change) plus
|
||||
the new ingest tests; `uv run ruff check .` exits 0, `uv run ruff format --check .`
|
||||
clean, `uv run mypy src` exits 0.
|
||||
8. **Honesty of artefacts:** no artefact delivered in I2 claims SQL/HTTP support,
|
||||
live-source integration, or commons sharing of the golden case. Concretely:
|
||||
case-insensitive grep for `sql`, `http`, `live`, `commons` over the new module's
|
||||
docstrings, new test files' docstrings, and any touched README/docs sections
|
||||
yields no support-claiming hits (mentions that explicitly DEFER — "I4", "I6",
|
||||
"extension point" — are fine); the plan fixes the exact grep invocation.
|
||||
|
||||
## Research Plan
|
||||
|
||||
No external research needed — the codebase and this brief contain sufficient context
|
||||
for planning. Rationale (recorded in lieu of an interview): the task is fully
|
||||
specified by the frozen `shared/ingest-spec.md`; the implementation surface is
|
||||
stdlib + pydantic patterns already established in this repo (`contracts.py`,
|
||||
`okf.py`, `verdicts.py`); the session is offline by rule, so there is no external
|
||||
API/library/version decision to research. Key assumption 4 (D7 SDK hooks) belongs to
|
||||
I3, not I2.
|
||||
|
||||
## Open Questions / Assumptions
|
||||
|
||||
- **[ASSUMPTION] Manifest schema breadth:** the pydantic contract is built
|
||||
polymorphic per spec §4 (validating `file`, `sql`, `http` source variants), while
|
||||
I2 implements ONLY the `file` connector execution path. This is the cheapest
|
||||
spec-complete reading; the plan may narrow it to `file`-only validation with
|
||||
justification (spec conformance for `sql` arrives in I4 either way).
|
||||
- **[ASSUMPTION] Invocation surface:** spec §9 requires ingest to be an explicit
|
||||
operator command. For I2 a public programmatic API (module function) satisfies the
|
||||
procedural gate for tests; whether to also add a `python -m` CLI entry in I2 is a
|
||||
plan decision (cheap, but not demanded by the I2 verification set).
|
||||
- **[ASSUMPTION] Module placement:** new module `src/portfolio_optimiser/ingest.py`
|
||||
(name/path is a plan decision; the constraint is only that `okf.py` stays unchanged
|
||||
and the module is MAF-free).
|
||||
- **[ASSUMPTION] Expected-bundle authoring:** `expected-bundle/` in the golden case is
|
||||
generated ONCE by the implementation after review of its content (then frozen in
|
||||
git), rather than hand-typed — acceptable because the golden test's job is
|
||||
regression pinning (spec §11), while correctness is carried by the unit/load-bearing
|
||||
tests. The plan should sequence this so the golden freeze happens LAST, after all
|
||||
behaviour tests are green.
|
||||
|
||||
## Prior Attempts
|
||||
|
||||
None as implementation — this is the first ingest implementation session. Two prior
|
||||
results are load-bearing context, not failures: (1) key assumption 1 was proven ad hoc
|
||||
on 2026-07-03 (a synthetic bundle with provenance frontmatter passed
|
||||
`navigate_bundle`/`bundle_context`; unknown fields preserved, context rendered) — I2
|
||||
makes that proof durable; (2) I1 delivered the frozen spec into `shared/` with the
|
||||
framework guard + structure test extended
|
||||
(`tests/test_method_spec_loadbearing.py`, commits `4df2140` + `7ba0fae`), so the
|
||||
contract I2 implements is already guarded in this repo.
|
||||
|
||||
## Metadata
|
||||
|
||||
- **Created:** 2026-07-03
|
||||
- **Interview turns:** 0 (brief derived from frozen, operator-approved plan + spec;
|
||||
see provenance note at top)
|
||||
- **Auto-research opted in:** no (0 topics)
|
||||
- **Source:** manual
|
||||
|
||||
---
|
||||
|
||||
## How to continue
|
||||
|
||||
Manual (default):
|
||||
|
||||
```bash
|
||||
# No research topics — go straight to plan:
|
||||
/trekplan --project .claude/projects/2026-07-03-i2-ingest-csv-maf/
|
||||
|
||||
# Then execute:
|
||||
/trekexecute --project .claude/projects/2026-07-03-i2-ingest-csv-maf/
|
||||
```
|
||||
|
|
@ -1,732 +0,0 @@
|
|||
# I2 — MAF reference implementation: ingest file catalogue/CSV (offline)
|
||||
|
||||
> **Plan quality: B+** (88/100) — APPROVE_WITH_NOTES
|
||||
>
|
||||
> Generated by trekplan v5.9 on 2026-07-03 — `plan_version: 1.7`
|
||||
|
||||
## Context
|
||||
|
||||
The 8-step optimiser loop consumes hand-curated OKF bundles today; the frozen ingest
|
||||
målbilde closes the gap to real data sources while preserving the architecture-defining
|
||||
rule that data reaches the model ONLY via OKF bundles (no RAG, no query-time retrieval —
|
||||
method spec §3). I2 is the first implementation session of that program: it proves the
|
||||
frozen contract (`shared/ingest-spec.md`, authored and guarded in I1) is implementable
|
||||
from the spec alone, on the reference (MAF) side, using the offline `file` source type so
|
||||
the session stays at zero model calls and zero network (cost discipline D6). It closes the
|
||||
session plan's key assumption 2 (fail-fast manifest validation without network, currently
|
||||
RISK-marked) and turns the planning session's ephemeral ad-hoc navigability proof into a
|
||||
durable load-bearing test. Source: brief Intent, verbatim intent trace.
|
||||
|
||||
Scope note (brief, Non-Goals): I2 is deliberately a **partial-conformance milestone** —
|
||||
spec §1 full conformance (`file` + `sql`) completes at I4. The manifest contract validates
|
||||
all three source variants; only the `file` connector executes in I2.
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "New in this plan"
|
||||
M[manifest.json] -->|"load_manifest() — pydantic fail-fast (§4), verdict reservation (§3)"| C[ingest.py contract models]
|
||||
C -->|"file connector: safe_resolve boundary + csv.reader + max_rows cap"| X[extraction rows in memory]
|
||||
X -->|"render: escaped markdown table + §5/§7 provenance frontmatter"| S[staged files in memory]
|
||||
S -->|"replacement semantics: delete stamped only, collision fail (§3)"| B[OKF bundle dir]
|
||||
S -->|"index create/update: bundle_summary body, managed-line removal, link_in_index append (§6)"| I[index.md]
|
||||
end
|
||||
subgraph "Existing, UNCHANGED"
|
||||
B --> N["okf.navigate_bundle / bundle_context"]
|
||||
I --> N
|
||||
V["verdicts.promote_verdict"] -->|"promoted-verdict-*.md + index link — must survive re-ingest"| B
|
||||
R["retrieval.safe_resolve / PathSecurityError"] -.reused by.-> C
|
||||
L["okf.link_in_index / render_frontmatter / parse_frontmatter"] -.reused by.-> S
|
||||
end
|
||||
```
|
||||
|
||||
## Codebase Analysis
|
||||
|
||||
- **Tech stack:** Python ≥3.10, pydantic ≥2.11, `uv`, pytest ≥8 (`asyncio_mode=auto`; ingest
|
||||
tests are plain sync), ruff (line-length 100), mypy (py3.10 target). No new runtime
|
||||
dependencies needed: stdlib `csv`, `hashlib`, `logging`, `pathlib` + pydantic suffice.
|
||||
- **Key patterns:** fail-fast pydantic startup contracts (`contracts.py:30-96`); frozen
|
||||
dataclasses for value objects, `BaseModel` for validated external input; PEP 604 unions,
|
||||
keyword-only options via bare `*`; condition-noun exceptions (`PromotionRefused`
|
||||
`verdicts.py:447`, `PathSecurityError` `retrieval.py:24`, dominant base `RuntimeError`);
|
||||
narrative module docstrings citing målbilde/spec sections; load-bearing tests with
|
||||
detach-RED docstrings (`tests/test_step8_promotion_loadbearing.py`).
|
||||
- **Relevant files:** `src/portfolio_optimiser/okf.py` (UNCHANGED — reuse
|
||||
`parse_frontmatter:31`, `render_frontmatter:146`, `link_in_index:168`,
|
||||
`navigate_bundle:113`, `bundle_context:133`), `src/portfolio_optimiser/retrieval.py`
|
||||
(`safe_resolve:63`, `is_within_dir:48`, `PathSecurityError:24`),
|
||||
`src/portfolio_optimiser/verdicts.py` (`promote_verdict:465` — structural analogue and
|
||||
the SC5 test's real promotion; SHA-256 `[:16]` idiom at `verdicts.py:95,461`),
|
||||
`tests/test_method_spec_loadbearing.py:156-174` (recorded I2 obligation: field-level
|
||||
cross-check), `tests/conftest.py` (no ingest-relevant fixtures — none needed),
|
||||
`shared/examples/bygg-energi-mikro/` (copytree fixture for the re-ingest safety test).
|
||||
- **Reusable code:** `safe_resolve` for the §4 root boundary (fail-closed);
|
||||
`render_frontmatter` for §5 key order + single-lining (collapses whitespace in ALL
|
||||
values — acceptable: §5 requires single-line values everywhere; golden pins it);
|
||||
`link_in_index` for §6 appends (idempotent by `](target)` — exactly the primitive §6
|
||||
references); `parse_frontmatter` for stamp detection (`generated == "true"` string
|
||||
compare — it returns strings, never booleans); `promote_verdict` as the shape template
|
||||
(explicit required `timestamp`, no wall-clock default).
|
||||
- **Deliberately NOT reused:** `okf.write_concept_file` for the final write — it delegates
|
||||
to `Path.write_text` (platform newline translation risk) and does not guarantee exactly
|
||||
one trailing newline (`okf.py:164`). The ingest module builds each file's full content as
|
||||
a string and writes bytes (`encode("utf-8")`) for the §5 LF-only guarantee on CONCEPT
|
||||
files. Honest limit: index APPENDS reuse the frozen `link_in_index` (`Path.write_text`,
|
||||
`okf.py:180`), so `index.md` LF-ness is platform-scoped — nil on the LF-only
|
||||
development/target platform; recorded as Assumption 6.
|
||||
`contracts.py` is NOT imported: it imports `backends` → `agent_framework`
|
||||
(`contracts.py:25`), which would break the ingest module's MAF-free guard. Manifest
|
||||
models live in `ingest.py` itself.
|
||||
- **External tech (researched):** none — offline task, frozen spec, established repo
|
||||
patterns (brief Research Plan: 0 topics).
|
||||
- **Recent git activity:** I1 landed the spec + guards 1h before planning (`4df2140`,
|
||||
`7ba0fae`); `tests/test_method_spec_loadbearing.py` is freshly touched — extend, don't
|
||||
conflict. All work is sequential on `main`, clean tree. Commons subtree is PULL-ONLY
|
||||
(violation observed + cleaned 2026-07-03); **nothing in this plan writes under `shared/`**.
|
||||
- **Codebase-analysis correction (honesty):** the brief's Research Plan rationale mentions
|
||||
JSON-Schema validation; exploration verified `datasource.py` contains no schema
|
||||
validation — the fail-fast validation pattern lives in `contracts.py` as pydantic. The
|
||||
plan follows the pydantic pattern (which the brief's Preferences already mandate).
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
Every step is TDD (Iron Law): the step's tests are written and observed RED before the
|
||||
production code that turns them green. `okf.py` and everything under `shared/` are
|
||||
untouchable in every step (see per-step `forbidden_paths`). All commands run offline.
|
||||
|
||||
### Step 1: Fail-fast manifest contract (pydantic, polymorphic, verdict reservation)
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/ingest.py` (new), `tests/test_ingest_manifest.py` (new)
|
||||
- **Changes:** Create the ingest module with its narrative docstring (purpose, spec §
|
||||
references, MAF-free invariant, D7-portability) and the manifest contract per spec §4:
|
||||
`FileSource` (`type: Literal["file"]`, `root: str`), `SqlSource` (`type: Literal["sql"]`,
|
||||
`connection_ref: str`), `HttpSource` (`type: Literal["http"]`, `base_url: str` — reject
|
||||
embedded credentials (userinfo `@`) via validator, optional `credential_ref: str`), all
|
||||
with common `id: str` constrained to `^[a-z0-9][a-z0-9-]*$`; `source` as a discriminated
|
||||
union on `type` (`Field(discriminator="type")`). `Extraction`: `id` (same grammar),
|
||||
`title` (non-empty, single-line — reject `\n`/`\r`), `query: str`, `okf_type` (non-empty,
|
||||
single-line, `@field_validator` rejecting `verdict` case-insensitively — spec §3, before
|
||||
any source call), `max_rows: int = Field(gt=0)`. `ManifestV1`: `manifest_version:
|
||||
Literal[1]`, `source`, `bundle_summary: str` (non-empty), `extractions` (`min_length=1`,
|
||||
`@model_validator` rejecting duplicate extraction ids). Public loader
|
||||
`load_manifest(path: str | Path) -> tuple[ManifestV1, str]` that reads raw bytes ONCE,
|
||||
computes the `{stem}@{sha256(raw)[:16]}` stamp (spec §5), `json.loads`, validates —
|
||||
raising `pydantic.ValidationError` / `json.JSONDecodeError` before ANY source access.
|
||||
Define `IngestError(RuntimeError)` (condition-noun family; used by later steps for
|
||||
materialization-time refusals). `title` is whitespace-normalized at validation
|
||||
(`" ".join(title.split())` after the single-line check) so the frontmatter rendering
|
||||
(which collapses runs, `okf.py:154`) and the index label are guaranteed identical.
|
||||
The `HttpSource` credential validator is defined precisely: reject when
|
||||
`urllib.parse.urlsplit(base_url)` yields a non-`None` `username` or `password`
|
||||
(userinfo is THE credential-embedding URL mechanism §4 targets; query-param token
|
||||
heuristics are out of scope — documented in the validator docstring).
|
||||
**Baseline capture (SC7):** BEFORE writing anything in this step, run `uv run pytest -q`
|
||||
on the clean tree and record the summary line in the session log — this is the
|
||||
pre-existing-suite baseline SC7's "unchanged-green" is checked against. (new file)
|
||||
- **Reuses:** pydantic contract idiom from `contracts.py:30-96` (`Field` constraints,
|
||||
`Literal` enums, `@model_validator(mode="after")` returning `self`); SHA-256 `[:16]`
|
||||
idiom from `verdicts.py:95`; docstring style from `okf.py:1-12`.
|
||||
- **Test first:**
|
||||
- File: `tests/test_ingest_manifest.py` (new)
|
||||
- Verifies: valid `file` manifest loads with correct stamp (stem + 16-hex of raw bytes);
|
||||
each missing top-level field raises `ValidationError`; `manifest_version != 1` rejected;
|
||||
empty `extractions` rejected; bad `id` grammar (uppercase, leading `-`, empty) rejected
|
||||
for both source and extraction ids; duplicate extraction ids rejected; `max_rows <= 0`
|
||||
rejected; unknown `source.type` rejected; `file` source missing `root` rejected;
|
||||
`okf_type` of `verdict`/`Verdict`/`VERDICT` rejected; multi-line `title` rejected;
|
||||
`base_url` with embedded credentials rejected; **fail-fast ordering proof**: a manifest
|
||||
that is malformed AND whose `source.root` does not exist raises `ValidationError`
|
||||
without touching the missing root (the ingest analogue of
|
||||
`test_no_chat_client_call_on_malformed_contract` in `tests/test_contracts.py`);
|
||||
`sql`/`http` variants VALIDATE (schema breadth per brief assumption) though no
|
||||
connector executes them in I2.
|
||||
- Pattern: `tests/test_contracts.py` (inline dict constants, one `pytest.raises` per
|
||||
malformation)
|
||||
- **Verify:** `uv run pytest tests/test_ingest_manifest.py -q` → expected: all pass (after
|
||||
observed RED on first run without implementation)
|
||||
- **On failure:** revert — `rm -f src/portfolio_optimiser/ingest.py tests/test_ingest_manifest.py`
|
||||
(both files are NEW and untracked until this step's checkpoint — `git checkout --` cannot
|
||||
remove untracked files)
|
||||
- **Checkpoint:** `git commit -m "feat(ingest): fail-fast manifest contract with verdict reservation (I2)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/ingest.py
|
||||
- tests/test_ingest_manifest.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(ingest\\): fail-fast manifest contract with verdict reservation \\(I2\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/okf.py
|
||||
- shared/ingest-spec.md
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/ingest.py
|
||||
pattern: "load_manifest"
|
||||
- path: src/portfolio_optimiser/ingest.py
|
||||
pattern: "discriminator"
|
||||
```
|
||||
|
||||
### Step 2: CSV connector — boundary check, streaming row cap, cell escaping, table body
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_materialize.py` (new)
|
||||
- **Changes:** Add the `file` connector + renderer. Connector: resolve the manifest's
|
||||
`root` against the **manifest file's parent directory** when relative (pinned decision —
|
||||
extraction must not depend on cwd), then `safe_resolve(root, extraction.query)` for the
|
||||
fail-closed §4 boundary check; open with `encoding="utf-8-sig"` (BOM never leaks into
|
||||
the first header cell) and `newline=""`; parse with `csv.reader`. First row = header;
|
||||
fail (`IngestError`) on empty file (no header). Stream data rows counting against
|
||||
`max_rows` — raise `IngestError` the moment the cap is exceeded (§8: error, never silent
|
||||
truncation; no fetch-all-then-count). Fail on ragged rows (data row width ≠ header
|
||||
width) — silent padding/truncation is coercion. Renderer: cell text **verbatim** with
|
||||
escaping in this exact order: `\` → `\\` FIRST, then `|` → `\|`, then any newline (CR,
|
||||
LF, CRLF) → single space (pinned decision: on the `file`/CSV path every cell is a
|
||||
string, so §5's integer/float/NULL clauses bite typed `sql` values in I4 — a spec-text
|
||||
reading recorded as Assumption 1). Newline replacement treats CRLF as ONE unit: replace
|
||||
`\r\n` first, then lone `\r`, then lone `\n` — each with a single space (a per-character
|
||||
replace would turn CRLF into two spaces). A well-formed manifest whose `root` directory
|
||||
or resolved `query` CSV does not exist raises `IngestError` with a clear message (never a
|
||||
bare `FileNotFoundError` leaking from `open()`). Body = markdown table: `| h1 | h2 |`
|
||||
header row, `| --- | --- |` separator, data rows in source order, LF line endings, ending
|
||||
in exactly one `\n`. **Header cells are escaped identically to data cells** (spec §5
|
||||
separates "column names" from "cell values" without giving headers their own rule —
|
||||
pinned decision: same escaping/newline-collapse, or a header containing `|` or an
|
||||
embedded newline breaks the table). The connector itself is PURE (returns header +
|
||||
rows); the §8 source-call logging lives in `materialize` (Step 3), which owns
|
||||
`ingested_at`.
|
||||
- **Reuses:** `retrieval.safe_resolve`/`PathSecurityError` (`retrieval.py:63,24`) —
|
||||
identical fail-closed semantics `okf.py` itself relies on.
|
||||
- **Test first:**
|
||||
- File: `tests/test_ingest_materialize.py` (new)
|
||||
- Verifies: table rendering (header/separator/rows, source order); escaping of `\`,
|
||||
`|`, embedded quoted newline (one assertion each, plus one cell containing BOTH `\`
|
||||
and `|` proving escape order, plus an explicit embedded-CRLF cell → exactly ONE
|
||||
space); a header cell containing `|` and one containing an embedded newline →
|
||||
escaped/collapsed identically to data cells; header-only CSV → header + separator,
|
||||
zero data rows; empty CSV file → `IngestError`; ragged row → `IngestError`; `max_rows` fixture with cap+1 rows →
|
||||
`IngestError`; `../escape` query → `PathSecurityError`; missing `root` dir and
|
||||
missing `query` file → `IngestError` (not bare `FileNotFoundError`); BOM fixture →
|
||||
first header cell clean.
|
||||
- Pattern: local `tmp_path` CSV-builder helper (conftest's LLM fixtures are irrelevant
|
||||
to ingest — no conftest changes)
|
||||
- **Verify:** `uv run pytest tests/test_ingest_materialize.py -q` → expected: all pass
|
||||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py && rm -f tests/test_ingest_materialize.py`
|
||||
(`ingest.py` is tracked after Step 1's checkpoint; the test file is new in this step)
|
||||
- **Checkpoint:** `git commit -m "feat(ingest): CSV connector with boundary check, row cap and escaped table body (I2)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/ingest.py
|
||||
- tests/test_ingest_materialize.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(ingest\\): CSV connector with boundary check, row cap and escaped table body \\(I2\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/okf.py
|
||||
- tests/conftest.py
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/ingest.py
|
||||
pattern: "safe_resolve"
|
||||
- path: src/portfolio_optimiser/ingest.py
|
||||
pattern: "utf-8-sig"
|
||||
```
|
||||
|
||||
### Step 3: Materialization — provenance frontmatter, LF byte-writer, in-memory staging
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_materialize.py`
|
||||
- **Changes:** Add the per-extraction concept-file builder and the public entry point
|
||||
`materialize(manifest_path: str | Path, bundle_dir: str | Path, *, ingested_at: str) ->
|
||||
list[Path]` (explicit required `ingested_at` keyword, NO wall-clock default — mirrors
|
||||
`promote_verdict`; validate against the regex `^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$` —
|
||||
NOT `datetime.fromisoformat`, which rejects the `Z` suffix on Python 3.10, the repo's
|
||||
version floor — then stamp the string verbatim). This programmatic function IS the I2
|
||||
invocation surface (recorded decision, resolving the brief's open question: a `python -m`
|
||||
CLI is not demanded by the I2 verification set and is deferred). `materialize` performs
|
||||
the §8 source-call log per extraction on logger `portfolio_optimiser.ingest`: source id,
|
||||
the `ingested_at` argument (deterministic "when" — never wall-clock), row count; never
|
||||
log cell contents or (in later source types) resolved secrets. It creates the bundle
|
||||
directory first: `Path(bundle_dir).mkdir(parents=True, exist_ok=True)` — `safe_resolve`
|
||||
never creates directories, and unlike `okf.write_concept_file` the byte-writer has no
|
||||
implicit mkdir. Frontmatter: ordered dict with EXACTLY the §5 keys in §5 order — `type`
|
||||
(= `okf_type`), `title`, `source_system` (= `source.id`), `source_query` (the query,
|
||||
whitespace-collapsed), `ingested_at`, `ingest_manifest` (the Step-1 stamp), `generated`
|
||||
(`true`) — rendered via `okf.render_frontmatter` (insertion-ordered, single-lining).
|
||||
Filenames: `ingest-{extraction.id}.md` (§5; grammar keeps the namespace disjoint from
|
||||
`index.md`/`promoted-verdict-*` by construction). File content assembled as ONE string
|
||||
`---\n{fm}\n---\n\n{body}` (body already ends in exactly one `\n`) and written as bytes
|
||||
(`content.encode("utf-8")`) to a `safe_resolve`d path — LF-only and exactly one trailing
|
||||
newline guaranteed on every platform (§5). **All extractions execute and render in
|
||||
memory BEFORE the first disk mutation** (crash-window mitigation for §5's non-atomic
|
||||
replace sequence; recovery = idempotent re-run, §10).
|
||||
- **Reuses:** `okf.render_frontmatter` (`okf.py:146`), `retrieval.safe_resolve`;
|
||||
`promote_verdict`'s explicit-timestamp shape (`verdicts.py:465-472`).
|
||||
- **Test first:**
|
||||
- File: `tests/test_ingest_materialize.py` (extend)
|
||||
- Verifies: generated file's RAW TEXT has the 7 frontmatter keys in exact §5 order (read
|
||||
the file text, not just the parsed dict); `okf.parse_frontmatter` round-trips the
|
||||
values (`generated == "true"` as string; `ingest_manifest == {stem}@{hash16}` computed
|
||||
independently in the test from the manifest's raw bytes; `ingested_at` verbatim;
|
||||
`source_query` whitespace-collapsed); file bytes are LF-only with exactly one trailing
|
||||
`\n` (`read_bytes()` assertions: no `\r`, endswith exactly one `b"\n"`);
|
||||
missing/invalid `ingested_at` (no argument has no default — non-`Z`/non-ISO string
|
||||
raises `ValueError`; the golden timestamp `2026-07-03T12:00:00Z` passes the regex);
|
||||
materializing into a NON-EXISTENT nested `bundle_dir` succeeds (mkdir path); §8 log
|
||||
record (via `caplog`) carries source id, the explicit `ingested_at` string, and row
|
||||
count; two `materialize` runs with identical inputs produce byte-identical files
|
||||
(§10 idempotence at file level).
|
||||
- Pattern: `tests/test_okf.py` frontmatter round-trip assertions
|
||||
- **Verify:** `uv run pytest tests/test_ingest_materialize.py -q` → expected: all pass
|
||||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_materialize.py`
|
||||
- **Checkpoint:** `git commit -m "feat(ingest): deterministic materialization with §5/§7 provenance stamp (I2)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/ingest.py
|
||||
- tests/test_ingest_materialize.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(ingest\\): deterministic materialization with .5/.7 provenance stamp \\(I2\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/okf.py
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/ingest.py
|
||||
pattern: "ingested_at"
|
||||
- path: src/portfolio_optimiser/ingest.py
|
||||
pattern: "ingest_manifest"
|
||||
```
|
||||
|
||||
### Step 4: Index generation + stamped-replacement semantics
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_materialize.py`
|
||||
- **Changes:** Complete `materialize` with §3/§5/§6 bundle semantics. (a) **Ownership
|
||||
scan:** classify a bundle file as ingest-owned iff `okf.parse_frontmatter` yields
|
||||
`generated == "true"` AND an `ingest_manifest` key (string comparisons; sorted directory
|
||||
scan for determinism). (b) **Collision gate:** if a target filename `ingest-{id}.md`
|
||||
exists WITHOUT the stamp → `IngestError`, nothing written (§3: never overwrite curated
|
||||
content). (c) **Replace:** delete every stamped file, then write the new staged set.
|
||||
(d) **Index:** if `index.md` missing → create it with `bundle_summary` as the body (no
|
||||
frontmatter — spec-minimal, pinned decision; content = `{bundle_summary}\n` written as
|
||||
bytes). If present → preserve every unmanaged line byte-for-byte; remove ONLY
|
||||
whole lines matching the managed-link form `- [<label>](<target>)` whose `<target>` is a
|
||||
previously-stamped file deleted in this run and NOT re-created (exact full-line
|
||||
target match, never bare-substring — a promoted verdict's or curated link must be
|
||||
unreachable by this filter). (e) **Link:** `okf.link_in_index(bundle_dir,
|
||||
"ingest-{id}.md", title)` per extraction in manifest order (idempotent by target: links
|
||||
for re-created targets keep their existing position — byte-stability across re-ingest).
|
||||
(f) **Label refresh:** for a re-created target whose existing managed line's label no
|
||||
longer equals the (normalized) extraction `title`, rewrite THAT line in place —
|
||||
`- [{new title}](ingest-{id}.md)` — preserving its position; §6 defines the label AS
|
||||
the title, so a title change must never leave a stale label behind. The refresh uses
|
||||
the SAME anchoring as the removal filter: only an exact full-line managed form
|
||||
`- [<label>](ingest-{id}.md)` targeting THIS run's ingest files is rewritten;
|
||||
non-conforming lines (curated prose mentioning the target inline, links to non-ingest
|
||||
files) are untouched. Index update is the LAST disk mutation.
|
||||
- **Reuses:** `okf.link_in_index` (`okf.py:168` — the exact primitive spec §6 cites),
|
||||
`okf.parse_frontmatter` (`okf.py:31`).
|
||||
- **Test first:**
|
||||
- File: `tests/test_ingest_materialize.py` (extend)
|
||||
- Verifies: fresh dir → `index.md` created with `bundle_summary` body + one link per
|
||||
extraction in manifest order; existing `index.md` with a curated line → line preserved
|
||||
byte-for-byte after materialization; re-materialization with an extraction REMOVED
|
||||
from the manifest → its file gone AND its index link gone, other links intact;
|
||||
re-materialization with a CHANGED extraction `title` (same id) → the managed link's
|
||||
label updated in place, position preserved (no stale label); unstamped
|
||||
`ingest-foo.md` present → `IngestError` and bundle unmodified (assert directory bytes
|
||||
unchanged); re-materialization twice with identical inputs → byte-identical bundle
|
||||
including `index.md` (§10).
|
||||
- Pattern: `tests/test_okf.py::test_link_in_index_makes_concept_file_navigable`
|
||||
- **Verify:** `uv run pytest tests/test_ingest_materialize.py -q` → expected: all pass
|
||||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_materialize.py`
|
||||
- **Checkpoint:** `git commit -m "feat(ingest): index generation and stamped-replacement semantics (I2)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/ingest.py
|
||||
- tests/test_ingest_materialize.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(ingest\\): index generation and stamped-replacement semantics \\(I2\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/okf.py
|
||||
- src/portfolio_optimiser/verdicts.py
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/ingest.py
|
||||
pattern: "link_in_index"
|
||||
- path: src/portfolio_optimiser/ingest.py
|
||||
pattern: "bundle_summary"
|
||||
```
|
||||
|
||||
### Step 5: Load-bearing seam quartet + MAF-free guard (detach-RED proofs)
|
||||
|
||||
- **Files:** `tests/test_ingest_loadbearing.py` (new)
|
||||
- **Changes:** The durable seam tests per spec §11's table, in the repo's load-bearing
|
||||
idiom (module docstring naming each seam + its detach-RED condition; scenarios built so
|
||||
the seam is the ONLY thing preventing the observable artifact). (1) **Verdict
|
||||
reservation:** an otherwise fully-valid manifest with `okf_type: "Verdict"` →
|
||||
`ValidationError` at load, zero files/links written (RED if the case-insensitive
|
||||
reservation validator is removed). Plus the defensive namespace invariant: the id
|
||||
grammar + `ingest-` prefix make `promoted-verdict-*`/`index.md` collisions structurally
|
||||
unreachable — asserted as invariant, with the honest note that its load-bearing bite is
|
||||
the `okf_type` check. (2) **Provenance stamping:** materialize, then assert the §7 layer
|
||||
on every generated file via UNCHANGED `okf.parse_frontmatter` (RED if the stamp layer is
|
||||
dropped). (3) **Navigability via UNCHANGED okf.py** (the durable version of the planning
|
||||
session's ad-hoc proof): `okf.navigate_bundle` reaches every `ingest-{id}.md` through
|
||||
index cross-links; each is classified by its `okf_type`; provenance fields ride through
|
||||
`frontmatter` (unknown-field preservation); `okf.bundle_context` renders the generated
|
||||
content (RED if index linking detaches — file unreachable). (4) **Re-ingest layer
|
||||
safety — with the removal filter ACTIVE:** on a `shutil.copytree` copy of
|
||||
`shared/examples/bygg-energi-mikro/`, run `materialize` (manifest with ≥2 extractions),
|
||||
then the REAL `verdicts.promote_verdict` (approved verdict, explicit timestamp), then
|
||||
`materialize` AGAIN with a REDUCED manifest (one extraction dropped) — the §6
|
||||
managed-line removal filter must actually FIRE while the promoted link and the bundle's
|
||||
curated managed-form links are present (an identical-manifest second run never removes
|
||||
anything, so it cannot catch a loose filter — the green-but-dead trap): the dropped
|
||||
extraction's file AND index link are gone; the promoted verdict file and its index link
|
||||
survive verbatim; every curated link survives verbatim; only stamped files were
|
||||
replaced (RED if the filter over-matches — `promote_verdict` writes exactly the
|
||||
managed-line shape a bare-substring filter would wrongly delete — or if replacement
|
||||
stops honouring the stamp ownership rule). (5) **MAF-free guard:** AST-parse
|
||||
`src/portfolio_optimiser/ingest.py` (pattern: `tests/test_okf.py::test_okf_is_maf_free`)
|
||||
asserting no import roots in `{agent_framework, mcp}` AND no `portfolio_optimiser.*`
|
||||
imports outside `{okf, retrieval}` (transitively stdlib-pure — `contracts`/`backends`
|
||||
would smuggle MAF in). **Execution rule:** each of (1)–(4) is proven RED by temporarily
|
||||
detaching its seam (comment out the validator / stamp keys / `link_in_index` call / the
|
||||
stamp filter), observing the failure, reattaching; the detach-proof log (seam → detached
|
||||
how → observed failure) goes into the session verification log for STATE/commit message.
|
||||
- **Reuses:** `_copy_bundle` copytree idiom (`tests/test_step8_promotion_loadbearing.py:51`),
|
||||
AST guard (`tests/test_okf.py:165-181`), real `promote_verdict` (`verdicts.py:465`).
|
||||
- **Test first:**
|
||||
- File: `tests/test_ingest_loadbearing.py` (new — this whole step IS tests; the seams it
|
||||
guards were built in Steps 1–4, so these must pass immediately and the detach-proofs
|
||||
demonstrate they can fail)
|
||||
- Verifies: the four §11 seams + module purity, as above
|
||||
- Pattern: `tests/test_step8_promotion_loadbearing.py` (docstring naming detach-RED per test)
|
||||
- **Verify:** `uv run pytest tests/test_ingest_loadbearing.py -q` → expected: all pass;
|
||||
then 4 documented detach→RED→reattach cycles each showing ≥1 failure
|
||||
- **On failure:** escalate — a quartet test that cannot go RED at detach means the seam is
|
||||
not actually load-bearing; stop and redesign the test before proceeding (do not weaken
|
||||
the assertion). Cleanup on abandon: `rm -f tests/test_ingest_loadbearing.py` (new file —
|
||||
untracked until this step's checkpoint)
|
||||
- **Checkpoint:** `git commit -m "test(ingest): load-bearing seam quartet + MAF-free guard, detach-proven (I2)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- tests/test_ingest_loadbearing.py
|
||||
min_file_count: 1
|
||||
commit_message_pattern: "^test\\(ingest\\): load-bearing seam quartet \\+ MAF-free guard, detach-proven \\(I2\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/okf.py
|
||||
- shared/examples/bygg-energi-mikro/index.md
|
||||
must_contain:
|
||||
- path: tests/test_ingest_loadbearing.py
|
||||
pattern: "promote_verdict"
|
||||
- path: tests/test_ingest_loadbearing.py
|
||||
pattern: "navigate_bundle"
|
||||
```
|
||||
|
||||
### Step 6: Spec §12 field cross-check (discharge the I1-recorded obligation)
|
||||
|
||||
- **Files:** `tests/test_method_spec_loadbearing.py`
|
||||
- **Changes:** Add `test_ingest_spec_documents_every_contract_field`, mirroring the
|
||||
existing `test_spec_documents_every_contract_field` (line 106): iterate the REAL pydantic
|
||||
models' `model_fields` (ManifestV1, recursively into the three source variants and
|
||||
Extraction) plus the seven §5 frontmatter keys plus the four golden-case entries
|
||||
(`manifest.json`, `fixture/`, `ingested-at.txt`, `expected-bundle/`), asserting each
|
||||
token appears in `shared/ingest-spec.md` (the §12 cross-check table's completeness,
|
||||
test-enforced at field level). This discharges the deferred note recorded at
|
||||
`tests/test_method_spec_loadbearing.py:156-162` ("arrives with the I2 contracts").
|
||||
RED when contract code and spec drift apart. Surgical: only ADD the test — the I1
|
||||
structure/neutrality guards in the file are untouched.
|
||||
- **Reuses:** the file's own test-3 pattern (`model_fields`-driven spec assertion).
|
||||
- **Test first:**
|
||||
- File: `tests/test_method_spec_loadbearing.py` (existing)
|
||||
- Verifies: every machine-readable contract field is documented in the frozen spec; a
|
||||
hypothetical new model field without a spec entry turns it RED (detach-proof: add a
|
||||
dummy field to a throwaway copy check — verified by temporarily adding a fake field
|
||||
name to the iteration list, observing RED, removing)
|
||||
- Pattern: `tests/test_method_spec_loadbearing.py::test_spec_documents_every_contract_field`
|
||||
- **Verify:** `uv run pytest tests/test_method_spec_loadbearing.py -q` → expected: all pass
|
||||
- **On failure:** escalate — a genuine field↔spec mismatch means either the contract
|
||||
deviates from the frozen spec (fix the contract) or the spec is defective (STOP: spec
|
||||
changes are a separate GATED commons round, never folded into I2)
|
||||
- **Checkpoint:** `git commit -m "test(spec-guard): ingest contract field cross-check vs spec §12 (I2)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- tests/test_method_spec_loadbearing.py
|
||||
min_file_count: 1
|
||||
commit_message_pattern: "^test\\(spec-guard\\): ingest contract field cross-check vs spec .12 \\(I2\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- shared/ingest-spec.md
|
||||
- shared/method-spec.md
|
||||
must_contain:
|
||||
- path: tests/test_method_spec_loadbearing.py
|
||||
pattern: "test_ingest_spec_documents_every_contract_field"
|
||||
```
|
||||
|
||||
### Step 7: Golden extraction case + byte-for-byte regression test
|
||||
|
||||
- **Files:** `examples/ingest-golden-file/` (new directory), `tests/test_ingest_golden.py` (new)
|
||||
- **Changes:** Author the golden case per spec §11 layout, LOCALLY in this repo (brief
|
||||
Non-Goal: commons sharing is a later gated round; `shared/` is untouchable). New
|
||||
top-level `examples/` directory (does not exist yet; outside ruff's `src` config —
|
||||
fixture data, not lint targets): `manifest.json` (file source, `root: "fixture"`
|
||||
resolved against the manifest's directory, 2 extractions: a plain cost table AND an
|
||||
edge-case table whose quoted CSV cells exercise `\`, `|`, an embedded newline, a cell
|
||||
with both `\` and `|`, AND number-shaped cells that DISCRIMINATE the text-verbatim
|
||||
reading — `007` and `1.50`, strings whose numeric normalization differs byte-wise
|
||||
(`7`, `1.5`); Assumption 1 is only pinned if a number-normalizing implementation CANNOT
|
||||
reproduce the golden — a cell like `30000.0` discriminates nothing since
|
||||
`repr(30000.0)` round-trips identically), `fixture/*.csv`, `ingested-at.txt`
|
||||
(`2026-07-03T12:00:00Z`, one line). `expected-bundle/` is generated ONCE by the (now fully behaviour-tested)
|
||||
implementation, its content REVIEWED against the spec §5/§6/§7 rules by hand, then
|
||||
frozen in git (brief Assumption 4: golden = regression pin; correctness is carried by
|
||||
Steps 1–6's tests; the freeze happens LAST). Golden test: read `ingested-at.txt`, run
|
||||
`materialize` into `tmp_path`, then assert (a) the produced file SET equals the
|
||||
`expected-bundle/` file set (catches extra AND missing files) and (b) every file's
|
||||
`read_bytes()` matches exactly. A second `materialize` run over the same output must
|
||||
leave every byte unchanged (golden-level idempotence, §10).
|
||||
- **Reuses:** `BUNDLE_DIR`-style module-level fixture-path idiom (`tests/test_okf.py`,
|
||||
`tests/test_step8_promotion_loadbearing.py`).
|
||||
- **Test first:**
|
||||
- File: `tests/test_ingest_golden.py` (new — written BEFORE `expected-bundle/` exists;
|
||||
observed RED against the empty expectation, then the reviewed freeze turns it green)
|
||||
- Verifies: byte-for-byte golden regression + file-set equality + repeat-run idempotence
|
||||
- Pattern: `tests/test_okf.py` fixture-path idiom
|
||||
- **Verify:** `uv run pytest tests/test_ingest_golden.py -q` → expected: all pass; then the
|
||||
full gate battery: `uv run pytest` (pre-existing suite unchanged-green vs the Step-1
|
||||
baseline + all new tests), `uv run ruff check .` → 0 issues,
|
||||
`uv run ruff format --check .` → clean, `uv run mypy src` → 0 errors; honesty grep
|
||||
(brief SC8 — scoped to PROSE per the brief: docstrings/comments of the new module and
|
||||
new test files, plus any touched docs):
|
||||
`grep -rinE "sql|http|live|commons" src/portfolio_optimiser/ingest.py tests/test_ingest_*.py` →
|
||||
every PROSE hit is a defer-mention (I4/I6/extension point/gated commons round) or a
|
||||
spec citation; hits that are polymorphic-contract SCHEMA IDENTIFIERS (`SqlSource`,
|
||||
`HttpSource`, `Literal["sql"]`, `Literal["http"]`, `connection_ref`, `base_url`,
|
||||
`credential_ref`) are expected and allowed — they validate variants per §4 and claim
|
||||
no delivered connector support
|
||||
- **On failure:** escalate — a golden mismatch after Steps 1–6 are green means a
|
||||
determinism defect (platform newline, ordering, escaping); diagnose root cause before
|
||||
re-freezing — NEVER re-freeze to make a mismatch disappear. Cleanup on abandon:
|
||||
`rm -rf examples/ tests/test_ingest_golden.py` (new, untracked until checkpoint)
|
||||
- **Checkpoint:** `git commit -m "test(ingest): golden extraction case examples/ingest-golden-file, byte-frozen (I2)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- examples/ingest-golden-file/manifest.json
|
||||
- examples/ingest-golden-file/ingested-at.txt
|
||||
- examples/ingest-golden-file/fixture/costs.csv
|
||||
- examples/ingest-golden-file/expected-bundle/index.md
|
||||
- tests/test_ingest_golden.py
|
||||
min_file_count: 5
|
||||
commit_message_pattern: "^test\\(ingest\\): golden extraction case examples/ingest-golden-file, byte-frozen \\(I2\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/okf.py
|
||||
- shared/ingest-spec.md
|
||||
must_contain:
|
||||
- path: tests/test_ingest_golden.py
|
||||
pattern: "read_bytes"
|
||||
- path: examples/ingest-golden-file/ingested-at.txt
|
||||
pattern: "2026-07-03T12:00:00Z"
|
||||
```
|
||||
|
||||
## Execution Strategy
|
||||
|
||||
Seven steps, strictly sequential — every step builds on the previous step's code and the
|
||||
session's operating model is single-session TDD (one session = one closed delivery). No
|
||||
parallel waves.
|
||||
|
||||
### Session 1: I2 complete (this session)
|
||||
- **Steps:** 1, 2, 3, 4, 5, 6, 7
|
||||
- **Wave:** 1
|
||||
- **Depends on:** none
|
||||
- **Scope fence:**
|
||||
- Touch: `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_*.py`,
|
||||
`tests/test_ingest_golden.py`, `tests/test_method_spec_loadbearing.py` (additive only),
|
||||
`examples/ingest-golden-file/**`
|
||||
- Never touch: `src/portfolio_optimiser/okf.py`, `src/portfolio_optimiser/run.py`,
|
||||
`src/portfolio_optimiser/verdicts.py`, `src/portfolio_optimiser/contracts.py`,
|
||||
`shared/**`, `tests/conftest.py`, `pyproject.toml` (no new deps needed)
|
||||
|
||||
### Execution Order
|
||||
- **Wave 1:** Session 1 (sequential steps 1→7)
|
||||
|
||||
### Grouping rules applied
|
||||
- All steps share `src/portfolio_optimiser/ingest.py` → one session by necessity.
|
||||
- TDD dependency chain (contract → connector → materializer → semantics → seams → guard →
|
||||
golden freeze) forbids reordering.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
| Approach | Pros | Cons | Why rejected |
|
||||
|----------|------|------|--------------|
|
||||
| Add a link-removal primitive to `okf.py` | Symmetric API next to `link_in_index` | Violates the brief's hard constraint (navigability criterion is "consumable via UNCHANGED okf.py"); widens D7-shared surface mid-program | Brief Constraints; removal lives in `ingest.py` |
|
||||
| Define manifest models in `contracts.py` | One contracts home | `contracts.py:25` imports `backends` → `agent_framework`; ingest module would fail its MAF-free guard; also violates surgical-scope | Brief Constraints (MAF-free ingest seam) |
|
||||
| Reuse `okf.write_concept_file` for file writes | Maximum reuse, path-safety built in | `Path.write_text` translates newlines per platform and doesn't guarantee exactly one trailing `\n` — §5 LF-only byte-determinism at risk | Risk assessment (High); ingest writes bytes itself, still via `safe_resolve` + `render_frontmatter` |
|
||||
| Validate only the `file` source variant in I2 | Smaller Step 1 | Spec §4's schema is polymorphic; I4 would retrofit the union; brief assumption already commits to schema breadth with connector-narrowness | Brief Assumption 1 (schema polymorphic, connector `file`-only) |
|
||||
| Hand-author `expected-bundle/` before implementing | "Purer" golden TDD | Byte-level hand-typing of hash stamps and escaping is error-prone; correctness is already carried by Steps 1–6 behaviour tests; golden is a regression PIN (spec §11) | Brief Assumption 4 (generate once, review, freeze LAST) |
|
||||
| Remove + re-append index links for re-created targets | Simpler §6 filter | Moves ingest links to index end whenever anything sits between; unnecessary byte-churn breaks repeat-run stability | Idempotence (§10): keep links in place for re-created targets — WITH the Step-4(f) in-place label refresh, so a changed title never leaves a stale label (plan-critic major finding) |
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- **Framework:** pytest ≥8, plain sync tests (ingest is deterministic — no asyncio), no new
|
||||
markers, no conftest changes, no mocking library (repo convention: hand-built doubles;
|
||||
ingest needs only `tmp_path` + small CSV builders).
|
||||
- **Existing patterns:** fail-fast contract tests (`tests/test_contracts.py`), detach-RED
|
||||
load-bearing modules (`tests/test_step8_promotion_loadbearing.py`), navigability
|
||||
round-trips (`tests/test_okf.py`), spec cross-check (`tests/test_method_spec_loadbearing.py`).
|
||||
- **New tests in this plan:** ~30 tests across 4 new files + 1 extended file.
|
||||
|
||||
### Tests to write
|
||||
|
||||
| Type | File | Verifies | Model test |
|
||||
|------|------|----------|------------|
|
||||
| Unit | `tests/test_ingest_manifest.py` | §4 schema fail-fast, verdict reservation, grammar, fail-before-source-access | `tests/test_contracts.py` |
|
||||
| Unit | `tests/test_ingest_materialize.py` | connector boundary/cap/escaping, §5 frontmatter order, LF bytes, §6 index semantics, collision gate, idempotence | `tests/test_okf.py` |
|
||||
| Load-bearing | `tests/test_ingest_loadbearing.py` | the four §11 seams + MAF-free module guard, each detach-RED-proven | `tests/test_step8_promotion_loadbearing.py` |
|
||||
| Spec guard | `tests/test_method_spec_loadbearing.py` | §12 field-level contract↔spec cross-check | its own test 3 |
|
||||
| Golden | `tests/test_ingest_golden.py` | byte-for-byte regression + file-set equality + repeat-run idempotence | `tests/test_okf.py` fixture idiom |
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Priority | Risk | Location | Impact | Mitigation |
|
||||
|----------|------|----------|--------|------------|
|
||||
| Critical | §6 link removal implemented as substring match deletes a promoted verdict's or curated link | new removal filter in `ingest.py`; `okf.py:168-181` is append-only | Corrupted `index.md`; §11 re-ingest seam broken | Exact whole-line managed-link match, only for targets deleted this run and not re-created (Step 4); re-ingest safety test uses the REAL `promote_verdict` (Step 5) |
|
||||
| Critical | Crash mid-replace leaves half-materialized bundle (§5 sequence is non-atomic) | `materialize` orchestration | Loop navigates a broken bundle until re-run | Full in-memory staging before first disk mutation; index update last; recovery = idempotent re-run (§10) — documented, not hidden |
|
||||
| High | Reformatting numeric-looking CSV text (e.g. `"30000.0"`, `"007"`) breaks golden bytes | renderer, spec §5 reading | Golden divergence; sibling (I3) mismatch | Pinned decision: CSV cells text-verbatim; §5 number rules are typed-`sql` territory (I4). Recorded as Assumption 1; golden fixture pins it |
|
||||
| High | Escape-order bug (`|` before `\`) double-escapes | renderer | Silent cell corruption | Fixed order `\`→`\\` then `|`→`\|`; both-chars cell in unit test AND golden fixture |
|
||||
| High | Platform newline translation / trailing-newline drift breaks byte-determinism | file writes | Golden mismatch cross-platform | Bytes writer with explicit `\n` for concept files (Step 3); `read_bytes()` assertions; `okf.write_concept_file` not used for writes. Index appends stay on the frozen `link_in_index` → LF-ness there is platform-scoped (Assumption 6), nil on macOS/Linux |
|
||||
| Medium | BOM/CRLF/embedded-newline CSV mishandling | connector | First header cell polluted; golden flaky | `utf-8-sig` + `newline=""` + `csv.reader` (pinned); BOM + CRLF + embedded-newline unit fixtures |
|
||||
| Medium | Stamp misclassification (string `"true"` vs bool) deletes curated file or strands stale ingest files | ownership scan; `okf.py:46` returns strings | §3 violation | Classify on `generated == "true"` AND `ingest_manifest` present; collision gate test (unstamped `ingest-foo.md` → error, bundle unmodified) |
|
||||
| Medium | `link_in_index` raises when `index.md` missing | `okf.py:176` | Mid-run crash on fresh bundle | Create `index.md` (bundle_summary body) before any linking (Step 4 order) |
|
||||
| Medium | Title containing `](x.md)` injects a parseable index link | index label = raw title; `okf.py:28` | Spurious navigation targets (skipped if missing — fail-closed) | Spec-exact validation only (single-line); limitation documented in module docstring — spec change would be a gated commons round |
|
||||
| Low | Golden re-frozen to hide a real determinism defect | Step 7 | Regression pin becomes a lie | Step 7 On-failure rule: diagnose root cause; NEVER re-freeze to green a mismatch |
|
||||
|
||||
## Assumptions
|
||||
|
||||
| # | Assumption | Why unverifiable | Impact if wrong |
|
||||
|---|-----------|-----------------|-----------------|
|
||||
| 1 | §5's integer/float/NULL rendering rules apply to typed (`sql`) values; `file`/CSV cells are text-verbatim (escape-only) | Spec prose lists both under one bullet; the frozen spec cannot be amended inside I2 | Golden bytes + sibling (I3) diverge; if I3 reads it differently, a GATED commons clarification round is triggered — the golden fixture makes the chosen reading explicit and shareable |
|
||||
| 2 | Fresh `index.md` is created WITHOUT frontmatter (body = `bundle_summary` only) — spec §6 is silent on index frontmatter | Spec silence; curated example (`bygg-energi-mikro/index.md`) has `type: index` but is hand-authored, not generated | Cosmetic divergence pinned by the golden; `navigate_bundle` handles both (frontmatter-less parse returns `{}`) — verified `okf.py:38-39` |
|
||||
| 3 | Relative `root` resolves against the manifest file's directory | Spec §4 does not name the resolution base | Golden becomes cwd-dependent if wrong; pinned decision documented in docstring + spec cross-check exercise for I3 |
|
||||
| 4 | `utf-8-sig` reading (BOM tolerated and stripped) is the correct connector-side choice | Spec silent on encoding | A BOM-carrying golden fixture in the sibling could differ; our golden contains no BOM in `fixture/`, so the shared surface is unaffected |
|
||||
| 5 | Empty CSV (no header), ragged rows, and missing `root`/`query` paths are `IngestError` fail-fast (never silent coercion/leaked `FileNotFoundError`) | Spec §5's MUST-fail clause addresses typed values; these input-shape failures are unaddressed spec silence | I3 sibling could choose differently; the decisions are documented in the module docstring and exercised by unit tests so the I3 cross-check surfaces any divergence — if the sibling disagrees, a GATED commons clarification round resolves it |
|
||||
| 6 | LF-only bytes are platform-scoped for `index.md`: concept files go through the ingest byte-writer (LF everywhere), but index appends REUSE the frozen `okf.link_in_index`, whose `Path.write_text` translates newlines per platform (and `read_text` universal-newline-decodes a CRLF curated index on round-trip) | The brief mandates reusing `link_in_index` and forbids changing `okf.py` — the platform behaviour is frozen with it | On the LF-only development/target platform (macOS/Linux) impact is nil and the golden holds; a Windows deployment would need its own golden verification — documented limitation, not a universal cross-platform claim |
|
||||
|
||||
*6 items — above the template's 3-item investigation threshold, accepted deliberately:
|
||||
all are spec-SILENCE or platform-scope points (not unknowns about this codebase) pinned
|
||||
by the golden fixture and/or documented decisions — exactly the mechanism the program
|
||||
uses to surface such choices to I3; none blocks execution.*
|
||||
|
||||
## Verification
|
||||
|
||||
*Per-step manifests verify automatically during execution. These are the end-to-end
|
||||
checks crossing step boundaries — they restate the brief's Success Criteria (SC1–SC8).*
|
||||
|
||||
- [ ] SC1: `uv run pytest tests/test_ingest_golden.py -q` → all pass (byte-for-byte +
|
||||
repeat-run idempotence)
|
||||
- [ ] SC2–SC5: `uv run pytest tests/test_ingest_loadbearing.py -q` → all pass, AND the
|
||||
session log records four detach→RED→reattach proofs (provenance, navigability,
|
||||
verdict reservation, re-ingest safety)
|
||||
- [ ] SC3 (unchanged okf.py): `git diff main@{session-start} -- src/portfolio_optimiser/okf.py`
|
||||
→ empty output
|
||||
- [ ] SC6: `uv run pytest tests/test_ingest_manifest.py -q` → all pass with no network
|
||||
available (key assumption 2 closed)
|
||||
- [ ] SC7: `uv run pytest` → exit 0, pre-existing suite unchanged-green vs the baseline
|
||||
recorded in Step 1 (pre-change `uv run pytest -q` summary in the session log) + new
|
||||
tests; `uv run ruff check .` → exit 0; `uv run ruff format --check .` → clean;
|
||||
`uv run mypy src` → exit 0
|
||||
- [ ] SC8: `grep -rinE "sql|http|live|commons" src/portfolio_optimiser/ingest.py tests/test_ingest*.py`
|
||||
→ every PROSE hit (docstrings/comments — the brief's scope) is a defer-mention
|
||||
(I4/I6/extension point/gated) or spec citation; polymorphic-contract schema
|
||||
identifiers (`SqlSource`/`HttpSource`/`Literal["sql"|"http"]`/`connection_ref`/
|
||||
`base_url`/`credential_ref`) are allowed hits claiming no delivered support
|
||||
- [ ] Layer hygiene: `uv run pytest tests/test_okf.py tests/test_method_spec_loadbearing.py -q`
|
||||
→ all pass (MAF-free + spec guards intact)
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Files to modify:** 1 (`tests/test_method_spec_loadbearing.py`, additive)
|
||||
- **Files to create:** 5 code/test files (`src/portfolio_optimiser/ingest.py`,
|
||||
`tests/test_ingest_manifest.py`, `tests/test_ingest_materialize.py`,
|
||||
`tests/test_ingest_loadbearing.py`, `tests/test_ingest_golden.py`) + the
|
||||
`examples/ingest-golden-file/` fixture tree (~6 small files)
|
||||
- **Complexity:** medium
|
||||
|
||||
## Plan Quality Score
|
||||
|
||||
| Dimension | Weight | Score | Notes |
|
||||
|-----------|--------|-------|-------|
|
||||
| Structural integrity | 0.15 | 90 | strict TDD dependency chain; Step-2/3 logging ownership clarified post-review |
|
||||
| Step quality | 0.20 | 88 | 1 module + 1 test file per step; all 3 critic majors fixed in revision |
|
||||
| Coverage completeness | 0.20 | 88 | all 8 SCs + §11 seams mapped; title-refresh, fresh-dir, missing-source cases added post-review |
|
||||
| Specification quality | 0.15 | 88 | 5 spec-silence points pinned as explicit assumptions |
|
||||
| Risk & pre-mortem | 0.15 | 88 | risk table incl. 2 critical collisions; review added revert/mkdir/staleness fixes |
|
||||
| Headless readiness | 0.10 | 85 | On-failure clauses corrected for untracked files; baseline capture now explicit in Step 1 |
|
||||
| Manifest quality | 0.05 | 88 | Step-7 manifest now pins the byte-frozen golden artifacts |
|
||||
| **Weighted total** | **1.00** | **88** | **Grade: B+** |
|
||||
|
||||
**Adversarial review:**
|
||||
- **Plan critic:** APPROVE_WITH_NOTES (81/100, grade B) — 0 blockers, 3 major, 10 minor;
|
||||
all 3 majors and 9 of 10 minors resolved in the revision below (the remaining minor —
|
||||
SC4's filename-namespace clause being structurally covered — is accepted as honest
|
||||
by-construction coverage, matching scope-guardian's reading).
|
||||
- **Scope guardian:** ALIGNED — 0 creep, 0 dependency issues, 3 gaps (1 major: SC8 grep
|
||||
scope; 2 minor: baseline step, SC4 structural note); the major and the baseline gap are
|
||||
resolved below.
|
||||
|
||||
## Revisions
|
||||
|
||||
*Added by adversarial review (plan-critic + scope-guardian, deduped 16 findings).*
|
||||
|
||||
| # | Finding | Severity | Resolution |
|
||||
|---|---------|----------|------------|
|
||||
| 1 | Step 1 On-failure `git checkout --` cannot remove untracked new files | major | Step 1 revert → `rm -f`; Step 2 revert split (tracked checkout + `rm` new test file); Steps 5/7 got explicit cleanup-on-abandon commands |
|
||||
| 2 | Changed extraction `title` on re-ingest left a stale index label (kept-in-place links) | major | Step 4 gained (f) in-place label refresh preserving line position + a changed-title test; Alternatives row updated |
|
||||
| 3 | `materialize` wrote bytes without creating `bundle_dir` (byte-writer has no mkdir) | major | Step 3 now mkdirs `bundle_dir` first + fresh-nested-dir test added |
|
||||
| 4 | SC8 whole-file grep unachievable — schema identifiers (`SqlSource` etc.) would false-positive | major | SC8 (Step 7 + Verification) rescoped to PROSE hits per the brief, with schema identifiers as an explicitly allowed category |
|
||||
| 5 | `ingested_at` validation via `fromisoformat` breaks on py3.10 (`Z` unsupported) | minor | Pinned to an explicit regex; py3.10 trap documented in Step 3 |
|
||||
| 6 | Missing `root`/`query` leaked bare `FileNotFoundError` | minor | Wrapped in `IngestError` + tests added (Step 2) |
|
||||
| 7 | `render_frontmatter` collapses title whitespace → frontmatter/label divergence | minor | `title` whitespace-normalized at validation so both representations agree (Step 1) |
|
||||
| 8 | CRLF could become two spaces | minor | CRLF replaced as one unit (`\r\n` first) + explicit CRLF test (Step 2) |
|
||||
| 9 | Step 2 logged `ingested_at` it does not own | minor | Connector is pure; §8 logging moved to `materialize` (Step 3) with the caplog test |
|
||||
| 10 | `HttpSource` credential validator under-defined | minor | Defined as `urlsplit` userinfo check; query-param heuristics explicitly out of scope (Step 1) |
|
||||
| 11 | Ragged/empty-file decisions unpinned vs I3 sibling | minor | Added as Assumption 5 (documented + unit-tested divergence surface) |
|
||||
| 12 | No baseline-capture step for SC7 | minor | Step 1 now opens with a pre-change `uv run pytest -q` baseline recording; SC7 references it |
|
||||
| 13 | Step 7 manifest omitted the byte-frozen golden artifacts | minor | `expected_paths` now includes `fixture/costs.csv` + `expected-bundle/index.md`, `min_file_count: 5` |
|
||||
| 14 | Invocation surface (API vs CLI) unstated | minor | Recorded in Step 3: programmatic `materialize` IS the I2 surface; CLI deferred |
|
||||
| 15 | SC4 filename-namespace clause covered by construction, not detach-testable | minor | Accepted as-is: honestly documented invariant (both reviewers concur); load-bearing bite is the `okf_type` check |
|
||||
|
||||
## Adversarial Pass 2 (independent second pass, v5.1.1 high-effort)
|
||||
|
||||
*Mandated by the brief's `effort: high` plan signal. The prescribed `gemini-bridge` pass
|
||||
was SUBSTITUTED with an independent fresh-context contrarian reviewer: Gemini deep-research
|
||||
result retrieval is verified broken as of 2026-07-03 (research completes, result
|
||||
unretrievable — operator memory), so a bridge run would burn time with no retrievable
|
||||
output. The substitute preserves the intent — an independent perspective that does not
|
||||
re-tread Pass 1's dedup space (it was given Pass 1's findings as excluded ground).*
|
||||
|
||||
Verdict: **PASS2_FINDINGS(5)** — 0 blocker, 3 major, 2 minor. All five incorporated:
|
||||
|
||||
| # | Finding | Severity | Resolution |
|
||||
|---|---------|----------|------------|
|
||||
| P2-1 | Re-ingest safety test ran `materialize` twice with an IDENTICAL manifest — the §6 removal filter never fired, so the plan's own Critical risk (loose filter deleting a promoted/curated link) was untested (green-but-dead) | major | Step 5(4) rewritten: second run uses a REDUCED manifest so removal fires with the promoted verdict + curated managed-form links present; over-match → RED |
|
||||
| P2-2 | Golden fixture had no cell discriminating text-verbatim vs number-normalization — Assumption 1 was not actually pinned (`30000.0` round-trips identically) | major | Step 7 fixture now mandates `007` and `1.50` cells whose normalized forms differ byte-wise |
|
||||
| P2-3 | Index writes route through the frozen `link_in_index` (`Path.write_text`) — the "LF-only on every platform" claim contradicted the plan's own reason for avoiding `write_concept_file` | major | Determinism claim rescoped: concept files LF-guaranteed via byte-writer; `index.md` LF-ness platform-scoped (nil on macOS/Linux) — Assumption 6 + risk-row update |
|
||||
| P2-4 | Header-cell escaping unspecified (spec §5 separates "column names" from "cell values") — a `|`/newline header would break the table | minor | Step 2: headers escaped identically to data cells (pinned) + header edge-case tests |
|
||||
| P2-5 | Step 4(f) label refresh did not restate the removal filter's full-line/ingest-target anchoring | minor | Step 4(f) now requires the same exact managed-line anchoring; non-conforming lines untouched |
|
||||
|
||||
Probed and clean (per the reviewer): §12 cross-check test CAN go red; file-deletion
|
||||
ownership seam holds against real `promote_verdict`; no invariant violations (okf.py
|
||||
unchanged, `shared/` untouched, MAF-free, no new deps); escape order correct; golden
|
||||
`tmp_path` second run does exercise re-ingest-over-existing.
|
||||
|
|
@ -1,181 +0,0 @@
|
|||
---
|
||||
type: trekbrief
|
||||
brief_version: "2.2"
|
||||
created: 2026-07-03
|
||||
task: "S11 — Sammenligningen + rapport: begge stacker målt på pinned commons-ref etter S3-protokollen; norsk forretningsrapport med verifiseringslogg"
|
||||
slug: s11-sammenligning-rapport
|
||||
project_dir: .claude/projects/2026-07-03-s11-sammenligning-rapport/
|
||||
research_topics: 0
|
||||
research_status: skipped
|
||||
auto_research: false
|
||||
interview_turns: 11
|
||||
source: interview
|
||||
framing: preserve
|
||||
phase_signals:
|
||||
- phase: research
|
||||
effort: high
|
||||
model: fable
|
||||
- phase: plan
|
||||
effort: high
|
||||
model: fable
|
||||
- phase: execute
|
||||
effort: high
|
||||
model: fable
|
||||
- phase: review
|
||||
effort: high
|
||||
model: fable
|
||||
---
|
||||
|
||||
# Task: S11 — Sammenligningen + rapport (program-leveransen)
|
||||
|
||||
> Generated by `/trekbrief` on 2026-07-03.
|
||||
> This brief is the contract between requirements and planning. `/trekplan`
|
||||
> reads it to produce the implementation plan. Every decision in the plan must
|
||||
> trace back to content in this brief.
|
||||
|
||||
## TL;DR
|
||||
|
||||
S11 er sesjonsplanens neste steg, uendret (framing: preserve): mål begge stacker
|
||||
(MAF her + Claude SDK-søskenet) på identisk pinned commons-ref `db86e15` etter den
|
||||
forhåndsbundne S3-protokollen (M1–M4), sammenstill med S10-artefaktene i søskenets
|
||||
`runs/s10/`, og skriv én norsk forretningsrapport med verifiseringslogg der hver
|
||||
tallpåstand har reproduserende kommando/kilde. Ingen kode-endringer i noen stack.
|
||||
|
||||
## Intent
|
||||
|
||||
S11 er program-SLUTTLEVERANSEN (målbilde §1): en rettferdig sammenligning av to
|
||||
uavhengige implementasjoner av samme metode — MAF-stacken i dette repoet og Claude
|
||||
Agent SDK-søskenet i `~/repos/portfolio-optimiser-claude` — på identisk delt kjerne.
|
||||
Resultatet dokumenteres i en norsk forretningsrapport som operatøren kan dele.
|
||||
Troverdigheten hviler på to ting: målestokken (sammenligningsprotokollen fra S3) ble
|
||||
definert FØR begge stacker fantes, slik at ingen stack kunne forme metrikken etter
|
||||
eget resultat, og hver tallpåstand i rapporten bærer sin egen reproduserende
|
||||
kommando/kilde (verifiseringslogg). Operatøren lever av tillit; rapporten må derfor
|
||||
være nøktern, uten salgsspråk, og aldri påstå mer enn bevisene bærer.
|
||||
|
||||
## Goal
|
||||
|
||||
Når S11 er ferdig finnes én norsk forretningsrapport i dette repoet
|
||||
(`docs/rapport/2026-07-03-sammenligningsrapport.md`) som måler begge stacker på
|
||||
pinned commons-ref `db86e15` etter protokollens fire metrikker (M1 validator-samsvar
|
||||
mot golden, M2 konvergens/runder, M3 tokenforbruk mot tak, M4 kvalitativ vurdering),
|
||||
gjengir liveness-asymmetri-erklæringen ordrett, sammenstiller S10-artefaktene fra
|
||||
søskenets `runs/s10/` (kun Claude-siden merket som live-observasjon), og har en
|
||||
verifiseringslogg der HVER tallpåstand peker på kommandoen/kilden som reproduserer
|
||||
den. Begge repo er verifisert grønne (pytest + ruff + mypy) på pinned ref som del av
|
||||
økta, og MAF offline-sim + D7 offline-speil er kjørt på samme ref som tallkilde for
|
||||
M1–M3 (offline-kategorien).
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **Ingen re-kjøring av S10.** Artefaktene i `runs/s10/` er fasit (protokoll §4.2/§4.5);
|
||||
re-kjøring er kun tillatt ved teknisk transportfeil, som ikke foreligger — kjøringen
|
||||
lyktes 2026-07-03.
|
||||
- **Ingen S12-/release-arbeid** (LICENSE, `[project.scripts]`, CHANGELOG-polering) —
|
||||
gated på operatør-enighet, egen økt.
|
||||
- **Ingen endringer i delt kjerne/commons og ingen re-pinning** — `db86e15` står.
|
||||
(Endres kjernen måtte begge sider re-pinnes og re-kjøres; det gjør vi ikke.)
|
||||
- **Ingen nye features eller refaktorering i noen av stackene** — S11 måler det som ER.
|
||||
- **Ingen statistiske påstander om modellatferd** — S10 er én kjøring; den beviser
|
||||
eksistens, ikke tendens (protokoll §4.1).
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Protokollen er normativ og fryst i økta:** `docs/plan/2026-07-03-sammenligningsprotokoll.md`
|
||||
kan ikke justeres under S11 — endring ville ugyldiggjøre «definert før begge
|
||||
stacker»-premisset som bærer rapportens troverdighet.
|
||||
- **Kostnadsdisiplin D6:** alt i S11 kjøres offline; null API-spend.
|
||||
- **Språk:** rapporten er norsk (forretningsdokument, jf. dokumentspråk-preferansen);
|
||||
kode, kommandoer og filnavn i verifiseringsloggen er engelsk som de er.
|
||||
- **Kategori-disiplin (protokoll §2):** M2/M3-tall fra offline-kjøringer sammenlignes
|
||||
KUN med offline-tall på motparten; kryss-kategori-sammenligning (offline vs live)
|
||||
er forbudt i rapporten.
|
||||
- **Skrivearbeid skjer i DETTE repoet.** Søskenrepoet er read-only i S11; eneste
|
||||
unntak er STATE.md-oppdatering der ved sesjonslutt (local-only fil).
|
||||
- **Push til Forgejo per driftsmodellen.** Push-frysen fra 2026-07-03 er observert
|
||||
opphevet (søskenets S10-commits er pushet; Forgejo 15.0.3 oppe) — dette repoets
|
||||
stale STATE.md korrigeres ved sesjonslutt.
|
||||
|
||||
## Preferences
|
||||
|
||||
- Rapport plasseres under `docs/rapport/` i dette repoet (ny katalog er akseptabelt).
|
||||
- Metrikk-tall i tabellform; begrunnelse og tolkning i prosa rundt tabellene.
|
||||
- Nøkternt språk uten superlativer — «fullt ut», «uten unntak» o.l. er røde flagg
|
||||
(verifiseringsplikten).
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
- **Reproducerbarhet:** hver M1–M3-tallrad i verifiseringsloggen skal kunne kjøres på
|
||||
nytt av en leser med begge repo klonet, og gi samme tall (deterministisk kjerne +
|
||||
fangede artefakter).
|
||||
- **Ærlighet:** ingen påstand i rapporten (eller i README/STATE etterpå) skal påstå
|
||||
mer enn koden/bevisene gjør — spesielt ingen antydning om live-atferd på MAF-siden.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- **SC1 — Rapport med ordrett erklæring:** `docs/rapport/2026-07-03-sammenligningsrapport.md`
|
||||
finnes, og liveness-asymmetri-erklæringen fra protokoll §3 gjengis ordrett —
|
||||
verifiserbart ved at blokksitatets tekst diff-er tomt mot protokollens §3-blokk.
|
||||
- **SC2 — Begge repo grønne på pinned ref:** `uv run pytest`, `uv run ruff check .`
|
||||
og `uv run mypy src` avslutter alle med exit 0 i BEGGE repo, og commons-ref er
|
||||
verifisert identisk på begge sider (subtree-commits fra `db86e15` — `git log
|
||||
--grep="db86e15"` viser squash-committen i begge).
|
||||
- **SC3 — Offline-kjøringer utført på pinned ref:** MAF offline-sim
|
||||
(`uv run python -m portfolio_optimiser.simulation`) og søskenets D7 offline-speil
|
||||
er kjørt i økta; M1–M3-tallene i rapporten stammer fra disse kjøringene og fra de
|
||||
committede S10-artefaktene (`runs/s10/*.json`) — aldri fra hukommelse.
|
||||
- **SC4 — Verifiseringslogg komplett:** hver tallpåstand i rapporten har en rad i
|
||||
verifiseringsloggen med reproduserende kommando/kilde; stikkprøve: hver M1–M3-rad
|
||||
kan kjøres på nytt og gir samme tall.
|
||||
- **SC5 — Kategori- og liveness-disiplin:** rapporten inneholder ingen påstand om
|
||||
live-atferd på MAF-siden og ingen kryss-kategori-sammenligning — kontrollert
|
||||
eksplisitt mot protokoll §2/§4 i review-fasen.
|
||||
|
||||
## Research Plan
|
||||
|
||||
No external research needed — the codebase and this brief contain sufficient
|
||||
context for planning. (Alle kilder er lokale og committet: begge repo, protokollen,
|
||||
sesjonsplanen §S11, `shared/method-spec.md`, S10-artefaktene i søskenets `runs/s10/`.
|
||||
Lokal kodeoppdagelse — f.eks. nøyaktig invokasjon av D7-offline-speilet i søskenrepoet —
|
||||
håndteres av /trekplan-utforskningen.)
|
||||
|
||||
## Open Questions / Assumptions
|
||||
|
||||
- **[ASSUMPTION] «D7 offline-speil»** antas å være søskenrepoets offline to-runs-bevis
|
||||
(sim-ekvivalenten fra S9-laget); eksakt kjørekommando avdekkes i plan-utforskningen
|
||||
av søskenrepoet — den er ikke navngitt i sesjonsplanen.
|
||||
- **[ASSUMPTION] Push-frysen er opphevet** — avledet av at søskenets S10-commits er
|
||||
pushet til Forgejo (0/0 mot origin) og at Forgejo 15.0.3 svarer. Om antakelsen er
|
||||
feil, parkeres push og operatør varsles; commits skjer uansett lokalt.
|
||||
- **[ASSUMPTION] Dette repoets STATE.md er stale** (sier «S6 er neste»); søskenets
|
||||
STATE (S10 fullført, S11 neste, arbeidssted = dette repoet) er autoritativ og lagt
|
||||
til grunn. Korrigeres ved sesjonslutt.
|
||||
|
||||
## Prior Attempts
|
||||
|
||||
None — fresh task. (Målestokken ble bevisst definert i S3, før begge stacker fantes;
|
||||
det er en forutsetning, ikke et tidligere forsøk.)
|
||||
|
||||
## Metadata
|
||||
|
||||
- **Created:** 2026-07-03
|
||||
- **Interview turns:** 11
|
||||
- **Auto-research opted in:** no
|
||||
- **Source:** trekbrief interview
|
||||
|
||||
---
|
||||
|
||||
## How to continue
|
||||
|
||||
Manual (default):
|
||||
|
||||
```bash
|
||||
# No research topics — go straight to plan:
|
||||
/trekplan --project .claude/projects/2026-07-03-s11-sammenligning-rapport/
|
||||
|
||||
# Then execute:
|
||||
/trekexecute --project .claude/projects/2026-07-03-s11-sammenligning-rapport/
|
||||
```
|
||||
|
||||
Auto (opt-in during `/trekbrief`): research and planning run
|
||||
automatically; only execution is manual.
|
||||
|
|
@ -1,443 +0,0 @@
|
|||
# S11 — Sammenligningen + rapport (program-leveransen)
|
||||
|
||||
> **Plan quality: B** (pending adversarial review) — APPROVE_WITH_NOTES pending
|
||||
>
|
||||
> Generated by trekplan v5.9.0 on 2026-07-03 — `plan_version: 1.7`
|
||||
|
||||
## Context
|
||||
|
||||
S11 er program-SLUTTLEVERANSEN (målbilde §1, briefens Intent): en rettferdig sammenligning av to
|
||||
uavhengige implementasjoner av samme metode — MAF-stacken i dette repoet og Claude Agent
|
||||
SDK-søskenet i `/Users/ktg/repos/portfolio-optimiser-claude` — på identisk delt kjerne, dokumentert
|
||||
i én norsk forretningsrapport med verifiseringslogg. Troverdigheten hviler på at målestokken
|
||||
(`docs/plan/2026-07-03-sammenligningsprotokoll.md`, S3) ble bundet FØR søskenrepoet fantes og FØR
|
||||
ekstraksjonen (S4/S5) — PRESIST det protokoll-headeren selv påstår, og ikke mer: MAF-implementasjonen
|
||||
forelå da protokollen ble skrevet, så pre-bindingen beskytter mot at CLAUDE-siden former målestokken
|
||||
etter eget resultat; denne residual-asymmetrien ERKLÆRES i rapporten (Adversarial Pass 2, B1) — og på
|
||||
at hver tallpåstand bærer sin egen reproduserende kommando/kilde. Rapporten skal være nøktern, uten
|
||||
salgsspråk, og aldri påstå mer enn bevisene bærer (briefens NFR «Ærlighet»; method-spec §1).
|
||||
|
||||
Protokollen er normativ og fryst i økta (briefens Constraint 1). Alt kjøres offline — null
|
||||
API-spend (Constraint 2, D6). Søskenrepoet er read-only: ingen tracked-fil endres der; transiente,
|
||||
gitignorerte artefakter fra testkjøring (`.pytest_cache/`, `__pycache__/`) tillates, operasjonalisert
|
||||
som identisk `git status --porcelain` før/etter (brief-review-funn a, avklart av risk-assessor).
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Delt kjerne (pinned db86e15, tre-hash fc58b0b9)"
|
||||
SHARED["shared/ — 13 filer<br/>golden.json + validator-input.json<br/>method-spec.md + OKF-bundle + persona"]
|
||||
end
|
||||
subgraph "MAF-repo (dette)"
|
||||
SIM["portfolio_optimiser.simulation<br/>(offline to-runs-bevis, stdout)"]
|
||||
MGOLD["tests/test_bygg_energi_mikro.py<br/>(M1: golden-frys)"]
|
||||
MSUITE["uv run pytest / ruff / mypy (SC2)"]
|
||||
end
|
||||
subgraph "SDK-søsken (read-only)"
|
||||
SGOLD["tests/test_bygg_energi_mikro.py<br/>(M1 + mutasjonskontroller)"]
|
||||
SMIRROR["tests/test_step7/8_*_loadbearing.py<br/>(offline-speil = testsuiten)"]
|
||||
S10["runs/s10/*.json<br/>(S10-artefakter = fasit, live)"]
|
||||
end
|
||||
SHARED --> SIM & MGOLD & SGOLD & SMIRROR
|
||||
SIM & MGOLD & MSUITE --> RAPPORT["docs/rapport/2026-07-03-sammenligningsrapport.md<br/>(M1–M4 + ordrett §3-erklæring + verifiseringslogg)"]
|
||||
SGOLD & SMIRROR & S10 --> RAPPORT
|
||||
```
|
||||
|
||||
## Codebase Analysis
|
||||
|
||||
- **Tech stack:** Python ≥3.10, `uv`, pytest/ruff/mypy begge repo. MAF-repo: `agent-framework-core`;
|
||||
søsken: `claude-agent-sdk`. MAF-venv Python 3.12.13; søsken-venv 3.14.0 (skew OK: begge
|
||||
validatorer bruker kun `random.Random(20260624).uniform(...)`, versjonsstabil sekvens; golden-tester
|
||||
grønne på hver side — python-versjonene loggføres i rapporten).
|
||||
- **Key patterns:** load-bearing tester (detach-bevis) begge repo; golden-suite som eneste fasit
|
||||
(seed 20260624, MAF `validator.py:41`, søsken `validator.py:25`); dokumentkonvensjon
|
||||
`YYYY-MM-DD-slug.md`, norsk forretningsprosa, verifiseringslogg-tabell (presedens
|
||||
`docs/2026-06-26-fot-i-bakken.md`).
|
||||
- **Relevant files (verifisert i utforskningen):**
|
||||
- MAF: `src/portfolio_optimiser/simulation.py` (294 LOC, `main()` :249; skriver INGEN artefakter —
|
||||
kun stdout + temp-katalog `po-sim-*`; exit 0 = løkke lukket), `budget.py` (`TokenMeter` :66–77),
|
||||
`run.py` (:317 `token_usage`), `validator.py`, `shared_root.py` (env `PORTFOLIO_SHARED_ROOT`),
|
||||
`tests/test_bygg_energi_mikro.py::test_validator_output_matches_golden` (:81–90).
|
||||
- Søsken (read-only): `tests/test_step7_async_loop_loadbearing.py` + `tests/test_step8_promotion_loadbearing.py`
|
||||
(= «D7 offline-speil»; INGEN kjørbar sim-modul finnes), `tests/test_bygg_energi_mikro.py`
|
||||
(golden + 2 mutasjonskontroller MAF mangler), `runs/s10/{proposal,provenance,run_result,usage}.json`
|
||||
(fasit, committet samlet i `0f8af04`), `src/portfolio_optimiser_claude/run_s10.py` (:46–53
|
||||
defaults = §4.3-konfigen; MÅ ALDRI kjøres i S11 — ville overskrive fasit + bruke API-penger).
|
||||
- Protokoll/plan: `docs/plan/2026-07-03-sammenligningsprotokoll.md` (M1–M4 :24–27, §3-erklæring,
|
||||
§4-regler), `docs/plan/2026-07-02-sesjonsplan.md` :183–193 (S11-kriterier).
|
||||
- **Reusable code:** ingen ny kode — S11 er kjøringer + dokument. Verbatim-sjekken gjenbruker
|
||||
test-strategistens python-oneliner; pin-sjekken gjenbruker `git rev-parse HEAD:shared`.
|
||||
- **External tech (researched):** ingen (0 research-topics, brief bekreftet).
|
||||
- **Recent git activity:** begge repo rene og pushet. MAF tip `aec95eb` (kun ny plan-fil for
|
||||
fremtidig ingest-lag — ortogonal til S11, verifisert `git show --stat`). Søsken tip `0f8af04`
|
||||
(eneste commit som rører `runs/s10/`). Commons-pin: MAF squash `7916e10`
|
||||
(«from 7d2b46c..db86e15»), søsken squash `97a9172` («content from commit db86e15»,
|
||||
`git-subtree-split: db86e159148d7d8553a38516e174370cf913f889`).
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
Ingen produksjonskode endres; «test first» erstattes av **Verify** med eksakte kommandoer
|
||||
(mal-tillatt for prosjekter der steget ikke er kode). Evidence-filer legges i prosjektkatalogens
|
||||
`evidence/`, som gjøres selv-ignorerende i Step 1 (`evidence/.gitignore` med innhold `*` —
|
||||
`.claude/projects/` er ellers IKKE gitignored, verifisert med `git check-ignore`); kommandoene i
|
||||
verifiseringsloggen er den reproduserbare kilden for lesere. Alle capture-kommandoer i Steps 1–3
|
||||
fanger stderr (`2>&1`) slik at feilmeldinger og EXIT-linjer havner i evidence-filene.
|
||||
|
||||
### Step 1: Preflight — pinned-ref-bevis og miljø-ground-truth
|
||||
|
||||
- **Files:** `.claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/01-preflight.txt` (new file)
|
||||
- **Changes:** Opprett først selv-ignorerende evidence-katalog:
|
||||
`mkdir -p evidence && printf '*\n' > evidence/.gitignore` (gjør at `git status --porcelain`
|
||||
forblir uendret av bevisfangsten). Fang deretter (alt med `2>&1`):
|
||||
(1) `git -C <begge repo> status --porcelain` + `rev-parse HEAD` (rene trær, tips dokumentert);
|
||||
(2) pin-identitet med `PIN: `-prefiks per linje:
|
||||
`git -C /Users/ktg/repos/portfolio-optimiser rev-parse 'HEAD:shared' 'db86e15^{tree}' | sed 's/^/PIN: /'`
|
||||
og samme i søskenet, PLUSS den TEMPORALE sjekken (Adversarial Pass 2, time-axis):
|
||||
`git -C /Users/ktg/repos/portfolio-optimiser-claude rev-parse '0f8af04:shared' | sed 's/^/PIN: /'`
|
||||
— beviser at shared/-treet ved S10-artefakt-committen er samme tre som dagens pin (kjernen endret
|
||||
seg ikke mellom live-kjøringen og sammenligningen). Alle FEM PIN-linjer skal vise
|
||||
`fc58b0b956e5f9871b29810f17284250e794abd5`;
|
||||
(3) `diff -rq /Users/ktg/repos/portfolio-optimiser/shared /Users/ktg/repos/portfolio-optimiser-claude/shared`
|
||||
(tom) + `find shared -type f | wc -l` (= 13); (4) `echo "PORTFOLIO_SHARED_ROOT=[$PORTFOLIO_SHARED_ROOT]"`
|
||||
(skal være tom — runtime-identitet, risk-assessors residual); (5) python-versjoner:
|
||||
`uv run --frozen python -V` i begge repo. (basert på kodebase-analyse)
|
||||
- **Reuses:** `git rev-parse HEAD:shared`-sjekken (test-strategist, verifisert i utforskningen);
|
||||
`shared_root.py`-sømmen dokumenterer env-varen.
|
||||
- **Verify:** `grep -c '^PIN: fc58b0b956e5f9871b29810f17284250e794abd5$' evidence/01-preflight.txt` → expected: `5` (prefikset teller kun rev-parse-output, ikke annoteringer; diff-linjen tom)
|
||||
- **On failure:** ulik tre-hash → escalate (pinned-ref-premisset brutt; re-pinning er eksplisitt Non-Goal — operatør må avgjøre). Satt `PORTFOLIO_SHARED_ROOT` → unset i sesjonens miljø, re-fang punkt (4), loggfør hendelsen i evidence; escalate kun hvis den ikke lar seg unsette
|
||||
- **Checkpoint:** ingen commit (read-only bevissteg; evidence er selv-gitignored)
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- .claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/.gitignore
|
||||
- .claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/01-preflight.txt
|
||||
min_file_count: 2
|
||||
commit_message_pattern: ""
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- shared/README.md
|
||||
- docs/plan/2026-07-03-sammenligningsprotokoll.md
|
||||
must_contain:
|
||||
- path: .claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/01-preflight.txt
|
||||
pattern: "PIN: fc58b0b956e5f9871b29810f17284250e794abd5"
|
||||
```
|
||||
|
||||
### Step 2: MAF-side kjøringer — gates (SC2) + offline-sim (SC3, M2/M3 offline)
|
||||
|
||||
- **Files:** `.claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/02-maf-gates.txt` (new file), `.claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/03-maf-sim.txt` (new file)
|
||||
- **Changes:** I MAF-repoet, fang til 02 (hver kommando med `2>&1`): `uv run --frozen pytest`
|
||||
(noter passed/skipped-tall og NAVNGI de 4 gatede live-skippene — «157 grønne» alene er upresist),
|
||||
`uv run ruff check .`, `uv run ruff format --check .`, `uv run mypy src`. Deretter fang til 03:
|
||||
`uv run --frozen python -m portfolio_optimiser.simulation 2>&1; echo "EXIT=$?"` — stdout-tracen ER
|
||||
M2/M3-kilden på MAF-siden (RUN A/PROMOTE/RUN B-linjer, «LEARNING LOOP CLOSED», syntetiske
|
||||
token-tellinger 8/reply, debatt kuttet av GroupChatOrchestrator ved max_rounds=3). Ferske tall,
|
||||
aldri STATE-prosa (premiss-verifisering). (basert på kodebase-analyse)
|
||||
- **Reuses:** `simulation.py` uendret; `--frozen` beskytter `uv.lock` (risk-assessor).
|
||||
- **Verify:** `grep -E "LEARNING LOOP CLOSED|EXIT=0" evidence/03-maf-sim.txt | wc -l` → expected: `2`
|
||||
- **On failure:** escalate — rød gate/åpen løkke på pinned ref er et rapport-nivå-funn operatøren må se før rapporten skrives
|
||||
- **Checkpoint:** ingen commit (kjøringer + selv-gitignored evidence; `git status --porcelain` skal være uendret)
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- .claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/02-maf-gates.txt
|
||||
- .claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/03-maf-sim.txt
|
||||
min_file_count: 2
|
||||
commit_message_pattern: ""
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/simulation.py
|
||||
- uv.lock
|
||||
must_contain:
|
||||
- path: .claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/03-maf-sim.txt
|
||||
pattern: "LEARNING LOOP CLOSED"
|
||||
```
|
||||
|
||||
### Step 3: Søsken-side gates (SC2/SC3) under read-only-protokoll
|
||||
|
||||
- **Files:** `.claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/04-sibling-gates.txt` (new file)
|
||||
- **Changes:** Fang (alt med `2>&1`): (1) status FØR med markør-prefiks:
|
||||
`git -C /Users/ktg/repos/portfolio-optimiser-claude status --porcelain | sed 's/^/BEFORE:/'`;
|
||||
(2) `PYTHONDONTWRITEBYTECODE=1 uv run --frozen --directory /Users/ktg/repos/portfolio-optimiser-claude pytest -p no:cacheprovider`
|
||||
(= «D7 offline-speil» kjørt: to-runs-beviset `test_step7_async_loop_loadbearing.py` +
|
||||
promoterings-beviset `test_step8_promotion_loadbearing.py` + golden `test_bygg_energi_mikro.py`
|
||||
inngår — noter ferskt passed-tall, forvent ~187); (3) ruff + mypy med søskenets egne konfigurerte
|
||||
kommandoer (`uv run --frozen --directory … ruff check .` og `… mypy src` — strictness styres av
|
||||
dets pyproject, per dets CLAUDE.md «mypy --strict rene»); (4) status ETTER med markør-prefiks:
|
||||
`git -C … status --porcelain | sed 's/^/AFTER:/'` — identisk med FØR (read-only-beviset, inn i
|
||||
verifiseringsloggen). ALDRI kjør `src/portfolio_optimiser_claude/run_s10.py` (overskriver
|
||||
fasit-artefaktene + ekte API-kost — farligste feilmodus). (basert på kodebase-analyse)
|
||||
- **Reuses:** søskenets egen testsuite uendret; cache-fri invokasjon fra test-strategist.
|
||||
- **Verify:** `bash -c 'diff <(grep "^BEFORE:" evidence/04-sibling-gates.txt | sed "s/^BEFORE://") <(grep "^AFTER:" evidence/04-sibling-gates.txt | sed "s/^AFTER://")'` → expected: tom diff, exit 0 (før/etter-status identisk; BSD-kompatibel — ingen `head -n -1`) og pytest-linjen viser `0 failed`
|
||||
- **On failure:** retry — én gang med `uv sync --frozen` først (miljø-drift); deretter escalate hvis fortsatt rødt
|
||||
- **Checkpoint:** ingen commit (ingen tracked filer endres — i noen av repoene)
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- .claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/04-sibling-gates.txt
|
||||
min_file_count: 1
|
||||
commit_message_pattern: ""
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: .claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/04-sibling-gates.txt
|
||||
pattern: "passed"
|
||||
```
|
||||
|
||||
### Step 4: Fasit-ekstraksjon — S10-artefakter (M2/M3 live) + M4-målinger
|
||||
|
||||
- **Files:** `.claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/05-s10-artefakter.txt` (new file), `.claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/06-m4.txt` (new file)
|
||||
- **Changes:** Til 05: `cat` av de fire `runs/s10/*.json` + `git -C <søsken> log --format="%H %cI %s" -- runs/s10/`
|
||||
(skal vise KUN `0f8af04`). Tall som siteres i rapporten leses HERFRA (36 791/150 000 tokens,
|
||||
2/12 runder, attempts 1, $0.127514, modell `claude-haiku-4-5-20251001`) — aldri fra STATE-prosa.
|
||||
§4.3-konfigen = defaultene i `src/portfolio_optimiser_claude/run_s10.py` (:46–53).
|
||||
M1-hovedpåstanden (live-utfallet reproduserer goldens persentiler bit-for-bit) BEVISES med denne
|
||||
kommandoen, som også gjengis i verifiseringsloggen:
|
||||
```
|
||||
python3 - <<'EOF'
|
||||
import json, pathlib
|
||||
g = json.loads(pathlib.Path("shared/examples/bygg-energi-mikro/golden.json").read_text())["validator"]
|
||||
r = json.loads(pathlib.Path("/Users/ktg/repos/portfolio-optimiser-claude/runs/s10/run_result.json").read_text())["outcome"]
|
||||
for k in ("p10", "p50", "p90"):
|
||||
assert g[k] == r[k], (k, g[k], r[k])
|
||||
print("OK: golden == run_result (bit-for-bit p10/p50/p90)")
|
||||
EOF
|
||||
```
|
||||
Til 06 (M4-råstoff, kommando per tall — FERSKE tall, forhåndstall fra utforskningen
|
||||
(~2 696 LOC/18 moduler vs ~1 696 LOC/15 moduler) er kun forventninger): LOC/moduler
|
||||
`find src -name '*.py' -exec wc -l {} +` i begge repo; load-bearing-inventar
|
||||
`ls tests/test_*loadbearing*.py` begge (+ søskenets `test_sdk_isolation.py`, mutasjonskontrollene
|
||||
MAF mangler); method-spec §12-tabellen som spec-troskaps-sjekkliste. (basert på kodebase-analyse)
|
||||
- **Reuses:** artefaktskjemaene kartlagt av dep-tracer (usage/run_result/provenance/proposal-felter).
|
||||
- **Verify:** `grep -c '"cost_usd": 0.127514' evidence/05-s10-artefakter.txt` → expected: `1`; M1-kommandoen over → `OK: golden == run_result (bit-for-bit p10/p50/p90)`
|
||||
- **On failure:** escalate — avvik mellom artefaktfil og forventet verdi betyr at fasit-premisset har driftet
|
||||
- **Checkpoint:** ingen commit (read-only ekstraksjon)
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- .claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/05-s10-artefakter.txt
|
||||
- .claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/06-m4.txt
|
||||
min_file_count: 2
|
||||
commit_message_pattern: ""
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: .claude/projects/2026-07-03-s11-sammenligning-rapport/evidence/05-s10-artefakter.txt
|
||||
pattern: "claude-haiku-4-5-20251001"
|
||||
```
|
||||
|
||||
### Step 5: Skriv rapporten, kjør mekaniske sjekker, commit + push
|
||||
|
||||
- **Files:** `docs/rapport/2026-07-03-sammenligningsrapport.md` (new file — ny katalog `docs/rapport/`, konsistent med sjanger-per-underkatalog)
|
||||
- **Changes:** Skriv rapporten på norsk etter konvensjons-hybriden: `> **Hva dette er:**`-blockquote →
|
||||
`## Hovedfunn` tidlig → nummererte seksjoner → verifiseringslogg-tabell sist. Innhold, bundet av
|
||||
protokollen OG Adversarial Pass 2-funnene:
|
||||
(1) **Bakgrunn/metode** — protokollens temporale status PRESIST (bundet før S4/S5, ETTER at
|
||||
MAF-implementasjonen forelå — residual-asymmetrien erklæres, B1); **interessekonflikt-erklæring**
|
||||
(forfatter er Anthropic-fokusert AI-rådgiver; begge implementasjoner + rapport skrevet med Claude —
|
||||
leverandøren av den ene komparanden; én setning, COI); pinned ref db86e15 med tre-hash-identiteten
|
||||
fra Step 1 INKL. den temporale (S10-commit-treet = dagens pin), python-versjoner, dato.
|
||||
(2) **M1 validator-samsvar** — RAMMES som EKVIVALENS-GATE, ikke differensierende metrikk (begge
|
||||
validatorer er ren python; golden ble generert av spec §7.2-referanseprosedyren fra MAF-linjen —
|
||||
generatoren NAVNGIS, så MAF-siden er selv-samsvar og kun Claude-siden er uavhengig reproduksjon).
|
||||
Begge sider grønne mot golden på pinned ref (pytest-node-id-er som kilde); søskenets live
|
||||
run_result reproduserer goldens persentiler bit-for-bit (Step 4-kommandoen som bevis); MAF-simens
|
||||
tall er IKKE golden-tall (skriptet forslag uten assumptions → degenerert MC, p90=90000) og siteres
|
||||
aldri som M1-kilde. Ekvivalens-påstanden SCOPES til det ene benchmark-bundelet (bygg-energi-mikro).
|
||||
(3) **M2/M3 — offline-kategori (BEGGE sider):** skriptede tall presenteres som DESIGNPARAMETRE +
|
||||
boolske håndhevelses-utfall (tak håndhevet ja/nei, refinement-sti utløst ja/nei) — ALDRI som målinger
|
||||
(forfatter-valgte inputs, synthetic≠measured); MAF-sim-tracen (løkke lukket, forsøk avledet av
|
||||
generation-prompt-telling) og søskenets offline-speil (testsuite — emitterer ikke runde-tall; sies
|
||||
eksplisitt). **Live- og offline-tall i SEPARATE tabeller** (ikke én tabell med kategorikolonne).
|
||||
(4) **S10 — live-kategori (KUN Claude-siden, merket):** **den feilede FØRSTE kjøringen rapporteres
|
||||
FØRSTEKLASSES med lik prominens** (B2) — protokoll §4.5 lister «tak nådd» som innholdsutfall som
|
||||
skal rapporteres, og unntaket dekker kun transportfeil; §4.5 SITERES, kategoriseringen
|
||||
(harness-/konfig-feil, operatør-godkjent re-kjøring etter fiks) begrunnes eksplisitt i et **datert
|
||||
protokoll-avviksnotat** i rapporten (avvik erklæres, absorberes aldri stille); **konfig-diffen
|
||||
mellom feilet og vellykket kjøring publiseres** (kilde: `git -C <søsken> diff 0238507..7637c6f --stat`
|
||||
+ run-lagets endringer — viser om suksess var utfalls-betinget tuning) samt den asymmetriske
|
||||
artefakt-retensjonen (suksess: 4 committede JSON; feil: kun prosa) med fremadrettet
|
||||
retensjonsregel. Verbatim-tall fra artefaktene; §4.3-konfig. Første kjørings tall finnes KUN i
|
||||
søskenets STATE.md:40–41 — raden merkes «disclosed-non-reproducible». Ingen statistiske ord
|
||||
(«typisk», «gjennomsnittlig», «stabilt»); «eksistensbevis» glosses for forretningspublikum
|
||||
(«én kjøring viser at det KAN skje, ikke at det VIL skje»).
|
||||
(5) **M4 kvalitativ** — RAMMES som CASE-STUDIE-observasjoner, aldri framework-egenskaper
|
||||
(Prechelt 2000: inter-programmerer-varians > inter-språk-varians; N=1 per rammeverk, samme
|
||||
forfatter+AI; spec-en ble ekstrahert FRA MAF-implementasjonen → MAF-spec-troskap er
|
||||
nær-tautologisk og søskenet har annenmanns-fordel — sies eksplisitt). LOC/moduler (ferske tall fra
|
||||
Step 4-kommandoene; forventet ~2 696/18 vs ~1 696/15), spec-troskap mot §12, load-bearing-dekning
|
||||
inkl. den ærlige asymmetrien (søskenet har mutasjonskontroller + attempts-felt MAF mangler;
|
||||
«kjørbar sim-modul vs testsuite-bevis» som formforskjell), utvikleropplevelse som
|
||||
ÉN-RATER-observasjon med eksplisitt disclaimer (ingen rubrikk-pretensjon).
|
||||
(6) **Liveness-asymmetri-erklæringen (§3) ORDRETT** som blockquote, PLUSS klarspråk-versjon i
|
||||
Hovedfunn (prominens, ikke bare tilstedeværelse).
|
||||
(7) **Begrensninger (threats to validity) — OBLIGATORISK seksjon:** N=1-implementasjon per
|
||||
rammeverk, ett benchmark-bundle, forfatter-COI, residual-asymmetrien fra B1, én live-kjøring;
|
||||
huser også liveness-erklæringens kontekst.
|
||||
(8) **Konklusjon** — nøktern; skriptede stand-ins merket som stand-ins overalt (method-spec §1).
|
||||
(9) **Verifiseringslogg med TRE-KLASSE-taksonomi** per rad: `reproduce` (kommando kan kjøres på
|
||||
nytt) / `recompute-from-artifact` (avledes av committede artefakter — live-tallene) /
|
||||
`disclosed-non-reproducible` (kun den feilede kjøringens STATE-prosa-rad). Rader: tre-hash x5,
|
||||
begge suiter (ferske tall m/ navngitte skips), sim-exit, S10-artefaktstier + single-commit-beviset,
|
||||
konfig-diff-kommandoen, LOC-kommandoer, verbatim-sjekken, read-only-beviset (status før/etter).
|
||||
Etter rapporten: **korriger dette repoets STATE.md** (briefens Constraint 6 / Assumption 3):
|
||||
overskriv med S11-utfallet (NESTE → S12 [GATED]), fjern ⛔ PUSH-FRYS-linjen, pek på rapporten.
|
||||
STATE.md er gitignored (local-only) — ingen commit av den.
|
||||
Mekaniske sjekker etter skriving: (a) verbatim-sjekken, KOMMANDOEN ER (whitespace-normalisert
|
||||
sammenligning; gjengis også i verifiseringsloggen):
|
||||
```
|
||||
python3 - <<'EOF'
|
||||
import re, pathlib
|
||||
proto = pathlib.Path("docs/plan/2026-07-03-sammenligningsprotokoll.md").read_text()
|
||||
block = re.search(r"> \*\*Liveness-asymmetri-erklæring:\*\*.*?(?=\n\n## )", proto, re.S).group(0)
|
||||
decl = "\n".join(l.lstrip("> ").rstrip() for l in block.splitlines())
|
||||
report = pathlib.Path("docs/rapport/2026-07-03-sammenligningsrapport.md").read_text()
|
||||
assert decl.replace("\n", " ") in " ".join(report.split()), "asymmetry declaration NOT verbatim"
|
||||
print("OK: declaration verbatim")
|
||||
EOF
|
||||
```
|
||||
(b) forbudt-ord-grep i ESCAPED form — `grep -inE 'typis[k]|gjennomsnittli[g]|stabil[t]' docs/rapport/2026-07-03-sammenligningsrapport.md`
|
||||
→ 0 treff; loggens egen rad gjengir kommandoen i samme escaped form slik at den aldri matcher seg
|
||||
selv; (c) manuell tabell-audit: ingen rad blander offline- og live-kategori. (basert på
|
||||
kodebase-analyse + protokollens §2–§5)
|
||||
- **Reuses:** presedens-strukturen fra `docs/2026-06-26-fot-i-bakken.md`; §3-teksten kopieres fra protokollen.
|
||||
- **Verify:** verbatim-kommandoen over → expected: `OK: declaration verbatim`; escaped grep → expected: exit 1 (0 treff); `grep -c "S11" STATE.md` → ≥ 1
|
||||
- **On failure:** retry — rett rapporten til sjekkene er grønne (dokumentfeil er alltid rettbar i økta); escalate kun hvis en KILDE viser seg å motsi en annen
|
||||
- **Checkpoint:** `git commit -m "docs(rapport): S11 — sammenligningsrapport begge stacker på pinned db86e15"`; deretter push KUN hvis Assumption 3 står (frys-oppheving operatør-bekreftet i økta — `git push origin main`); hvis operatøren i mellomtiden har gjeninnført frys: parker push, informer
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- docs/rapport/2026-07-03-sammenligningsrapport.md
|
||||
- STATE.md
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^docs\\(rapport\\): S11 — "
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- docs/plan/2026-07-03-sammenligningsprotokoll.md
|
||||
- docs/plan/2026-07-02-sesjonsplan.md
|
||||
- shared/method-spec.md
|
||||
must_contain:
|
||||
- path: docs/rapport/2026-07-03-sammenligningsrapport.md
|
||||
pattern: "Liveness-asymmetri-erklæring"
|
||||
- path: STATE.md
|
||||
pattern: "S11"
|
||||
```
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
| Approach | Pros | Cons | Why rejected |
|
||||
|----------|------|------|--------------|
|
||||
| Permanent load-bearing test som vokter rapportens §3-sitat | Automatisk drift-vern | Rapporten er terminal artefakt ingenting konsumerer — ingen søm detacher; bryter 90 %-prinsippet | One-off verbatim-kommando i verifiseringsloggen dekker behovet (test-strategist) |
|
||||
| Frisk S10-re-kjøring for «bedre» live-tall | Ferskere data | Forbudt: §4.2 (artefakter er fasit) + §4.5 (re-rull av utfall) + D6 (ny spend krever godkjenning) + overskriver committet fasit | Protokollbrudd; Non-Goal (a) i briefen |
|
||||
| Rapport i `docs/plan/` i stedet for ny `docs/rapport/` | Ingen ny katalog | Blander leveranse med planverk; sjanger-per-underkatalog er etablert konvensjon | Brief/Preference sier `docs/rapport/`; konvensjonsskann støtter |
|
||||
| Commit-melding-grep som SC2-ref-sjekk (briefens opprinnelige formulering) | Enkelt | Beviser omtale, ikke innhold — squash-meldingene er ulike; en lokal shared/-endring ville passert | Erstattet av tre-hash-identitet `fc58b0b9…` x4 (brief-review-funn b, verifisert av to agenter) |
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- **Framework:** pytest (begge repo). Ingen nye testfiler i S11 — leveransen er et dokument;
|
||||
mekanisk verifisering skjer via kommandoer som selv føres i verifiseringsloggen.
|
||||
- **Existing patterns:** golden-frys (`test_validator_output_matches_golden`), load-bearing
|
||||
detach-bevis (`test_*loadbearing*.py`), env-gatede live-skips (de 4 i MAF-repoet — navngis i loggen).
|
||||
- **New tests in this plan:** 0 (bevisst — se Alternatives rad 1).
|
||||
- **Mekanisk rapport-verifisering:** verbatim-sjekk (python, whitespace-normalisert), forbudt-ord-grep,
|
||||
kategori-tabell-audit (separate tabeller), read-only-bevis (status før/etter), tre-hash-identitet x5.
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Priority | Risk | Location | Impact | Mitigation |
|
||||
|----------|------|----------|--------|------------|
|
||||
| Critical | `run_s10.py` kjøres «for å speile» → overskriver committet fasit + ekte API-kost | søsken `src/portfolio_optimiser_claude/run_s10.py` (default `--out runs/s10`) | Fasit ødelagt; §4.2/§4.5/D6 brutt | Step 3/4 leser KUN filer; forbudet står i Step 3-prosaen; API-nøkkel i miljøet gjør kallet reelt mulig — aldri utfør |
|
||||
| High | `uv run` re-låser `uv.lock` (tracked) i et av repoene | begge repo | Read-only brutt i søskenet; støy-commit her | `--frozen` på ALLE `uv run`-kall (Step 1–3); status-før/etter-bevis |
|
||||
| High | Kryss-kategori-lekkasje: S10-live-tall havner i samme tabellrad som MAF-offline-tall | rapportens M2/M3-seksjoner | Protokoll §2-brudd — rapportens gyldighet | To adskilte seksjoner + kategori-kolonne; eksplisitt audit i Step 5-sjekkene |
|
||||
| High | §4.5-kategorisering av S10-re-kjøringen fremstilles glatt («transportfeil») | rapportens S10-seksjon | Ærlighetsbrudd; protokollens ordlyd dekker ikke konfig-feil | Loggfør BEGGE kjøringer med kost/exit; drøft avviket eksplisitt (arch-mapper/risk-assessor) |
|
||||
| Medium | `PORTFOLIO_SHARED_ROOT` satt i miljøet re-peker MAF-kjernen utenom pinningen | `shared_root.py` | «Identisk kjerne» usann runtime | Step 1 sjekker at varen er tom og loggfører det |
|
||||
| Medium | Tall siteres fra STATE-prosa i stedet for ferske kjøringer/artefakter | rapporten | Premiss-verifiseringsbrudd; STATE var alt stale én gang i dag | Steps 2–4 fanger alt ferskt; rapporten siterer kun evidence/kommandoer |
|
||||
| Low | MC-determinisme på tvers av python 3.12/3.14 | begge validatorer | Golden-avvik | Kun `Random.uniform` (versjonsstabil); golden-testene ER sjekken; versjoner loggføres |
|
||||
| Low | Mid-sesjon-commits fra andre sesjoner (aec95eb-mønsteret) | dette repoet | Plan-premisser drifter | Step 1 re-dokumenterer tips; commit i Step 5 skjer fra verifisert tilstand |
|
||||
|
||||
## Assumptions
|
||||
|
||||
| # | Assumption | Why unverifiable | Impact if wrong |
|
||||
|---|-----------|-----------------|-----------------|
|
||||
| 1 | Operatørens S10-re-kjøringsgodkjenning (dokumentert i søskenets STATE-logg) aksepteres som §4.5-loggføringsgrunnlag | Beslutningen ligger i STATE-prosa, ikke i en committet protokolltekst | Rapportens §4.5-drøfting må skjerpes/eskaleres til operatør |
|
||||
| 2 | Søskenets pyproject kjører mypy i strict-modus via konfig (STATE/CLAUDE.md sier «--strict rene») | Ikke lest ut av pyproject i utforskningen | Step 3 bruker uansett repoets egne konfigurerte kommandoer; loggen siterer faktisk invokasjon |
|
||||
| 3 | Push-frys er opphevet (operatør-bekreftet i økta; sibling pushet 10:57) | Muntlig/chat-bekreftelse, ikke skriftlig direktiv-endring i STATE ennå | Step 5 push parkeres; commit står lokalt; STATE-korrigering uansett ved sesjonslutt |
|
||||
|
||||
## Verification
|
||||
|
||||
*Per-step manifests over; dette er end-to-end-sjekkene (= briefens SC1–SC5).*
|
||||
|
||||
- [ ] SC1: `python3`-verbatim-sjekken (Step 5) → `OK: declaration verbatim`
|
||||
- [ ] SC2: `uv run --frozen pytest` exit 0 i BEGGE repo (ferske tall, navngitte skips) + ruff + mypy exit 0 + tre-hash `fc58b0b9…` x5 identisk (inkl. temporal S10-commit-sjekk)
|
||||
- [ ] SC3: `evidence/03-maf-sim.txt` inneholder `LEARNING LOOP CLOSED` + `EXIT=0`; søsken-suiten (inkl. step7/step8-loadbearing) grønn i `evidence/04-sibling-gates.txt`; alle M1–M3-tall i rapporten sporer til evidence/artefakter
|
||||
- [ ] SC4: hver tallpåstand i rapporten har rad i verifiseringsloggen; ALLE M1–M3-rader kjøres på nytt → samme tall (briefens ordlyd; det eneste erklærte unntaket er den feilede S10-kjøringens STATE-prosa-rad, merket ikke-reproduserbar)
|
||||
- [ ] SC5: `grep -inE 'typisk|gjennomsnittlig|stabilt' docs/rapport/…` → 0 treff; tabell-audit: ingen rad blander kategori; ingen live-påstand om MAF-siden
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Files to modify:** 0
|
||||
- **Files to create:** 1 tracked (rapporten) + 6 untracked evidence-filer
|
||||
- **Complexity:** medium (ingen kode, men høy presisjonsbyrde: hver setning i rapporten er en verifiserbar påstand)
|
||||
|
||||
## Plan Quality Score
|
||||
|
||||
| Dimension | Weight | Score | Notes |
|
||||
|-----------|--------|-------|-------|
|
||||
| Structural integrity | 0.15 | 95 | Lineær avhengighet 1→5; ingen sirkularitet |
|
||||
| Step quality | 0.20 | 90 | Eksakte kommandoer; Verify per steg; read-only operasjonalisert |
|
||||
| Coverage completeness | 0.20 | 95 | Alle SC1–SC5 + begge brief-review-funn adressert |
|
||||
| Specification quality | 0.15 | 90 | Rapportinnhold seksjon-for-seksjon; ingen «etter behov» |
|
||||
| Risk & pre-mortem | 0.15 | 95 | 8 risikoer m/ mitigering; farligste (run_s10) eksplisitt |
|
||||
| Headless readiness | 0.10 | 80 | Steps 1–4 har ingen commit (bevisst — read-only); manifester kompenserer |
|
||||
| Manifest quality | 0.05 | 85 | must_contain på nøkkelverdier; tomme commit-patterns på read-only-steg |
|
||||
| **Weighted total** | **1.00** | **91** | **Grade: A** |
|
||||
|
||||
**Adversarial review:**
|
||||
- **Plan critic:** 1 blocker, 5 major, 6 minor — alle adressert i revisjon 1 (se Revisions)
|
||||
- **Scope guardian:** ALIGNED — 0 creep, 2 minor gaps + 1 sti-feil, alle adressert i revisjon 1
|
||||
|
||||
## Revisions
|
||||
|
||||
| # | Finding | Severity | Resolution |
|
||||
|---|---------|----------|------------|
|
||||
| 1 | Step 3 Verify brukte `head -n -1` (ulovlig i BSD-head) og udefinerte markørlinjer | blocker | BEFORE:/AFTER:-prefiks definert i Changes; Verify omskrevet til BSD-kompatibel grep/sed-diff |
|
||||
| 2 | evidence/ ikke gitignored → status-støy + commit-sweep-risiko | major | Step 1 oppretter selv-ignorerende `evidence/.gitignore` (`*`); manifest håndhever |
|
||||
| 3 | Feilet S10-kjørings tall (162 250/exit 3/$0.331506) hadde ingen tillatt kilde | major | Eksplisitt kilde-unntak i Step 5: raden merkes «ikke-reproduserbar kilde: STATE-prosa» ærlig |
|
||||
| 4 | SC1-verbatim-sjekkens kommando sto ikke i planen | major | Full python-kommando innfelt i Step 5 (whitespace-normalisert semantikk definert) |
|
||||
| 5 | Forbudt-ord-grep ville matche sin egen logg-rad | major | Escaped form `typis[k]\|gjennomsnittli[g]\|stabil[t]` foreskrevet i både sjekk og logg |
|
||||
| 6 | M1-bit-for-bit-påstanden manglet beviskommando | major | Python-sammenligning golden↔run_result innfelt i Step 4 + Verify |
|
||||
| 7 | `ls shared/ -R \| wc` teller feil (28 linjer, ikke 13 filer) | minor | Byttet til `find shared -type f \| wc -l` |
|
||||
| 8 | `run_s10.py`-sti feil (repo-rot vs `src/portfolio_optimiser_claude/`) | minor | Rettet i Codebase Analysis, Step 3/4 og risikotabellen (reist av begge reviewere) |
|
||||
| 9 | Stalt søsken-modultall (16; ground truth 15) | minor | Rettet til ~15 og alle forhåndstall degradert til «forventninger» — kommandoen er kilden |
|
||||
| 10 | Step 1-grep skjør + manglende On-failure for satt env-var | minor | PIN:-prefiks ankret grep; unset/re-fang/escalate-sti lagt til |
|
||||
| 11 | `2>&1` uspesifisert i captures | minor | Foreskrevet globalt i plan-preamble + eksplisitt i Steps 1–3 |
|
||||
| 12 | Push-betingelsen bodde kun i Assumptions | minor | Flyttet inn i Step 5-checkpointen (push kun hvis A3 står; ellers parker + informer) |
|
||||
| 13 | SC4-stikkprøve (3 rader) svakere enn briefens «hver rad» | minor | Verification SC4 skjerpet til ALLE M1–M3-rader; STATE-prosa-unntaket erklært |
|
||||
| 14 | STATE.md-korrigering manglet som steg (falt bort headless) | minor | Lagt i Step 5 Changes + manifest (`STATE.md` must_contain «S11») |
|
||||
|
||||
## Adversarial Pass 2 (v5.1.1 high-effort — gemini-bridge feilet; contrarian-researcher substitutt)
|
||||
|
||||
**Prosessnotat (ærlighet):** Gemini Deep Research-jobben ble startet og fullførte server-side, men
|
||||
resultatet var uhentbart pga. to uavhengige verktøyfeil i gemini-MCP-serveren
|
||||
(`get_research_result`: `'Interaction' object has no attribute 'outputs'`, to forsøk;
|
||||
`research_followup`: 404 på pensjonert modell). Passet ble derfor gjennomført av
|
||||
`contrarian-researcher` (uavhengig web-grunnlagt metodestress-test — annet perspektiv enn
|
||||
plan-critic, samme hensikt som gemini-passet). Funnene under er innarbeidet i Context og Step 5.
|
||||
|
||||
| # | Finding | Severity | Resolution |
|
||||
|---|---------|----------|------------|
|
||||
| B1 | «Definert før begge stacker» er falsifiserbar — protokollen predaterer kun S4/S5; MAF-impl forelå | blocker (restatement) | Context + Step 5(1): presis temporal påstand + residual-asymmetri erklært |
|
||||
| B2 | Vellykket live-kjøring ser ut til å bryte §4.5 («tak nådd» = innholdsutfall; unntak kun transportfeil); klassifiseringen var post-hoc | blocker (restatement) | Step 5(4): feilet kjøring rapporteres førsteklasses, §4.5 siteres, datert avviksnotat |
|
||||
| M1g | M1 er asymmetrisk/sirkulær (golden generert fra MAF-linjen; begge validatorer framework-frie) — ekvivalens-gate, ikke komparativ metrikk | major | Step 5(2): M1 omrammet; generator navngis; scope til bundelet |
|
||||
| MPr | N=1 per rammeverk, samme forfatter+AI (Prechelt 2000); spec ekstrahert fra MAF → M4 unattributabel til rammeverk | major | Step 5(5): M4 = case-studie-observasjoner, aldri framework-egenskaper |
|
||||
| MCOI | Udeklarert interessekonflikt (Anthropic-fokusert forfatter; rapport skrevet med Claude) | major | Step 5(1): COI-erklæring, én setning |
|
||||
| MCfg | Konfig-diff feilet↔vellykket kjøring + asymmetrisk artefakt-retensjon må publiseres | major | Step 5(4): diff-kommando + retensjonsregel |
|
||||
| MTid | §1 krever re-pin+re-kjør ved kjerneendring, men Claude-siden kan ikke re-kjøres — temporal identitet må BEVISES | major | Step 1: femte PIN-linje `rev-parse '0f8af04:shared'` (verifisert = fc58b0b9…) |
|
||||
| MSyn | Skriptede M2/M3-tall er forfatter-valgte inputs, ikke målinger | major | Step 5(3): designparametre + boolske håndhevelses-utfall; separate tabeller |
|
||||
| MVer | Verifiseringsloggens «reproduserende kommando»-regel selv-bryter for live-tall | major | Step 5(9): tre-klasse-taksonomi (reproduce / recompute-from-artifact / disclosed-non-reproducible) |
|
||||
| min1–5 | Ett bundle-scope · Begrensninger-seksjon obligatorisk · erklæringens prominens (Hovedfunn) · «eksistensbevis» glosses · DX én-rater-disclaimer | minor | Step 5(2)(5)(6)(7): alle innarbeidet |
|
||||
|
|
@ -1,76 +0,0 @@
|
|||
# I4 — SQL-kilde på MAF-siden (brief)
|
||||
|
||||
> **Økt-type:** kode-økt, offline, IKKE gatet (lokalt/offline; driftsmodellens klarsignal
|
||||
> dekker den). Andre kildetype i ingest-programmet, mot I1s polymorfe manifest-skjema UTEN
|
||||
> spec-endring. Speiler I2 (`file`/CSV) for `sql`.
|
||||
>
|
||||
> **Kilder:** [ingest-spec](../../../shared/ingest-spec.md) (normativ, FROSSET) · [målbilde
|
||||
> ingest](../../../docs/plan/2026-07-03-maalbilde-ingest-lag.md) · [sesjonsplan §I4](../../../docs/plan/2026-07-03-sesjonsplan-ingest.md).
|
||||
|
||||
## Mål (én lukket leveranse)
|
||||
|
||||
sqlite-konnektor på MAF-siden mot det polymorfe manifest-skjemaet: `manifest (source.type:
|
||||
sql)` → read-only sqlite-uttrekk → materialisert OKF-bundle (samme §5–§7 provenance + §6
|
||||
index som `file`), hele veien load-bearing- og golden-testet. `materialize()` går fra å
|
||||
avvise alt ≠ `FileSource` til å dispatche `FileSource → read_csv`, `SqlSource → read_sql`
|
||||
(`http` forblir I6-avvist).
|
||||
|
||||
## Suksesskriterier (hver = verifiserbar)
|
||||
|
||||
1. `read_sql(connection_ref, query, *, max_rows)` eksekverer et read-only SELECT mot en
|
||||
sqlite-fil resolvet fra env-varen `connection_ref` navner, med §5-typet celle-rendering.
|
||||
2. `materialize()` materialiserer et `sql`-manifest til en bundle bit-identisk med en golden
|
||||
(`examples/ingest-golden-sql/`), idempotent ved re-run.
|
||||
3. Nye load-bearing-tester (SQL-navigerbarhet + §5-type-sømmer) er RØDE ved detach (bevist).
|
||||
4. Full suite grønn uten nøkkel/nettverk; `okf.py`/`shared/`/`run.py` urørt; ruff+format+mypy
|
||||
grønne; null modellkall.
|
||||
|
||||
## Non-goals (bygg IKKE)
|
||||
|
||||
- Ingen spec-/commons-endring (spec FROSSET; oppdages drift → STOPP, egen GATET runde).
|
||||
- Ingen `http`-konnektor (I6), ingen `python -m`-CLI, ingen live-DB, ingen ORDER-BY-håndhevelse
|
||||
i kode (manifest-forfatters SHOULD, §4), ingen per-rad-filsplitting, ingen inkrementell
|
||||
re-ingest, ingen index-typing-RFC (åpen, utsatt).
|
||||
- Ingen søsken-repo-endring (I5, GATET).
|
||||
|
||||
## Låste beslutninger — SQL §5-type/tall-regler (delegert HIT av STATE; pinnes i kode+golden)
|
||||
|
||||
STATE: «CSV-celle-verbatim-regelen utsatte tall-/type-håndtering eksplisitt til sql/I4 →
|
||||
SQL-kolonners type/tall-regler avgjøres HER.» Konsistente med §5, ikke en spec-endring:
|
||||
|
||||
| Python-type (sqlite default) | §5-tekst | Begrunnelse |
|
||||
|---|---|---|
|
||||
| `None` (SQL NULL) | `""` (tom) | §5 «SQL NULL as the empty string» |
|
||||
| `int` (INTEGER) | `str(v)` plain decimal | §5 «integers in plain decimal» |
|
||||
| `float` (REAL) | `repr(v)` korteste round-trip | §5 «non-integral numbers … shortest round-trip decimal form»; `repr` er Pythons korteste round-trip (ekstrem magnitude → E-notasjon, akseptert som korteste form) |
|
||||
| `str` (TEXT) | verbatim (escapes i `render_table`) | §5 «text verbatim» |
|
||||
| `bool` | `IngestError` | sqlite emitter den ikke; guard hindrer `str(True)`→`"True"` silent-coercion |
|
||||
| `bytes` (BLOB) / annet | `IngestError` | §5 «any other value type MUST fail (never silent coercion)» |
|
||||
|
||||
**Konnektor-regler:**
|
||||
- `connection_ref` = env-var-NAVN → verdi = **sti til sqlite-fil** (I4-scope: sqlite-fixture).
|
||||
Env-var usatt → `IngestError` fail-fast; fil mangler → `IngestError` (speiler `read_csv`).
|
||||
- Read-only: åpnes `file:{quote(path)}?mode=ro` med `uri=True` → et skrive-forsøk feiler på
|
||||
DB-nivå (§4 «SHOULD enforce read-only», robust — ikke skjør streng-parsing).
|
||||
- Single-statement: `Connection.execute` avviser flere statements naturlig (§4 «one statement»).
|
||||
- `max_rows`: streaming-tak, `IngestError` ved overskridelse (§8, aldri silent truncation —
|
||||
speiler `read_csv`).
|
||||
- sqlite3-feil (syntaks, skrive-forsøk, korrupt db) → wrappes som `IngestError` (fail-fast,
|
||||
ærlig — som `read_csv`s `FileNotFoundError`-wrapping).
|
||||
|
||||
## Antakelser (med test)
|
||||
|
||||
- **A1: sqlite3 default-typing gir int/float/str/bytes/None** (ingen registrerte konvertere)
|
||||
→ testet i `test_ingest_sql.py` (typet fixture-db, én rad per type).
|
||||
- **A2: `mode=ro` håndhever read-only** → test: skrive-query mot ro-connection raiser.
|
||||
- **A3: golden-db-bytene påvirker ikke bundle-determinismen** (kun DB-INNHOLD via query;
|
||||
env-sti lekker aldri inn i frontmatter) → golden bit-deterministisk uansett checkout-sti.
|
||||
|
||||
## Verifisering (kommandoer)
|
||||
|
||||
- `uv run pytest -q` — 211+ passerer, ingen ny skip, ingen nett/nøkkel.
|
||||
- Detach-bevis (manuelt, revertert): bryt hver ny load-bearing-søm → bekreft RØD → revert.
|
||||
- `uv run ruff check . && uv run ruff format --check . && uv run mypy src` — grønne.
|
||||
- `git diff --stat` — `src/portfolio_optimiser/okf.py`, `shared/`, `src/portfolio_optimiser/run.py`
|
||||
IKKE i diffen.
|
||||
- MAF-fri: `test_ingest_module_is_maf_free_and_context_layer_pure` grønn (kun stdlib lagt til).
|
||||
|
|
@ -1,72 +0,0 @@
|
|||
# I4 — SQL-kilde på MAF-siden (plan)
|
||||
|
||||
TDD (Iron Law: ingen produksjonskode uten rød test først). Ett `ingest.py` endres; ny konnektor
|
||||
+ dispatch + typet §5-rendering. Ren stdlib-tillegg (`os`, `sqlite3`, `contextlib.closing`,
|
||||
`urllib.parse.quote`) → MAF-fri-guarden urørt.
|
||||
|
||||
## Steg
|
||||
|
||||
### S1 — Røde tester (skriv FØR impl)
|
||||
- `tests/test_ingest_sql.py` (konnektor + `materialize`-dispatch):
|
||||
- env-var usatt → `IngestError`; env peker på manglende fil → `IngestError`.
|
||||
- typet rendering via tmp sqlite-db: INTEGER→decimal, REAL→`repr`, TEXT→verbatim, NULL→`""`.
|
||||
- `float` integral (`4200.0`) → `"4200.0"` (bevart, ikke `"4200"`).
|
||||
- BLOB → `IngestError` (never silent coercion); `bool`-guard (konstruér via `SELECT`? sqlite
|
||||
gir int — dekk `bytes` som «other»; bool-guarden er defensiv, dekkes med en direkte
|
||||
`_sql_value_to_text(True)`-kall-assert).
|
||||
- `max_rows` overskredet → `IngestError`.
|
||||
- read-only (A2): skrive-query mot ro-connection → `IngestError`.
|
||||
- happy path: `sql`-manifest → bundle med korrekt provenance + body.
|
||||
- `tests/test_ingest_sql_loadbearing.py` (SQL-spesifikke sømmer, detach-RED):
|
||||
- `test_sql_generated_bundle_is_navigable_via_unchanged_okf` (speiler seam 3: `navigate_bundle`
|
||||
+ `bundle_context` mot SQL-generert bundle; RØD hvis index-linking detaches).
|
||||
- `test_sql_null_renders_as_empty_not_none` (§5-load-bearing: RØD hvis None→`str()`).
|
||||
- `test_sql_real_preserves_float_form` (§5: `4200.0`→`"4200.0"`; RØD hvis int-coerced).
|
||||
- `test_sql_blob_fails_never_silent_coercion` (§5: BLOB→`IngestError`; RØD hvis silent str).
|
||||
- `test_sql_manifest_mapping_to_verdict_is_rejected` (§3-reservasjon fyrer på `sql`-source).
|
||||
- Oppdater 2 eksisterende:
|
||||
- `test_ingest_materialize.py::test_sql_source_has_no_connector_in_i2` → repekes til `http`
|
||||
(`test_http_source_has_no_connector` — den gjenstående I6-avvisningen; sql eksekverer nå).
|
||||
- `test_ingest_manifest.py::test_sql_and_http_variants_validate` — kommentar «sql → I4»
|
||||
oppdateres til at `file`+`sql` eksekverer, `http`→I6.
|
||||
- **Kjør → RØD** (read_sql finnes ikke; materialize avviser sql).
|
||||
|
||||
### S2 — Implementasjon (`ingest.py`)
|
||||
- Import: `os`, `sqlite3`, `from contextlib import closing`, `from urllib.parse import quote`.
|
||||
- `_sql_value_to_text(value) -> str` (rekkefølge: None → bool-guard → int → float → str → fail).
|
||||
- `read_sql(connection_ref, query, *, max_rows) -> tuple[list[str], list[list[str]]]`:
|
||||
env-resolusjon → fil-eksistens → ro-connection (`closing`) → `execute` → header fra
|
||||
`cursor.description` → streaming max_rows-tak → `_sql_value_to_text` per celle → wrap
|
||||
`sqlite3.Error/Warning` som `IngestError`.
|
||||
- `materialize`: erstatt `if not isinstance(..FileSource): raise` med dispatch:
|
||||
`http`→refuse (I6); `file`→root-resolusjon+`read_csv`; `sql`→`read_sql`. §8-logg per uttrekk.
|
||||
- Docstring-oppdatering: modul-docstring (SQL nå implementert, §5-tallregler pinnet);
|
||||
`render_table` parentes (SQL-verdier pre-konvertert til §5-tekst i `read_sql`).
|
||||
- **Kjør → GRØNN** (unit + loadbearing).
|
||||
|
||||
### S3 — Golden-fixture
|
||||
- `examples/ingest-golden-sql/`:
|
||||
- `fixture/portefolje.sqlite` — committet db (gen-script i denne prosjektmappa; schema i brief).
|
||||
Tabeller: `costs`(id INTEGER, item TEXT, amount REAL, note TEXT-nullable) med NULL + `4200.0`;
|
||||
`meta`(k TEXT, v TEXT) med pipe/backslash for escape-bevis.
|
||||
- `manifest.json` — `source.type:sql`, `connection_ref:"PORTEFOLJE_SQL_DSN"`, to uttrekk med
|
||||
ORDER BY.
|
||||
- `ingested-at.txt` — fast timestamp.
|
||||
- `expected-bundle/` — generert av `materialize`, HÅNDVERIFISERT mot §5 celle-for-celle, frosset.
|
||||
- `tests/test_ingest_golden_sql.py` — speiler `test_ingest_golden.py`; setter env-var til
|
||||
fixture-db (monkeypatch), asserterer bit-likhet + idempotent re-run.
|
||||
- **Kjør → GRØNN**.
|
||||
|
||||
### S4 — Detach-bevis (hver ny load-bearing, revertert)
|
||||
- null→str, float→int-cast, BLOB→str, index-link-detach, verdict-validator-fjern → bekreft RØD.
|
||||
|
||||
### S5 — Full verifisering + commit
|
||||
- `pytest -q` grønn · ruff+format+mypy grønne · `git diff --stat` (okf/shared/run urørt) ·
|
||||
MAF-fri-guard grønn.
|
||||
- Overskriv STATE.md (I4 levert, NESTE = I5 gated søsken). Commits (Conventional, `(I4)`-scope):
|
||||
test(røde) → feat(impl) → test(golden+fixture) → docs(brief/plan) → evt. style(ruff). Push Forgejo.
|
||||
|
||||
## Verifisering (plan som helhet)
|
||||
Program-sluttkriteriets SQL-halvdel: golden bit-deterministisk fra delt spec, load-bearing røde
|
||||
ved detach, suite grønn uten nøkkel/nett, ingen artefakt påstår mer enn koden gjør (sqlite-fixture,
|
||||
ikke live-DB). Ingen §-endring i spec.
|
||||
|
|
@ -1,220 +0,0 @@
|
|||
---
|
||||
type: trekbrief
|
||||
brief_version: "2.2"
|
||||
created: 2026-07-04
|
||||
task: "I6 — wire the HTTP source connector behind an opt-in network flag (MAF-only), tested against a local mock, no live source"
|
||||
slug: i6-ingest-http-maf
|
||||
project_dir: .claude/projects/2026-07-04-i6-ingest-http-maf/
|
||||
research_topics: 0
|
||||
research_status: skipped
|
||||
auto_research: false
|
||||
interview_turns: 1
|
||||
source: interview
|
||||
framing: refine
|
||||
phase_signals:
|
||||
- phase: research
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: plan
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: execute
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: review
|
||||
effort: high
|
||||
model: opus
|
||||
---
|
||||
|
||||
# Task: I6 — HTTP source connector as a demonstrated extension point (MAF-only)
|
||||
|
||||
> Generated by `/trekbrief` on 2026-07-04.
|
||||
> This brief is the contract between requirements and planning. `/trekplan`
|
||||
> reads it to produce the implementation plan. Every decision in the plan must
|
||||
> trace back to content in this brief.
|
||||
|
||||
## TL;DR
|
||||
|
||||
Refine of the frozen ingest program (I1–I5). Activate the `http` source family —
|
||||
already declared an OPTIONAL extension point in the spec — by wiring the connector
|
||||
behind an explicit per-run network opt-in flag, tested ONLY against a local mock
|
||||
(no live source; that gate stays unconsumed). Proves the extension-point pattern
|
||||
and the "no silent egress" frame. MAF-only; `okf.py`/`shared/` and the spec untouched.
|
||||
|
||||
## Intent
|
||||
|
||||
The ingest layer now implements the `file`/CSV (I2/I3) and `sql` (I4/I5) source
|
||||
families on both stacks, built from the shared frozen spec. That spec (§1, §4)
|
||||
declares `type: "http"` an OPTIONAL extension point: the schema validates
|
||||
`HttpSource` today, but `materialize` refuses it fail-fast
|
||||
(`src/portfolio_optimiser/ingest.py:394`, "gated extension point (I6) with no
|
||||
connector"). I6 activates that stub to demonstrate — honestly and end-to-end —
|
||||
HOW a third, network-transport source family plugs into the same connector /
|
||||
materialization / gate contracts, WITHOUT consuming the platform's "first live
|
||||
source" milestone. This realizes two locked framework principles at once: the
|
||||
90%-principle (build the generic core plus clear extension points, do not chase
|
||||
the last 10%) and the security frame's "local-only default, no silent egress"
|
||||
(§8) — showing a network source can be added strictly behind an explicit gate
|
||||
that the manifest itself cannot grant.
|
||||
|
||||
## Goal
|
||||
|
||||
`materialize` (and the surrounding run path) accept an explicit opt-in network
|
||||
flag that is a RUN argument, never a manifest field (§8). When a manifest's
|
||||
source is `HttpSource`: without the flag the connector refuses fail-fast with
|
||||
`IngestError`; with the flag it fetches each extraction (the extraction `query`
|
||||
is a path joined to `base_url`, retrieved with GET — §4/§5), resolves an optional
|
||||
`credential_ref` from the environment by name (mirroring the I4 `connection_ref`
|
||||
seam), and materializes one `ingest-{id}.md` per extraction whose body is the
|
||||
**response body verbatim inside a fenced code block** (§5, distinct from the
|
||||
markdown table used for `file`/`sql`), enforcing `max_rows` fail-fast. A golden
|
||||
case `examples/ingest-golden-http/` reproduces byte-deterministic output against
|
||||
committed mock payloads — with no network and no credentials (§9/§11).
|
||||
`docs/extending.md` documents the extension-point pattern and points at the D7
|
||||
sibling hook. `okf.py` and `shared/` are untouched; the frozen spec is unchanged.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **No live HTTP source.** Tested only against a local/in-process (or injected)
|
||||
mock. The "first live source" gate in the ingest brief remains unconsumed.
|
||||
- **No MCP client wired into the run path.** MCP stays the documented
|
||||
"extension of this family" framing (§4) plus the existing server-side
|
||||
demonstration in `datasource.py` (`build_mcp_server`); I6 does not add a new
|
||||
MCP-over-HTTP client connector.
|
||||
- **No spec change.** The ingest spec is FROZEN; any change is a separate gated
|
||||
commons round. I6 implements, it does not amend.
|
||||
- **No change to `okf.py` or `shared/`.** The MAF-free, D7-portable context seam
|
||||
and the pull-only commons subtree stay untouched.
|
||||
- **No change to the D7 sibling repo this session.** MAF-only; the D7 http mirror
|
||||
is a later, separate slice.
|
||||
- **No manifest-level network grant.** The flag is a run argument only.
|
||||
- **Not building "typed index everywhere."** That OKF-standard remains a deferred,
|
||||
separately-coordinated commons RFC.
|
||||
|
||||
## Constraints
|
||||
|
||||
- The frozen ingest spec (`shared/ingest-spec.md`) is authoritative. §4/§5 define
|
||||
the http query (path + GET), body rendering (verbatim in a fenced code block),
|
||||
frontmatter keys, and credential-by-name; §8/§9 define the network gate and the
|
||||
HITL layers; §11 defines the golden format. I6 conforms; it does not extend the
|
||||
spec.
|
||||
- Test suite MUST run **without network access and without credentials** (§9,
|
||||
§11) — full green offline.
|
||||
- Prefer **zero new runtime dependencies** — the connector uses the stdlib for
|
||||
GET, and tests use a stdlib/injected mock.
|
||||
- **Cost discipline (D6):** null API-spend; no model calls in the ingest program.
|
||||
- **Commons is pull-only.** Do not touch `shared/`; never `git subtree push`.
|
||||
|
||||
## Preferences
|
||||
|
||||
- Mirror the I4 SQL implementation shape: a per-source-type value/body render
|
||||
dispatch, credential/connection resolution by env-name, a committed golden case,
|
||||
and load-bearing tests that go RED when a seam is detached.
|
||||
- **Transport seam:** an injectable GET callable (defaulting to a stdlib
|
||||
`urllib` GET) so golden tests supply canned mock payloads with no socket —
|
||||
the hermetic reading of §11 "without network access".
|
||||
- TDD (Iron Law: failing test first). Surgical changes; match existing
|
||||
`ingest.py` style and docstring-pinning convention.
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
- **Hermetic tests:** the full `pytest` suite is green with no network and no
|
||||
credentials.
|
||||
- **Determinism:** the golden http extraction is byte-identical on repeat runs
|
||||
(fixed `ingested_at`, committed mock payloads).
|
||||
- **Fail-fast security:** no silent egress; flag absent → refuse before any
|
||||
source call.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- **Network gate (load-bearing):** without the opt-in flag, an `http` manifest is
|
||||
refused fail-fast — a test asserts `IngestError`, and reverting the gate seam
|
||||
turns that test RED. (`uv run pytest -k <gate test>` passes; detach → fail.)
|
||||
- **Golden determinism:** with the flag, the golden case materializes
|
||||
byte-identically — `uv run pytest -k <golden http test>` passes and output
|
||||
matches committed `examples/ingest-golden-http/` bytes.
|
||||
- **Offline suite:** `uv run pytest` exits 0 with no network and no credentials
|
||||
set (mock in-process / injected transport).
|
||||
- **max_rows fail-fast:** an http extraction whose response exceeds `max_rows` is
|
||||
an `IngestError` (never silent truncation) — asserted by a test.
|
||||
- **No live claim:** `grep -rin "live" docs/ src/ examples/` surfaces only
|
||||
"local mock" / "no live source" framing; the "first live source" gate is still
|
||||
documented as unconsumed.
|
||||
- **Context seam untouched:** `git diff --stat` shows no change to `okf.py` or
|
||||
`shared/`.
|
||||
- **Lint/format/type green:** `uv run ruff check .`, `uv run ruff format --check .`,
|
||||
and `uv run mypy src` each exit 0.
|
||||
- **Null API-spend:** the run makes no model or network calls (the golden run uses
|
||||
the injected mock).
|
||||
- **Docs:** `docs/extending.md` exists and documents the extension-point pattern
|
||||
with the D7 `create_sdk_mcp_server` hook pointer.
|
||||
|
||||
Do NOT accept vague criteria ("http works", "the connector is implemented") —
|
||||
each criterion above is command-checkable or detach-provable.
|
||||
|
||||
## Research Plan
|
||||
|
||||
No external research needed — the codebase and this brief contain sufficient
|
||||
context for planning. Verified basis: the frozen spec (`shared/ingest-spec.md`
|
||||
§4/§5/§8/§9/§11) fully defines the http source semantics; the implementation
|
||||
mirrors the already-landed I4 SQL patterns (env-resolved credential-by-name,
|
||||
per-type render dispatch, golden-case structure, detach-proven load-bearing
|
||||
tests); the local mock uses the Python stdlib (no obscure or platform-specific
|
||||
behavior — the search-first rule is not triggered). The single external-adjacent
|
||||
item, the current `create_sdk_mcp_server` API for the `docs/extending.md` D7
|
||||
pointer, is a low-stakes documentation pointer for the sibling repo, recorded as
|
||||
an assumption to verify at doc-writing time — it does not gate the plan.
|
||||
|
||||
## Open Questions / Assumptions
|
||||
|
||||
- **[ASSUMPTION] `max_rows` for http.** The response body renders verbatim (no
|
||||
rows). The plan must pin the enforced interpretation — most consistent with §8
|
||||
("source calls are logged … row count"; caps "enforced fail-fast"): the
|
||||
response body's line count is the enforced "size", fail-fast on exceed. To be
|
||||
locked in the plan like the I4 SQL type/number rules, and pinned in the
|
||||
`ingest.py` docstring + golden.
|
||||
- **[ASSUMPTION] Local-mock strategy.** An injectable GET callable (defaulting to
|
||||
a stdlib `urllib` GET) so golden tests supply canned payloads with no socket —
|
||||
the hermetic reading of §11. A localhost `http.server` thread is a documented
|
||||
fallback; the plan decides.
|
||||
- **[ASSUMPTION] `source_system` frontmatter value for http.** Mirror the
|
||||
existing `file`/`sql` renderer convention (plan to confirm from the current
|
||||
materializer — likely the source `type` or a `base_url`-derived value; must
|
||||
match the golden bytes).
|
||||
- **[ASSUMPTION] Opt-in flag threading + name.** The flag threads through
|
||||
`materialize` and `run_project` as a run argument (e.g. `allow_network: bool =
|
||||
False`); the exact name/signature is pinned in the plan.
|
||||
- **[ASSUMPTION] `create_sdk_mcp_server` D7 pointer.** Assumed current Claude
|
||||
Agent SDK primitive for the docs pointer; verify at doc-writing time (cheap,
|
||||
non-gating).
|
||||
|
||||
## Prior Attempts
|
||||
|
||||
None for the http family specifically. I2 (CSV, `e4ee8bd`→`994f5be`) and I4 (SQL,
|
||||
`d7e5f2f`→`1b7612b`) are the direct precedents on this stack — identical
|
||||
connector / materialization / golden / load-bearing pattern — and I6 mirrors them
|
||||
for the network-transport family. The `HttpSource` schema and its fail-fast
|
||||
rejection already exist (`ingest.py:84` and `:394-396`) as the gated stub that I6
|
||||
activates behind the new flag.
|
||||
|
||||
## Metadata
|
||||
|
||||
- **Created:** 2026-07-04
|
||||
- **Interview turns:** 1 (framing); required-section content pre-populated from
|
||||
the detailed task directive + the frozen spec, then reviewed.
|
||||
- **Auto-research opted in:** no
|
||||
- **Source:** trekbrief interview
|
||||
|
||||
---
|
||||
|
||||
## How to continue
|
||||
|
||||
Manual (default):
|
||||
|
||||
```bash
|
||||
# No research topics — go straight to planning:
|
||||
/trekplan --project .claude/projects/2026-07-04-i6-ingest-http-maf/
|
||||
|
||||
# Then execute:
|
||||
/trekexecute --project .claude/projects/2026-07-04-i6-ingest-http-maf/
|
||||
```
|
||||
|
|
@ -1,565 +0,0 @@
|
|||
# I6 — HTTP source connector as a demonstrated extension point (MAF-only)
|
||||
|
||||
> **Plan quality: A** (92/100) — APPROVE_WITH_NOTES
|
||||
>
|
||||
> Generated by trekplan v5.9.1 on 2026-07-04 — `plan_version: 1.7`
|
||||
|
||||
## Context
|
||||
|
||||
The ingest layer implements the `file`/CSV (I2/I3) and `sql` (I4/I5) source families on
|
||||
both stacks, built from the frozen `shared/ingest-spec.md`. That spec (§1, §4) declares
|
||||
`type: "http"` an OPTIONAL extension point: the schema already validates `HttpSource`
|
||||
(`src/portfolio_optimiser/ingest.py:84-100`), but `materialize` refuses it fail-fast
|
||||
(`ingest.py:394-398`, "gated extension point (I6) with no connector"). I6 activates that stub
|
||||
to demonstrate — honestly and end-to-end — HOW a third, network-transport source family plugs
|
||||
into the same connector / materialization / gate contracts, WITHOUT consuming the platform's
|
||||
"first live source" milestone. It realizes two locked framework principles at once: the
|
||||
90%-principle (generic core + clear extension points, do not chase the last 10%) and the
|
||||
security frame's "local-only default, no silent egress" (§8) — a network source added strictly
|
||||
behind an explicit run-flag the manifest itself cannot grant. MAF-only; `okf.py`, `shared/`,
|
||||
and the frozen spec are untouched.
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "I6 changes in ingest.py"
|
||||
M["materialize(...,<br/>allow_network=False,<br/>http_get=None)"]:::mod
|
||||
GATE{"isinstance(HttpSource)<br/>AND not allow_network?"}:::new
|
||||
DISP["dispatch loop<br/>(file / sql / http branch)"]:::mod
|
||||
RH["read_http(base_url, query,<br/>*, max_rows, credential_ref, get)"]:::new
|
||||
UG["_urllib_get<br/>(stdlib GET — ONLY socket path)"]:::new
|
||||
FENCE["_render_fenced_block<br/>(verbatim, ``` fence)"]:::new
|
||||
end
|
||||
subgraph "reused UNCHANGED"
|
||||
RC["_render_concept_file<br/>(7-key frontmatter)"]:::keep
|
||||
WB["_write_bytes / ownership /<br/>collision / index"]:::keep
|
||||
OKF["okf.render_frontmatter /<br/>link_in_index"]:::keep
|
||||
end
|
||||
TEST["golden + load-bearing tests<br/>inject a canned http_get<br/>(no socket, no creds)"]:::test
|
||||
|
||||
M --> GATE
|
||||
GATE -->|"no flag → IngestError"| REFUSE["fail-fast (spec §8)"]:::new
|
||||
GATE -->|"allow_network=True"| DISP
|
||||
DISP --> RH --> UG
|
||||
RH -->|"body text"| FENCE --> RC --> WB --> OKF
|
||||
TEST -.injects.-> RH
|
||||
|
||||
classDef new fill:#2d6,stroke:#164,color:#000
|
||||
classDef mod fill:#fd6,stroke:#a70,color:#000
|
||||
classDef keep fill:#ddd,stroke:#888,color:#000
|
||||
classDef test fill:#8cf,stroke:#048,color:#000
|
||||
```
|
||||
|
||||
## Codebase Analysis
|
||||
|
||||
- **Tech stack:** Python ≥3.10, Pydantic (validation/IR), stdlib `urllib` (transport),
|
||||
`pytest` (+ byte-for-byte golden harness), `ruff`/`mypy`, `uv`. Zero new runtime deps.
|
||||
- **Key patterns:** one connector fn per source type returning renderable content; a
|
||||
per-source-type dispatch inside `materialize`; credential/connection resolution by
|
||||
**env-var name** (never from the manifest); a committed golden case per type; load-bearing
|
||||
tests that go RED when a seam is detached; spec-silent decisions pinned in the module
|
||||
docstring + frozen in golden bytes.
|
||||
- **Relevant files (verified):**
|
||||
- `src/portfolio_optimiser/ingest.py` — `HttpSource` schema (`:84-100`), the gate to
|
||||
replace (`:394-398`), the dispatch loop (`:408-426`), `read_sql` template (`:233-278`),
|
||||
`_render_concept_file` 7-key frontmatter (`:304-318`, `source_system = manifest.source.id`),
|
||||
`_write_bytes` (`:321-327`), ownership/collision/index machinery (`:428-470`), module
|
||||
docstring pin-block (`:1-31`).
|
||||
- `tests/test_ingest_golden_sql.py` — byte-for-byte golden harness (`_bundle_bytes`, file-set
|
||||
equality, §10 second-run idempotence).
|
||||
- `tests/test_ingest_sql_loadbearing.py` / `tests/test_ingest_sql.py` — detach-RED and
|
||||
connector-unit templates.
|
||||
- `tests/test_ingest_materialize.py:245-253` — `test_http_source_has_no_connector` (stale
|
||||
after I6: still green under default-no-flag, but its "no connector" comment becomes false).
|
||||
- `examples/ingest-golden-sql/` — golden layout to mirror (`manifest.json`, `fixture/`,
|
||||
`ingested-at.txt`, `expected-bundle/{index.md, ingest-*.md}`).
|
||||
- `.claude/projects/2026-07-04-i4-ingest-sql-maf/build_fixture.py` — throwaway fixture-builder
|
||||
pattern.
|
||||
- `docs/extending.md` — **already exists** (bilingual dev doc: Norwegian headers, English
|
||||
prose); I6 **appends** an http source-family section + the D7 hook pointer.
|
||||
- **Reusable code (leverage unchanged):** `_render_concept_file`, `_write_bytes`, the ownership
|
||||
scan / collision gate / index generation (`ingest.py:428-470`), `load_manifest`,
|
||||
`okf.render_frontmatter` / `okf.link_in_index`. **Explicitly NOT reused for http:**
|
||||
`render_table` / `_escape_cell` / `_sql_value_to_text` (http body is verbatim inside a
|
||||
fenced code block, not a table).
|
||||
- **External tech (researched):** none — 0 research topics (frozen spec fully defines http
|
||||
semantics; I4 SQL is the direct in-repo precedent; stdlib mock). The single external-adjacent
|
||||
item (`create_sdk_mcp_server` currency for the docs pointer) is a doc-time verification, not a
|
||||
planning input.
|
||||
- **Recent git activity:** `bc52c12` (I6 brief), `1b7612b`/`d7e5f2f` (I4 SQL, the direct
|
||||
precedent), `e4ee8bd`→`994f5be` (I2 CSV). Branch `main`, clean.
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
Each step is test-first (Iron Law). Steps are ordered by dependency: connector → gate+dispatch →
|
||||
golden → docs.
|
||||
|
||||
### Step 1: Implement `read_http` connector + injectable transport seam (docstring-pinned)
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_http.py` *(new)*
|
||||
- **Changes:** Add, near `read_sql` (`ingest.py:233`):
|
||||
- A transport-seam type alias `HttpGet = Callable[[str, str | None], str]` (`(url, credential) -> response body text`); add `from collections.abc import Callable` and
|
||||
`from urllib.request import Request, urlopen` + `from urllib.error import URLError` to imports.
|
||||
- `def _urllib_get(url: str, credential: str | None) -> str` — the stdlib GET and the **only**
|
||||
socket path: build a `Request(url, headers=..., method="GET")`, attach
|
||||
`Authorization: Bearer {credential}` iff `credential is not None`, `urlopen`, read, decode
|
||||
UTF-8. Wrap `URLError`/`OSError`/`UnicodeDecodeError` as `IngestError`. Do NOT pre-add a
|
||||
`# noqa: S310` — `pyproject.toml` selects no `flake8-bandit` (`S`) rules, so S310 never
|
||||
fires and a dead noqa would trip `RUF100` if it is ever enabled; add the noqa ONLY if
|
||||
`uv run ruff check .` actually flags the `urlopen` at Verify.
|
||||
- `def read_http(base_url, query, *, max_rows, credential_ref=None, get: HttpGet = _urllib_get) -> str`
|
||||
mirroring `read_sql`: resolve `credential_ref` via `os.environ.get` (present-but-unset/empty →
|
||||
`IngestError` fail-fast, mirroring `ingest.py:249`; `None` → no auth); build
|
||||
`url = base_url.rstrip("/") + "/" + query.lstrip("/")`; call `get(url, credential)`;
|
||||
normalize CRLF→LF; **fence-collision fail-fast** — if any normalized line, after stripping
|
||||
up to its leading spaces (`line.lstrip(" ")`), begins with ` ``` ` (a fence marker — note
|
||||
CommonMark permits a closing fence indented ≤3 spaces, so a leading-space check is required),
|
||||
raise `IngestError` (a body that cannot be safely embedded in a fenced code block is an
|
||||
ERROR, never silently-corrupted markdown — the module's no-silent-coercion discipline + §8
|
||||
fail-fast); enforce the cap on the **`\n`-only line count** of the CRLF-normalized text —
|
||||
`n = normalized.count("\n"); line_count = n if (normalized == "" or normalized.endswith("\n")) else n + 1`;
|
||||
`line_count > max_rows → IngestError` (never truncation, §8). **Do NOT use `str.splitlines()`**
|
||||
for the cap: CPython `splitlines()` splits on the extended Unicode set
|
||||
(`\v`, `\f`, `\x1c`-`\x1e`, `\x85`, `\u2028`, `\u2029`), so a body with a vertical-tab or line-separator would be over-counted vs the
|
||||
LF-only rendered file (verified 2026-07-04). The `\n`-only count matches the rendered body and
|
||||
still treats a single trailing newline as no phantom line. Return normalized text. Pure: no
|
||||
writes, no logging (materialize owns §8 log).
|
||||
- `def _render_fenced_block(body: str) -> str` → `f"\`\`\`\n{body.rstrip(chr(10))}\n\`\`\`\n"`
|
||||
(verbatim inside a fenced code block; LF-only; exactly one trailing newline).
|
||||
- **Docstring pin (module docstring, `ingest.py:23-30` block):** append the http (I6) pinned
|
||||
decisions, mirroring the I4 sql sentence: transport is an injectable `get` callable
|
||||
(default `_urllib_get`, the only socket); `credential_ref` names an env secret resolved at
|
||||
run time (Bearer), never from the manifest, never logged, never stamped in frontmatter;
|
||||
URL = `base_url` (one trailing slash stripped) + `/` + `query` (leading slash stripped);
|
||||
body = response decoded UTF-8, CRLF→LF, trailing newlines stripped, rendered verbatim in a
|
||||
``` fenced block (NOT table-escaped); `max_rows` caps response **line count**
|
||||
(`len(normalized.splitlines())`), fail-fast; a response line beginning with ` ``` ` →
|
||||
`IngestError` fail-fast (cannot be safely fenced — never silent corruption, consistent with
|
||||
the no-silent-coercion frame; the fixed 3-backtick fence is deliberate, NOT a silent
|
||||
fallback).
|
||||
- **Reuses:** `read_sql` structure (`ingest.py:233-278`) — env-resolve → fetch → cap →
|
||||
`IngestError` wrap; the CSV/SQL streaming-cap idiom (`ingest.py:193-197`, `:270-274`).
|
||||
- **Test first:**
|
||||
- File: `tests/test_ingest_http.py` (new)
|
||||
- Verifies (each injects a canned `get`, no socket, no creds unless monkeypatched):
|
||||
`read_http` returns the injected body verbatim and calls `get` with the joined URL;
|
||||
a body of exactly `max_rows` lines → returns normally (boundary pass);
|
||||
`max_rows + 1` lines → `IngestError` (never truncated) **[SC max_rows]**;
|
||||
a body with a line beginning ` ``` ` → `IngestError` (fence-collision fail-fast);
|
||||
`credential_ref` present but env unset → `IngestError`; `credential_ref` set → the injected
|
||||
`get` receives the resolved secret (secret comes from env, never the manifest);
|
||||
`read_http`'s `get` default is `_urllib_get` (`inspect.signature` — proves the stdlib
|
||||
default without a socket).
|
||||
- Pattern: `tests/test_ingest_sql.py` (connector-unit style; local `get`-builder mirroring
|
||||
`_make_db` locality — do NOT add to `conftest.py`).
|
||||
- **Verify:** `uv run pytest tests/test_ingest_http.py -q` → expected: all pass (offline).
|
||||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_http.py`
|
||||
- **Checkpoint:** `git commit -m "feat(ingest): http read_http connector + injectable transport seam (I6)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/ingest.py
|
||||
- tests/test_ingest_http.py
|
||||
min_file_count: 2
|
||||
commit_message_pattern: "^feat\\(ingest\\): http read_http connector \\+ injectable transport seam \\(I6\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/okf.py
|
||||
- shared/ingest-spec.md
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/ingest.py
|
||||
pattern: "def read_http"
|
||||
- path: src/portfolio_optimiser/ingest.py
|
||||
pattern: "_render_fenced_block"
|
||||
- path: tests/test_ingest_http.py
|
||||
pattern: "read_http"
|
||||
```
|
||||
|
||||
### Step 2: Gate http behind the `allow_network` run-flag + wire the dispatch branch
|
||||
|
||||
- **Files:** `src/portfolio_optimiser/ingest.py`, `tests/test_ingest_http_loadbearing.py`
|
||||
*(new)*, `tests/test_ingest_materialize.py`
|
||||
- **Changes:**
|
||||
- `materialize` signature (`ingest.py:373-375`): add keyword-only `allow_network: bool = False`
|
||||
and `http_get: HttpGet | None = None` (both defaulted → all existing file/sql callers stay
|
||||
backward-compatible; `run_project`/`run_portfolio` are NOT touched — verified they never call
|
||||
`materialize`). Extend the docstring to note the §8 network opt-in flag is a run argument.
|
||||
- Replace the unconditional refusal (`ingest.py:393-398`) with the gate:
|
||||
`if isinstance(source, HttpSource) and not allow_network: raise IngestError(...)` — refuse
|
||||
fail-fast **before any source call**, message citing §8 ("the manifest cannot grant itself
|
||||
network access"). This exact `and not allow_network` clause is the load-bearing seam.
|
||||
- Restructure the dispatch loop (`ingest.py:408-426`) so each branch sets `body` and a
|
||||
`row_count`: `FileSource` → `read_csv` + `render_table`, `row_count = len(rows)`;
|
||||
`elif isinstance(source, HttpSource)` → `get = http_get if http_get is not None else _urllib_get`;
|
||||
`text = read_http(source.base_url, extraction.query, max_rows=extraction.max_rows, credential_ref=source.credential_ref, get=get)`;
|
||||
`body = _render_fenced_block(text)`; `row_count = len(text.splitlines())`; `else` (Sql) →
|
||||
`read_sql` + `render_table`, `row_count = len(rows)`. Keep the single `_LOGGER.info` §8 log
|
||||
(`ingest.py:416-421`) logging `rows=row_count` (line count for http). The stale
|
||||
"http already refused above" comment (`ingest.py:412`) is removed.
|
||||
- `tests/test_ingest_materialize.py:245-253`: update `test_http_source_has_no_connector`'s
|
||||
comment/intent — the default-no-flag path still raises `IngestError`, but the wording
|
||||
"no connector" is now false; reword to "http is refused without the per-run network flag"
|
||||
(assertion stays green; comment corrected). Do not delete — it is the default-refuse smoke.
|
||||
- **Reuses:** `_render_concept_file`, `_write_bytes`, ownership/collision/index machinery
|
||||
(`ingest.py:428-470`) — all unchanged; the http bundle flows through the identical §3/§6 disk
|
||||
phase.
|
||||
- **Test first:**
|
||||
- File: `tests/test_ingest_http_loadbearing.py` (new)
|
||||
- Verifies (detach-RED seams, each maps to a Success Criterion):
|
||||
1. **Network gate [SC1] — asserts BOTH branches on the SAME manifest + SAME recording `get`
|
||||
(this is what makes it genuinely detach-RED):**
|
||||
- *Refuse branch:* `materialize(http_manifest, out, ingested_at=…, allow_network=False, http_get=recording_get)`
|
||||
→ assert `IngestError` **and** `recording_get` was **never called** (refuse before any
|
||||
source access, §8).
|
||||
- *Allow branch:* `materialize(http_manifest, out, ingested_at=…, allow_network=True, http_get=recording_get)`
|
||||
→ assert it **succeeds** (writes `ingest-*.md`) **and** `recording_get` **was called**.
|
||||
Why both: the seam is `if isinstance(source, HttpSource) and not allow_network: raise`.
|
||||
Removing the whole clause → the *refuse* assertion fails (get called, no error) → RED;
|
||||
removing just `and not allow_network` (reverting to the *unconditional* refusal that
|
||||
exists today) → the *allow* assertion fails (raises when it should fetch) → RED. A
|
||||
refuse-only test would false-green on the second detach. *(Inject a recording `get` that
|
||||
returns a valid payload — never the default `_urllib_get`, whose dead socket would
|
||||
false-green the refuse branch.)*
|
||||
2. **Fenced-not-table [§5 http seam]:** payload with a raw `|` and `\`; assert the body is
|
||||
``` -fenced and the pipe survives **un-escaped** (`"\\|" not in body`, raw `|` present).
|
||||
RED if the http path degrades to `render_table`.
|
||||
3. **Navigability [§2/§11]:** drive the http bundle through unchanged
|
||||
`okf.navigate_bundle`/`bundle_context`; reachable via index link, classified by
|
||||
`okf_type`, provenance rides through. RED when index linking detaches.
|
||||
4. **Over-cap writes nothing [SC max_rows]:** over-cap payload → `IngestError` **and** no
|
||||
`ingest-*.md` written (memory-staging). RED if silent truncation replaces the raise.
|
||||
- Pattern: `tests/test_ingest_sql_loadbearing.py` (detach discipline) + the ordering proof in
|
||||
`tests/test_ingest_manifest.py:178`.
|
||||
- **Verify:** `uv run pytest tests/test_ingest_http_loadbearing.py tests/test_ingest_materialize.py -q`
|
||||
→ expected: all pass. Detach-proof (manual, at review): the gate test goes RED on BOTH
|
||||
detaches — (a) delete the whole `if isinstance(source, HttpSource) and not allow_network:`
|
||||
block → refuse branch RED; (b) reduce it to the unconditional `if isinstance(source, HttpSource):`
|
||||
→ allow branch RED.
|
||||
- **On failure:** revert — `git checkout -- src/portfolio_optimiser/ingest.py tests/test_ingest_http_loadbearing.py tests/test_ingest_materialize.py`
|
||||
- **Checkpoint:** `git commit -m "feat(ingest): gate http behind allow_network run-flag + dispatch (I6)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- src/portfolio_optimiser/ingest.py
|
||||
- tests/test_ingest_http_loadbearing.py
|
||||
- tests/test_ingest_materialize.py
|
||||
min_file_count: 3
|
||||
commit_message_pattern: "^feat\\(ingest\\): gate http behind allow_network run-flag \\+ dispatch \\(I6\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/okf.py
|
||||
- src/portfolio_optimiser/run.py
|
||||
- shared/ingest-spec.md
|
||||
must_contain:
|
||||
- path: src/portfolio_optimiser/ingest.py
|
||||
pattern: "allow_network"
|
||||
- path: src/portfolio_optimiser/ingest.py
|
||||
pattern: "and not allow_network"
|
||||
- path: tests/test_ingest_http_loadbearing.py
|
||||
pattern: "allow_network=False"
|
||||
```
|
||||
|
||||
### Step 3: Author the byte-deterministic golden case `examples/ingest-golden-http/`
|
||||
|
||||
- **Files:** `examples/ingest-golden-http/manifest.json` *(new)*,
|
||||
`examples/ingest-golden-http/fixture/status` *(new)*,
|
||||
`examples/ingest-golden-http/fixture/report` *(new)*,
|
||||
`examples/ingest-golden-http/ingested-at.txt` *(new)*,
|
||||
`examples/ingest-golden-http/expected-bundle/` *(new, generated + hand-reviewed)*,
|
||||
`tests/test_ingest_golden_http.py` *(new)*,
|
||||
`.claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py` *(new, throwaway)*
|
||||
- **Changes:**
|
||||
- `manifest.json`: `source` = `{"type": "http", "id": "status-api", "base_url": "https://api.example.test/v1", "credential_ref": null}`
|
||||
(`.test` reserved TLD + `credential_ref: null` → runs with **no network, no credentials**,
|
||||
proving SC offline); `bundle_summary`; two extractions —
|
||||
`{"id": "status", "title": "Service status", "query": "status", "okf_type": "dataset", "max_rows": 10}`,
|
||||
`{"id": "report", "title": "Ops report", "query": "report", "okf_type": "reference", "max_rows": 10}`.
|
||||
- `fixture/status` (1 line, discriminates fenced-verbatim vs table-escape — contains raw `|`
|
||||
and `\`): `{"service": "billing", "state": "degraded | partial", "path": "c:\temp\cache"}`
|
||||
- `fixture/report` (3 lines, ≤ max_rows): `baseline ok` / `pipe | and backslash \ kept verbatim` /
|
||||
`end of report`.
|
||||
- `ingested-at.txt`: `2026-07-04T12:00:00Z` + trailing newline (byte-write, §5 format).
|
||||
- `build_fixture.py`: writes the two payload files + `ingested-at.txt` (mirrors the I4 builder;
|
||||
payloads are LF-only so the golden is platform-stable). Documents regenerability. **Author
|
||||
the payload bytes as raw bytes / raw-string literals** — in a normal Python string literal
|
||||
`\t` is a TAB and `\c` a literal backslash, so the intended verbatim backslashes in
|
||||
`c:\temp\cache` (and `backslash \`) would be silently corrupted; write with `rb"..."` /
|
||||
`.write_bytes(b"...")` or `r"..."` so the byte the fenced-not-escaped test depends on
|
||||
survives exactly.
|
||||
- `expected-bundle/`: **generate** by running `materialize` against the fixture with a canned
|
||||
`get` (map `url.rsplit("/",1)[-1]` → `fixture/<name>`), `allow_network=True`, then **verify
|
||||
the generated bytes against the spec-derived expectation below BEFORE committing** — this is
|
||||
an agent check against an independent oracle, not a human gate, so a headless `claude -p`
|
||||
run performs it too. The non-circular correctness oracle is carried here in the plan (and,
|
||||
independently, by the Step-2 load-bearing tests, which hand-assert the fenced-not-escaped
|
||||
body, the 7-key frontmatter, and navigability — so correctness is proven by hand-written
|
||||
assertions, not by comparing code output to code output). If the generated `ingest-status.md`
|
||||
diverges from the expectation below in any way OTHER than the `ingest_manifest` hash,
|
||||
**escalate** (do not commit a wrong oracle).
|
||||
|
||||
Hand-derived `expected-bundle/ingest-status.md` (from §5/§7 + `fixture/status`; every field
|
||||
except the manifest-hash is independently derived):
|
||||
```
|
||||
---
|
||||
type: dataset
|
||||
title: Service status
|
||||
source_system: status-api
|
||||
source_query: status
|
||||
ingested_at: 2026-07-04T12:00:00Z
|
||||
ingest_manifest: manifest@<sha256(manifest.json)[:16]>
|
||||
generated: true
|
||||
---
|
||||
|
||||
```‹fence›
|
||||
{"service": "billing", "state": "degraded | partial", "path": "c:\temp\cache"}
|
||||
```‹fence›
|
||||
```
|
||||
Note the pipe and backslashes are **un-escaped** (verbatim) inside the fence — the exact
|
||||
discriminator vs `render_table`. `index.md` = `bundle_summary` + `- [Service status](ingest-status.md)`
|
||||
+ `- [Ops report](ingest-report.md)`. The `ingest_manifest` hash is the only computed field
|
||||
(SHA-256 of the committed `manifest.json` bytes) — frozen once committed.
|
||||
- **Reuses:** golden layout + harness from `examples/ingest-golden-sql/` and
|
||||
`tests/test_ingest_golden_sql.py`; builder shape from the I4 `build_fixture.py`.
|
||||
- **Test first:**
|
||||
- File: `tests/test_ingest_golden_http.py` (new)
|
||||
- Verifies **[SC golden determinism]**: build the canned `get` from `fixture/`; `materialize(GOLDEN/manifest.json, out, ingested_at=<ingested-at.txt>, allow_network=True, http_get=fake)`;
|
||||
file-set equality, per-file byte compare against `expected-bundle/`, second-run idempotence
|
||||
(§10). No credential env set (`credential_ref: null`).
|
||||
- Pattern: `tests/test_ingest_golden_sql.py:30-47` (swap the `monkeypatch.setenv` DSN for the
|
||||
injected `get`).
|
||||
- **Verify:** `uv run pytest tests/test_ingest_golden_http.py -q` → expected: pass, byte-identical.
|
||||
- **On failure:** revert — `git checkout -- examples/ingest-golden-http tests/test_ingest_golden_http.py .claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py && rm -rf examples/ingest-golden-http`
|
||||
- **Checkpoint:** `git commit -m "test(ingest): http golden extraction, byte-deterministic (I6)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- examples/ingest-golden-http/manifest.json
|
||||
- examples/ingest-golden-http/ingested-at.txt
|
||||
- examples/ingest-golden-http/fixture/status
|
||||
- examples/ingest-golden-http/fixture/report
|
||||
- examples/ingest-golden-http/expected-bundle/index.md
|
||||
- examples/ingest-golden-http/expected-bundle/ingest-status.md
|
||||
- examples/ingest-golden-http/expected-bundle/ingest-report.md
|
||||
- tests/test_ingest_golden_http.py
|
||||
- .claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py
|
||||
min_file_count: 9
|
||||
commit_message_pattern: "^test\\(ingest\\): http golden extraction, byte-deterministic \\(I6\\)$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/okf.py
|
||||
- shared/ingest-spec.md
|
||||
must_contain:
|
||||
- path: examples/ingest-golden-http/expected-bundle/ingest-status.md
|
||||
pattern: "```"
|
||||
- path: examples/ingest-golden-http/expected-bundle/ingest-status.md
|
||||
pattern: "generated: true"
|
||||
- path: examples/ingest-golden-http/manifest.json
|
||||
pattern: "\"type\": \"http\""
|
||||
```
|
||||
|
||||
### Step 4: Document the http extension-point pattern + D7 SDK hook pointer in `docs/extending.md`
|
||||
|
||||
- **Files:** `docs/extending.md`
|
||||
- **Changes:** Append a new section (bilingual house style: Norwegian header, English prose)
|
||||
documenting the http source family as the worked extension-point example: the manifest
|
||||
`type: "http"` contract (`base_url` no embedded credentials, `credential_ref` by env-name),
|
||||
the **run-argument** network opt-in (`materialize(..., allow_network=True)`; the manifest
|
||||
cannot grant network — §8), the injectable transport seam (`http_get`, default stdlib
|
||||
`urllib`), and the "verbatim in a fenced code block" body. Add the D7 sibling-hook pointer:
|
||||
an MCP-based connector is "an extension of this family" (§4), and on the Claude Agent SDK side
|
||||
the in-process server hook is **`create_sdk_mcp_server(name, version="1.0.0", tools=...) ->
|
||||
McpSdkServerConfig`** (package `claude-agent-sdk`) — **VERIFIED 2026-07-04 against the official
|
||||
Claude Agent SDK Python docs** (code.claude.com/docs/en/agent-sdk/python + docs.claude.com
|
||||
agent-sdk/mcp; the name is confirmed, not assumed — pin it). Note MCP is documented as an
|
||||
extension of this family, NOT a new client wired into the run path (Non-Goal). Frame
|
||||
everything as "local mock / no live source"; do not claim a live integration.
|
||||
- **Reuses:** the existing "Bevisst ikke bygget (90 %-kuttlista)" seam-naming style in
|
||||
`docs/extending.md:55-74`; the honesty framing in `docs/extending.md:7-13`.
|
||||
- **Test first:** *(docs — no unit test; command-checkable at Verify)*
|
||||
- Verifies: `grep -q create_sdk_mcp_server docs/extending.md` exits 0, and a new http
|
||||
source-family section is present; the new I6 artifacts carry no affirmative live-source claim.
|
||||
- **Verify:** `grep -q create_sdk_mcp_server docs/extending.md && echo OK` → expected: `OK`;
|
||||
`grep -rin "live" docs/extending.md examples/ingest-golden-http/ src/portfolio_optimiser/ingest.py`
|
||||
→ expected: any hit is only "local mock" / "no live source" negated framing (scoped to the
|
||||
I6 artifacts — the repo-wide grep is not a clean gate: `docs/`, `src/persona.py`,
|
||||
`simulation.py` etc. carry dozens of unrelated pre-existing "live"/"delivery" substrings).
|
||||
- **On failure:** revert — `git checkout -- docs/extending.md`
|
||||
- **Checkpoint:** `git commit -m "docs(i6): http source extension-point + D7 create_sdk_mcp_server pointer"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- docs/extending.md
|
||||
min_file_count: 1
|
||||
commit_message_pattern: "^docs\\(i6\\): http source extension-point \\+ D7 create_sdk_mcp_server pointer$"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- src/portfolio_optimiser/okf.py
|
||||
- shared/ingest-spec.md
|
||||
must_contain:
|
||||
- path: docs/extending.md
|
||||
pattern: "create_sdk_mcp_server"
|
||||
- path: docs/extending.md
|
||||
pattern: "allow_network"
|
||||
```
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
| Approach | Pros | Cons | Why rejected |
|
||||
|----------|------|------|--------------|
|
||||
| Thread `allow_network` through `run_project`/`run_portfolio` (as the brief assumed) | Matches the brief's literal assumption | `run.py` never imports `ingest`/calls `materialize` (verified by 2 agents + grep) — the param would be dead code | Scope creep + speculative feature; violates surgical-change. Flag scoped to `materialize` only. |
|
||||
| Reuse `render_table` for the http body | Maximal "mirror I4" symmetry | §5 requires the http body **verbatim in a fenced code block**, not a table; `render_table` escapes `\|`/`\\` | Spec-nonconforming; the fenced-not-table load-bearing test would fail. New `_render_fenced_block`. |
|
||||
| `localhost http.server` thread for tests | "Real" HTTP round-trip | Opens a socket → violates §11 "without network access"; flaky/slow; hermeticity risk | Injectable `get` callable (canned payloads, no socket) is the hermetic §11 reading. Documented fallback only. |
|
||||
| Variable-length fence (count backticks in body) | Silently "handles" a body containing ``` | Extra determinism surface; papers over an ambiguous body the model would later read | Rejected in favour of **fail-fast**: a body line beginning ` ``` ` → `IngestError` (no silent corruption; consistent with the module's no-silent-coercion frame). Fixed 3-backtick fence is deliberate, not a fallback. |
|
||||
| `urllib.parse.urljoin` for the URL | Stdlib canonical | Surprising path semantics (absolute paths, `..`) → non-deterministic joins | Explicit `rstrip("/") + "/" + lstrip("/")` — predictable, pinned. |
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- **Framework:** `pytest` (`uv run pytest`); byte-for-byte golden harness; detach-RED
|
||||
load-bearing regime (spec §11).
|
||||
- **Existing patterns:** three-file-per-source-type convention
|
||||
(`test_ingest_http.py` connector-unit, `_loadbearing.py` detach-RED, `_golden_http.py`
|
||||
byte-for-byte) — mirrors the sql trio. Canned-`get` builders local to the test files (never
|
||||
`conftest.py`, which is MAF/LLM-only).
|
||||
- **New tests in this plan:** ~11 across 3 new files + 1 comment correction.
|
||||
|
||||
### Tests to write
|
||||
|
||||
| Type | File | Verifies | Model test |
|
||||
|------|------|----------|------------|
|
||||
| Unit | `tests/test_ingest_http.py` | verbatim fetch; URL join; `max_rows`; credential env-resolve (set/unset); urllib default | `tests/test_ingest_sql.py` |
|
||||
| Load-bearing | `tests/test_ingest_http_loadbearing.py` | network gate (never-called + IngestError); fenced-not-table; navigability; over-cap-writes-nothing | `tests/test_ingest_sql_loadbearing.py` |
|
||||
| Golden | `tests/test_ingest_golden_http.py` | byte-identical materialization; §10 idempotence; no creds | `tests/test_ingest_golden_sql.py` |
|
||||
| Comment fix | `tests/test_ingest_materialize.py:245` | default-no-flag still refuses (intent reworded) | — |
|
||||
|
||||
**Offline property (SC):** satisfied by construction — every http test injects `http_get` (no
|
||||
socket) and needs no credential. Acceptance: `uv run pytest` exits 0 with the network cable
|
||||
pulled. Convention to enforce in review: **no ingest test calls `materialize`/`read_http` for an
|
||||
http source without passing `http_get`** (the default `_urllib_get` is the only socket path and
|
||||
is never exercised).
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Priority | Risk | Location | Impact | Mitigation |
|
||||
|----------|------|----------|--------|------------|
|
||||
| Critical | Silent egress if the gate sits after connector construction, or `_urllib_get` is reachable without the flag | `ingest.py:393` gate / dispatch | Network hit with no operator opt-in — defeats the task | Gate in `materialize` BEFORE the staging loop; `_urllib_get` only reached inside the http branch, which the gate guards. Load-bearing test asserts `get` never called when flag absent. |
|
||||
| High | Credential echo in the verbatim body | http body render | A reflecting endpoint could echo `Authorization` into a committed bundle file | Inherent to "verbatim body" — do NOT auto-redact (breaks determinism). Golden mock never echoes a secret; document as a residual deployer-owned risk (§1 boundary). |
|
||||
| High | Credential leak via error/log text | `read_http` IngestError, `_LOGGER.info` | Secret in an exception or log line | Mirror `read_sql`: wrap transport errors WITHOUT the secret; keep §8 log to `id`/`ingested_at`/`row_count` only. (Optional sentinel-secret grep test — nice-to-have, not gating.) |
|
||||
| High | Determinism: verbatim-vs-LF-only tension, trailing newline, fence collision | `_render_fenced_block` / `read_http` | Golden passes locally, diverges on other input/platform; or silently-broken markdown in the bundle | Pin (docstring + golden): UTF-8 decode fail-fast; CRLF→LF; strip trailing newlines outside the fence; `max_rows = len(splitlines())`; **a body line beginning ` ``` ` → `IngestError` fail-fast** (never a silently-corrupted fenced block — consistent with the no-silent-coercion frame). |
|
||||
| Medium | `max_rows` line-count semantics (extended-Unicode / trailing / blank lines) | `read_http` | Spurious pass/fail vs golden | Count `\n`-only on the CRLF-normalized text (NOT `str.splitlines()`, which also splits on `\v \f \x1c-\x1e \x85 \u2028 \u2029` — verified): `n = normalized.count("\n"); line_count = n if normalized.endswith("\n") else n+1`. Matches the LF-only body; a trailing newline adds no phantom line; internal blank lines count. Test exactly `max_rows` (pass) and `max_rows+1` (raise). |
|
||||
| Medium | `base_url`+`query` join edge cases | `read_http` | Non-deterministic joins; scheme/host escape | Pinned `rstrip("/")+"/"+lstrip("/")`; `base_url` userinfo already rejected (`ingest.py:92-100`); the joined URL is never stamped in frontmatter. |
|
||||
| Low | Stale `test_http_source_has_no_connector` intent | `tests/test_ingest_materialize.py:245` | Comment says "no connector" (now false) | Reword in Step 2; keep the default-refuse assertion. |
|
||||
| Low | Temptation to touch `okf.py`/`shared/` | `okf.py`, `shared/` | Breaks D7-portability invariant / commons pull-only | `forbidden_paths` on every code step; SC `git diff --stat` shows no change. |
|
||||
| Low | ruff `S310` (urlopen) / mypy `Callable` typing | `ingest.py` | Lint/type red | `# noqa: S310` on the gated `urlopen`; `Callable` from `collections.abc`; confirm at Verify. |
|
||||
|
||||
## Assumptions
|
||||
|
||||
| # | Assumption | Why unverifiable at plan time | Impact if wrong |
|
||||
|---|-----------|-----------------|-----------------|
|
||||
| 1 | The default auth scheme for a resolved `credential_ref` is `Authorization: Bearer {secret}` | The spec names credential-by-ref but not the wire scheme; endpoint-specific | A real deployer's endpoint may need a different scheme — acceptable for a demo extension point; documented, and the `get` seam is swappable. Golden uses `credential_ref: null` so this is not frozen in golden bytes. |
|
||||
|
||||
*(RESOLVED, not open — the brief's `[ASSUMPTION] flag threads through materialize and
|
||||
run_project` was verified FALSE: `run.py` never calls `materialize`. The flag is scoped to
|
||||
`materialize` only. `source_system` = `manifest.source.id` confirmed from
|
||||
`_render_concept_file:312`; `max_rows` = response line count pinned per §8. The D7 pointer
|
||||
`create_sdk_mcp_server` was verified 2026-07-04 against the official Claude Agent SDK Python docs
|
||||
— it is the confirmed in-process MCP-server primitive, not an assumption.)*
|
||||
|
||||
## Verification
|
||||
|
||||
*Per-step manifests are checked by trekexecute. These are the brief's Success Criteria as
|
||||
end-to-end checks (run from repo root):*
|
||||
|
||||
- [ ] `uv run pytest -q` → expected: exits 0 (full suite green, no network, no credentials set).
|
||||
- [ ] Network gate detach-proof: comment out `and not allow_network` in `ingest.py`, run
|
||||
`uv run pytest tests/test_ingest_http_loadbearing.py -k gate` → expected: **RED**; restore → green.
|
||||
- [ ] `uv run pytest tests/test_ingest_golden_http.py -q` → expected: pass; output byte-matches
|
||||
`examples/ingest-golden-http/expected-bundle/`.
|
||||
- [ ] `uv run pytest tests/test_ingest_http.py -k max_rows` → expected: pass (over-cap → IngestError).
|
||||
- [ ] `grep -rin "live" docs/extending.md examples/ingest-golden-http/ src/portfolio_optimiser/ingest.py`
|
||||
→ expected: any hit is only "local mock" / "no live source" negated framing (scoped to the
|
||||
NEW I6 artifacts — a repo-wide `grep "live"` is not a pass/fail gate: dozens of unrelated
|
||||
pre-existing hits in `docs/`, `persona.py`, `simulation.py`, `test_*_live.py`).
|
||||
- [ ] `git diff --stat` (vs pre-I6) → expected: NO change to `src/portfolio_optimiser/okf.py` or `shared/`.
|
||||
- [ ] `uv run ruff check .` → 0; `uv run ruff format --check .` → 0; `uv run mypy src` → 0.
|
||||
- [ ] `grep -q create_sdk_mcp_server docs/extending.md` → exits 0; new http section present.
|
||||
- [ ] Null API-spend: no model calls, no real network (golden/gate/unit all use injected `get`).
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Files to modify:** 3 (`ingest.py`, `tests/test_ingest_materialize.py`, `docs/extending.md`).
|
||||
- **Files to create:** 11 (3 test files + 8 golden/fixture/builder: `manifest.json`,
|
||||
`ingested-at.txt`, `fixture/status`, `fixture/report`, 3 × `expected-bundle/*.md`,
|
||||
`build_fixture.py`).
|
||||
- **Complexity:** medium (one genuinely new render path + a security gate; the rest mirrors I4).
|
||||
|
||||
## Plan Quality Score
|
||||
|
||||
| Dimension | Weight | Score | Notes |
|
||||
|-----------|--------|-------|-------|
|
||||
| Structural integrity | 0.15 | 95 | 4 steps, strict dependency order (connector→gate→golden→docs) |
|
||||
| Step quality | 0.20 | 92 | test-first, exact file:line, explicit reuse; Step 2 touches 3 files (cohesive) |
|
||||
| Coverage completeness | 0.20 | 95 | every Success Criterion → a step + a Verification check |
|
||||
| Specification quality | 0.15 | 90 | pinned decisions concrete; 2 honest doc-time assumptions remain |
|
||||
| Risk & pre-mortem | 0.15 | 92 | silent-egress, credential-leak, determinism all mitigated |
|
||||
| Headless readiness | 0.10 | 90 | On-failure + Checkpoint + Manifest per step |
|
||||
| Manifest quality | 0.05 | 88 | must_contain + forbidden_paths guard okf.py/shared/ |
|
||||
| **Weighted total** | **1.00** | **92** | **Grade: A** |
|
||||
|
||||
**Adversarial review:**
|
||||
- **Plan critic:** REVISE → addressed. 1 blocker (gate false-green), 5 major (live-grep
|
||||
unsatisfiable, first-live-source circular, create_sdk_mcp_server contradiction, fence-collision
|
||||
silent, golden circular/hand-review), 6 minor — all resolved in Revisions below.
|
||||
- **Scope guardian:** ALIGNED. 0 creep, 0 Non-Goal violations, 0 dependency issues; independently
|
||||
confirmed the `materialize`-only flag scoping is a correct scope fix (not a gap). Its one minor
|
||||
(over-broad live-grep) coincides with plan-critic MAJOR-1 and is fixed.
|
||||
|
||||
## Revisions
|
||||
|
||||
*Added by adversarial review (Phase 9). All blocker + major findings addressed; the plan was
|
||||
revised once for the merged set.*
|
||||
|
||||
| # | Finding | Severity | Resolution |
|
||||
|---|---------|----------|------------|
|
||||
| 1 | Gate load-bearing test would false-green: reverting `and not allow_network` to the unconditional refusal still passes a refuse-only test | blocker | Step 2 gate test now asserts BOTH branches on the same manifest + same recording `get` (refuse: flag off → IngestError + get never called; allow: flag on → succeeds + get called). Removing the whole gate → refuse branch RED; removing just the clause → allow branch RED. Verify wording fixed. |
|
||||
| 2 | `grep -rin "live" docs/ src/ examples/` unsatisfiable (dozens of pre-existing hits) | major | Scoped the "no live claim" check to the NEW I6 artifacts (`docs/extending.md`, `examples/ingest-golden-http/`, `ingest.py`) in both Step 4 and Verification. |
|
||||
| 3 | `git grep "first live source"` circular (matches only plan/brief) | major | Removed the circular sub-check. |
|
||||
| 4 | `create_sdk_mcp_server` pinned in 3 gates while flagged "may be stale" | major | Verified 2026-07-04 against official Claude Agent SDK Python docs — confirmed correct; pinned confidently, Assumption downgraded to a verified fact. |
|
||||
| 5 | Fence-collision (body containing a ` ``` ` line) silently corrupts the bundle | major | Changed to fail-fast: a body line beginning ` ``` ` → `IngestError` (Step 1 connector + docstring + risk + alternatives + a new unit test). |
|
||||
| 6 | Golden expected-bundle circular + "hand-review" is a non-headless human step | major | Step 3 now carries the hand-derived `ingest-status.md` expectation IN the plan (non-circular, headless-usable), names the Step-2 load-bearing tests as the independent correctness oracle, and adds an escalate-on-divergence clause. |
|
||||
| 7 | `max_rows` counts lines pre-strip; blank-line nuance | minor | Pinned: count on CRLF-normalized text before the trailing-strip; blank lines count as content lines (deterministic). Risk row + Step 1 updated. |
|
||||
| 8 | `c:\temp\cache` backslash/tab trap in `build_fixture.py` | minor | Pinned raw-bytes / raw-string authoring so the verbatim backslash survives. |
|
||||
| 9 | `# noqa: S310` likely dead (no bandit config) | minor | Do not pre-add; add only if `ruff check` flags the `urlopen` at Verify. |
|
||||
| 10 | File-count off by one (10 vs 11 created) | minor | Corrected to 11 created / 3 modified. |
|
||||
| 11 | Line-ref drift (`:393` vs `:394`) | minor | Standardized the gate reference to `:394-398`. |
|
||||
| 12 | Pass-at-`max_rows` boundary test only in risk table | minor | Added the boundary-pass + `max_rows+1`-raise pair to the Step 1 test enumeration. |
|
||||
|
||||
## Adversarial Pass 2 (gemini-bridge, v5.1.1 high-effort)
|
||||
|
||||
The high-effort plan phase runs an additional independent gemini-bridge review. **Outcome:
|
||||
gemini-bridge is non-functional (verified 2026-07-04, not assumed).** The deep-research query ran
|
||||
to `status: completed`, but every retrieval path failed: `gemini_get_research_result` →
|
||||
`'Interaction' object has no attribute 'outputs'` (server-side bug, 3× retry); the followup
|
||||
re-emit → `404 model gemini-3-pro-preview no longer available` (deprecated). This confirms the
|
||||
prior recorded outage — the operator's expectation that it was fixed did not hold. No findings
|
||||
were retrievable; the agent correctly refused to fabricate a second opinion.
|
||||
|
||||
**Substitute triangulation (per the standing gemini-outage handling):**
|
||||
- An independent high-effort **plan-critic confirmation pass** traced the blocker fix against both
|
||||
detach cases and verified revisions #2–#6 — plan **sound**, no new blocker/major; folded in one
|
||||
minor (CommonMark ≤3-space indented closing fence → `line.lstrip(" ").startswith("```")`).
|
||||
- The two decisions gemini flagged as spec-checkable were resolved by **primary-source verification**
|
||||
(no gemini needed): **CommonMark §4.5** fenced-code semantics (closing fence ≥ opening length,
|
||||
≤3-space indent) → the fence-collision check is adequate; **CPython `str.splitlines()`** splits on
|
||||
the extended Unicode set → the `max_rows` cap was switched to a **`\n`-only** count so it matches
|
||||
the LF-only rendered body (Step 1 + risk row updated).
|
||||
- Decision 4 ("config cannot grant its own capability; capability is an explicit runtime grant")
|
||||
is the **capability-security / POLA** (Principle of Least Authority) pattern — the manifest is
|
||||
data that cannot escalate its own authority; only the `allow_network` runtime grant can.
|
||||
|
||||
Net: no external second opinion available, but every decision gemini would have triangulated is
|
||||
covered by an independent internal pass or an authoritative primary-source check.
|
||||
|
|
@ -1,67 +0,0 @@
|
|||
---
|
||||
type: trekreview
|
||||
review_version: "1.0"
|
||||
created: 2026-07-04
|
||||
task: "I6 — wire the HTTP source connector behind an opt-in network flag (MAF-only), tested against a local mock, no live source"
|
||||
slug: i6-ingest-http-maf
|
||||
project_dir: .claude/projects/2026-07-04-i6-ingest-http-maf/
|
||||
brief_path: .claude/projects/2026-07-04-i6-ingest-http-maf/brief.md
|
||||
scope_sha_start: 2208cdf
|
||||
scope_sha_end: 696f19a
|
||||
reviewed_files_count: 15
|
||||
verdict: WARN
|
||||
findings:
|
||||
- a1f4c9e07b3d264f8e5106bd9c2a7f31e0d4b8a6
|
||||
---
|
||||
|
||||
# Review: I6 — HTTP source connector (MAF-only)
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Verdict: **WARN** — one MAJOR finding, zero BLOCKERs. The code-correctness pass was clean (0 defects across all 7 dimensions), and the brief-conformance pass ran (not skipped): all 9 Success Criteria trace and all 7 Non-Goals are honored. The single MAJOR is a scope-boundary observation — a concurrent operator commit (`fe4ae69`) edited an unrelated session-plan model directive inside the reviewed range. Under high-effort mode this is retained for operator judgment rather than dropped; look at it first. Note: this review ran in high-effort mode (brief `phase_signals.review.effort: high`), so Pass 3 (reasonableness filter) was skipped by design.
|
||||
|
||||
## Coverage
|
||||
|
||||
| File | Treatment | Reason |
|
||||
|------|-----------|--------|
|
||||
| build_fixture.py | summary-only | In I6 delivery range; supporting fixture builder |
|
||||
| docs/extending.md | summary-only | In I6 delivery range; http source extension-point docs |
|
||||
| docs/plan/2026-07-03-sesjonsplan-ingest.md | summary-only | In range via concurrent commit `fe4ae69`; flagged (see Findings) |
|
||||
| examples/ingest-golden-http/expected-bundle/index.md | summary-only | Golden expected-bundle artifact |
|
||||
| examples/ingest-golden-http/expected-bundle/ingest-report.md | summary-only | Golden expected-bundle artifact |
|
||||
| examples/ingest-golden-http/expected-bundle/ingest-status.md | summary-only | Golden expected-bundle artifact |
|
||||
| examples/ingest-golden-http/fixture/report | summary-only | Golden fixture input |
|
||||
| examples/ingest-golden-http/fixture/status | summary-only | Golden fixture input |
|
||||
| examples/ingest-golden-http/ingested-at.txt | summary-only | Golden determinism marker |
|
||||
| examples/ingest-golden-http/manifest.json | summary-only | Golden manifest |
|
||||
| src/portfolio_optimiser/ingest.py | summary-only | Core connector + gate implementation |
|
||||
| tests/test_ingest_golden_http.py | summary-only | Golden byte-deterministic extraction test |
|
||||
| tests/test_ingest_http.py | summary-only | HTTP connector unit test |
|
||||
| tests/test_ingest_http_loadbearing.py | summary-only | Load-bearing gate/dispatch test |
|
||||
| tests/test_ingest_materialize.py | summary-only | Materialization test |
|
||||
|
||||
No files were marked `skip` — all 15 files in the range received coverage. No uncommitted working-tree changes were present.
|
||||
|
||||
## Findings (MAJOR)
|
||||
|
||||
### a1f4c9e07b3d264f8e5106bd9c2a7f31e0d4b8a6
|
||||
- file: docs/plan/2026-07-03-sesjonsplan-ingest.md
|
||||
- line: 11
|
||||
- rule_key: SCOPE_CREEP_BUILT
|
||||
- brief_ref: none
|
||||
- title: Unrelated session-plan model-directive edit bundled into the I6 delivery range
|
||||
- detail: The reviewed range (2208cdf..HEAD) includes an edit to docs/plan/2026-07-03-sesjonsplan-ingest.md:11-13 flipping the program-level model directive from 'Fable 5' to 'Opus 4.8 ... ingen Fable 5' (commit fe4ae69, a concurrent operator session). This file is not in the I6 plan's declared modify/create scope and traces to no I6 brief Success Criterion or Non-Goal — it concerns model selection for all ingest sessions, unrelated to the HTTP source connector deliverable.
|
||||
- recommended_action: Split the model-directive correction into its own commit outside the I6 deliverable (or record it as an explicitly in-scope housekeeping item), keeping the I6 range to the contracted connector/gate/golden/docs work.
|
||||
|
||||
## Remediation Summary
|
||||
|
||||
- BLOCKER: 0
|
||||
- MAJOR: 1
|
||||
- MINOR: 0
|
||||
- SUGGESTION: 0
|
||||
|
||||
`/trekplan --brief review.md` will consume the single MAJOR scope-boundary finding: the concurrent model-directive edit that landed inside the I6 range. No blocking defects were found in the connector/gate/golden/docs deliverable itself.
|
||||
|
||||
```json
|
||||
{"verdict":"WARN","counts":{"BLOCKER":0,"MAJOR":1,"MINOR":0,"SUGGESTION":0},"findings":[{"id":"a1f4c9e07b3d264f8e5106bd9c2a7f31e0d4b8a6","severity":"MAJOR","rule_key":"SCOPE_CREEP_BUILT","file":"docs/plan/2026-07-03-sesjonsplan-ingest.md","line":11,"brief_ref":"none","title":"Unrelated session-plan model-directive edit bundled into the I6 delivery range","detail":"The reviewed range (2208cdf..HEAD) includes an edit to docs/plan/2026-07-03-sesjonsplan-ingest.md:11-13 flipping the program-level model directive from 'Fable 5' to 'Opus 4.8 ... ingen Fable 5' (commit fe4ae69, a concurrent operator session). This file is not in the I6 plan's declared modify/create scope and traces to no I6 brief Success Criterion or Non-Goal — it concerns model selection for all ingest sessions, unrelated to the HTTP source connector deliverable.","recommended_action":"Split the model-directive correction into its own commit outside the I6 deliverable (or record it as an explicitly in-scope housekeeping item), keeping the I6 range to the contracted connector/gate/golden/docs work."}]}
|
||||
```
|
||||
|
|
@ -1,123 +0,0 @@
|
|||
---
|
||||
type: trekbrief
|
||||
brief_version: "2.2"
|
||||
created: 2026-07-04
|
||||
task: "I7 — program-avslutning ingest: README/extending.md-finpuss (MAF) + felles norsk statusrapport"
|
||||
slug: i7-program-avslutning
|
||||
project_dir: .claude/projects/2026-07-04-i7-program-avslutning/
|
||||
research_topics: 0
|
||||
research_status: skipped
|
||||
auto_research: false
|
||||
interview_turns: 3
|
||||
source: interview
|
||||
framing: preserve
|
||||
phase_signals:
|
||||
- phase: research
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: plan
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: execute
|
||||
effort: high
|
||||
model: opus
|
||||
- phase: review
|
||||
effort: high
|
||||
model: opus
|
||||
---
|
||||
|
||||
# Task: I7 — program-avslutning (ingest): dokumentasjon + felles statusrapport
|
||||
|
||||
> Generated by `/trekbrief` on 2026-07-04.
|
||||
> This brief is the contract between requirements and planning. `/trekplan`
|
||||
> reads it to produce the implementation plan. Every decision in the plan must
|
||||
> trace back to content in this brief.
|
||||
|
||||
## TL;DR
|
||||
|
||||
- I7 lukker ingest-programmet: finpuss `README.md` + `docs/extending.md` (MAF-stacken) og skriv **én felles norsk statusrapport** (bevist/ikke-bevist + verifiseringslogg) som dekker begge stacker.
|
||||
- **Framing: preserve** — planlagt siste steg (§I7) i den frosne, godkjente ingest-sesjonsplanen; ingen scope-endring.
|
||||
- Bindende: **ærlighets-regelen** (målbilde §11 «Stack-mapping (ærlighets-avgrenset)» + §0) — dokumentér KUN det koden gjør. D7-omtale avgrenset til CSV+SQL; HTTP/MCP kun pekere; ingen live-kilde-påstand.
|
||||
- Docs-økt: ingen produksjonskode, ingen D7-repo-commit (polyrepo-grense), null modell-spend.
|
||||
|
||||
## Intent
|
||||
|
||||
Ingest-programmet (I1–I6) bygde og load-bearing-testet ingest-laget på tvers av begge stacker: den framework-nøytrale spec-en i commons, CSV/filkatalog og SQL på begge repo (bit-identiske golden-uttrekk fra delt spec), og HTTP/MCP demonstrert som extension point MAF-only mot lokal mock. I7 er det planlagte avslutningssteget: gjøre dokumentasjonen sannferdig og komplett, og produsere én ærlig norsk statusrapport av hva programmet faktisk beviste — og hva det ikke beviste. Motivasjonen er tillit: rammeverket publiseres på Forgejo for alle som har MAF, og en artefakt som overselger (antyder at D7 har HTTP/MCP-støtte, eller at en live-kilde er integrert) eroderer tilliten operatøren lever av. Ærlighets-regelen (målbilde §11 «Stack-mapping (ærlighets-avgrenset)» — som eksplisitt sier «HTTP/MCP demonstrert som extension point KUN på MAF-siden» og «Ingen live-kilde forekommer noensinne i programmet»; scope-ærlighet i §0) er den bindende føringen — den skriptede offline-simuleringen beviser rørlegging og deterministisk ryggrad, ikke levende LLM-atferd, og rapporten må si nettopp det.
|
||||
|
||||
## Goal
|
||||
|
||||
Når I7 er ferdig: (1) `README.md` + `docs/extending.md` i DETTE repoet (MAF-stacken) er finpusset slik at all D7-omtale er strengt avgrenset til det D7 faktisk har (CSV + SQL); HTTP/MCP omtales KUN som MAF-demonstrert extension point / D7-krok-peker, aldri som D7-kapabilitet. (2) Én felles norsk program-statusrapport bor i `docs/rapport/` og dekker BEGGE stacker i hybrid form — kort leder-/executive-sammendrag øverst (for KI-seksjon/ledelse), så en teknisk verifiseringslogg-seksjon under (hver påstand → kommando/kilde), med en eksplisitt «bevist / ikke bevist»-inndeling. (3) Ingen artefakt i repoet påstår mer enn koden gjør. Begge suiter er verifisert offline (MAF grønn; D7 read-only — rødt/ikke-kjørbart rapporteres som «ikke bevist», ikke fikset her).
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **Ingen endring i D7-søskenrepoet** (`portfolio-optimiser-claude`) fra denne økta — polyrepo-grense. Søskenets README/`extending.md` gjøres i søskenets EGEN økt, drevet av dets egen STATE. (D7-suiten verifiseres her kun read-only og siteres.)
|
||||
- **Ingen egen D7-lokal statusrapport** — operatør valgte «én felles rapport her».
|
||||
- **Ingen produksjonskode / ingen TDD** — ren docs-økt (Iron Law gjelder produksjonskode, ikke dokumentasjon).
|
||||
- **Ingen spec-/commons-endring** — spec er frossen, PULL-ONLY.
|
||||
- **Ingen S12/release** — uavhengig av ingest-programmet, eget klarsignal.
|
||||
- **Ingen endring i `okf.py` / `shared/`** — framework-nøytral kjerne forblir urørt.
|
||||
- **Ingen påstand om live-kilde-integrasjon eller D7 HTTP/MCP-støtte** noe sted.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Språk:** statusrapporten på **norsk** (docs-språkregel: forretnings-/leder-docs norsk); `README.md` forblir **engelsk** (utvikler-vendt).
|
||||
- **Ærlighets-regel (målbilde §11 + §0; CLAUDE.md framework-/simulering-invariant):** dokumentér kun det koden gjør; skriptet offline-sim beviser plumbing, ikke levende LLM-atferd.
|
||||
- **Commons PULL-ONLY** — aldri subtree push.
|
||||
- **Offline, null modell-spend** (D6 kostnadsdisiplin) — ingen nettverk, ingen nøkkel.
|
||||
- **Modell:** Opus 4.8 xhigh alle faser/subagenter; **aldri Fable 5** (stående global regel).
|
||||
|
||||
## Preferences
|
||||
|
||||
- Named markdown-lenker til lokale filer (`[navn](file:///absolutt/sti)`) per CLAUDE.md communication-pattern.
|
||||
- Referér eksisterende docs (`docs/rapport/2026-07-03-sammenligningsrapport.md`, målbilde, sesjonsplan) framfor å duplisere innhold.
|
||||
- Rapporten skal tåle å bli lest av noen som ikke kjenner koden — men uten salgsspråk (KTG-verifiseringsplikt: ingen «fullt ut», «uten unntak»).
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
- MAF-repoets suite grønn offline: `uv run pytest` → 0 failed, uten nøkkel/nettverk (baseline: 237 passed / 4 skipped).
|
||||
- `uv run ruff check .` og `uv run mypy src` exit 0 i MAF-repoet.
|
||||
- D7-søskensuiten grønn offline — verifisert read-only + sitert (ingen commit der).
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- **SC1 — Ingen overselgende artefakt (grep-gate):** `grep -rniE '(d7|claude[- ]?sdk).{0,60}(http|mcp)' README.md docs/` gjennomgått manuelt → hvert treff er enten (a) eksplisitt «MAF-only / demonstrert extension point», eller (b) en ren D7-krok-peker (`create_sdk_mcp_server`) — INGEN treff påstår D7 HTTP/MCP-*støtte*. I tillegg: `grep -rniE '(live|produksjon|production).{0,30}(kilde|source|integrasjon|integration)' README.md docs/rapport/ docs/extending.md` → ingen påstand om live-kilde-integrasjon.
|
||||
- **SC2 — Statusrapport eksisterer og har rett form:** `docs/rapport/2026-07-04-program-statusrapport-ingest.md` finnes og inneholder (a) executive-sammendrag, (b) eksplisitt «bevist / ikke bevist»-seksjon, (c) verifiseringslogg-seksjon.
|
||||
- **SC3 — Hver påstand er sporet:** hver rad i rapportens verifiseringslogg har enten en kjørbar kommando ELLER en fil-/commit-sitering (ingen ubelagte påstander). Sjekkes ved gjennomlesning: 0 påstander uten kilde.
|
||||
- **SC4 — MAF-suite grønn offline:** `uv run pytest` exit 0, 0 failed (uten nøkkel/nettverk).
|
||||
- **SC5 — Lint/type rent:** `uv run ruff check .` exit 0 og `uv run mypy src` exit 0.
|
||||
- **SC6 — D7-suite grønn offline (read-only, sitert):** i `/Users/ktg/repos/portfolio-optimiser-claude`: `uv run pytest` exit 0 offline, forutsatt repoet er `uv sync`-et — ellers skiller rapporten eksplisitt «suite rød» fra «kunne ikke kjøres offline». Kjørt read-only og sitert i rapporten; INGEN commit i søsknet. Hvis rød/ikke-kjørbar: flagges som «ikke bevist», ikke fikset her.
|
||||
|
||||
## Research Plan
|
||||
|
||||
No external research needed — the codebase and this brief contain sufficient context for planning. (Operatør bekreftet: alt kildemateriale er internt — kode, git-historikk, eksisterende docs. Ingen nye teknologi-/bibliotekvalg.)
|
||||
|
||||
## Open Questions / Assumptions
|
||||
|
||||
- **[ASSUMPTION]** D7-søskenrepoets suite er for tiden grønn offline. I7 verifiserer dette read-only og siterer det; hvis rødt, rapporteres det som «ikke bevist» framfor å fikses her (søsknets egen økt eier fikser). SC6 dekker dette.
|
||||
- **[ASSUMPTION]** Eksisterende `sammenligningsrapport.md` (2026-07-03) dekker stack-sammenligningen; den nye I7-rapporten er program-avslutnings-rapporten (bevist/ikke-bevist på tvers av I1–I7) som REFERERER, ikke duplisererer, sammenligningsrapporten.
|
||||
- **[ASSUMPTION]** README/`extending.md` trenger «finpuss», ikke omskriving — I6 la allerede http-familien + D7-peker i `extending.md`. Planfasen bekrefter faktisk delta ved å lese begge filene først.
|
||||
|
||||
## Prior Attempts
|
||||
|
||||
None — fresh task. I7 er det planlagte siste steget i ingest-sesjonsplanen; I1–I6 er levert (se STATE.md-loggen og git-historikk).
|
||||
|
||||
## Metadata
|
||||
|
||||
- **Created:** 2026-07-04
|
||||
- **Interview turns:** 3
|
||||
- **Auto-research opted in:** no
|
||||
- **Source:** trekbrief interview
|
||||
|
||||
---
|
||||
|
||||
## How to continue
|
||||
|
||||
Manual (default):
|
||||
|
||||
```bash
|
||||
# No research topics — go straight to planning:
|
||||
/trekplan --project .claude/projects/2026-07-04-i7-program-avslutning
|
||||
|
||||
# Then execute:
|
||||
/trekexecute --project .claude/projects/2026-07-04-i7-program-avslutning
|
||||
```
|
||||
|
|
@ -1,286 +0,0 @@
|
|||
# I7 — Program-avslutning (ingest): dokumentasjon + felles statusrapport
|
||||
|
||||
> **Plan quality: APPROVE_WITH_NOTES** — plan-critic ga REVISE (73/C) pre-revisjon på 1 blocker
|
||||
> (README real-source-overclaim); blocker + alle 2 major/4 minor adressert (se Revisions). Scope-
|
||||
> guardian ALIGNED (0 creep). Post-revisjon utførbar.
|
||||
>
|
||||
> Generated by trekplan v5.9.1 on 2026-07-04 — `plan_version: 1.7`
|
||||
|
||||
## Context
|
||||
|
||||
Ingest-programmet (I1–I6) er levert: den framework-nøytrale spec-en i commons, file/CSV (I2/I3)
|
||||
og SQL (I4/I5) på begge stacker med bit-identiske golden-uttrekk, og HTTP demonstrert som
|
||||
extension point MAF-only mot lokal mock (I6). I7 er det planlagte avslutningssteget (sesjonsplan
|
||||
§I7): gjøre dokumentasjonen sannferdig og lukke programmet med én ærlig norsk statusrapport av
|
||||
hva som er bevist — og ikke bevist. Motivasjonen er tillit (brief Intent): rammeverket publiseres
|
||||
på Forgejo for alle som har MAF, og en artefakt som overselger eroderer tilliten. Bindende føring
|
||||
er ærlighets-regelen (målbilde §11 «Stack-mapping (ærlighets-avgrenset)» + §0). Ground-truth-
|
||||
utforsking avdekket at README-status er STALE («ingest specified but not yet implemented») og at
|
||||
en presis to-akset «live»-distinksjon må inn i rapporten (live data-kilde vs live modell-kjøring).
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "I7 — docs-only, MAF-repo + felles rapport"
|
||||
R[README.md<br/>fiks stale ingest-status] -->|honest status| OUT[Ingen overselgende artefakt]
|
||||
E[docs/extending.md<br/>skjerp D7-grense: CSV+SQL bygget,<br/>HTTP/MCP MAF-only + peker] -->|SC1 grep-gate| OUT
|
||||
REP[docs/rapport/2026-07-04-<br/>program-statusrapport-ingest.md<br/>NY: exec-sammendrag + bevist/ikke-bevist<br/>+ verifiseringslogg] -->|SC2/SC3| OUT
|
||||
EV[Evidens read-only:<br/>pytest/ruff/mypy MAF · D7-suite · D7 S10-live-run] -.cites.-> REP
|
||||
end
|
||||
style REP fill:#2d5,color:#000
|
||||
style EV fill:#eee,color:#000
|
||||
```
|
||||
|
||||
## Codebase Analysis
|
||||
|
||||
- **Tech stack:** Python ≥3.10, MAF, `uv`, pytest/ruff/mypy. Docs i markdown.
|
||||
- **Key patterns:** ærlighets-avgrenset dokumentasjon (målbilde §0/§11); named markdown-lenker;
|
||||
norsk for leder-/forretningsdocs, engelsk for README/utviklerdocs (docs-språkregel).
|
||||
- **Relevant files (verifisert under utforsking):**
|
||||
- `README.md` — Status-blokka (linje 9–13) sier «The ingest layer (real data sources) is
|
||||
specified but **not yet implemented**»; linje 90–92 sier «will materialize» (futurum). Begge
|
||||
STALE etter I1–I6.
|
||||
- `docs/extending.md` — har allerede http-familien (linje 55–101) + D7-MCP-peker
|
||||
(`create_sdk_mcp_server`, linje 92–101) fra I6. D7-grensa kan skjerpes (eksplisitt «CSV+SQL
|
||||
bygget; HTTP/MCP MAF-only demo, ikke shipped D7-connector»).
|
||||
- `docs/rapport/2026-07-03-sammenligningsrapport.md` — eksisterende stack-sammenligning
|
||||
(refereres, dupliseres ikke).
|
||||
- `docs/plan/2026-07-03-sesjonsplan-ingest.md` (§I7) + `docs/plan/2026-07-03-maalbilde-ingest-lag.md`
|
||||
(§0/§11) — kilder for scope + ærlighets-regel.
|
||||
- **Reusable code:** ingen kode — docs-økt. Gjenbruker eksisterende rapport-mønster
|
||||
(sammenligningsrapporten) + verifiseringslogg-formen fra KTG-verifiseringsplikten.
|
||||
- **External tech (researched):** ingen (brief: 0 research-topics).
|
||||
- **Recent git activity:** I6 lukket på `72cb15f` (review WARN akseptert). I1–I6-commits
|
||||
`e4ee8bd..696f19a` (CSV/SQL/HTTP + golden + load-bearing). Ren tree ved oppstart.
|
||||
|
||||
### D7-søsken-fakta (read-only, grunner rapporten — INGEN skriving/commit der)
|
||||
|
||||
- `/Users/ktg/repos/portfolio-optimiser-claude/README.md` sier: **programmets ene budsjetterte
|
||||
live *modell*-kjøring (S10) er utført + validert 2026-07-03** (linje 13, 76, 121–125), og at
|
||||
http-kilden er «an optional extension point, **not built in** [...] this repo ships no network
|
||||
connector and no live-source integration» (linje 114–118). D7 har CSV+SQL-connectorer (I3/I5).
|
||||
- D7 har **ingen `docs/rapport/`-mappe** → bekrefter «én felles rapport her» (brief-valg).
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
Docs-økt: ingen tester (`Test first` utelatt), `Verify` er konkrete grep/kjør-kommandoer. TDD
|
||||
Iron Law gjelder produksjonskode, ikke dokumentasjon (brief Non-Goal).
|
||||
|
||||
### Step 1: Fiks stale ingest-status i README (specified→implementert, uten overclaim)
|
||||
|
||||
- **Files:** `README.md`
|
||||
- **Changes:** Oppdater Status-blokka (linje 9–13): erstatt «The ingest layer (real data sources)
|
||||
is specified but not yet implemented» med en formulering som skiller **kapabilitet-implementert**
|
||||
fra **kjørt-mot-ekte-kilde**: ingest-laget er implementert — file/CSV og SQL på begge stacker
|
||||
(bit-identiske golden-uttrekk fra delt spec), HTTP som **MAF-only demonstrert extension point mot
|
||||
en lokal mock** — og **kun øvd mot committede fixtures/golden; ingen bundle er materialisert fra
|
||||
en live-kilde**. Behold «no live-model run yet» for MAF-siden (fortsatt sant — offline skriptet
|
||||
sim). **(BLOCKER-fiks, plan-critic):** linje 90–92 — IKKE endre «will materialize» til et bart
|
||||
«materializes … today» (det ville motsi «hand-curated today» og overselge: bundelene er fortsatt
|
||||
hand-curated, connectorene har kun kjørt mot fixtures). Skriv i stedet en reconciled, ærlig
|
||||
setning: de medfølgende eksemplene er hand-curated; ingest-laget som materialiserer en bundle fra
|
||||
en kilde (file/CSV + SQL, HTTP som MAF-only demonstrert extension point) via et deterministisk,
|
||||
schema-validert manifest FØR loopen er **implementert og øvd mot committede fixtures — ingen
|
||||
bundle er ennå materialisert fra en live-kilde**. Ikke rør Karpathy/OKF-seksjonene. Ingen påstand
|
||||
om at D7 har HTTP/MCP. Sibling-presiseringen «file/CSV and SQL mirror the MAF stack» er valgfri —
|
||||
ta den KUN med hvis den ikke trigger Step 1s egen `(claude).{0,40}(http|mcp)`-grep (den gjør den
|
||||
ikke: ingen http/mcp-nærhet).
|
||||
- **Reuses:** eksisterende Status-blockquote-stil i `README.md`; ærlighets-formuleringen fra
|
||||
`docs/extending.md` linje 88–90 («local mock — no live source»).
|
||||
- **Verify:** `grep -c "not yet implemented" README.md` → `0`; `grep -c "local mock" README.md`
|
||||
→ `≥1`; `grep -niE '(d7|claude).{0,40}(http|mcp).{0,20}(support|støtte|built)' README.md` →
|
||||
ingen treff som påstår D7 HTTP/MCP-støtte. **Honesty-gate (major, plan-critic):**
|
||||
`grep -niE '(materializ|materialiser)[^.]{0,40}(real|live|ekte)[^.]{0,20}(source|kilde)' README.md`
|
||||
→ ingen treff som påstår materialisering fra ekte/live kilde som gjort; gjennomlesning bekrefter
|
||||
«no bundle … materialized from a live source»-klausulen er til stede (kapabilitet ≠ live-bruk).
|
||||
- **On failure:** revert — `git checkout -- README.md`
|
||||
- **Checkpoint:** `git commit -m "docs(i7): refresh stale README ingest status — implemented (CSV+SQL both stacks, HTTP MAF-only demo)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- README.md
|
||||
min_file_count: 1
|
||||
commit_message_pattern: "^docs\\(i7\\): refresh stale README ingest status"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- docs/extending.md
|
||||
must_contain:
|
||||
- path: README.md
|
||||
pattern: "local mock"
|
||||
```
|
||||
|
||||
### Step 2: Skjerp D7-grensa i extending.md (CSV+SQL bygget; HTTP/MCP MAF-only + peker)
|
||||
|
||||
- **Files:** `docs/extending.md`
|
||||
- **Changes:** I «D7 sibling hook»-underseksjonen (linje 92–101), gjør grensa eksplisitt per
|
||||
målbilde §11: D7 bygde **file/CSV og SQL**-connectorene (speiler I3/I5); **HTTP/MCP er
|
||||
demonstrert KUN på MAF-siden** mot lokal mock, og `create_sdk_mcp_server` er en **dokumentert
|
||||
krok** en deployer ville brukt — **ikke en shipped D7-connector**; ingen D7-HTTP-økt planlagt.
|
||||
Legg til «MAF-only» eksplisitt der HTTP/MCP nevnes for D7. Ikke endre http-manifest-avsnittene
|
||||
(linje 55–90) — de er korrekte fra I6.
|
||||
- **Reuses:** eksisterende D7-hook-avsnitt (linje 92–101); målbilde §11-formuleringen verbatim
|
||||
som kilde.
|
||||
- **Verify:** `grep -c "MAF-only" docs/extending.md` → `≥1`;
|
||||
`grep -niE 'D7.{0,30}(built|shipped|implement).{0,20}(http|mcp)' docs/extending.md` → ingen
|
||||
treff som påstår D7 bygde HTTP/MCP.
|
||||
- **On failure:** revert — `git checkout -- docs/extending.md`
|
||||
- **Checkpoint:** `git commit -m "docs(i7): sharpen D7 boundary in extending.md — CSV+SQL built, HTTP/MCP MAF-only"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- docs/extending.md
|
||||
min_file_count: 1
|
||||
commit_message_pattern: "^docs\\(i7\\): sharpen D7 boundary"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- README.md
|
||||
must_contain:
|
||||
- path: docs/extending.md
|
||||
pattern: "MAF-only"
|
||||
- path: docs/extending.md
|
||||
pattern: "create_sdk_mcp_server"
|
||||
```
|
||||
|
||||
### Step 3: Skriv felles norsk program-statusrapport (hybrid, verifiseringslogg)
|
||||
|
||||
- **Files:** `docs/rapport/2026-07-04-program-statusrapport-ingest.md` (new file)
|
||||
- **Changes:** Skriv rapporten på norsk i hybrid form:
|
||||
1. **Executive-sammendrag** (KI-seksjon/ledelse): hva ble bygget (ingest-lag på begge stacker),
|
||||
hva er bevist, hva gjenstår — kort, uten salgsspråk.
|
||||
2. **Bevist / ikke bevist** (ærlighets-regel, målbilde §11). MÅ inneholde den **to-aksede
|
||||
«live»-distinksjonen**: (a) **live data-KILDE** (http mot ekte endpoint) — ALDRI gjort,
|
||||
ingen stack; «første live kilde»-gaten ukonsumert. (b) **live MODELL-kjøring** (ekte LLM) —
|
||||
IKKE på MAF (offline skriptet sim beviser plumbing + deterministisk ryggrad, ikke levende
|
||||
LLM-atferd), **DONE én gang på D7 (S10, budsjettert, validert 2026-07-03)**. Per connector:
|
||||
CSV+SQL bevist begge stacker (golden bit-identisk); HTTP MAF-only demo mot mock; MCP kun
|
||||
dokumentert krok.
|
||||
3. **Verifiseringslogg**: tabell der HVER påstand → kjørbar kommando ELLER fil-/commit-sitering.
|
||||
Kjør og fang faktisk output for: MAF-suite (`uv run pytest`), `uv run ruff check .`,
|
||||
`uv run mypy src`, SC1-grep-gaten, og D7-suite **read-only** i søsknet
|
||||
(`cd /Users/ktg/repos/portfolio-optimiser-claude && uv run pytest`; skill «suite rød» fra
|
||||
«kunne ikke kjøres offline» hvis ikke `uv sync`-et). Siter D7 S10-live-run fra D7s egen
|
||||
README (read-only). Referér `sammenligningsrapport.md` (ikke dupliser).
|
||||
- Ingen salgsspråk (KTG-verifiseringsplikt): unngå «fullt ut», «uten unntak». Named
|
||||
markdown-lenker til lokale filer.
|
||||
- **Reuses:** rapport-mønster fra `docs/rapport/2026-07-03-sammenligningsrapport.md`;
|
||||
verifiseringslogg-formen fra KTG-global «Verifiseringsplikt».
|
||||
- **Verify:** fila finnes;
|
||||
`grep -c "Verifiseringslogg" docs/rapport/2026-07-04-program-statusrapport-ingest.md` → `≥1`;
|
||||
`grep -ciE "ikke bevist" docs/rapport/2026-07-04-program-statusrapport-ingest.md` → `≥1`;
|
||||
`grep -niE '(live)[^.]{0,40}(kilde|source|integrasjon)' docs/rapport/2026-07-04-program-statusrapport-ingest.md`
|
||||
→ hvert treff er negasjon («ingen live-kilde»), ingen påstand om live-kilde-integrasjon;
|
||||
gjennomlesning: 0 rader i verifiseringsloggen uten kommando/kilde. **SC5 per-steg (minor):** kjør
|
||||
og fang `uv run ruff check .` (exit 0) + `uv run mypy src` (exit 0) her, siter faktiske exit-koder
|
||||
i loggen (ikke bare i ende-til-ende-sjekken).
|
||||
- **On failure:** revert — `git checkout -- docs/rapport/2026-07-04-program-statusrapport-ingest.md` (slett fila hvis untracked).
|
||||
**Circuit breaker (minor, plan-critic):** hvis MAF-suiten (SC4) er RØD ved evidens-innsamling —
|
||||
STOPP og eskaler; skriv ALDRI en rapport som påstår «grønn offline» over en rød suite. (D7 rød/
|
||||
ikke-kjørbar er derimot forventet-håndtert: rapporteres som «ikke bevist» per SC6, ikke en stopp.)
|
||||
- **Checkpoint:** `git commit -m "docs(i7): program status report — bevist/ikke-bevist + verifiseringslogg (I7)"`
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- docs/rapport/2026-07-04-program-statusrapport-ingest.md
|
||||
min_file_count: 1
|
||||
commit_message_pattern: "^docs\\(i7\\): program status report"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths:
|
||||
- README.md
|
||||
- docs/extending.md
|
||||
- src/portfolio_optimiser/okf.py
|
||||
must_contain:
|
||||
- path: docs/rapport/2026-07-04-program-statusrapport-ingest.md
|
||||
pattern: "Verifiseringslogg"
|
||||
- path: docs/rapport/2026-07-04-program-statusrapport-ingest.md
|
||||
pattern: "S10"
|
||||
```
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
| Approach | Pros | Cons | Why rejected |
|
||||
|----------|------|------|--------------|
|
||||
| Full 7-agent exploration-swarm | Prosess-symmetri | Analyserer Python-arkitektur irrelevant for docs; token-sløsing | /trekplan hard rule «Adaptive/Honesty» — docs-økt, målrettet utforsking i stedet |
|
||||
| Egen D7-lokal rapport + MAF-rapport | Polyrepo-symmetri | Dupliserer «bevist»-narrativ, tråding mellom to | Operatør valgte «én felles rapport her» (brief) |
|
||||
| Slå README+extending sammen til én commit | Færre commits | Blander to distinkte finpuss; svakere headless-revert | To steg = to rene reverts + tydeligere manifest-gate |
|
||||
| Skriv rapport FØR docs-fiks | — | Rapporten ville sitert ustabile/stale docs | Rapport sist (Step 3): siterer ferdig-finpusset README/extending |
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- **Framework:** ingen nye tester — docs-økt. Eksisterende suite er regresjonsvernet.
|
||||
- **Existing patterns:** load-bearing-tester (I1–I6) forblir grønne; I7 rører ikke `src/`,
|
||||
`okf.py` eller `shared/`.
|
||||
- **New tests in this plan:** 0. Verifikasjon er grep-gater + suite-grønn-sjekk (brief SC1–SC6).
|
||||
- **Manuell verifisering:** SC1/SC3 har en menneskelig klassifiseringssteg (docs-prosa) med
|
||||
bundet objektiv proxy (grep + hver logg-rad har kilde) — akseptert av brief-reviewer.
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Priority | Risk | Location | Impact | Mitigation |
|
||||
|----------|------|----------|--------|------------|
|
||||
| Medium | D7-suite ikke `uv sync`-et → pytest feiler av miljøgrunn | søsken-repo | «D7 rød» feilrapporteres | SC6: skill «rød» fra «kunne ikke kjøres offline» eksplisitt i rapporten |
|
||||
| Medium | Conflate live-kilde vs live-modell-kjøring | rapport §bevist | Ærlighets-brudd (over- eller under-claim) | Eksplisitt to-akset distinksjon påkrevd i Step 3; D7 S10 sitert fra D7-README |
|
||||
| Low | Utilsiktet endring i søsken-repo | D7 | Polyrepo-grensebrudd | Hard rule: D7 er READ-ONLY; ingen commit/write der; forbidden i scope-fence |
|
||||
| Low | README-edit rører OKF/Karpathy-prosa | README.md | Scope-drift | Step 1 Changes avgrenser eksakt til Status-blokk + linje 90–92 |
|
||||
| Low | Premiss «237/4» sitert uverifisert | rapport | Feil tall i logg | Step 3 kjører `uv run pytest` og fanger FAKTISK output (premiss-verifisering) |
|
||||
|
||||
## Assumptions
|
||||
|
||||
| # | Assumption | Why unverifiable | Impact if wrong |
|
||||
|---|-----------|-----------------|-----------------|
|
||||
| 1 | D7-suiten er grønn offline (uv-synced) | Ikke kjørt ennå (execute-tid) | SC6 rapporterer «ikke bevist» framfor å fikse — dekket |
|
||||
| 2 | Sammenligningsrapporten dekker stack-sammenligning | Antatt fra tittel/størrelse | Rapport må evt. si mer selv; leses i execute før referanse |
|
||||
| 3 | README/extending trenger finpuss, ikke omskriving | Bekreftet: I6 la http+D7-peker | Ingen — begge filer lest, delta bekreftet |
|
||||
|
||||
## Verification
|
||||
|
||||
*Ende-til-ende integrasjonssjekker på tvers av steg (per-steg manifest kjøres av trekexecute).*
|
||||
|
||||
- [ ] `grep -c "not yet implemented" README.md` → `0` (SC1: stale-status fjernet)
|
||||
- [ ] **SC1-grep (a), briefens fulle scope** `grep -rniE '(d7|claude[- ]?sdk).{0,60}(http|mcp)' README.md docs/` → manuell klassifisering: HVERT treff er enten (i) eksplisitt «MAF-only / demonstrert extension point», (ii) ren D7-krok-peker (`create_sdk_mcp_server`), eller (iii) plan-/målbilde-prosa som FORBYR/negerer D7-HTTP (§11, sesjonsplan) — INGEN treff påstår D7 HTTP/MCP-*støtte* som bygget
|
||||
- [ ] **SC1-grep (b), live-source, briefens verbatim** `grep -rniE '(live|produksjon|production).{0,30}(kilde|source|integrasjon|integration)' README.md docs/rapport/ docs/extending.md` → hvert treff er negasjon/forbud («ingen live-kilde», «no live source», «not built in»), INGEN påstand om live-kilde-integrasjon (dekker README + de to ed.-filene, ikke bare rapporten)
|
||||
- [ ] `test -f docs/rapport/2026-07-04-program-statusrapport-ingest.md` → fila finnes med exec-sammendrag + «bevist/ikke bevist» + «Verifiseringslogg» (SC2)
|
||||
- [ ] Gjennomlesning: 0 rader i verifiseringsloggen uten kjørbar kommando eller fil-/commit-sitering (SC3)
|
||||
- [ ] `uv run pytest` → exit 0, 0 failed (uten nøkkel/nettverk) (SC4)
|
||||
- [ ] `uv run ruff check .` exit 0 og `uv run mypy src` exit 0 (SC5)
|
||||
- [ ] `cd /Users/ktg/repos/portfolio-optimiser-claude && uv run pytest` → exit 0 offline, sitert read-only, INGEN commit der (SC6)
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Files to modify:** 2 (`README.md`, `docs/extending.md`)
|
||||
- **Files to create:** 1 (`docs/rapport/2026-07-04-program-statusrapport-ingest.md`)
|
||||
- **Complexity:** low (docs-økt; ingen kode, ingen nye tester; hovedrisiko = ærlighets-presisjon)
|
||||
|
||||
## Plan Quality Score
|
||||
|
||||
| Dimension | Weight | Score | Notes |
|
||||
|-----------|--------|-------|-------|
|
||||
| Structural integrity | 0.15 | 92 | 3 steg, ren avhengighet (1,2 → 3), ≤5 så ingen Execution Strategy |
|
||||
| Step quality | 0.20 | 90 | Kirurgiske changes, eksakte linjer, manifest per steg |
|
||||
| Coverage completeness | 0.20 | 94 | SC1–SC6 alle sporet til steg + Verification |
|
||||
| Specification quality | 0.15 | 90 | Konkrete grep/kjør-kommandoer, ingen placeholders |
|
||||
| Risk & pre-mortem | 0.15 | 90 | Live-akse-conflate + D7-miljø fanget |
|
||||
| Headless readiness | 0.10 | 90 | On failure + Checkpoint per steg |
|
||||
| Manifest quality | 0.05 | 88 | Gyldige manifester; innholds-absens dekkes av Verify (grep -c 0) |
|
||||
| **Weighted total** | **1.00** | **91** | **Grade: A** |
|
||||
|
||||
**Adversarial review:**
|
||||
- **Plan critic:** REVISE (pre-revisjon 73/C) — 1 blocker, 2 major, 4 minor. Blocker (README
|
||||
real-source-overclaim) + alle major/minor adressert i Revisions under. Post-revisjon: blocker lukket.
|
||||
- **Scope guardian:** ALIGNED — 0 creep, 2 minor (SC1-dekning). Adressert (Rev 3). Polyrepo-grense
|
||||
(D7 read-only) bekreftet respektert.
|
||||
|
||||
## Revisions
|
||||
|
||||
| # | Finding (severity) | Kilde | Resolution |
|
||||
|---|--------------------|-------|------------|
|
||||
| 1 | README linje 90–92 «materializes today» overselger real-source, motsier «hand-curated» (**blocker**) | plan-critic | Step 1 Changes omskrevet: skill kapabilitet-implementert fra kjørt-mot-ekte-kilde; reconciled setning «implementert og øvd mot committede fixtures — ingen bundle materialisert fra live-kilde». Ikke bar nåtid. |
|
||||
| 2 | Grep-gater dekker ikke live-*bruk*-overclaim-vektoren (**major**) | plan-critic | Step 1 Verify: ny honesty-gate-grep (`materializ…real/live…source` → 0) + gjennomlesnings-sjekk på «no live source»-klausul. |
|
||||
| 3 | SC1 narrowet fra briefens `docs/` til 3 filer; live-source-grep manglet i top-level Verification (**major/minor ×3**) | plan-critic + scope-guardian | Verification: begge SC1-grep gjenopprettet verbatim fra brief (hele `docs/`-treet + live-source over README/extending/rapport), med manuell klassifiserings-regel. |
|
||||
| 4 | Step 3 Verify `<fil>`-placeholder ikke headless-kjørbar (**minor**) | plan-critic | Erstattet med literal sti i alle tre grep. |
|
||||
| 5 | Ingen circuit breaker hvis MAF-suite rød ved Step 3 (**minor**) | plan-critic | Step 3 On failure: STOPP+eskaler hvis MAF-suite rød; aldri «grønn»-rapport over rød suite. |
|
||||
| 6 | SC5 (ruff/mypy) kun ende-til-ende, ikke per-steg (**minor**) | plan-critic | Step 3 Verify: kjør + siter faktiske exit-koder i loggen. |
|
||||
| 7 | Step 1 valgfri sibling-CSV+SQL-claim = scope-nudge (**minor**) | plan-critic | Step 1 Changes: markert valgfri, betinget av at den ikke trigger egen http/mcp-grep. |
|
||||
12
.gitignore
vendored
12
.gitignore
vendored
|
|
@ -24,10 +24,14 @@ STATE.md
|
|||
.session-state.local.json
|
||||
.voyage/
|
||||
|
||||
# Voyage genererte render-derivater under .claude/projects/ (kildene .md trackes; HTML
|
||||
# er operatør-lokal annotering m/ localStorage, progress.json er execute-tracker)
|
||||
.claude/projects/**/*.html
|
||||
.claude/projects/**/progress.json
|
||||
# Voyage prosess-artefakter — LOCAL-ONLY (operatørbeslutning 2026-08-05, avgjør (g)).
|
||||
# Briefer/planer/reviews/research som ØKTENE våre genererer er arbeidsbenk, ikke leveranse, og
|
||||
# plandokumenter skal ikke være offentlige. Dette repoet har et offentlig speil (`open/`), så
|
||||
# samme regel som STATE.md over gjelder: privat remote → local-only, aldri tracked.
|
||||
# NB: dette er IKKE en regel mot kuraterte docs — `docs/plan/`, `docs/research/` og `docs/rapport/`
|
||||
# er bevisst tracket og README-lenket som beslutningsloggen. Grensen går ved hvem som skrev dem:
|
||||
# øktmaskineriet (her) vs. et dokument skrevet for repoets lesere (der).
|
||||
.claude/projects/
|
||||
|
||||
# Voyage execute-tracker (efemert per kjøring)
|
||||
.trekexecute-progress-*
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# Azure AI Foundry auth recipe (S4.1)
|
||||
|
||||
> Verified against Microsoft Learn and the pinned `agent-framework-foundry==1.8.2` in this repo's
|
||||
> `.venv` (see `.claude/projects/2026-07-15-s41-azure-preflight/research/01-foundry-auth-recipe.md`).
|
||||
> `.venv`.
|
||||
> This note is the operator's manual auth recipe for the single planned live Foundry run (M1). The
|
||||
> offline `python -m portfolio_optimiser.preflight --profile azure` checks the config; this doc
|
||||
> covers what preflight cannot: the actual authentication.
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ so a raw ``|`` and ``\`` survive un-escaped — the exact discriminator vs ``ren
|
|||
ANY byte of the expected bundle diverges (spec §11 "Golden regression" seam).
|
||||
|
||||
The canned ``get`` returns the committed fixture payloads (rebuild:
|
||||
``.claude/projects/2026-07-04-i6-ingest-http-maf/build_fixture.py``) — no socket, no credentials
|
||||
``examples/ingest-golden-http/build_fixture.py``) — no socket, no credentials
|
||||
(``credential_ref`` is null), so the golden runs offline. The base_url / query never enter the
|
||||
bundle, so the golden is checkout-location independent.
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ preserved, not integer-coerced), non-integral 1200.5/89.9, and the escape order
|
|||
§11 "Golden regression" seam).
|
||||
|
||||
The fixture db is committed under ``fixture/`` (rebuild:
|
||||
``.claude/projects/2026-07-04-i4-ingest-sql-maf/build_fixture.py``); the ``connection_ref`` env
|
||||
``examples/ingest-golden-sql/build_fixture.py``); the ``connection_ref`` env
|
||||
var is set to it (the §4 run-time resolution). The db PATH never enters the bundle, so the
|
||||
golden is checkout-location independent.
|
||||
"""
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue