From 78e03c3254dd00eaadec79e0f2bfd529ca2975c3 Mon Sep 17 00:00:00 2001 From: Kjell Tore Guttormsen Date: Wed, 24 Jun 2026 11:56:57 +0200 Subject: [PATCH] =?UTF-8?q?docs(fase2):=20/trekbrief=20=E2=80=94=20gated?= =?UTF-8?q?=20MVP=20vertical-slice=20brief=20(6/6)=20[skip-docs]?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fase 2 started. Brief for the MVP vertical slice (one synthetic project end-to-end on MAF 1.9.0: debate -> blocking validator -> two-layer HITL + provenance -> ExpeL learning) produced via /trekbrief from documented intent (incremental-plan §Fase 2 + capability-map §1/§5 + Fase 1 findings), not live Q&A per the project operating model. framing=preserve; phase_signals=high/opus. Brief-reviewer gate passed 6/6 (all dimensions 5; verdict PROCEED_WITH_RISKS) after 2 iterations: iter 1 flagged a no-op token-accounting criterion and a false-positive on research-field order; 3 derivable refinements applied (retargeted the UsageDetails assertion, added Open Questions for the concrete local model + Foundry deployment names, added a topic-count note). Validator green (0 errors/warnings). 3 research topics identified (gaps the capability-map did not resolve): native HITL in MAF workflows, local_folder via MCPStdioTool + citation provenance, and a real local-profile chat client (spikes used only FakeChatClient). Manual path: stopped at the brief as a gated checkpoint; research must run before /trekplan. .gitignore: ignore generated .html annotation + progress.json render-derivatives under .claude/projects/ (the .md sources stay tracked). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH --- .../brief.md | 339 ++++++++++++++++++ .gitignore | 5 + 2 files changed, 344 insertions(+) create mode 100644 .claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md diff --git a/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md b/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md new file mode 100644 index 0000000..6ef34b5 --- /dev/null +++ b/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md @@ -0,0 +1,339 @@ +--- +type: trekbrief +brief_version: "2.2" +created: 2026-06-24 +task: "Fase 2 MVP vertical slice — one synthetic project end-to-end on MAF 1.9.0: debate → block-validate → HITL → learn" +slug: fase2-mvp-vertical-slice +project_dir: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ +research_topics: 3 +research_status: pending +auto_research: false +interview_turns: 0 +source: interview +framing: preserve +phase_signals: + - phase: research + effort: high + model: opus + - phase: plan + effort: high + model: opus + - phase: execute + effort: high + model: opus + - phase: review + effort: high + model: opus +--- + +# Task: Fase 2 MVP vertical slice — one synthetic project end-to-end on MAF + +> Generated by `/trekbrief` on 2026-06-24. +> This brief is the contract between requirements and planning. `/trekplan` +> reads it to produce the implementation plan. Every decision in the plan must +> trace back to content in this brief. +> +> **Interview note (project driftsmodell):** this brief was driven from +> documented operator intent — `docs/plan/2026-06-23-incremental-plan.md` §Fase 2, +> `docs/research/2026-06-24-maf-capability-map.md` §1+§5, Fase 1 findings, and +> `CLAUDE.md` invariants — not live Q&A, per the project's standing operating +> model (operator's only input is the trigger phrase; the session executes the +> next backbone step autonomously). `interview_turns: 0` reflects that. The +> operator can correct any derived decision via the annotation HTML. + +## TL;DR + +Fase 2 **preserves** the locked plan, composing the verified Fase 1 spikes into one +MVP vertical slice: ONE synthetic project end-to-end on MAF 1.9.0 — debate → blocking +hybrid validator → two-layer HITL + provenance → ExpeL learning, in both profiles. MAF +GA features wired per the capability-map (adopt tokens/budget-middleware/round-caps/@tool/MCP; keep hand-rolled VerdictStore + validator; promote `fresh_workflow()`). Fan-out over N projects is Fase 3. + +## Intent + +Fase 1 proved the four riskiest assumptions in isolation with throwaway spikes +(maker-checker convergence, fan-out state-bleed, the hybrid validator, ExpeL +retrieval) and produced a verified MAF 1.9.0 capability map that says, per need, +whether to adopt a MAF feature or keep our own. We now need the first thing that +is actually a *system*: a single project flowing through the whole method — +candidate measures debated, a deterministic validator deciding the value, a domain +expert judging via HITL, and the judgment fed back so the next run is smarter. This +slice is the MVP backbone every later phase (generalisation in Fase 3, open-source +in Fase 4, the D7 Claude-SDK sibling) builds on, so getting the seams right — token +budget as middleware, fresh-workflow isolation factory, the inline validator gate, +the ExpeL injection seam — matters more than breadth. The danger we are buying down +is integration risk: each spike worked alone; the open question is whether they +compose into one fail-fast, budget-bounded, provenance-stamped, learning loop on +real (not faked) chat clients in both profiles. + +## Goal + +A runnable vertical slice in which one synthetic "anleggskostnad" project is taken +end-to-end: load its data via a local-folder MCP source with citations, run the +debate (single-agent baseline, escalating to Group Chat maker-checker under a +deterministic termination contract with native round caps and a shared +middleware-driven token budget), pass every candidate through the obligatory +blocking hybrid validator (LLM → Pydantic IR → solver → Monte-Carlo P10/P50/P90) +that returns either a `ValidatedProposal` or a structural `Rejection`, emit exactly +one provenance-stamped saving proposal, capture a domain expert's verdict through a +two-layer HITL, persist it to the hand-rolled VerdictStore, and have the *next* run +retrieve that verdict via the ExpeL `ContextProvider` seam (with the corrected +two-arg `extend_instructions(source_id, instructions)`). Nothing runs without +valid, fail-fast-validated contracts (data-source schema, model-map, termination +contract, feedback schema). The slice runs on the local profile (primary) and is +verified minimally on the Azure/Foundry profile. The hand-rolled `Budget`/ +`TokenMeter` shrinks to a thin shared meter that a `ChatMiddleware` drives off real +`UsageDetails` token counts; word-count proxies are gone. + +## Non-Goals + +- **Fan-out over N projects (Fase 3).** Fase 2 *builds* `fresh_workflow()` as the + core isolation factory (the documented MS State-Isolation pattern) but runs ONE + project through it. Orchestrated fan-out and adding a second project via config + only are Fase 3. +- **Compliance functions (D3).** No DPIA/ROS/behandlingsformål features — the + deployer owns those. We ship only the technical preconditions (local-only by + default, provenance, no silent egress) plus a prominent disclaimer. +- **Preview MAF features on the critical path (D2/D6).** mem0/redis memory, + Cosmos/Redis history, and declarative YAML agent factory are Preview — kept OFF + the MVP critical path. GA core only (sessions, middleware, observability, tools, + MCP, checkpointing). +- **Magentic orchestration.** Experimental; Group Chat maker-checker is the debate + default (CLAUDE.md invariant). +- **Replacing the hand-rolled VerdictStore or validator with MAF memory/eval.** + Both MAF features are the wrong shape (bag-of-words / offline quality score) — + capability-map verdict is KEEP ours. +- **Chasing the last 10% (D5).** Build ~90% generic core + clear extension points; + competent integrators configure the last mile. +- **D7 Claude Agent SDK sibling.** Starts only after this slice works end-to-end. + +## Constraints + +- **Stack:** Python ≥3.10, `uv`, `ruff` (lint+format), `mypy`, `pytest`, Pydantic. +- **MAF:** `agent-framework-core` 1.9.0 with GA packages pinned explicitly (NOT the + `agent-framework[all]` meta). Promote `agent-framework-orchestrations` 1.0.0 from + dev → core (it was throwaway-dev in Fase 1). +- **API truth:** verified against *installed* 1.9.0 source via introspection; docs + via the `microsoft-learn` MCP (Learn pages are NOT version-pinned — where source + and docs disagree, installed source wins). +- **Platform:** Intel mac (no Apple Silicon). CBC solver confirmed working in Fase 1 + (Spike C); PuLP via `pulp[cbc]` / `COIN_CMD`. +- **Two profiles, one core API (D2):** local (fallback, primary for dev) + + Azure/Foundry (full). Pluggable backend; same core API. +- **Cost discipline (D6):** develop on the local profile (free); Foundry/Azure only + for targeted, minimal verification; cheapest models, tiny synthetic data, hard + token/round caps. No heavy test runs. +- **Determinism is blocking:** the validator is obligatory and blocking — never an + optional plugin; it cannot be bypassed. +- **Fail-fast at startup:** no run begins without all contracts validated. +- **STATE.md is local-only** (gitignored); the repo's private Forgejo remote is the + only push target until Fase 4. + +## Preferences + +- **Adopt MAF GA features instead of reinventing (capability-map §1/§5):** real + `UsageDetails` token counts (delete `len(text.split())`); budget cap as a shared + `ChatMiddleware` short-circuit; native `GroupChatBuilder.with_max_rounds`; GA + `@tool`/`FunctionTool` with explicit `schema=`; `MCPStdioTool` (local) / + `MCPStreamableHTTPTool` (remote); observability via `gen_ai.client.token.usage` + (derive cost = tokens × per-model pricing — there is no native cost metric). +- **Promote `fresh_workflow()`** (Spike B) to a core fan-out factory — it IS the + documented Microsoft isolation pattern; never reuse a built workflow across tasks. +- **Keep hand-rolled** VerdictStore (structural Jaccard over typed cost-codes + + measure-type + magnitude bucket) and validator (inline gate returning a domain + object the system branches on). Promote the `ExpeLContextProvider` seam to core + with the two-arg fix `extend_instructions(source_id, instructions)`. +- **Debate ladder:** single-agent baseline first; escalate to Group Chat + maker-checker only under documented need, under a deterministic termination + contract (token-disiplin). +- **Method as Agent Skill:** one `SKILL.md` + deterministic `scripts/` validator; + MAF natively consumes `SKILL.md` via `SkillsProvider.from_paths(...)` — pin the + experimental Skills surface and watch for breaks. +- **Roll a tiny role→deployment map** (a dict/YAML → chat-client ctor); MAF + declarative is Preview + not installed. +- **Solver:** reuse Spike C's PuLP/CBC path in the skill's `scripts/` validator + (proven on Intel mac) unless research surfaces a clearly better fit. + +## Non-Functional Requirements + +- **Hard caps enforced, fail-closed:** a token budget overrun short-circuits via the + `ChatMiddleware`; a round overrun halts via `with_max_rounds`. No unbounded loop + can start (Fase 1 B4: `max_round_count=None` does NOT self-terminate). +- **No silent egress (D3 technical precondition):** on the local profile no project + data leaves the machine; any egress is explicit and configured. +- **Provenance on every emitted proposal (B6):** source citations + model/role + + validator decision + token usage traceable on the single proposal. +- **State isolation:** every project run gets a fresh workflow/session — zero + cross-run conversation bleed (Fase 1 B7, verified on received-message content). +- **Reproducible determinism:** the validator's accept/reject and the Monte-Carlo + percentiles are deterministic for a fixed seed and input. +- **Repo hygiene:** `uv run mypy src`, `uv run ruff check .` clean; tests pass. + +## Success Criteria + +- **Suite green:** `uv run pytest` exits 0 (including new Fase 2 tests); + `uv run ruff check .` and `uv run mypy src` exit 0. +- **End-to-end, one valid proposal:** a single-command run of the synthetic project + on the local profile produces exactly one `ValidatedProposal` (not `Rejection`) + carrying a populated provenance stamp (assert provenance fields: ≥1 citation, + model/role, validator decision, token usage). +- **Structural block works:** an out-of-range / infeasible candidate is blocked by + the validator and returns a `Rejection` (assert type is `Rejection`, with reason). +- **Verdict captured + persisted:** an expert verdict entered via the two-layer HITL + is written to the VerdictStore (assert a verdict record exists for the proposal). +- **Learning closes the loop:** a second run on a structurally similar proposal + retrieves the prior verdict via the ExpeL `ContextProvider` (assert the retrieved + verdict id == the one persisted), exercising `extend_instructions(source_id, …)`. +- **Fail-fast contracts:** running with a malformed data-source / model-map / + termination / feedback contract raises a validation error at startup BEFORE any + agent call (assert it raises; assert no chat-client call was made). +- **Hard-stop respected:** with a tiny budget/round cap, the run terminates via the + middleware/round-cap path (assert `BudgetExceeded`-equivalent or round-cap halt; + assert the loop did not exceed the cap). +- **Real token accounting:** the core token meter reads token usage from + `UsageDetails` (`response.usage_details["total_token_count"]`, `None`-safe) — + assert the meter is populated from `UsageDetails` on a real run (positive, + load-bearing assertion), and assert no `len(... .split())` word-count token proxy + is introduced into `src/` (the Fase 1 proxy lives in `spikes/_harness.py`; `src/` + must stay at 0 such proxies). +- **Both profiles exercised:** the slice runs on the local profile (full run) and a + trivial agent responds on the Azure/Foundry profile (targeted minimal check) — + assert a successful response from each chat-client path. + +## Research Plan + +Three load-bearing topics the Fase 1 capability-map did NOT resolve. Each is +answerable and gates a specific set of plan steps. (`research_topics: 3` in +frontmatter == the three `### Topic` headings below; `research_status: pending` is +correct — these run via the manual `/trekresearch` calls in *How to continue* +BEFORE `/trekplan`; the orchestrator must not auto-advance to planning first.) + +### Topic 1: Native human-in-the-loop in MAF workflows + +- **Why this matters:** the "expert verdict captured via HITL" success criterion + needs a concrete mechanism. If MAF has a native workflow pause/resume for human + input (e.g. a `RequestInfoExecutor` / request-response executor or a + checkpoint-and-resume seam), the two-layer HITL should use it; if not, we capture + the verdict out-of-band post-run and feed it forward via the VerdictStore. The + choice changes the workflow graph and the HITL design substantially. +- **Research question:** "Does `agent-framework` 1.9.0 (core + orchestrations 1.0.0) + provide a native human-in-the-loop primitive to pause a workflow for external + input and resume it, and how does it interact with checkpointing and session + state?" +- **Suggested invocation:** `/trekresearch --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ --external "Does Microsoft Agent Framework 1.9.0 support native human-in-the-loop pause/resume in workflows (RequestInfoExecutor / request-response), and how does it interact with checkpointing and session state?"` +- **Required for plan steps:** HITL design; per-project workflow graph; checkpointing + integration; the verdict-capture step. +- **Confidence needed:** high +- **Estimated cost:** standard +- **Scope hint:** both + +### Topic 2: Local-folder data access via MCP with citation provenance + +- **Why this matters:** the slice needs a `local_folder` data source surfaced as an + MCP tool whose retrieved chunks carry enough metadata for a citation-aware + `AIContextProvider` and the provenance stamp. We must know whether to use an + existing filesystem MCP server (e.g. the official `@modelcontextprotocol/ + server-filesystem`) via `MCPStdioTool`, or build a thin custom local-folder MCP + server that returns citation-able chunks. This decides a build-vs-reuse step and + the provenance data model. +- **Research question:** "What is the best way to expose a local document folder to + a MAF agent via `MCPStdioTool` such that retrieved content carries citation + metadata (file + locator) — is there a suitable existing filesystem MCP server, or + is a thin custom server required, and what citation shape does an AIContextProvider + expect?" +- **Suggested invocation:** `/trekresearch --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ --external "Best way to expose a local document folder to a Microsoft Agent Framework agent via MCPStdioTool with citation metadata — existing filesystem MCP server vs thin custom server, and the citation shape an AIContextProvider expects?"` +- **Required for plan steps:** data-access step; citation-aware context provider; + provenance-stamping; build-vs-reuse decision for the local-folder server. +- **Confidence needed:** high +- **Estimated cost:** standard +- **Scope hint:** both + +### Topic 3: Real local-profile chat client for agent-framework 1.9.0 + +- **Why this matters:** every Fase 1 spike used `FakeChatClient` — no real LLM. The + "both profiles" criterion needs a REAL local model on the primary (free) profile. + We must know how `agent-framework` 1.9.0 talks to a local model (an + OpenAI-compatible local endpoint via `OpenAIChatClient`, an Ollama integration, or + another local `BaseChatClient`) and how `UsageDetails` is populated there (some + providers return `None` — which our middleware must handle). +- **Research question:** "How does `agent-framework-core` 1.9.0 run agents against a + local model on the free/local profile (OpenAI-compatible endpoint, Ollama, or + other local chat client), and does that path populate `UsageDetails` token counts + or return `None`?" +- **Suggested invocation:** `/trekresearch --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ --external "How does Microsoft agent-framework-core 1.9.0 run agents against a local model (OpenAI-compatible endpoint, Ollama, or local chat client) on a free local profile, and does it populate UsageDetails token counts?"` +- **Required for plan steps:** backend-profile abstraction (local path); the budget + middleware's `None`-handling; the end-to-end run; cost-discipline verification. +- **Confidence needed:** high +- **Estimated cost:** standard +- **Scope hint:** both + +## Open Questions / Assumptions + +- **[ASSUMPTION]** Solver stays PuLP/CBC in the skill's `scripts/` validator (Spike + C proved it on Intel mac); `mcp-solver` is not pursued unless Topic research shows + a clear win. +- **[ASSUMPTION]** The role→deployment model-map is a small hand-rolled dict/YAML → + chat-client constructor; the available Foundry deployment names are + tenant-specific (operator-supplied), not externally researchable. +- **[ASSUMPTION]** "1 turn = 1 round" matches our intent for `with_max_rounds` on the + Group Chat (capability-map flagged this to confirm during implementation). +- **[ASSUMPTION]** The synthetic reference domain from Fase 0 (small fictional + "anleggskostnad" projects) is reused as the single project for this slice; no new + domain data is invented. +- **[OPEN]** Whether checkpointing is on the Fase 2 critical path or deferred — + capability-map said "ADOPT (later)"; default is defer past MVP unless Topic 1 + shows HITL needs checkpoint-resume. +- **[OPEN]** Exact two-layer HITL semantics (synchronous review vs async + + notification stub) — annenrangs risk in the plan; the notification stub (B11) is a + stub only in Fase 2. +- **[OPEN]** Which concrete local model runs on the Intel mac for the local profile. + Topic 3 settles the *mechanism* (how 1.9.0 talks to a local model); the concrete + installed model/endpoint is machine-specific and operator-supplied. The whole + free-local-profile run hinges on a working local endpoint existing. +- **[OPEN]** Which Foundry deployment names back the role→deployment map for the + Azure/Foundry-profile check (tenant-specific, operator-supplied). The "both + profiles exercised" success criterion depends on this operator input. + +## Prior Attempts + +Fase 0 (skeleton + synthetic domain + pluggable-backend shell) and Fase 1 (four +throwaway de-risk spikes A–D) are complete and pushed. Fase 1 closed via +`/trekreview` round 2 = ALLOW after a real BLOCKER was found and fixed: Spike B(b)'s +fan-out "state isolation" had been confirmed with a tautological call-counter; the +rebuilt experiment measured real conversation bleed on received-message content and +confirmed the footgun, validated against Microsoft's own Workflows "State Isolation" +doc. The MAF 1.9.0 capability map (the front-end input to this brief) was then +produced from two grounded research passes (installed-source introspection + MS +Learn MCP). Key Fase 1 facts carried forward: `BaseChatClient` drives the GA +builders; `max_round_count=None` does NOT self-terminate (external guard required); +reused workflow accumulates the conversation thread (use a fresh instance per task); +`ContextProvider` requires `source_id`; `extend_instructions` is two-arg; real token +usage lives on `UsageDetails`, not word-counts. + +## Metadata + +- **Created:** 2026-06-24 +- **Interview turns:** 0 (driven from documented operator intent per project driftsmodell) +- **Auto-research opted in:** no +- **Source:** trekbrief interview + +--- + +## How to continue + +Manual (default): + +```bash +# Run each research topic (order does not matter): +/trekresearch --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ --external "Does Microsoft Agent Framework 1.9.0 support native human-in-the-loop pause/resume in workflows (RequestInfoExecutor / request-response), and how does it interact with checkpointing and session state?" +/trekresearch --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ --external "Best way to expose a local document folder to a Microsoft Agent Framework agent via MCPStdioTool with citation metadata — existing filesystem MCP server vs thin custom server, and the citation shape an AIContextProvider expects?" +/trekresearch --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ --external "How does Microsoft agent-framework-core 1.9.0 run agents against a local model (OpenAI-compatible endpoint, Ollama, or local chat client) on a free local profile, and does it populate UsageDetails token counts?" + +# Then plan: +/trekplan --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ + +# Then execute: +/trekexecute --project .claude/projects/2026-06-24-fase2-mvp-vertical-slice/ +``` + +Auto (opt-in during `/trekbrief`): research and planning run +automatically; only execution is manual. diff --git a/.gitignore b/.gitignore index 2700daa..a5df369 100644 --- a/.gitignore +++ b/.gitignore @@ -23,3 +23,8 @@ STATE.md # Voyage execution-bokholderi (efemert per kjøring; STATE.md er kanonisk kontinuitet) .session-state.local.json .voyage/ + +# Voyage genererte render-derivater under .claude/projects/ (kildene .md trackes; HTML +# er operatør-lokal annotering m/ localStorage, progress.json er execute-tracker) +.claude/projects/**/*.html +.claude/projects/**/progress.json