diff --git a/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/research/01-native-hitl-maf-workflows.md b/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/research/01-native-hitl-maf-workflows.md new file mode 100644 index 0000000..68fcfcf --- /dev/null +++ b/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/research/01-native-hitl-maf-workflows.md @@ -0,0 +1,273 @@ +--- +type: trekresearch-brief +created: 2026-06-24 +question: "Does Microsoft Agent Framework 1.9.0 (core + orchestrations 1.0.0) provide a native human-in-the-loop primitive to pause a workflow for external input and resume it, and how does it interact with checkpointing and session state?" +confidence: 0.85 +dimensions: 6 +mcp_servers_used: [microsoft-learn, tavily] +local_agents_used: [installed-source-introspection (orchestrator, main context)] +external_agents_used: [docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (unavailable)] +topic: 1 +brief: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md +--- + +# Native human-in-the-loop in MAF 1.9.0 workflows + +> Generated by trekresearch (Voyage 5.6.0) on 2026-06-24. Topic 1 of 3 for the +> Fase 2 MVP-vertical-slice brief. Scope: external swarm + installed-source +> introspection (API truth from the installed 1.9.0 package wins over Learn docs, +> per CLAUDE.md). Gemini second opinion was unavailable (MCP SDK broke on +> Google's May-2026 Interactions API change) — triangulation rests on docs + +> community + contrarian + installed source. + +## Research Question + +Does `agent-framework` 1.9.0 (`agent-framework-core` + `agent-framework-orchestrations` 1.0.0) +provide a native human-in-the-loop (HITL) mechanism to pause a workflow for +external/human input and resume it, and how does it interact with checkpointing +and session/conversation state? Decision it feeds: the Fase 2 "expert verdict +captured via HITL" success criterion — design the two-layer HITL + verdict-capture ++ feed-forward seam, and the per-project workflow graph. + +## Executive Summary + +**Yes — native HITL exists and is GA in 1.9.0**, in three distinct surfaces: +(1) a workflow-level request/response primitive (`ctx.request_info()` + `@response_handler`, +resume via `run(responses=...)`), (2) durable checkpoint-and-resume that persists +pending requests, and (3) an orchestration-level human-approval gate that — verified +in installed source — **works on our Group Chat maker-checker default** via +`GroupChatBuilder.with_request_info(agents=[...])`. **Confidence on the mechanism is +HIGH (installed-source-verified); confidence on durable checkpoint-resume RELIABILITY +is LOW/contradictory** — official docs promise lossless durable resume, but the bug +tracker shows the *pause* half is solid while the *persist-and-resume* half has +multiple open/silent-failure bugs surviving into the 1.9.0 line (#5818, #6127, #6372, +#5621) plus a pickle type-allowlist that rejects custom Pydantic types on restore (#5810). +**Key caveat:** our learned-verdict path is an *indefinite, out-of-band, never-resume-the- +original-run* decision whose durable artifact is the hand-rolled VerdictStore — so the +recommendation is to use the GA *in-run* approval gate for synchronous Layer-1 review, +keep the durable learned verdict OUT-OF-BAND in the VerdictStore, and **defer +checkpointing off the MVP critical path**. + +## Dimensions + +### 1. Native HITL primitive — Confidence: high + +**Installed-source findings (ground truth, 1.9.0):** +- `WorkflowContext.request_info(request_data: object, response_type: type, *, request_id: str | None = None)` — any executor calls this to suspend the workflow and request external input (`agent_framework/_workflows/_workflow_context.py:393`). +- Every `Executor` inherits `RequestInfoMixin` (`_workflows/_executor.py:30`); a method decorated `@response_handler` receives the typed response on resume; handlers are matched by the request/response **type annotations** (`_workflows/_request_info_mixin.py`). +- `RequestInfoExecutor` is **NOT present** in installed 1.9.0 (only `RequestInfoMixin` + `response_handler` + `ctx.request_info`). `send_responses`/`send_responses_streaming` are **NOT public** on `Workflow` — internal `_send_responses_internal` only. + +**External findings:** +- Docs + AutoGen→MAF migration guide confirm "workflows can pause execution and wait for external input before continuing" — a capability AutoGen's `Team` lacked. +- The versioned upgrade guides explain why the installed surface looks as it does: `RequestInfoExecutor` removed (python-1.0.0b251104) in favour of `ctx.request_info()` + `@response_handler`; `send_responses*` removed (python-1.0.0b260210, PR #3720) in favour of `run(responses=...)`. Both predate 1.9.0 (2026-06-18). , + +**Contradictions:** +- The auto-generated **API-reference** page (`agent-framework-python-latest`) still lists `send_responses*` and prose-mentions `RequestInfoExecutor`. This **conflicts** with both the dated changes-guide and the installed source. Resolution: **installed source wins** (CLAUDE.md invariant) — those names are doc-lag / the C# (`RequestPort`) idiom. Do not write them into Python 1.9.0 code. + +### 2. Pause/resume mechanics — Confidence: high + +**Installed-source findings:** +- `Workflow.run(message=None, *, responses=None, checkpoint_id=None, checkpoint_storage=None)` — three intents, exactly one of `message` / `responses` / `checkpoint_id` per call (`_workflows/_workflow.py:675+`). `responses` is `Mapping[str, Any]` keyed by `request_id`; mutually exclusive with `message`; **can be combined with `checkpoint_id`** ("restore then send responses in a single call" per the docstring). +- Run-state enum (`_workflows/_events.py:58-65`): `IDLE`, `IDLE_WITH_PENDING_REQUESTS` (paused awaiting input — non-terminal), `IN_PROGRESS_PENDING_REQUESTS`. `result.get_final_state()` returns it. +- `RunnerContext.send_request_info_response(request_id, response)` validates the response **type** against the original request and raises `ValueError` on unknown `request_id` or type mismatch (`_workflows/_runner_context.py:457-471`). +- Events collapsed to a generic `WorkflowEvent[DataT]` carrying `.request_id` / `.data` / `.type` (no `RequestInfoEvent` subclass). + +**External findings:** +- Canonical loop (official sample `guessing_game_with_human_input.py`): `run(..., stream=True)` → collect `request_info` events → `run(stream=True, responses=pending)` → repeat until no pending requests. **In-process resume needs NO checkpoint** — "state is preserved across multiple calls to run." +- Request IDs are caller-supplied or auto-UUID; responses strongly typed via `response_type`. + +### 3. Checkpointing interaction — Confidence: high (mechanism) / contradictory (reliability) + +**Installed-source + docs findings (mechanism — high):** +- `WorkflowCheckpoint.pending_request_info_events: dict[str, ...]` — pending HITL requests ARE serialized into the checkpoint (`_workflows/_checkpoint.py:81`). On restore the runner rehydrates them (`_runner_context.py:424-425`) and **re-emits** them as `request_info` events; you then answer with a *separate* `run(responses=...)` call (you cannot inject responses during the restore call itself). +- Storage backends (all implement the `CheckpointStorage` protocol, swap without code change): `InMemoryCheckpointStorage` (ephemeral), `FileCheckpointStorage` (local disk, explicit `storage_path`, **pickle** + restricted unpickler), `CosmosCheckpointStorage` (Azure, preview — egress). Checkpoints fire at superstep boundaries, so a HITL pause lands on one cleanly. + +**Contradictions (reliability — LOW):** +- **Docs** present durable "checkpoint → exit process → restore → respond" as a supported happy path. +- **Community / bug tracker** shows persist-and-resume is the fragile half: + - #5818 (OPEN, Magentic + `AgentSession` resume): same `request_id` sent back, workflow does not resume. + - #6127 (unanswered, Sequential): resume re-prompts the same approval **and** re-executes the function; duplicate `CallId` in streaming chunks. + - #6372 (fixed PR #6491, but affected 1.1.0–**1.9.0**): fan-in barrier silently loses buffered messages across checkpoint/resume — **no error raised**. + - #5621 (OPEN, Handoff restore): `Expected exactly one update for key 'SharedState'`, unmatched ToolApproval. + - #5810 (fixed PR #6049): restore **type-allowlist** blocks non-safe types (`Checkpoint deserialization blocked for type … MessageRole`) — custom/Pydantic payloads must be registered via `allowed_checkpoint_types`. + - #3255 (fixed PR #3689): sub-workflow restore re-sent already-answered requests → `Response provided for unknown request ID`. +- **Version-skew (documented):** PR #3744 — "**Existing checkpoints cannot be resumed between versions.**" A MAF bump invalidates stored checkpoints. + +### 4. Session / conversation-state interaction — Confidence: high + +**External findings:** +- The built-in `AgentExecutor` (wraps an agent inside a workflow) serializes on checkpoint: internal message cache, **full conversation history**, agent session state, and pending requests/responses — and restores them. So a HITL-paused agent-bearing workflow does NOT lose chat history. +- **Material gap for the Azure/Foundry profile:** "Checkpointing with agents that use **server-side sessions** (e.g. `FoundryAgent`) has limitations. Server-side session state is **not captured in checkpoints**." A durable HITL pause with a Foundry-hosted agent will not have its conversation reliably restored from the checkpoint alone. (Same doc.) +- For pure custom-executor workflows (no `AgentExecutor`): only shared `state`, in-transit messages, and pending requests are captured; executor-local fields persist only if you override `on_checkpoint_save()` / `on_checkpoint_restore()`. +- Pre-1.9.0 rename relevant to code around a pause: `SharedState`→`State`, `ctx.shared_state`→`ctx.state`, state getters/setters now **synchronous** (PR #3667). + +**Cross-link:** this is the Fase 1 B7 bleed vector — cross-run conversation state lives in `AgentSession.state` + `InMemoryHistoryProvider`. The HITL/checkpoint state is workflow-level and distinct, but `fresh_workflow()` isolation still governs whether a restored conversation contaminates the next project run. (See `docs/research/2026-06-24-maf-capability-map.md`, Fase 1 B7.) + +### 5. Group-chat / orchestration-level approval gate — Confidence: high + +**Installed-source findings (decisive for our debate default):** +- `GroupChatBuilder.with_request_info(*, agents: Sequence[str | SupportsAgentRun] | None = None)` **EXISTS** (`agent_framework_orchestrations/_group_chat.py:882`). It pauses after the named agent(s) respond and emits a `request_info` event (`type='request_info'`) "that allows the caller to review the conversation and optionally [approve/edit] … the standard response_handler/request_info pattern." Same method exists on `SequentialBuilder` (`_sequential.py:154`). +- Participants matching the filter are wrapped as `AgentApprovalExecutor(WorkflowExecutor)` (`_orchestration_request_info.py:168`), constructed `allow_direct_output=True` so the user-approved final response surfaces as workflow output. +- The human reply object is `AgentRequestInfoResponse` (public export) with `.approve()` (accept as-is), `.from_strings([text])`, `.from_messages([...])` (`_orchestration_request_info.py:44-85`). Supplied via `run(responses={request_id: AgentRequestInfoResponse...})`. +- `resolve_request_info_filter(agents)` selects which agents pause for approval — e.g. pause only before the **checker** in maker-checker. +- A separate, lighter gate also exists: tool-approval via `@tool(approval_mode="always_require")` → `function_approval_request` content → approve/deny (in-run, no graph). + +**External findings:** +- Docs/maintainer confirm there is **no single universal "approve/edit/reject" object**; the documented routes are (a) `with_request_info` + `AgentRequestInfoResponse`, (b) tool-approval, (c) Magentic plan-review (`enable_plan_review`) — Magentic is experimental and OFF our path. For arbitrary maker-checker over agent output, the intended route is `with_request_info` (now installed-source-confirmed for GroupChat) or a custom executor calling `ctx.request_info()`. +- **Caveat:** community shows the orchestration approval *resume* path is where bugs cluster (#5818, #6127, #6006) — the gate fires reliably; persisting/resuming the approval across serialization is fragile. + +### 6. Design implication for our two-layer HITL + verdict capture — Confidence: high (recommendation) + +This is decision-relevant — see **Recommendation** below. Short form: the brief's +fallback assumption ("if MAF lacks native HITL, capture out-of-band") is **partly +inverted**: MAF *has* native HITL, and the right split is to use the GA *in-run* +pieces for synchronous review but keep the *durable learned verdict* out-of-band. + +## External Knowledge + +### Best Practice +- Canonical 1.9.0 HITL = `ctx.request_info()` + `@response_handler` + `run(responses=)`; detect pause via `IDLE_WITH_PENDING_REQUESTS` / `get_request_info_events()`. Official samples: `guessing_game_with_human_input.py`, `sequential_request_info.py`, `checkpoint/checkpoint_with_human_in_the_loop.py`, `magentic_human_plan_review.py` (all under `microsoft/agent-framework` `python/samples/03-workflows/`). + +### Security (relevant to D3 no-silent-egress + local-only) +- `FileCheckpointStorage` writes **unencrypted pickle** blobs containing conversation history + pending HITL payloads + shared state. Encryption-at-rest / permissions / ACLs are **the developer's responsibility** (no built-in encryption). Lock the `storage_path` down (dedicated dir, `0700`/`0600`, encrypted volume; macOS FileVault helps). +- Restore is hardened: restricted unpickler **ON by default** since 1.0.1 (we're on 1.9.0); non-safe types throw `WorkflowCheckpointException` unless registered in `allowed_checkpoint_types`. Treat it as a safety net, not the control — "never load checkpoints from untrusted sources." +- **No CVEs** against `agent-framework*` (GitHub Security Advisories empty; OSV empty). Two transitive Starlette advisories on deps.dev — confirm via `uv run pip-audit`. Semantic Kernel CVEs (CVE-2026-26030/-25592) are a *different package* and not in the checkpoint path; only relevant if `semantic-kernel` is pulled in (it is not, per our pinned tree). +- **Telemetry OFF by default** — no exporter ships; `ENABLE_INSTRUMENTATION`/`ENABLE_SENSITIVE_DATA`/`ENABLE_CONSOLE_EXPORTERS` all default false. No-silent-egress holds out of the box if we (a) don't set those, (b) set no `OTEL_EXPORTER_OTLP_*`, (c) use File/InMemory (never Cosmos). One subtlety: MAF auto-propagates OTel trace context into MCP `tools/call` `_meta` when a span is active — inert with instrumentation off; keep MCP servers local (stdio) regardless. + +### Known Issues +- See Dimension 3 contradictions. Plus: `with_request_info()` naming is opaque and there is no event-type filter for which events trigger a HITL pause (#3534, open). Local-model + HITL is **unverified by anyone** — zero community signal; several worst resume bugs are Azure-server-side-persistence-specific and may simply not apply locally, but then we own conversation-history persistence ourselves. Spike locally; do not rely on precedent. + +## Gemini Second Opinion + +Unavailable. The `gemini-mcp` server's client SDK predates Google's May-2026 +Interactions API breaking change and returned `400 BadRequestError` before any +research ran. No independent Gemini triangulation was obtained for this topic; +treat the second opinion as absent (not negative). To restore: upgrade the +gemini-mcp server's client SDK to ≥ 2.0.0. + +## Synthesis + +The triangulation surfaces an insight no single source states: **MAF 1.9.0 has +three different HITL surfaces, and the strongest one for *our* need is the +lightest one — while the heaviest one (durable checkpoint-resume) is both the +shakiest in practice and a poor fit for the problem.** + +1. **In-run synchronous review** (`GroupChatBuilder.with_request_info` → + `AgentApprovalExecutor` → `run(responses={id: AgentRequestInfoResponse...})`) + is GA, installed-source-confirmed for our Group Chat maker-checker default, and + needs **no checkpointing** (state persists across `run()` calls in-process). + This is the solid, happy-path piece. + +2. **Durable cross-process pause** (checkpoint + restore + responses) is where the + docs promise and the bug tracker diverge hardest: open/silent-failure resume + bugs into the 1.9.0 line, a pickle type-allowlist that fights our Pydantic IR, + and "checkpoints cannot be resumed between versions." The contrarian pass is + right that coupling our **highest-value data path** (the verdict the system + *learns* from) to MAF's **most-churned, least-durable, Azure-favoring** surface + is a self-inflicted risk — and a lock-in against the D7 Claude-SDK sibling, + which has no executor/checkpoint model and can only share a *framework-agnostic* + verdict seam. + +3. The actual shape of "fagekspert enters a verdict the next run learns from" is + an **indefinite, out-of-band, never-resume-the-original-run** decision. Its + durable artifact is the hand-rolled **VerdictStore**, not an in-flight workflow + checkpoint. A workflow checkpoint is engineered for "pause seconds-to-minutes, + resume the same process" — the wrong tool for "pause indefinitely, decide + elsewhere." + +So the brief's binary ("native HITL → use it; else out-of-band") resolves to a +**split**: adopt the GA in-run approval gate for the *synchronous Layer-1 review* +where it fits; keep the *durable, learning-loop verdict* out-of-band in the +VerdictStore; and **defer checkpointing off the MVP critical path** (matches the +brief's `[OPEN]` default — and Topic 1 confirms the default rather than overturning +it, because durable checkpoint-resume is the fragile part). + +## Open Questions + +- **Layer-1 review: native gate vs. simplest possible?** `with_request_info` is + GA and fits, but a custom executor calling `ctx.request_info()` gives full + control over the request payload (the `ValidatedProposal` + provenance). Decide + in /trekplan: native `with_request_info(agents=[checker])` vs custom request-info + executor. Either way: in-process, no checkpoint. +- **Is Layer-1 even in the MVP, or is it async-only?** The brief marks two-layer + HITL semantics (sync review vs async+notification stub) as `[OPEN]`. If Fase 2 + ships only the async/out-of-band verdict + notification *stub* (B11), the native + in-run gate may be deferred too — fewer moving parts. Resolve in /trekplan. +- **Local-profile HITL behaviour** — unverified by anyone. Needs a self-spike: + run the approval gate / `request_info` loop against the local OpenAI-compatible + endpoint (Topic 3) and confirm it fires and resumes. Tie to Topic 3's outcome. +- **If durable pause is ever needed:** the acceptance gate must be a + resume-integrity test (pause → checkpoint → restore → assert pending requests + match + conversation intact), given #6372/#5621-class silent failures. Out of + MVP scope but record the condition. + +## Recommendation + +**For the Fase 2 MVP, capture the expert verdict OUT-OF-BAND and keep the durable +artifact in the hand-rolled VerdictStore; do NOT couple the learned-verdict path +to native checkpoint-resume.** Concretely: + +1. **Layer 2 (durable, learning loop) = out-of-band VerdictStore.** The emitted + `ValidatedProposal` + provenance is presented to the fagekspert out-of-band; the + verdict is written to the VerdictStore; the next run retrieves it via the ExpeL + `ContextProvider` seam (`extend_instructions(source_id, instructions)`). This is + framework-agnostic, D7-portable, and the success-criterion ("second run + retrieves the prior verdict") is satisfied without any MAF checkpoint. +2. **Layer 1 (optional in-run synchronous review) = GA native gate IF included.** + If Fase 2 ships a synchronous review, use `GroupChatBuilder.with_request_info( + agents=[checker])` (or a custom `ctx.request_info()` executor for a richer typed + payload) — in-process, resume via `run(responses=...)`, **no checkpointing**. + Register any custom Pydantic response type's expectations now so it survives if + checkpointing is ever added. +3. **Defer checkpointing off the MVP critical path** (brief `[OPEN]` default + upheld). `InMemoryCheckpointStorage` is fine for tests; do not put + `FileCheckpointStorage` durable HITL on the critical path. If added later: + File/InMemory only (never Cosmos — egress), locked-down `storage_path`, + `allowed_checkpoint_types` for our IR, a resume-integrity acceptance test, and + awareness of cross-version checkpoint invalidation. +4. **Provenance, not checkpoint, carries the audit trail** — the single emitted + proposal's provenance stamp (citations + model/role + validator decision + token + usage) is the durable record, consistent with the no-silent-egress + provenance + NFRs. + +Risks to carry into the plan: (a) MAF HITL/checkpoint API churn → any MAF-native +HITL code carries upgrade cost; pin the surface and watch the changes guide. +(b) Local-profile HITL is unverified → spike with Topic 3's local client. +(c) If a synchronous in-run gate is used, the orchestration *resume* path is where +community bugs cluster — keep it in-process (no serialization boundary) to dodge +that whole class. + +## Sources + +| # | Source | Type | Quality | Used in | +|---|--------|------|---------|---------| +| 1 | `.venv/.../agent_framework/_workflows/_workflow_context.py:393` (`request_info`) | codebase | high | Dim 1, 2 | +| 2 | `.venv/.../agent_framework/_workflows/_request_info_mixin.py` (`response_handler`, `RequestInfoMixin`) | codebase | high | Dim 1 | +| 3 | `.venv/.../agent_framework/_workflows/_workflow.py:675+` (`run(responses=, checkpoint_id=)`) | codebase | high | Dim 2, 3 | +| 4 | `.venv/.../agent_framework/_workflows/_events.py:58-65` (`WorkflowRunState`) | codebase | high | Dim 2 | +| 5 | `.venv/.../agent_framework/_workflows/_runner_context.py:424,457-471` (rehydrate + `send_request_info_response`) | codebase | high | Dim 2, 3 | +| 6 | `.venv/.../agent_framework/_workflows/_checkpoint.py:81` (`pending_request_info_events`) | codebase | high | Dim 3 | +| 7 | `.venv/.../agent_framework_orchestrations/_group_chat.py:882` (`GroupChatBuilder.with_request_info`) | codebase | high | Dim 5 | +| 8 | `.venv/.../agent_framework_orchestrations/_orchestration_request_info.py:44-85,168` (`AgentRequestInfoResponse`, `AgentApprovalExecutor`) | codebase | high | Dim 5 | +| 9 | | official | high | Dim 1,2,3,5 | +| 10 | | official | high | Dim 3, Security | +| 11 | | official | high | Dim 4 | +| 12 | | official | high | Dim 1,2,3 | +| 13 | | official | high | Dim 1 | +| 14 | | official | high | Dim 1 | +| 15 | (Magentic AgentSession resume, OPEN) | community | high | Dim 3,5 | +| 16 | (Sequential re-prompt+re-exec) | community | medium | Dim 3,5 | +| 17 | (fan-in barrier silent loss, 1.1.0–1.9.0) | community | high | Dim 3 | +| 18 | (Handoff restore fails, OPEN) | community | high | Dim 3 | +| 19 | (checkpoint type-allowlist) | community | high | Dim 3, Security | +| 20 | (sub-workflow dup request, fixed) | community | high | Dim 3 | +| 21 | (with_request_info naming/filter, OPEN) | community | medium | Known Issues | +| 22 | (no universal approval object; custom executor) | community | high | Dim 5 | +| 23 | | community | medium | Synthesis | +| 24 | (checkpoint limitations) | community | medium | Synthesis | +| 25 | (stability/production timeline, unanswered) | community | medium | Synthesis | +| 26 | (no advisories) | official | high | Security | +| 27 | (no results) | official | high | Security | +| 28 | (telemetry off by default) | official | high | Security | +| 29 | (local provider exists) | official | medium | Known Issues | +| 30 | (1.9.0 = 2026-06-18) | official | high | Exec summary | diff --git a/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/research/02-local-folder-mcp-citation-provenance.md b/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/research/02-local-folder-mcp-citation-provenance.md new file mode 100644 index 0000000..2015ac8 --- /dev/null +++ b/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/research/02-local-folder-mcp-citation-provenance.md @@ -0,0 +1,245 @@ +--- +type: trekresearch-brief +created: 2026-06-24 +question: "What is the best way to expose a local document folder to a MAF 1.9.0 agent via MCPStdioTool such that retrieved content carries citation metadata (file + locator) — existing filesystem MCP server vs thin custom server, and the citation shape a context provider expects?" +confidence: 0.82 +dimensions: 6 +mcp_servers_used: [microsoft-learn, tavily] +local_agents_used: [installed-source-introspection (orchestrator, main context)] +external_agents_used: [docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (unavailable)] +topic: 2 +brief: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md +--- + +# Local-folder data access via MCP with citation provenance (MAF 1.9.0) + +> Generated by trekresearch (Voyage 5.6.0) on 2026-06-24. Topic 2 of 3 for the +> Fase 2 MVP-vertical-slice brief. Scope: external swarm + installed-source +> introspection (installed 1.9.0 API truth wins over Learn docs). Gemini +> unavailable (MCP SDK broke on Google's May-2026 API change). + +## Research Question + +What is the best way to expose a LOCAL document folder to a MAF 1.9.0 agent via +`MCPStdioTool` so retrieved content carries citation metadata (file + locator) — +reuse an existing filesystem MCP server, or build a thin custom server — and what +citation shape does MAF's context/annotation model expect? Decision it feeds: the +data-access step, the citation-aware context provider, provenance-stamping, and the +build-vs-reuse decision for the local-folder server. + +## Executive Summary + +**The brief's lean ("reuse the official `@modelcontextprotocol/server-filesystem` ++ attach citations via `parse_tool_results`") does not survive contact with the +requirement and should be dropped.** The official filesystem server returns raw +text and bare path strings only — **no line/char offsets, no chunk IDs; its +`search_files` is filename-glob, not content search** — so it physically cannot +yield a `(file + locator + snippet)` citation; you would have to re-read, re-chunk +and re-locate inside the callback (i.e. build the whole citation engine anyway, in +the worst place, behind a Node subprocess). MAF also does **not** auto-create +citation annotations from MCP results. **Recommendation (honouring the CLAUDE.md +"data access via MCP, JSON-Schema-validated, fail-fast" convention): build a THIN +CUSTOM local-folder MCP server that returns citation-ready structured chunks +(`{file, locator, snippet, score}`), wrapping a framework-agnostic in-process +retriever core that is the D7-portable seam.** Carry provenance as first-class +Pydantic data on the emitted proposal, independent of MAF's annotation propagation +(which has an open Python streaming-drop bug, #4316). Confidence 0.82: the +capability facts are HIGH (installed-source + official-README verified); the +exact MVP retrieval depth (keyword vs local embeddings) is a deferrable design +choice. + +## Dimensions + +### 1. Can the official filesystem server carry citations? — Confidence: high (NO) + +**External findings (docs + community + contrarian agree):** +- `@modelcontextprotocol/server-filesystem` tools: `read_text_file` (whole file, or `head`/`tail` by line *count* only), `read_multiple_files`, `list_directory`, `directory_tree`, `get_file_info`, `search_files`, `write_file`, `edit_file`. +- Read results = **raw UTF-8 text, no line numbers, no byte/char offsets, no chunk IDs.** `search_files` returns **matching file PATHS only** (filename glob — NOT content grep). `get_file_info` = size/timestamps, not a locator. +- Net: the only locator it gives is the file path you already passed in. Enhanced community forks advertise `read_file_lines` / grep-with-line-numbers / chunking precisely because the official server lacks them. + +**Conclusion:** the official server cannot produce `(file + locator + snippet)` citations. It is fine only for "let the agent open named files," not citation-backed retrieval. + +### 2. Does MAF auto-create citations from MCP results? — Confidence: high (NO) + +**Installed-source findings (1.9.0):** +- Default parser `_parse_contents_from_mcp_tool_result` / `_parse_tool_result_from_mcp` maps MCP content → `Content.from_text/from_data/from_uri`; **attaches no annotations** (`agent_framework/_mcp.py:507-517`). +- `structuredContent` IS now serialized to text Content (`_mcp.py:586-587`) — so the #3313 "structuredContent dropped → None" bug **is fixed in installed 1.9.0**. But it lands as JSON *text*, not as typed/annotated content — you still parse it yourself for provenance. +- The seam to attach citations is `MCPTool.__init__(parse_tool_results: Callable[[CallToolResult], str | list[Content]])` — your callback fully replaces the default parse. + +**External findings:** +- Confirmed: docs/RAG paths use **prompt-level** "cite your sources" (a `string_mapper`), not structured annotations. The one citation-aware RAG provider, `TextSearchProvider`, is **.NET-only — no Python equivalent**. +- **Open bug #4316:** file-citation annotations are **silently dropped in the Python streaming path** (`annotations=None`); .NET fixed, Python lagging. → do NOT rely on MAF's `Annotation` propagation for load-bearing provenance. + +### 3. Citation shape MAF expects — Confidence: high + +**Installed-source findings (1.9.0, ground truth):** +- `Annotation` is a **TypedDict** (`agent_framework/_types.py:374`): `type: Literal["citation"]`, `title`, `url`, `file_id`, `tool_name`, `snippet`, `annotated_regions: Sequence[TextSpanRegion]`, `additional_properties`, `raw_representation`. +- `TextSpanRegion` TypedDict (`_types.py:366`): `type: Literal["text_span"]`, `start_index`, `end_index`. +- The class form `CitationAnnotation` was replaced by this TypedDict in 1.0.0b260123 (PR #3252) — installed 1.9.0 = **TypedDict** form. Build citations as dicts: `{"type":"citation","file_id":...,"title":...,"snippet":...,"annotated_regions":[{"start_index":..,"end_index":..}]}`. +- Carrier: `FunctionResultContent.annotations: list[... ] | None` (accepts dict/TypedDict). + +**Use:** our provenance stamp's "≥1 citation" maps cleanly onto this `Annotation`. But because of #4316, treat `Annotation` as a *display/serialization* surface and keep the authoritative provenance as our own Pydantic object. + +### 4. Context provider injection seam — Confidence: high (corrects an external error) + +**Installed-source findings (1.9.0) — and a flagged docs-vs-source conflict:** +- `class ContextProvider` (`agent_framework/_sessions.py:351`), `__init__(self, source_id: str)` — **source_id REQUIRED** (confirms Fase 1). +- Hooks: `async def before_run(...)` (`:370`) and `async def after_run(...)` (`:391`) — add messages/instructions/tools in `before_run`, process/store in `after_run`. +- `Context.extend_instructions(self, source_id: str, instructions: str | Sequence[str])` (`:253`) and `extend_tools(source_id, tools)` (`:266`) — **the two-arg seam the brief and Fase 1 specified, confirmed verbatim.** +- `MemoryContextProvider` / `InMemoryHistoryProvider` exist but are conversation memory, **not** retrievers. + +> **Premiss correction (important for /trekplan):** the docs-researcher reported, with high confidence from the Learn API-ref page, that Python uses `invoking`/`invoked` hooks and that `source_id` / `extend_instructions` are "C#-only, not in Python." **That is wrong for installed 1.9.0** — the installed source has `before_run`/`after_run`, `ContextProvider(source_id)`, and `extend_instructions(source_id, …)`. Per CLAUDE.md, installed source wins. Do NOT let the planning phase adopt the `invoking`/`invoked` / no-source_id surface. + +**Design fit:** a custom `ExpeLContextProvider(ContextProvider)` (Fase 1 seam, promote to core) injects retrieved+cited content + prior verdicts in `before_run` via `extend_instructions(source_id, …)`, and the same provider family is where the ExpeL learning loop reads VerdictStore. No built-in local-folder retrieval provider exists in Python — retrieval is ours to build. + +### 5. Build vs reuse (the decision) — Confidence: high (recommendation) + +**Triangulated picture:** +- **Reuse official server:** cannot cite (Dim 1); adds a Node runtime + `npx` cold-start + version-pin burden; npx PATH/nvm handshake failure is THE most-reported stdio blocker on macOS (B5); supply-chain surface (chalk/debug + Shai-Hulud hit MCP-SDK transitives). Security: usable only if pinned ≥ `2025.7.1` (EscapeRoute CVE-2025-53109/53110, fixed there; current `2026.1.14` clean), vendored (never `npx @latest`), allowlist scoped to the doc folder (never `$HOME`). +- **Thin custom MCP server:** removes the npm supply-chain surface, emits provenance natively (`{file, line_range/char_span, snippet, score}` via `structuredContent`), honours the CLAUDE.md "data access via MCP, JSON-Schema-validated, fail-fast" convention, and is portable to the D7 Claude-SDK sibling's in-process-MCP idiom. Cost: **we own** path-canonicalisation + symlink-realpath validation (the exact bug class the official server had fixed twice — TDD against the EscapeRoute scenarios) and input validation (no shelling out; use native file APIs). +- **In-process retriever, no MCP** (contrarian's first choice): simplest + lowest latency + best D7 fit, but conflicts with the project's "data access via MCP" convention. Keep as the fallback if that convention is relaxed. +- Reference design: `shinpr/mcp-local-rag` (LanceDB file-based + local embeddings; returns `path + chunk index + title + score + chunk text` + `read_chunk_neighbors` context expansion) — the closest local-only, citation-grade pattern. + +### 6. Retrieval depth for the MVP — Confidence: medium + +- The MVP uses a **tiny synthetic "anleggskostnad" domain** (Fase 0). A full vector stack is not required for D5 (90%) / D6 (cost). A simple in-process retriever (keyword/substring + chunk-at-ingest with exact locators) likely suffices and keeps deps minimal. +- The Python "real RAG" path (SK `VectorStore.create_search_function().as_agent_framework_tool()`) **requires pulling in `semantic-kernel ≥ 1.38`** — a different package with its own RCE CVE (CVE-2026-26030, eval in in-memory vector-store filter, < 1.39.4). Adding it conflicts with the "GA-pakker pinnet eksplisitt, ikke metaen" dependency discipline and adds CVE surface. **Defer / avoid for MVP** unless semantic retrieval proves necessary; if adopted, pin ≥ 1.39.4. + +## External Knowledge + +### Best Practice +- Canonical `MCPStdioTool` wiring (official docstring + Learn): `MCPStdioTool(name="filesystem", command="npx", args=["-y","@modelcontextprotocol/server-filesystem", ])`, used as `async with`; requires `pip install mcp --pre`. +- MAF can also expose an *agent* as an MCP server (`agent.as_mcp_server()`) — not relevant to local-folder retrieval. Generic server-building is the MCP Python SDK / FastMCP domain. + +### Security (D3 no-silent-egress + local-only) +- Stdio MCP server makes **no network calls**; only inherent egress in the path is the `npx` fetch at launch (removed by vendoring/pinning). OTel `_meta` trace-context injection into `tools/call` is inert with instrumentation off and carries only random W3C IDs (no project data). +- EscapeRoute CVEs (CVE-2025-53110 prefix-collision, CVE-2025-53109 symlink-escape) — both **fixed in 2025.7.1**; exploit trigger is prompt injection of the agent (relevant: our maker-checker + blocking validator are compensating controls). Never run the server with elevated privileges. +- npx supply chain: pin exact version + lockfile, verify tarball for `postinstall`, quarantine fresh versions, consider containerization. A thin custom dependency-light server removes this surface entirely. + +### Known Issues +- MAF MCP bugs — **verified against installed 1.9.0**: #3313 structuredContent parsing **FIXED** (`_mcp.py:586`); #2884 stdio session-invalidation handled (`is_connected` reset + `ClosedResourceError` catch, `_mcp.py:935/1355/1544`). Still: #2284 `_meta` dropped (don't rely on `_meta` for provenance — use `structuredContent`/`content`); #4316 Python streaming citation-annotation drop (open). +- npx/nvm PATH handshake failure on macOS (servers#64) — use absolute `command` path + explicit `env` PATH/NODE_PATH, prefer pre-installed/`uvx`-pinned over cold `npx -y`. Filesystem server fails to start if any allowed dir is unavailable (servers#3232). +- Don't split a citation across `content` vs `structuredContent` vs `_meta` — clients (MAF included) forward them inconsistently; put it in one reliably-forwarded channel. + +## Gemini Second Opinion + +Unavailable (same MCP-SDK/Interactions-API failure as Topic 1). No independent +Gemini triangulation for this topic; treat as absent, not negative. + +## Synthesis + +Three sources converge on a conclusion the brief did not anticipate: **the +citation requirement, not the data-access requirement, drives the design — and it +rules out the off-the-shelf filesystem server.** Citations of the form +`(file + locator + snippet)` must be *exact by construction*, which means +chunk-and-locate **at ingest**, in code we own. Bolting them on afterward — over a +third-party server's lossy whole-file text, through a `parse_tool_results` callback, +relying on a MAF annotation path with an open streaming-drop bug — stacks three +fragilities to recreate something we could have produced cleanly up front. + +The right shape is a **two-layer seam**: (1) a framework-agnostic in-process +retriever core — `retrieve(query) -> list[RetrievedChunk]`, where `RetrievedChunk` +is a Pydantic type carrying `file`, `locator` (line range / char span), `snippet`, +`score` — which is the **D7-portable contract** (MAF and Claude Agent SDK both +satisfy it); and (2) a **thin custom MCP server** wrapping that core, returning the +chunks as structured `structuredContent`, which honours the project's "data access +via MCP, JSON-Schema-validated, fail-fast" convention and ports to the Claude SDK's +in-process-MCP idiom. Provenance lives as first-class Pydantic data on the emitted +proposal; the MAF `Annotation` TypedDict is only a display/serialisation view of it, +so #4316 never sits on the load-bearing path. The official filesystem server is +demoted to a possible *future* "browse named files" convenience, pinned and scoped. + +This also threads the CLAUDE.md needle: the contrarian's "skip MCP, in-process +only" is the simplest and the best D7 fit, but it breaks the MCP-data-access +convention; the thin-custom-MCP-server-over-an-in-process-core keeps the convention +*and* the D7 portability *and* citation fidelity — at the cost of owning sandbox +correctness (testable, bounded). + +## Open Questions + +- **MVP retrieval depth:** keyword/substring + ingest-time chunking (no extra deps, + D5/D6-aligned) vs local embeddings (LanceDB/SK VectorStore, heavier + CVE surface). + Default to keyword for the tiny synthetic domain; revisit only if recall is poor. + Resolve in /trekplan. +- **Locator format:** line range vs char span. It is the shared D7 contract — define + and test it in the golden-suite up front. Default: char span (`start_index`/ + `end_index`) to map straight onto `TextSpanRegion`; carry line numbers in + `additional_properties` for human display. +- **Does the MVP even need MCP, or is the in-process retriever enough?** The CLAUDE.md + convention says data access via MCP. The thin-custom-server honours it; confirm the + operator wants the MCP boundary in Fase 2 vs deferring it (the in-process core is + needed either way). Surface in /trekplan. +- **Chunk-at-ingest vs at-query:** ingest-time chunking gives exact, cacheable + locators; decide whether the MVP ingests once or reads live. Tie to determinism NFR. + +## Recommendation + +**Build a thin custom local-folder MCP server that returns citation-ready +structured chunks, wrapping a framework-agnostic in-process retriever core. Do NOT +reuse `@modelcontextprotocol/server-filesystem` for citation-backed retrieval.** + +1. **Retriever core (D7 seam):** `retrieve(query, top_k) -> list[RetrievedChunk]`; + `RetrievedChunk(file: str, locator: TextSpan, snippet: str, score: float)`. + Chunk + assign locators at ingest so citations are exact by construction. + Start with keyword/substring retrieval (no heavy deps; D5/D6). Pure Python, + no MAF/MCP imports — portable to the Claude SDK sibling. +2. **Thin custom MCP server (MAF-facing, honours the convention):** a minimal + `mcp`/FastMCP stdio server exposing a `search(query)` tool that returns the + chunks via `structuredContent` (JSON-Schema-validated, fail-fast on bad config). + Path handling is security-critical: canonicalise + boundary-check (no + `startsWith`), resolve symlink realpath and fail closed, native file APIs (no + shelling out) — TDD against the EscapeRoute bypass patterns. +3. **Wire into MAF:** `MCPStdioTool(name="docfolder", command=, args=[...])` + with a `parse_tool_results` callback that maps `structuredContent` chunks into + `FunctionResultContent` + `Annotation(type="citation", file_id, snippet, + annotated_regions=[TextSpanRegion])`. On Intel mac, pass absolute `command` + + explicit `env` PATH; pre-install the server (no cold `npx`/uvx fetch on the path). +4. **Provenance = first-class Pydantic data** on the emitted `ValidatedProposal` + (≥1 citation: file + locator + snippet, plus model/role + validator decision + + token usage). The MAF `Annotation` TypedDict is only its display view — route + around #4316 by never depending on MAF annotation propagation for the assertion. +5. **Citation-aware context injection:** a custom `ExpeLContextProvider(ContextProvider)` + (`source_id` required) uses `before_run` + `extend_instructions(source_id, …)` to + inject cited content (and prior verdicts) — the confirmed two-arg seam. +6. **If the official filesystem server is ever used** (future "browse named files"): + pin ≥ `2025.7.1` vendored (never `npx @latest`), allowlist the doc folder only + (never `$HOME`), absolute node path + explicit env. + +Fallback (only if the MCP-data-access convention is relaxed for the MVP): skip the +MCP server and call the in-process retriever directly via a `@tool`/`ContextProvider` +— simplest, lowest-latency, best D7 fit. + +Risks to carry into the plan: (a) we own MCP-server sandbox correctness — treat +path/symlink validation as security-critical, TDD it; (b) MAF Python citation +annotation propagation is buggy in streaming (#4316) — provenance must be our own +data; (c) adding `semantic-kernel` for vector RAG pulls a CVE-bearing package — defer. + +## Sources + +| # | Source | Type | Quality | Used in | +|---|--------|------|---------|---------| +| 1 | `.venv/.../agent_framework/_mcp.py:507-517,586-587,655` (default parser, structuredContent) | codebase | high | Dim 2, Known Issues | +| 2 | `.venv/.../agent_framework/_mcp.py:911,935,1330,1355,1544` (session lifecycle, ClosedResourceError) | codebase | high | Known Issues | +| 3 | `.venv/.../agent_framework/_types.py:366,374` (`TextSpanRegion`, `Annotation` TypedDict) | codebase | high | Dim 3 | +| 4 | `.venv/.../agent_framework/_sessions.py:351,370,391,253,266` (`ContextProvider`, `before_run`/`after_run`, `extend_instructions`) | codebase | high | Dim 4 | +| 5 | (tool surface, no locators) | official | high | Dim 1 | +| 6 | | official | high | Dim 1,5 | +| 7 | | official | high | Dim 5, Best Practice | +| 8 | | official | high | Best Practice | +| 9 | (TextSearchProvider .NET-only; SK VectorStore path) | official | high | Dim 2, 6 | +| 10 | (Annotation TypedDict PR #3252) | official | high | Dim 3 | +| 11 | (Python streaming citation drop, OPEN) | community | high | Dim 2, Synthesis | +| 12 | (structuredContent dropped, fixed) | community | high | Dim 2, Known Issues | +| 13 | (stdio session not invalidated, fixed #3154) | community | high | Known Issues | +| 14 | (`_meta` discarded) | community | medium | Known Issues | +| 15 | (npx/nvm PATH handshake) | community | high | Dim 5, Known Issues | +| 16 | (startup fails on unavailable dir) | community | medium | Known Issues | +| 17 | (EscapeRoute CVEs) | community | high | Dim 5, Security | +| 18 | (CVE-2025-53109) | official | high | Security | +| 19 | (CVE-2025-53110) | official | high | Security | +| 20 | (no advisories > 2025.7.1) | community | high | Security | +| 21 | (chalk/debug, Shai-Hulud) | community | high | Dim 5, Security | +| 22 | (local-only citation-grade RAG reference) | community | medium | Dim 5 | +| 23 | (Claude SDK in-process MCP idiom) | official | high | Dim 5, Synthesis | +| 24 | (hosted FileSearch cites service-side) | official | medium | Dim 2 | +| 25 | (MCP token bloat) | community | medium | Known Issues | diff --git a/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/research/03-local-profile-chat-client.md b/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/research/03-local-profile-chat-client.md new file mode 100644 index 0000000..5f14d4f --- /dev/null +++ b/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/research/03-local-profile-chat-client.md @@ -0,0 +1,237 @@ +--- +type: trekresearch-brief +created: 2026-06-24 +question: "How does agent-framework-core 1.9.0 run agents against a local model on the free/local profile (OpenAI-compatible endpoint, Ollama, or other local chat client), and does that path populate UsageDetails token counts or return None?" +confidence: 0.80 +dimensions: 6 +mcp_servers_used: [microsoft-learn, tavily] +local_agents_used: [installed-source-introspection (orchestrator, main context)] +external_agents_used: [docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (unavailable)] +topic: 3 +brief: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md +--- + +# Real local-profile chat client for agent-framework 1.9.0 + UsageDetails + +> Generated by trekresearch (Voyage 5.6.0) on 2026-06-24. Topic 3 of 3 for the +> Fase 2 MVP-vertical-slice brief. Scope: external swarm + installed-source +> introspection (installed API truth wins). Gemini unavailable (MCP SDK broke on +> Google's May-2026 API change). + +## Research Question + +How does `agent-framework-core` 1.9.0 (+ `agent-framework-openai`) run agents +against a LOCAL model on the free/local profile, and does that path populate +`UsageDetails` token counts or return `None`? Decision it feeds: the backend-profile +abstraction (local path), the budget middleware's `None`-handling, the end-to-end +run, and cost-discipline verification. + +## Executive Summary + +**Two local paths exist, and the brief's framing needs two corrections.** Path A — +`OpenAIChatCompletionClient(base_url=)` — is **installed (1.8.2), needs +zero new dependencies, and populates `UsageDetails` None-safely from the local +server's `usage` object**. Path B — the native `OllamaChatClient` — is a **`--pre` +beta package (`agent-framework-ollama 1.0.0b260521`) that is NOT installed** and +adds the `ollama` client dependency. **Correction 1 (client):** use +`OpenAIChatCompletionClient` (Chat Completions API — the surface Ollama/LM Studio +`/v1` exposes), NOT `OpenAIChatClient` (Responses API). **Correction 2 (streaming):** +run the debate/validator path **NON-STREAMING** — that simultaneously dodges the +well-documented `/v1` streaming+tool-calling drop bugs AND the streaming +`usage=None` problem (`UsageDetails` is reliably populated only non-streaming). +**Token accounting works** (confirmed in installed source), so the "real token +accounting" success criterion is satisfiable on the local profile. The real limits +are model capability and Intel-Mac CPU speed: small CPU-runnable models are weak at +tool-calling, and an Intel Mac (no Metal/MLX, ~4–6 tok/s on 7B) makes a multi-round +debate take minutes — so local is for plumbing/smoke verification, with +representative-quality runs pushed to Foundry. Confidence 0.80. + +## Dimensions + +### 1. Local chat-client paths — Confidence: high + +**Installed-source findings (ground truth):** +- `agent-framework-openai` 1.8.2 exports **both** `OpenAIChatClient` (Responses API) and `OpenAIChatCompletionClient` (Chat Completions API) — both import OK in the venv. `RawOpenAIChatClient`/`RawOpenAIChatCompletionClient` are the lower-level variants. +- Both accept `base_url` (→ env `OPENAI_BASE_URL`) and `api_key` (required even for local — pass a dummy like `"ollama"`/`"not-needed"`). `model_id` selects the model. +- The native Ollama path: `agent_framework/ollama/__init__.pyi` re-exports `OllamaChatClient`, `OllamaChatOptions`, `OllamaEmbeddingClient`, … from `agent_framework_ollama` — but **`agent_framework_ollama` is NOT installed** (`import agent_framework_ollama` → ModuleNotFoundError; only the lazy re-export stub exists). Installing it (`uv add agent-framework-ollama --prerelease=allow`) pulls `ollama >=0.5.3,<0.5.4`. + +**External findings:** +- **Correct client for local `/v1`:** Ollama/LM Studio/vLLM expose a Chat Completions surface (`/v1/chat/completions`), so `OpenAIChatCompletionClient` is the technically correct client; `OpenAIChatClient` targets the Responses API which local servers don't expose. Two official MS docs disagree on which to use for Ollama — **flagged conflict; the Chat-Completions client is correct for `/v1`.** +- Official local-endpoint table: Ollama `http://localhost:11434/v1/`, LM Studio `http://localhost:1234/v1/`, vLLM `http://localhost:8000/v1/`. The `/v1/` suffix matters; the native `OllamaChatClient` uses `:11434` WITHOUT `/v1`. +- Native `OllamaChatClient` is **preview/beta** (`1.0.0b260521`, "4-Beta", `--pre`), wraps `ollama.AsyncClient` (native `/api/chat`), `OTEL_PROVIDER_NAME='ollama'`. MS docs describe it as having "full support for function tools and streaming." + +### 2. UsageDetails population — Confidence: high + +**Installed-source findings (the load-bearing answer):** +- `OpenAIChatCompletionClient`: `_chat_completion_client.py:705` — `usage_details=self._parse_usage_from_openai(response.usage) if response.usage else None`. **None-safe.** `_parse_usage_from_openai` (`:757-776`) maps `usage.prompt_tokens`→`input_token_count`, `usage.completion_tokens`→`output_token_count`, `usage.total_tokens`→`total_token_count` (+ reasoning/cached/audio details). +- `UsageDetails` is the TypedDict in `agent_framework/_types.py` (`input/output/total_token_count: int | None`); `add_usage_details(u1, u2)` helper exists (`_types.py:417`) — use it to accumulate across calls in the shared budget meter. Lands on `response.usage_details` (`_types.py:547`); read `response.usage_details["total_token_count"]` (None-safe). +- Streaming: `_chat_completion_client.py:724-726` — `if chunk.usage: Content.from_usage(...)` — usage only arrives if the provider emits a usage chunk. + +**External findings:** +- **Non-streaming:** Ollama/LM Studio/llama.cpp `/v1` all return a populated `usage` object → `UsageDetails` populated. Confirmed multi-source. +- **Streaming:** OpenAI semantics require `stream_options={"include_usage": true}`; whether MAF sets it automatically is **unverified** — so streamed `UsageDetails` may be `None`. → run non-streaming for the accounting path. +- **Native `OllamaChatClient`:** maps Ollama's `prompt_eval_count`→input, `eval_count`→output; **`total_token_count` is NOT set (derive it)**. Streaming updates carry no usage. +- MAF usage plumbing is young: "Fix OTel usage detail attributes" landed in python-1.8.0 (already in our 1.9.0). + +### 3. Tool-calling / structured output on local models — Confidence: high (load-bearing risk) + +**External findings (strong multi-source pattern):** +- Small CPU-runnable models are markedly weak at tool-calling: 7–8B tool-selection F1 ≈ 0.48–0.57; failure modes = never calling the tool (answers from memory), emitting the tool call as **plain text/JSON** instead of a structured `tool_calls` object, and argument type drift (`"5"` vs `5`). +- Qwen3-class is the most reliable small family (F1 ≈ 0.93 in the same eval); MAF docs explicitly bless `qwen3:4b` / `llama3.2` for tools and warn "not all models support function calling." `num_ctx ≥ 32k` improves reliability. +- Streaming + tools on `/v1` historically silently dropped/leaked tool calls (5+ repos; fixed 2025-05-28 but a 0.12.3 regression remains) — **another reason to run non-streaming.** +- Structured output: `response_format: type[BaseModel]` is available on both local-path clients, but reliability is model-dependent. +- MAF bug #1772: `OpenAIChatClient` + Ollama + ChatContext middleware inserts the system prompt twice — verify against 1.9.0; the native client may avoid it. + +**Design fit:** our **blocking Pydantic validator becomes the mitigation** — it must retry/repair on parse failure, turning model unreliability into a bounded retry loop rather than silent corruption. + +### 4. Intel-Mac CPU performance reality — Confidence: medium-high + +**External findings:** +- Intel Macs run Ollama **CPU-only** (no Metal/MLX — Apple-Silicon-only). Reported ~4–6 tok/s on a 7B model (i9 2019); 3–8 tok/s CPU-only generally — "usable for testing, painful for actual work." +- A maker-checker debate (N agents × M rounds × tool round-trips × validator retries) at single-digit tok/s runs into minutes per candidate. + +**Conclusion:** Intel-Mac local is a **plumbing-verification + smoke-test** environment (does the slice wire up, do tool calls fire, does the validator parse, does usage come back?), not an iterative/batch run environment — exactly matching D6 ("develop on local, verify minimally on Foundry"). Keep local runs tiny: 1 maker + 1 checker, ≤3 rounds, tiny synthetic data, hard token caps, a 3–4B model. + +### 5. Determinism boundary — Confidence: high + +- Local LLM output is non-deterministic even at temperature 0 / fixed seed (documented edge cases). This does **not** conflict with the success criterion **iff** the determinism criterion is scoped to the **validator + Monte-Carlo** (fixed seed + fixed candidate input → identical accept/reject + identical P10/P50/P90), NOT to the LLM that produces the candidate. +- **Trap to avoid:** never assert end-to-end determinism *through* the LLM. Golden tests pin the validator's *inputs*, not the LLM's *outputs*. Our architecture already separates "agents propose (non-deterministic) → validator decides (deterministic)", so the boundary is sound — make it explicit in the test design. + +### 6. Security / no-egress posture — Confidence: high + +**External findings:** +- `OpenAIChatCompletionClient(base_url=localhost)` keeps inference on-box; the OpenAI SDK sends only to `base_url` and never contacts `api.openai.com`; a dummy `api_key` is safe. **No new dependency** (uses already-installed `agent-framework-openai`). This is the lower-supply-chain-cost path. +- Native `agent-framework-ollama` is official MS but **beta** and adds the `ollama` client dep (pinned `<0.5.4`) — a supply-chain + GA-discipline cost. +- Ollama daemon: bind **`127.0.0.1`** (never `0.0.0.0` — it has NO auth by design, CVE-2025-63389); pin Ollama **≥ 0.17.1** (clears CVE-2024-37032 "Probllama" RCE, the 39719-22 set, and CVE-2026-7482 "Bleeding Llama" GGUF heap-read); keep `OLLAMA_DEBUG` unset (else prompts are written to `server.log`). +- **Model-pull is explicit egress** (downloads from a registry) — document it in provenance; not "silent" but real. Air-gap option: pre-stage model blobs offline. LM Studio persists chats/RAG locally by default (treat as data-at-rest). +- No native telemetry/phone-home from MAF client (instrumentation off by default), Ollama (no inference telemetry), or LM Studio (off by default). + +## External Knowledge + +### Best Practice +- Local wiring: `OpenAIChatCompletionClient(base_url="http://127.0.0.1:11434/v1/", api_key="ollama", model_id="qwen3:4b")` (non-streaming); or native `OllamaChatClient(host=..., model_id=...)` after `--pre` install. Cost: no native cost metric — derive `cost = tokens × per-model price` (local = 0); token usage on OTel `gen_ai.client.token.usage`. + +### Known Issues +- Streaming+tools `/v1` drop/leak (run non-streaming); `#1772` double-system-prompt with OpenAIChatClient+Ollama+middleware (verify vs 1.9.0); native client `total_token_count` not set (derive); `prompt_eval_count` historically flaky for cached/large prompts. **No community report exists of MAF *Python* against a *local* model on an *Intel Mac* with token usage — this scenario is under-documented; verify empirically.** + +## Gemini Second Opinion + +Unavailable (same MCP-SDK/Interactions-API failure as Topics 1–2). No independent +Gemini triangulation; treat as absent, not negative. + +## Synthesis + +The brief's premise — "run locally via `OpenAIChatClient(base_url=...)`; `UsageDetails` +works because the local server returns OpenAI-style `usage`" — is *directionally +right but wrong in two specifics that would fail success criteria on the very +profile we develop on*. First, the **Chat Completions** client +(`OpenAIChatCompletionClient`), not the **Responses** client (`OpenAIChatClient`), +is the one that talks to a local `/v1` endpoint and the one whose verified +`_parse_usage_from_openai` populates `UsageDetails`. Second, **streaming is the +trap**: it is exactly where local `/v1` silently drops/leaks tool calls AND where +`usage` comes back `None` unless `include_usage` is set (uncertain in MAF). Running +the debate/validator path **non-streaming** fixes both at once — and non-streaming +is what we want anyway for deterministic accounting. + +The contrarian's "use the native `OllamaChatClient` instead" is a real option (MS +docs route you to it; it sidesteps `/v1` translation edges and gets +`prompt_eval_count`/`eval_count` directly), but it conflicts with the project's +**GA-pin dependency discipline** (it is a `--pre` beta + a new `ollama` dep). The +resolution is a **spike-gate**: start with the zero-new-dep Chat-Completions +`base_url` path non-streaming; if a quick spike shows tool-calls leaking as text, or +`usage=None`, or the #1772 middleware bug biting, switch to the native client and +accept the scoped `--pre` dependency. Either way the model-map → chat-client ctor +abstraction (D2) hides the choice behind the profile. + +The deeper truth the swarm surfaces: on Intel-Mac CPU, "local primary" means +**local proves the slice RUNS** (plumbing, tool-calls fire, validator parses, usage +populated) — it does **not** prove the agents produce *good* candidates. "Both +profiles exercised" therefore means: local = it runs; one minimal Foundry pass +(cheapest model, hard cap) = the candidates are substantively sound. And the model's +tool-calling weakness is absorbed by our **blocking validator-as-retry** — the +obligatory validator stops being just a gate and becomes the reliability mechanism +that makes a weak local model usable. + +## Open Questions + +- **Concrete local model + endpoint** (brief `[OPEN]`, operator-supplied): default + to `qwen3:4b` via Ollama on `127.0.0.1:11434` for dev/smoke; `qwen2.5:7b`/`qwen3:8b` + for reliability checks. The whole free-local run hinges on a working local endpoint + existing on the machine. +- **Foundry deployment names** (brief `[OPEN]`, tenant-specific, operator-supplied) — + back the role→deployment map for the minimal Azure/Foundry-profile check. +- **base_url vs native client** — resolve via a one-call spike in /trekexecute: + does `OpenAIChatCompletionClient(base_url)` non-streaming return populated + `UsageDetails` and structured tool-calls on the chosen model? If yes, no new dep; + if no, switch to native `OllamaChatClient`. +- **Does MAF set `include_usage` on streaming?** Unverified — irrelevant if we run + non-streaming, but confirm if streaming is ever needed for narration. + +## Recommendation + +**Run the local profile via `OpenAIChatCompletionClient(base_url=)`, +non-streaming, behind the D2 model-map abstraction; keep the native `OllamaChatClient` +as a spike-gated fallback.** + +1. **Client:** `OpenAIChatCompletionClient(base_url="http://127.0.0.1:11434/v1/", + api_key="ollama", model_id="qwen3:4b")` — installed, **zero new deps**, honours the + GA-pin discipline. NOT `OpenAIChatClient` (Responses API). +2. **Non-streaming for debate + validator** — dodges `/v1` streaming+tools drop AND + streaming `usage=None` in one move. Streaming only for non-tool narration, if ever. +3. **Budget middleware: treat `usage_details is None` as a HARD FAIL in dev** + (assertion) so a usage-reporting regression can never silently disable the budget + cap. Maps directly to the "real token accounting" success criterion (assert the + meter is populated from `UsageDetails` on a real run). Accumulate via + `add_usage_details(...)`. No `len(...split())` proxy in `src/`. +4. **Validator-as-retry:** the blocking Pydantic validator must retry/repair on parse + failure — small models emit unparseable/text-leaked tool calls; this turns the + obligatory validator into the reliability mechanism. +5. **Model:** `qwen3:4b` for dev/smoke (MAF-blessed, tool-capable, CPU-tolerable); + `qwen2.5:7b`/`qwen3:8b` for reliability checks; `num_ctx=32k`. +6. **Determinism golden tests:** pin validator + Monte-Carlo inputs (fixed seed → + identical decision/percentiles); never assert determinism through the LLM. +7. **"Both profiles exercised":** local (full run) proves the slice RUNS; one minimal + Foundry pass (cheapest model, hard token cap, per D6) proves candidates are + substantively sound. Keep local runs tiny (1 maker + 1 checker, ≤3 rounds). +8. **Spike-gate the client choice** in /trekexecute: if `base_url` non-streaming shows + tool-call text-leakage, `usage=None`, or #1772, switch to native `OllamaChatClient` + (accept the `--pre` `agent-framework-ollama` dep + verify its `prompt_eval_count`/ + `eval_count` mapping; derive `total_token_count`). +9. **Security hardening:** Ollama bind `127.0.0.1`, pin **≥ 0.17.1**, `OLLAMA_DEBUG` + unset; document model-pull as explicit egress; dummy `api_key`. + +Risks to carry into the plan: (a) Intel-CPU latency → local is plumbing/smoke, not +fast iteration; (b) small-model tool-calling weakness → lean on validator-retry + +Qwen3; (c) native-client fallback is `--pre` (GA-discipline tension) — adopt only if +spike-gated. + +## Sources + +| # | Source | Type | Quality | Used in | +|---|--------|------|---------|---------| +| 1 | `.venv/.../agent_framework_openai/__init__.py:33-43` (`OpenAIChatClient`, `OpenAIChatCompletionClient`) | codebase | high | Dim 1 | +| 2 | `.venv/.../agent_framework_openai/_chat_completion_client.py:705,724-726,757-776` (UsageDetails mapping, None-safe) | codebase | high | Dim 2 | +| 3 | `.venv/.../agent_framework/_types.py:402-417,547` (`UsageDetails`, `add_usage_details`, `response.usage_details`) | codebase | high | Dim 2 | +| 4 | `.venv/.../agent_framework_openai/_chat_client.py:376,382` (`base_url`, `api_key`) | codebase | high | Dim 1 | +| 5 | `agent_framework/ollama/__init__.pyi` + `import agent_framework_ollama` fails (not installed) | codebase | high | Dim 1, 6 | +| 6 | (base_url table, ChatCompletion client) | official | high | Dim 1, 2 | +| 7 | (native client, tool caveat, qwen3:4b) | official | high | Dim 1,3 | +| 8 | | official | high | Dim 2 | +| 9 | (1.0.0b260521 beta, --pre) | official | high | Dim 1, 6 | +| 10 | (deps: ollama <0.5.4) | official | high | Dim 1, 6 | +| 11 | (usage; streaming needs include_usage) | official | high | Dim 2 | +| 12 | (prompt_eval_count/eval_count) | official | high | Dim 2 | +| 13 | (1.8.0 OTel usage fix; 1.9.0 ollama tools fix) | official | high | Dim 2 | +| 14 | (double system prompt, Ollama+middleware) | community | medium | Dim 3 | +| 15 | (streaming+tools fixed 2025-05-28) | official | high | Dim 3 | +| 16 | (streaming tool-call regression 0.12.3) | community | medium | Dim 3 | +| 17 | (tool-call F1 by model) | community | high | Dim 3 | +| 18 | (Qwen3 best tool-calling) | community | medium | Dim 3 | +| 19 | (Intel i9 4-6 tok/s, no MLX) | community | medium | Dim 4 | +| 20 | (seed+temp0 not deterministic) | community | medium | Dim 5 | +| 21 | (CVE-2024-37032) | community | high | Dim 6 | +| 22 | (CVE-2025-63389 no-auth by design) | official | high | Dim 6 | +| 23 | (CVE-2026-7482 Bleeding Llama, < 0.17.1) | community | high | Dim 6 | +| 24 | (bind 127.0.0.1) | community | high | Dim 6 | +| 25 | (no native cost metric) | official | high | Best Practice | +| 26 | (LM Studio local persistence, no telemetry default) | official | medium | Dim 6 |