docs(fase2): 3 /trekresearch briefs — installed-source-verified [skip-docs]
Topic 1 (HITL): native HITL is GA (ctx.request_info/@response_handler/
run(responses=); GroupChatBuilder.with_request_info → AgentApprovalExecutor),
but durable checkpoint-resume is fragile (open #5818/#6127/#6372 into 1.9.0)
→ capture verdict out-of-band in VerdictStore, defer checkpointing off MVP path.
Topic 2 (MCP citation): REVERSES brief lean — official server-filesystem cannot
cite (raw text + bare paths) → build thin custom local-folder MCP server
returning {file,locator,snippet,score} over a framework-agnostic in-process
retriever (D7 seam). Corrected docs error: ContextProvider(source_id) +
before_run/after_run + extend_instructions(source_id,...) DO exist in 1.9.0.
Topic 3 (local chat client): use OpenAIChatCompletionClient(base_url) NON-STREAMING
(not OpenAIChatClient/Responses) — installed, 0 new deps, UsageDetails None-safe
and populated non-streaming. Native OllamaChatClient is --pre fallback (spike-gated).
validator-as-retry mitigates weak small-model tool-calling; Intel-CPU = plumbing only.
All grounded in installed 1.9.0 source (source wins over Learn docs). Gemini
bridge unavailable (MCP SDK predates Google May-2026 API change).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
This commit is contained in:
parent
78e03c3254
commit
c72730941e
3 changed files with 755 additions and 0 deletions
|
|
@ -0,0 +1,273 @@
|
|||
---
|
||||
type: trekresearch-brief
|
||||
created: 2026-06-24
|
||||
question: "Does Microsoft Agent Framework 1.9.0 (core + orchestrations 1.0.0) provide a native human-in-the-loop primitive to pause a workflow for external input and resume it, and how does it interact with checkpointing and session state?"
|
||||
confidence: 0.85
|
||||
dimensions: 6
|
||||
mcp_servers_used: [microsoft-learn, tavily]
|
||||
local_agents_used: [installed-source-introspection (orchestrator, main context)]
|
||||
external_agents_used: [docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (unavailable)]
|
||||
topic: 1
|
||||
brief: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md
|
||||
---
|
||||
|
||||
# Native human-in-the-loop in MAF 1.9.0 workflows
|
||||
|
||||
> Generated by trekresearch (Voyage 5.6.0) on 2026-06-24. Topic 1 of 3 for the
|
||||
> Fase 2 MVP-vertical-slice brief. Scope: external swarm + installed-source
|
||||
> introspection (API truth from the installed 1.9.0 package wins over Learn docs,
|
||||
> per CLAUDE.md). Gemini second opinion was unavailable (MCP SDK broke on
|
||||
> Google's May-2026 Interactions API change) — triangulation rests on docs +
|
||||
> community + contrarian + installed source.
|
||||
|
||||
## Research Question
|
||||
|
||||
Does `agent-framework` 1.9.0 (`agent-framework-core` + `agent-framework-orchestrations` 1.0.0)
|
||||
provide a native human-in-the-loop (HITL) mechanism to pause a workflow for
|
||||
external/human input and resume it, and how does it interact with checkpointing
|
||||
and session/conversation state? Decision it feeds: the Fase 2 "expert verdict
|
||||
captured via HITL" success criterion — design the two-layer HITL + verdict-capture
|
||||
+ feed-forward seam, and the per-project workflow graph.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Yes — native HITL exists and is GA in 1.9.0**, in three distinct surfaces:
|
||||
(1) a workflow-level request/response primitive (`ctx.request_info()` + `@response_handler`,
|
||||
resume via `run(responses=...)`), (2) durable checkpoint-and-resume that persists
|
||||
pending requests, and (3) an orchestration-level human-approval gate that — verified
|
||||
in installed source — **works on our Group Chat maker-checker default** via
|
||||
`GroupChatBuilder.with_request_info(agents=[...])`. **Confidence on the mechanism is
|
||||
HIGH (installed-source-verified); confidence on durable checkpoint-resume RELIABILITY
|
||||
is LOW/contradictory** — official docs promise lossless durable resume, but the bug
|
||||
tracker shows the *pause* half is solid while the *persist-and-resume* half has
|
||||
multiple open/silent-failure bugs surviving into the 1.9.0 line (#5818, #6127, #6372,
|
||||
#5621) plus a pickle type-allowlist that rejects custom Pydantic types on restore (#5810).
|
||||
**Key caveat:** our learned-verdict path is an *indefinite, out-of-band, never-resume-the-
|
||||
original-run* decision whose durable artifact is the hand-rolled VerdictStore — so the
|
||||
recommendation is to use the GA *in-run* approval gate for synchronous Layer-1 review,
|
||||
keep the durable learned verdict OUT-OF-BAND in the VerdictStore, and **defer
|
||||
checkpointing off the MVP critical path**.
|
||||
|
||||
## Dimensions
|
||||
|
||||
### 1. Native HITL primitive — Confidence: high
|
||||
|
||||
**Installed-source findings (ground truth, 1.9.0):**
|
||||
- `WorkflowContext.request_info(request_data: object, response_type: type, *, request_id: str | None = None)` — any executor calls this to suspend the workflow and request external input (`agent_framework/_workflows/_workflow_context.py:393`).
|
||||
- Every `Executor` inherits `RequestInfoMixin` (`_workflows/_executor.py:30`); a method decorated `@response_handler` receives the typed response on resume; handlers are matched by the request/response **type annotations** (`_workflows/_request_info_mixin.py`).
|
||||
- `RequestInfoExecutor` is **NOT present** in installed 1.9.0 (only `RequestInfoMixin` + `response_handler` + `ctx.request_info`). `send_responses`/`send_responses_streaming` are **NOT public** on `Workflow` — internal `_send_responses_internal` only.
|
||||
|
||||
**External findings:**
|
||||
- Docs + AutoGen→MAF migration guide confirm "workflows can pause execution and wait for external input before continuing" — a capability AutoGen's `Team` lacked. <https://learn.microsoft.com/agent-framework/migration-guide/from-autogen/>
|
||||
- The versioned upgrade guides explain why the installed surface looks as it does: `RequestInfoExecutor` removed (python-1.0.0b251104) in favour of `ctx.request_info()` + `@response_handler`; `send_responses*` removed (python-1.0.0b260210, PR #3720) in favour of `run(responses=...)`. Both predate 1.9.0 (2026-06-18). <https://learn.microsoft.com/agent-framework/support/upgrade/requests-and-responses-upgrade-guide-python>, <https://learn.microsoft.com/agent-framework/support/upgrade/python-2026-significant-changes>
|
||||
|
||||
**Contradictions:**
|
||||
- The auto-generated **API-reference** page (`agent-framework-python-latest`) still lists `send_responses*` and prose-mentions `RequestInfoExecutor`. This **conflicts** with both the dated changes-guide and the installed source. Resolution: **installed source wins** (CLAUDE.md invariant) — those names are doc-lag / the C# (`RequestPort`) idiom. Do not write them into Python 1.9.0 code.
|
||||
|
||||
### 2. Pause/resume mechanics — Confidence: high
|
||||
|
||||
**Installed-source findings:**
|
||||
- `Workflow.run(message=None, *, responses=None, checkpoint_id=None, checkpoint_storage=None)` — three intents, exactly one of `message` / `responses` / `checkpoint_id` per call (`_workflows/_workflow.py:675+`). `responses` is `Mapping[str, Any]` keyed by `request_id`; mutually exclusive with `message`; **can be combined with `checkpoint_id`** ("restore then send responses in a single call" per the docstring).
|
||||
- Run-state enum (`_workflows/_events.py:58-65`): `IDLE`, `IDLE_WITH_PENDING_REQUESTS` (paused awaiting input — non-terminal), `IN_PROGRESS_PENDING_REQUESTS`. `result.get_final_state()` returns it.
|
||||
- `RunnerContext.send_request_info_response(request_id, response)` validates the response **type** against the original request and raises `ValueError` on unknown `request_id` or type mismatch (`_workflows/_runner_context.py:457-471`).
|
||||
- Events collapsed to a generic `WorkflowEvent[DataT]` carrying `.request_id` / `.data` / `.type` (no `RequestInfoEvent` subclass).
|
||||
|
||||
**External findings:**
|
||||
- Canonical loop (official sample `guessing_game_with_human_input.py`): `run(..., stream=True)` → collect `request_info` events → `run(stream=True, responses=pending)` → repeat until no pending requests. **In-process resume needs NO checkpoint** — "state is preserved across multiple calls to run." <https://learn.microsoft.com/agent-framework/workflows/human-in-the-loop>
|
||||
- Request IDs are caller-supplied or auto-UUID; responses strongly typed via `response_type`.
|
||||
|
||||
### 3. Checkpointing interaction — Confidence: high (mechanism) / contradictory (reliability)
|
||||
|
||||
**Installed-source + docs findings (mechanism — high):**
|
||||
- `WorkflowCheckpoint.pending_request_info_events: dict[str, ...]` — pending HITL requests ARE serialized into the checkpoint (`_workflows/_checkpoint.py:81`). On restore the runner rehydrates them (`_runner_context.py:424-425`) and **re-emits** them as `request_info` events; you then answer with a *separate* `run(responses=...)` call (you cannot inject responses during the restore call itself).
|
||||
- Storage backends (all implement the `CheckpointStorage` protocol, swap without code change): `InMemoryCheckpointStorage` (ephemeral), `FileCheckpointStorage` (local disk, explicit `storage_path`, **pickle** + restricted unpickler), `CosmosCheckpointStorage` (Azure, preview — egress). Checkpoints fire at superstep boundaries, so a HITL pause lands on one cleanly. <https://learn.microsoft.com/agent-framework/workflows/checkpoints>
|
||||
|
||||
**Contradictions (reliability — LOW):**
|
||||
- **Docs** present durable "checkpoint → exit process → restore → respond" as a supported happy path.
|
||||
- **Community / bug tracker** shows persist-and-resume is the fragile half:
|
||||
- #5818 (OPEN, Magentic + `AgentSession` resume): same `request_id` sent back, workflow does not resume. <https://github.com/microsoft/agent-framework/issues/5818>
|
||||
- #6127 (unanswered, Sequential): resume re-prompts the same approval **and** re-executes the function; duplicate `CallId` in streaming chunks. <https://github.com/microsoft/agent-framework/discussions/6127>
|
||||
- #6372 (fixed PR #6491, but affected 1.1.0–**1.9.0**): fan-in barrier silently loses buffered messages across checkpoint/resume — **no error raised**. <https://github.com/microsoft/agent-framework/issues/6372>
|
||||
- #5621 (OPEN, Handoff restore): `Expected exactly one update for key 'SharedState'`, unmatched ToolApproval. <https://github.com/microsoft/agent-framework/issues/5621>
|
||||
- #5810 (fixed PR #6049): restore **type-allowlist** blocks non-safe types (`Checkpoint deserialization blocked for type … MessageRole`) — custom/Pydantic payloads must be registered via `allowed_checkpoint_types`. <https://github.com/microsoft/agent-framework/issues/5810>
|
||||
- #3255 (fixed PR #3689): sub-workflow restore re-sent already-answered requests → `Response provided for unknown request ID`. <https://github.com/microsoft/agent-framework/issues/3255>
|
||||
- **Version-skew (documented):** PR #3744 — "**Existing checkpoints cannot be resumed between versions.**" A MAF bump invalidates stored checkpoints.
|
||||
|
||||
### 4. Session / conversation-state interaction — Confidence: high
|
||||
|
||||
**External findings:**
|
||||
- The built-in `AgentExecutor` (wraps an agent inside a workflow) serializes on checkpoint: internal message cache, **full conversation history**, agent session state, and pending requests/responses — and restores them. So a HITL-paused agent-bearing workflow does NOT lose chat history. <https://learn.microsoft.com/agent-framework/workflows/advanced/agent-executor>
|
||||
- **Material gap for the Azure/Foundry profile:** "Checkpointing with agents that use **server-side sessions** (e.g. `FoundryAgent`) has limitations. Server-side session state is **not captured in checkpoints**." A durable HITL pause with a Foundry-hosted agent will not have its conversation reliably restored from the checkpoint alone. (Same doc.)
|
||||
- For pure custom-executor workflows (no `AgentExecutor`): only shared `state`, in-transit messages, and pending requests are captured; executor-local fields persist only if you override `on_checkpoint_save()` / `on_checkpoint_restore()`.
|
||||
- Pre-1.9.0 rename relevant to code around a pause: `SharedState`→`State`, `ctx.shared_state`→`ctx.state`, state getters/setters now **synchronous** (PR #3667).
|
||||
|
||||
**Cross-link:** this is the Fase 1 B7 bleed vector — cross-run conversation state lives in `AgentSession.state` + `InMemoryHistoryProvider`. The HITL/checkpoint state is workflow-level and distinct, but `fresh_workflow()` isolation still governs whether a restored conversation contaminates the next project run. (See `docs/research/2026-06-24-maf-capability-map.md`, Fase 1 B7.)
|
||||
|
||||
### 5. Group-chat / orchestration-level approval gate — Confidence: high
|
||||
|
||||
**Installed-source findings (decisive for our debate default):**
|
||||
- `GroupChatBuilder.with_request_info(*, agents: Sequence[str | SupportsAgentRun] | None = None)` **EXISTS** (`agent_framework_orchestrations/_group_chat.py:882`). It pauses after the named agent(s) respond and emits a `request_info` event (`type='request_info'`) "that allows the caller to review the conversation and optionally [approve/edit] … the standard response_handler/request_info pattern." Same method exists on `SequentialBuilder` (`_sequential.py:154`).
|
||||
- Participants matching the filter are wrapped as `AgentApprovalExecutor(WorkflowExecutor)` (`_orchestration_request_info.py:168`), constructed `allow_direct_output=True` so the user-approved final response surfaces as workflow output.
|
||||
- The human reply object is `AgentRequestInfoResponse` (public export) with `.approve()` (accept as-is), `.from_strings([text])`, `.from_messages([...])` (`_orchestration_request_info.py:44-85`). Supplied via `run(responses={request_id: AgentRequestInfoResponse...})`.
|
||||
- `resolve_request_info_filter(agents)` selects which agents pause for approval — e.g. pause only before the **checker** in maker-checker.
|
||||
- A separate, lighter gate also exists: tool-approval via `@tool(approval_mode="always_require")` → `function_approval_request` content → approve/deny (in-run, no graph).
|
||||
|
||||
**External findings:**
|
||||
- Docs/maintainer confirm there is **no single universal "approve/edit/reject" object**; the documented routes are (a) `with_request_info` + `AgentRequestInfoResponse`, (b) tool-approval, (c) Magentic plan-review (`enable_plan_review`) — Magentic is experimental and OFF our path. For arbitrary maker-checker over agent output, the intended route is `with_request_info` (now installed-source-confirmed for GroupChat) or a custom executor calling `ctx.request_info()`. <https://github.com/microsoft/agent-framework/discussions/1287>
|
||||
- **Caveat:** community shows the orchestration approval *resume* path is where bugs cluster (#5818, #6127, #6006) — the gate fires reliably; persisting/resuming the approval across serialization is fragile.
|
||||
|
||||
### 6. Design implication for our two-layer HITL + verdict capture — Confidence: high (recommendation)
|
||||
|
||||
This is decision-relevant — see **Recommendation** below. Short form: the brief's
|
||||
fallback assumption ("if MAF lacks native HITL, capture out-of-band") is **partly
|
||||
inverted**: MAF *has* native HITL, and the right split is to use the GA *in-run*
|
||||
pieces for synchronous review but keep the *durable learned verdict* out-of-band.
|
||||
|
||||
## External Knowledge
|
||||
|
||||
### Best Practice
|
||||
- Canonical 1.9.0 HITL = `ctx.request_info()` + `@response_handler` + `run(responses=)`; detect pause via `IDLE_WITH_PENDING_REQUESTS` / `get_request_info_events()`. Official samples: `guessing_game_with_human_input.py`, `sequential_request_info.py`, `checkpoint/checkpoint_with_human_in_the_loop.py`, `magentic_human_plan_review.py` (all under `microsoft/agent-framework` `python/samples/03-workflows/`).
|
||||
|
||||
### Security (relevant to D3 no-silent-egress + local-only)
|
||||
- `FileCheckpointStorage` writes **unencrypted pickle** blobs containing conversation history + pending HITL payloads + shared state. Encryption-at-rest / permissions / ACLs are **the developer's responsibility** (no built-in encryption). Lock the `storage_path` down (dedicated dir, `0700`/`0600`, encrypted volume; macOS FileVault helps).
|
||||
- Restore is hardened: restricted unpickler **ON by default** since 1.0.1 (we're on 1.9.0); non-safe types throw `WorkflowCheckpointException` unless registered in `allowed_checkpoint_types`. Treat it as a safety net, not the control — "never load checkpoints from untrusted sources."
|
||||
- **No CVEs** against `agent-framework*` (GitHub Security Advisories empty; OSV empty). Two transitive Starlette advisories on deps.dev — confirm via `uv run pip-audit`. Semantic Kernel CVEs (CVE-2026-26030/-25592) are a *different package* and not in the checkpoint path; only relevant if `semantic-kernel` is pulled in (it is not, per our pinned tree).
|
||||
- **Telemetry OFF by default** — no exporter ships; `ENABLE_INSTRUMENTATION`/`ENABLE_SENSITIVE_DATA`/`ENABLE_CONSOLE_EXPORTERS` all default false. No-silent-egress holds out of the box if we (a) don't set those, (b) set no `OTEL_EXPORTER_OTLP_*`, (c) use File/InMemory (never Cosmos). One subtlety: MAF auto-propagates OTel trace context into MCP `tools/call` `_meta` when a span is active — inert with instrumentation off; keep MCP servers local (stdio) regardless.
|
||||
|
||||
### Known Issues
|
||||
- See Dimension 3 contradictions. Plus: `with_request_info()` naming is opaque and there is no event-type filter for which events trigger a HITL pause (#3534, open). Local-model + HITL is **unverified by anyone** — zero community signal; several worst resume bugs are Azure-server-side-persistence-specific and may simply not apply locally, but then we own conversation-history persistence ourselves. Spike locally; do not rely on precedent.
|
||||
|
||||
## Gemini Second Opinion
|
||||
|
||||
Unavailable. The `gemini-mcp` server's client SDK predates Google's May-2026
|
||||
Interactions API breaking change and returned `400 BadRequestError` before any
|
||||
research ran. No independent Gemini triangulation was obtained for this topic;
|
||||
treat the second opinion as absent (not negative). To restore: upgrade the
|
||||
gemini-mcp server's client SDK to ≥ 2.0.0.
|
||||
|
||||
## Synthesis
|
||||
|
||||
The triangulation surfaces an insight no single source states: **MAF 1.9.0 has
|
||||
three different HITL surfaces, and the strongest one for *our* need is the
|
||||
lightest one — while the heaviest one (durable checkpoint-resume) is both the
|
||||
shakiest in practice and a poor fit for the problem.**
|
||||
|
||||
1. **In-run synchronous review** (`GroupChatBuilder.with_request_info` →
|
||||
`AgentApprovalExecutor` → `run(responses={id: AgentRequestInfoResponse...})`)
|
||||
is GA, installed-source-confirmed for our Group Chat maker-checker default, and
|
||||
needs **no checkpointing** (state persists across `run()` calls in-process).
|
||||
This is the solid, happy-path piece.
|
||||
|
||||
2. **Durable cross-process pause** (checkpoint + restore + responses) is where the
|
||||
docs promise and the bug tracker diverge hardest: open/silent-failure resume
|
||||
bugs into the 1.9.0 line, a pickle type-allowlist that fights our Pydantic IR,
|
||||
and "checkpoints cannot be resumed between versions." The contrarian pass is
|
||||
right that coupling our **highest-value data path** (the verdict the system
|
||||
*learns* from) to MAF's **most-churned, least-durable, Azure-favoring** surface
|
||||
is a self-inflicted risk — and a lock-in against the D7 Claude-SDK sibling,
|
||||
which has no executor/checkpoint model and can only share a *framework-agnostic*
|
||||
verdict seam.
|
||||
|
||||
3. The actual shape of "fagekspert enters a verdict the next run learns from" is
|
||||
an **indefinite, out-of-band, never-resume-the-original-run** decision. Its
|
||||
durable artifact is the hand-rolled **VerdictStore**, not an in-flight workflow
|
||||
checkpoint. A workflow checkpoint is engineered for "pause seconds-to-minutes,
|
||||
resume the same process" — the wrong tool for "pause indefinitely, decide
|
||||
elsewhere."
|
||||
|
||||
So the brief's binary ("native HITL → use it; else out-of-band") resolves to a
|
||||
**split**: adopt the GA in-run approval gate for the *synchronous Layer-1 review*
|
||||
where it fits; keep the *durable, learning-loop verdict* out-of-band in the
|
||||
VerdictStore; and **defer checkpointing off the MVP critical path** (matches the
|
||||
brief's `[OPEN]` default — and Topic 1 confirms the default rather than overturning
|
||||
it, because durable checkpoint-resume is the fragile part).
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **Layer-1 review: native gate vs. simplest possible?** `with_request_info` is
|
||||
GA and fits, but a custom executor calling `ctx.request_info()` gives full
|
||||
control over the request payload (the `ValidatedProposal` + provenance). Decide
|
||||
in /trekplan: native `with_request_info(agents=[checker])` vs custom request-info
|
||||
executor. Either way: in-process, no checkpoint.
|
||||
- **Is Layer-1 even in the MVP, or is it async-only?** The brief marks two-layer
|
||||
HITL semantics (sync review vs async+notification stub) as `[OPEN]`. If Fase 2
|
||||
ships only the async/out-of-band verdict + notification *stub* (B11), the native
|
||||
in-run gate may be deferred too — fewer moving parts. Resolve in /trekplan.
|
||||
- **Local-profile HITL behaviour** — unverified by anyone. Needs a self-spike:
|
||||
run the approval gate / `request_info` loop against the local OpenAI-compatible
|
||||
endpoint (Topic 3) and confirm it fires and resumes. Tie to Topic 3's outcome.
|
||||
- **If durable pause is ever needed:** the acceptance gate must be a
|
||||
resume-integrity test (pause → checkpoint → restore → assert pending requests
|
||||
match + conversation intact), given #6372/#5621-class silent failures. Out of
|
||||
MVP scope but record the condition.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**For the Fase 2 MVP, capture the expert verdict OUT-OF-BAND and keep the durable
|
||||
artifact in the hand-rolled VerdictStore; do NOT couple the learned-verdict path
|
||||
to native checkpoint-resume.** Concretely:
|
||||
|
||||
1. **Layer 2 (durable, learning loop) = out-of-band VerdictStore.** The emitted
|
||||
`ValidatedProposal` + provenance is presented to the fagekspert out-of-band; the
|
||||
verdict is written to the VerdictStore; the next run retrieves it via the ExpeL
|
||||
`ContextProvider` seam (`extend_instructions(source_id, instructions)`). This is
|
||||
framework-agnostic, D7-portable, and the success-criterion ("second run
|
||||
retrieves the prior verdict") is satisfied without any MAF checkpoint.
|
||||
2. **Layer 1 (optional in-run synchronous review) = GA native gate IF included.**
|
||||
If Fase 2 ships a synchronous review, use `GroupChatBuilder.with_request_info(
|
||||
agents=[checker])` (or a custom `ctx.request_info()` executor for a richer typed
|
||||
payload) — in-process, resume via `run(responses=...)`, **no checkpointing**.
|
||||
Register any custom Pydantic response type's expectations now so it survives if
|
||||
checkpointing is ever added.
|
||||
3. **Defer checkpointing off the MVP critical path** (brief `[OPEN]` default
|
||||
upheld). `InMemoryCheckpointStorage` is fine for tests; do not put
|
||||
`FileCheckpointStorage` durable HITL on the critical path. If added later:
|
||||
File/InMemory only (never Cosmos — egress), locked-down `storage_path`,
|
||||
`allowed_checkpoint_types` for our IR, a resume-integrity acceptance test, and
|
||||
awareness of cross-version checkpoint invalidation.
|
||||
4. **Provenance, not checkpoint, carries the audit trail** — the single emitted
|
||||
proposal's provenance stamp (citations + model/role + validator decision + token
|
||||
usage) is the durable record, consistent with the no-silent-egress + provenance
|
||||
NFRs.
|
||||
|
||||
Risks to carry into the plan: (a) MAF HITL/checkpoint API churn → any MAF-native
|
||||
HITL code carries upgrade cost; pin the surface and watch the changes guide.
|
||||
(b) Local-profile HITL is unverified → spike with Topic 3's local client.
|
||||
(c) If a synchronous in-run gate is used, the orchestration *resume* path is where
|
||||
community bugs cluster — keep it in-process (no serialization boundary) to dodge
|
||||
that whole class.
|
||||
|
||||
## Sources
|
||||
|
||||
| # | Source | Type | Quality | Used in |
|
||||
|---|--------|------|---------|---------|
|
||||
| 1 | `.venv/.../agent_framework/_workflows/_workflow_context.py:393` (`request_info`) | codebase | high | Dim 1, 2 |
|
||||
| 2 | `.venv/.../agent_framework/_workflows/_request_info_mixin.py` (`response_handler`, `RequestInfoMixin`) | codebase | high | Dim 1 |
|
||||
| 3 | `.venv/.../agent_framework/_workflows/_workflow.py:675+` (`run(responses=, checkpoint_id=)`) | codebase | high | Dim 2, 3 |
|
||||
| 4 | `.venv/.../agent_framework/_workflows/_events.py:58-65` (`WorkflowRunState`) | codebase | high | Dim 2 |
|
||||
| 5 | `.venv/.../agent_framework/_workflows/_runner_context.py:424,457-471` (rehydrate + `send_request_info_response`) | codebase | high | Dim 2, 3 |
|
||||
| 6 | `.venv/.../agent_framework/_workflows/_checkpoint.py:81` (`pending_request_info_events`) | codebase | high | Dim 3 |
|
||||
| 7 | `.venv/.../agent_framework_orchestrations/_group_chat.py:882` (`GroupChatBuilder.with_request_info`) | codebase | high | Dim 5 |
|
||||
| 8 | `.venv/.../agent_framework_orchestrations/_orchestration_request_info.py:44-85,168` (`AgentRequestInfoResponse`, `AgentApprovalExecutor`) | codebase | high | Dim 5 |
|
||||
| 9 | <https://learn.microsoft.com/agent-framework/workflows/human-in-the-loop> | official | high | Dim 1,2,3,5 |
|
||||
| 10 | <https://learn.microsoft.com/agent-framework/workflows/checkpoints> | official | high | Dim 3, Security |
|
||||
| 11 | <https://learn.microsoft.com/agent-framework/workflows/advanced/agent-executor> | official | high | Dim 4 |
|
||||
| 12 | <https://learn.microsoft.com/agent-framework/support/upgrade/python-2026-significant-changes> | official | high | Dim 1,2,3 |
|
||||
| 13 | <https://learn.microsoft.com/agent-framework/support/upgrade/requests-and-responses-upgrade-guide-python> | official | high | Dim 1 |
|
||||
| 14 | <https://learn.microsoft.com/agent-framework/migration-guide/from-autogen/> | official | high | Dim 1 |
|
||||
| 15 | <https://github.com/microsoft/agent-framework/issues/5818> (Magentic AgentSession resume, OPEN) | community | high | Dim 3,5 |
|
||||
| 16 | <https://github.com/microsoft/agent-framework/discussions/6127> (Sequential re-prompt+re-exec) | community | medium | Dim 3,5 |
|
||||
| 17 | <https://github.com/microsoft/agent-framework/issues/6372> (fan-in barrier silent loss, 1.1.0–1.9.0) | community | high | Dim 3 |
|
||||
| 18 | <https://github.com/microsoft/agent-framework/issues/5621> (Handoff restore fails, OPEN) | community | high | Dim 3 |
|
||||
| 19 | <https://github.com/microsoft/agent-framework/issues/5810> (checkpoint type-allowlist) | community | high | Dim 3, Security |
|
||||
| 20 | <https://github.com/microsoft/agent-framework/issues/3255> (sub-workflow dup request, fixed) | community | high | Dim 3 |
|
||||
| 21 | <https://github.com/microsoft/agent-framework/issues/3534> (with_request_info naming/filter, OPEN) | community | medium | Known Issues |
|
||||
| 22 | <https://github.com/microsoft/agent-framework/discussions/1287> (no universal approval object; custom executor) | community | high | Dim 5 |
|
||||
| 23 | <https://www.diagrid.io/blog/still-not-durable-how-microsoft-agent-framework-and-strands-agents-repeat-the-same-mistake> | community | medium | Synthesis |
|
||||
| 24 | <https://github.com/microsoft/agent-framework/discussions/2305> (checkpoint limitations) | community | medium | Synthesis |
|
||||
| 25 | <https://github.com/microsoft/agent-framework/issues/4078> (stability/production timeline, unanswered) | community | medium | Synthesis |
|
||||
| 26 | <https://github.com/microsoft/agent-framework/security/advisories> (no advisories) | official | high | Security |
|
||||
| 27 | <https://osv.dev/list?ecosystem=PyPI&q=agent-framework> (no results) | official | high | Security |
|
||||
| 28 | <https://learn.microsoft.com/agent-framework/agents/observability> (telemetry off by default) | official | high | Security |
|
||||
| 29 | <https://learn.microsoft.com/agent-framework/agents/providers/ollama> (local provider exists) | official | medium | Known Issues |
|
||||
| 30 | <https://pypi.org/project/agent-framework-core/> (1.9.0 = 2026-06-18) | official | high | Exec summary |
|
||||
|
|
@ -0,0 +1,245 @@
|
|||
---
|
||||
type: trekresearch-brief
|
||||
created: 2026-06-24
|
||||
question: "What is the best way to expose a local document folder to a MAF 1.9.0 agent via MCPStdioTool such that retrieved content carries citation metadata (file + locator) — existing filesystem MCP server vs thin custom server, and the citation shape a context provider expects?"
|
||||
confidence: 0.82
|
||||
dimensions: 6
|
||||
mcp_servers_used: [microsoft-learn, tavily]
|
||||
local_agents_used: [installed-source-introspection (orchestrator, main context)]
|
||||
external_agents_used: [docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (unavailable)]
|
||||
topic: 2
|
||||
brief: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md
|
||||
---
|
||||
|
||||
# Local-folder data access via MCP with citation provenance (MAF 1.9.0)
|
||||
|
||||
> Generated by trekresearch (Voyage 5.6.0) on 2026-06-24. Topic 2 of 3 for the
|
||||
> Fase 2 MVP-vertical-slice brief. Scope: external swarm + installed-source
|
||||
> introspection (installed 1.9.0 API truth wins over Learn docs). Gemini
|
||||
> unavailable (MCP SDK broke on Google's May-2026 API change).
|
||||
|
||||
## Research Question
|
||||
|
||||
What is the best way to expose a LOCAL document folder to a MAF 1.9.0 agent via
|
||||
`MCPStdioTool` so retrieved content carries citation metadata (file + locator) —
|
||||
reuse an existing filesystem MCP server, or build a thin custom server — and what
|
||||
citation shape does MAF's context/annotation model expect? Decision it feeds: the
|
||||
data-access step, the citation-aware context provider, provenance-stamping, and the
|
||||
build-vs-reuse decision for the local-folder server.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**The brief's lean ("reuse the official `@modelcontextprotocol/server-filesystem`
|
||||
+ attach citations via `parse_tool_results`") does not survive contact with the
|
||||
requirement and should be dropped.** The official filesystem server returns raw
|
||||
text and bare path strings only — **no line/char offsets, no chunk IDs; its
|
||||
`search_files` is filename-glob, not content search** — so it physically cannot
|
||||
yield a `(file + locator + snippet)` citation; you would have to re-read, re-chunk
|
||||
and re-locate inside the callback (i.e. build the whole citation engine anyway, in
|
||||
the worst place, behind a Node subprocess). MAF also does **not** auto-create
|
||||
citation annotations from MCP results. **Recommendation (honouring the CLAUDE.md
|
||||
"data access via MCP, JSON-Schema-validated, fail-fast" convention): build a THIN
|
||||
CUSTOM local-folder MCP server that returns citation-ready structured chunks
|
||||
(`{file, locator, snippet, score}`), wrapping a framework-agnostic in-process
|
||||
retriever core that is the D7-portable seam.** Carry provenance as first-class
|
||||
Pydantic data on the emitted proposal, independent of MAF's annotation propagation
|
||||
(which has an open Python streaming-drop bug, #4316). Confidence 0.82: the
|
||||
capability facts are HIGH (installed-source + official-README verified); the
|
||||
exact MVP retrieval depth (keyword vs local embeddings) is a deferrable design
|
||||
choice.
|
||||
|
||||
## Dimensions
|
||||
|
||||
### 1. Can the official filesystem server carry citations? — Confidence: high (NO)
|
||||
|
||||
**External findings (docs + community + contrarian agree):**
|
||||
- `@modelcontextprotocol/server-filesystem` tools: `read_text_file` (whole file, or `head`/`tail` by line *count* only), `read_multiple_files`, `list_directory`, `directory_tree`, `get_file_info`, `search_files`, `write_file`, `edit_file`. <https://github.com/modelcontextprotocol/servers/blob/main/src/filesystem/README.md>
|
||||
- Read results = **raw UTF-8 text, no line numbers, no byte/char offsets, no chunk IDs.** `search_files` returns **matching file PATHS only** (filename glob — NOT content grep). `get_file_info` = size/timestamps, not a locator.
|
||||
- Net: the only locator it gives is the file path you already passed in. Enhanced community forks advertise `read_file_lines` / grep-with-line-numbers / chunking precisely because the official server lacks them.
|
||||
|
||||
**Conclusion:** the official server cannot produce `(file + locator + snippet)` citations. It is fine only for "let the agent open named files," not citation-backed retrieval.
|
||||
|
||||
### 2. Does MAF auto-create citations from MCP results? — Confidence: high (NO)
|
||||
|
||||
**Installed-source findings (1.9.0):**
|
||||
- Default parser `_parse_contents_from_mcp_tool_result` / `_parse_tool_result_from_mcp` maps MCP content → `Content.from_text/from_data/from_uri`; **attaches no annotations** (`agent_framework/_mcp.py:507-517`).
|
||||
- `structuredContent` IS now serialized to text Content (`_mcp.py:586-587`) — so the #3313 "structuredContent dropped → None" bug **is fixed in installed 1.9.0**. But it lands as JSON *text*, not as typed/annotated content — you still parse it yourself for provenance.
|
||||
- The seam to attach citations is `MCPTool.__init__(parse_tool_results: Callable[[CallToolResult], str | list[Content]])` — your callback fully replaces the default parse.
|
||||
|
||||
**External findings:**
|
||||
- Confirmed: docs/RAG paths use **prompt-level** "cite your sources" (a `string_mapper`), not structured annotations. The one citation-aware RAG provider, `TextSearchProvider`, is **.NET-only — no Python equivalent**. <https://learn.microsoft.com/agent-framework/agents/rag>
|
||||
- **Open bug #4316:** file-citation annotations are **silently dropped in the Python streaming path** (`annotations=None`); .NET fixed, Python lagging. → do NOT rely on MAF's `Annotation` propagation for load-bearing provenance. <https://github.com/microsoft/agent-framework/issues/4316>
|
||||
|
||||
### 3. Citation shape MAF expects — Confidence: high
|
||||
|
||||
**Installed-source findings (1.9.0, ground truth):**
|
||||
- `Annotation` is a **TypedDict** (`agent_framework/_types.py:374`): `type: Literal["citation"]`, `title`, `url`, `file_id`, `tool_name`, `snippet`, `annotated_regions: Sequence[TextSpanRegion]`, `additional_properties`, `raw_representation`.
|
||||
- `TextSpanRegion` TypedDict (`_types.py:366`): `type: Literal["text_span"]`, `start_index`, `end_index`.
|
||||
- The class form `CitationAnnotation` was replaced by this TypedDict in 1.0.0b260123 (PR #3252) — installed 1.9.0 = **TypedDict** form. Build citations as dicts: `{"type":"citation","file_id":...,"title":...,"snippet":...,"annotated_regions":[{"start_index":..,"end_index":..}]}`.
|
||||
- Carrier: `FunctionResultContent.annotations: list[... ] | None` (accepts dict/TypedDict).
|
||||
|
||||
**Use:** our provenance stamp's "≥1 citation" maps cleanly onto this `Annotation`. But because of #4316, treat `Annotation` as a *display/serialization* surface and keep the authoritative provenance as our own Pydantic object.
|
||||
|
||||
### 4. Context provider injection seam — Confidence: high (corrects an external error)
|
||||
|
||||
**Installed-source findings (1.9.0) — and a flagged docs-vs-source conflict:**
|
||||
- `class ContextProvider` (`agent_framework/_sessions.py:351`), `__init__(self, source_id: str)` — **source_id REQUIRED** (confirms Fase 1).
|
||||
- Hooks: `async def before_run(...)` (`:370`) and `async def after_run(...)` (`:391`) — add messages/instructions/tools in `before_run`, process/store in `after_run`.
|
||||
- `Context.extend_instructions(self, source_id: str, instructions: str | Sequence[str])` (`:253`) and `extend_tools(source_id, tools)` (`:266`) — **the two-arg seam the brief and Fase 1 specified, confirmed verbatim.**
|
||||
- `MemoryContextProvider` / `InMemoryHistoryProvider` exist but are conversation memory, **not** retrievers.
|
||||
|
||||
> **Premiss correction (important for /trekplan):** the docs-researcher reported, with high confidence from the Learn API-ref page, that Python uses `invoking`/`invoked` hooks and that `source_id` / `extend_instructions` are "C#-only, not in Python." **That is wrong for installed 1.9.0** — the installed source has `before_run`/`after_run`, `ContextProvider(source_id)`, and `extend_instructions(source_id, …)`. Per CLAUDE.md, installed source wins. Do NOT let the planning phase adopt the `invoking`/`invoked` / no-source_id surface.
|
||||
|
||||
**Design fit:** a custom `ExpeLContextProvider(ContextProvider)` (Fase 1 seam, promote to core) injects retrieved+cited content + prior verdicts in `before_run` via `extend_instructions(source_id, …)`, and the same provider family is where the ExpeL learning loop reads VerdictStore. No built-in local-folder retrieval provider exists in Python — retrieval is ours to build.
|
||||
|
||||
### 5. Build vs reuse (the decision) — Confidence: high (recommendation)
|
||||
|
||||
**Triangulated picture:**
|
||||
- **Reuse official server:** cannot cite (Dim 1); adds a Node runtime + `npx` cold-start + version-pin burden; npx PATH/nvm handshake failure is THE most-reported stdio blocker on macOS (B5); supply-chain surface (chalk/debug + Shai-Hulud hit MCP-SDK transitives). Security: usable only if pinned ≥ `2025.7.1` (EscapeRoute CVE-2025-53109/53110, fixed there; current `2026.1.14` clean), vendored (never `npx @latest`), allowlist scoped to the doc folder (never `$HOME`).
|
||||
- **Thin custom MCP server:** removes the npm supply-chain surface, emits provenance natively (`{file, line_range/char_span, snippet, score}` via `structuredContent`), honours the CLAUDE.md "data access via MCP, JSON-Schema-validated, fail-fast" convention, and is portable to the D7 Claude-SDK sibling's in-process-MCP idiom. Cost: **we own** path-canonicalisation + symlink-realpath validation (the exact bug class the official server had fixed twice — TDD against the EscapeRoute scenarios) and input validation (no shelling out; use native file APIs).
|
||||
- **In-process retriever, no MCP** (contrarian's first choice): simplest + lowest latency + best D7 fit, but conflicts with the project's "data access via MCP" convention. Keep as the fallback if that convention is relaxed.
|
||||
- Reference design: `shinpr/mcp-local-rag` (LanceDB file-based + local embeddings; returns `path + chunk index + title + score + chunk text` + `read_chunk_neighbors` context expansion) — the closest local-only, citation-grade pattern. <https://github.com/shinpr/mcp-local-rag>
|
||||
|
||||
### 6. Retrieval depth for the MVP — Confidence: medium
|
||||
|
||||
- The MVP uses a **tiny synthetic "anleggskostnad" domain** (Fase 0). A full vector stack is not required for D5 (90%) / D6 (cost). A simple in-process retriever (keyword/substring + chunk-at-ingest with exact locators) likely suffices and keeps deps minimal.
|
||||
- The Python "real RAG" path (SK `VectorStore.create_search_function().as_agent_framework_tool()`) **requires pulling in `semantic-kernel ≥ 1.38`** — a different package with its own RCE CVE (CVE-2026-26030, eval in in-memory vector-store filter, < 1.39.4). Adding it conflicts with the "GA-pakker pinnet eksplisitt, ikke metaen" dependency discipline and adds CVE surface. **Defer / avoid for MVP** unless semantic retrieval proves necessary; if adopted, pin ≥ 1.39.4.
|
||||
|
||||
## External Knowledge
|
||||
|
||||
### Best Practice
|
||||
- Canonical `MCPStdioTool` wiring (official docstring + Learn): `MCPStdioTool(name="filesystem", command="npx", args=["-y","@modelcontextprotocol/server-filesystem", <dir>])`, used as `async with`; requires `pip install mcp --pre`. <https://learn.microsoft.com/agent-framework/agents/tools/local-mcp-tools>
|
||||
- MAF can also expose an *agent* as an MCP server (`agent.as_mcp_server()`) — not relevant to local-folder retrieval. Generic server-building is the MCP Python SDK / FastMCP domain. <https://github.com/modelcontextprotocol/python-sdk>
|
||||
|
||||
### Security (D3 no-silent-egress + local-only)
|
||||
- Stdio MCP server makes **no network calls**; only inherent egress in the path is the `npx` fetch at launch (removed by vendoring/pinning). OTel `_meta` trace-context injection into `tools/call` is inert with instrumentation off and carries only random W3C IDs (no project data).
|
||||
- EscapeRoute CVEs (CVE-2025-53110 prefix-collision, CVE-2025-53109 symlink-escape) — both **fixed in 2025.7.1**; exploit trigger is prompt injection of the agent (relevant: our maker-checker + blocking validator are compensating controls). Never run the server with elevated privileges. <https://cymulate.com/blog/cve-2025-53109-53110-escaperoute-anthropic/>
|
||||
- npx supply chain: pin exact version + lockfile, verify tarball for `postinstall`, quarantine fresh versions, consider containerization. A thin custom dependency-light server removes this surface entirely.
|
||||
|
||||
### Known Issues
|
||||
- MAF MCP bugs — **verified against installed 1.9.0**: #3313 structuredContent parsing **FIXED** (`_mcp.py:586`); #2884 stdio session-invalidation handled (`is_connected` reset + `ClosedResourceError` catch, `_mcp.py:935/1355/1544`). Still: #2284 `_meta` dropped (don't rely on `_meta` for provenance — use `structuredContent`/`content`); #4316 Python streaming citation-annotation drop (open).
|
||||
- npx/nvm PATH handshake failure on macOS (servers#64) — use absolute `command` path + explicit `env` PATH/NODE_PATH, prefer pre-installed/`uvx`-pinned over cold `npx -y`. Filesystem server fails to start if any allowed dir is unavailable (servers#3232).
|
||||
- Don't split a citation across `content` vs `structuredContent` vs `_meta` — clients (MAF included) forward them inconsistently; put it in one reliably-forwarded channel.
|
||||
|
||||
## Gemini Second Opinion
|
||||
|
||||
Unavailable (same MCP-SDK/Interactions-API failure as Topic 1). No independent
|
||||
Gemini triangulation for this topic; treat as absent, not negative.
|
||||
|
||||
## Synthesis
|
||||
|
||||
Three sources converge on a conclusion the brief did not anticipate: **the
|
||||
citation requirement, not the data-access requirement, drives the design — and it
|
||||
rules out the off-the-shelf filesystem server.** Citations of the form
|
||||
`(file + locator + snippet)` must be *exact by construction*, which means
|
||||
chunk-and-locate **at ingest**, in code we own. Bolting them on afterward — over a
|
||||
third-party server's lossy whole-file text, through a `parse_tool_results` callback,
|
||||
relying on a MAF annotation path with an open streaming-drop bug — stacks three
|
||||
fragilities to recreate something we could have produced cleanly up front.
|
||||
|
||||
The right shape is a **two-layer seam**: (1) a framework-agnostic in-process
|
||||
retriever core — `retrieve(query) -> list[RetrievedChunk]`, where `RetrievedChunk`
|
||||
is a Pydantic type carrying `file`, `locator` (line range / char span), `snippet`,
|
||||
`score` — which is the **D7-portable contract** (MAF and Claude Agent SDK both
|
||||
satisfy it); and (2) a **thin custom MCP server** wrapping that core, returning the
|
||||
chunks as structured `structuredContent`, which honours the project's "data access
|
||||
via MCP, JSON-Schema-validated, fail-fast" convention and ports to the Claude SDK's
|
||||
in-process-MCP idiom. Provenance lives as first-class Pydantic data on the emitted
|
||||
proposal; the MAF `Annotation` TypedDict is only a display/serialisation view of it,
|
||||
so #4316 never sits on the load-bearing path. The official filesystem server is
|
||||
demoted to a possible *future* "browse named files" convenience, pinned and scoped.
|
||||
|
||||
This also threads the CLAUDE.md needle: the contrarian's "skip MCP, in-process
|
||||
only" is the simplest and the best D7 fit, but it breaks the MCP-data-access
|
||||
convention; the thin-custom-MCP-server-over-an-in-process-core keeps the convention
|
||||
*and* the D7 portability *and* citation fidelity — at the cost of owning sandbox
|
||||
correctness (testable, bounded).
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **MVP retrieval depth:** keyword/substring + ingest-time chunking (no extra deps,
|
||||
D5/D6-aligned) vs local embeddings (LanceDB/SK VectorStore, heavier + CVE surface).
|
||||
Default to keyword for the tiny synthetic domain; revisit only if recall is poor.
|
||||
Resolve in /trekplan.
|
||||
- **Locator format:** line range vs char span. It is the shared D7 contract — define
|
||||
and test it in the golden-suite up front. Default: char span (`start_index`/
|
||||
`end_index`) to map straight onto `TextSpanRegion`; carry line numbers in
|
||||
`additional_properties` for human display.
|
||||
- **Does the MVP even need MCP, or is the in-process retriever enough?** The CLAUDE.md
|
||||
convention says data access via MCP. The thin-custom-server honours it; confirm the
|
||||
operator wants the MCP boundary in Fase 2 vs deferring it (the in-process core is
|
||||
needed either way). Surface in /trekplan.
|
||||
- **Chunk-at-ingest vs at-query:** ingest-time chunking gives exact, cacheable
|
||||
locators; decide whether the MVP ingests once or reads live. Tie to determinism NFR.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Build a thin custom local-folder MCP server that returns citation-ready
|
||||
structured chunks, wrapping a framework-agnostic in-process retriever core. Do NOT
|
||||
reuse `@modelcontextprotocol/server-filesystem` for citation-backed retrieval.**
|
||||
|
||||
1. **Retriever core (D7 seam):** `retrieve(query, top_k) -> list[RetrievedChunk]`;
|
||||
`RetrievedChunk(file: str, locator: TextSpan, snippet: str, score: float)`.
|
||||
Chunk + assign locators at ingest so citations are exact by construction.
|
||||
Start with keyword/substring retrieval (no heavy deps; D5/D6). Pure Python,
|
||||
no MAF/MCP imports — portable to the Claude SDK sibling.
|
||||
2. **Thin custom MCP server (MAF-facing, honours the convention):** a minimal
|
||||
`mcp`/FastMCP stdio server exposing a `search(query)` tool that returns the
|
||||
chunks via `structuredContent` (JSON-Schema-validated, fail-fast on bad config).
|
||||
Path handling is security-critical: canonicalise + boundary-check (no
|
||||
`startsWith`), resolve symlink realpath and fail closed, native file APIs (no
|
||||
shelling out) — TDD against the EscapeRoute bypass patterns.
|
||||
3. **Wire into MAF:** `MCPStdioTool(name="docfolder", command=<abs python>, args=[...])`
|
||||
with a `parse_tool_results` callback that maps `structuredContent` chunks into
|
||||
`FunctionResultContent` + `Annotation(type="citation", file_id, snippet,
|
||||
annotated_regions=[TextSpanRegion])`. On Intel mac, pass absolute `command` +
|
||||
explicit `env` PATH; pre-install the server (no cold `npx`/uvx fetch on the path).
|
||||
4. **Provenance = first-class Pydantic data** on the emitted `ValidatedProposal`
|
||||
(≥1 citation: file + locator + snippet, plus model/role + validator decision +
|
||||
token usage). The MAF `Annotation` TypedDict is only its display view — route
|
||||
around #4316 by never depending on MAF annotation propagation for the assertion.
|
||||
5. **Citation-aware context injection:** a custom `ExpeLContextProvider(ContextProvider)`
|
||||
(`source_id` required) uses `before_run` + `extend_instructions(source_id, …)` to
|
||||
inject cited content (and prior verdicts) — the confirmed two-arg seam.
|
||||
6. **If the official filesystem server is ever used** (future "browse named files"):
|
||||
pin ≥ `2025.7.1` vendored (never `npx @latest`), allowlist the doc folder only
|
||||
(never `$HOME`), absolute node path + explicit env.
|
||||
|
||||
Fallback (only if the MCP-data-access convention is relaxed for the MVP): skip the
|
||||
MCP server and call the in-process retriever directly via a `@tool`/`ContextProvider`
|
||||
— simplest, lowest-latency, best D7 fit.
|
||||
|
||||
Risks to carry into the plan: (a) we own MCP-server sandbox correctness — treat
|
||||
path/symlink validation as security-critical, TDD it; (b) MAF Python citation
|
||||
annotation propagation is buggy in streaming (#4316) — provenance must be our own
|
||||
data; (c) adding `semantic-kernel` for vector RAG pulls a CVE-bearing package — defer.
|
||||
|
||||
## Sources
|
||||
|
||||
| # | Source | Type | Quality | Used in |
|
||||
|---|--------|------|---------|---------|
|
||||
| 1 | `.venv/.../agent_framework/_mcp.py:507-517,586-587,655` (default parser, structuredContent) | codebase | high | Dim 2, Known Issues |
|
||||
| 2 | `.venv/.../agent_framework/_mcp.py:911,935,1330,1355,1544` (session lifecycle, ClosedResourceError) | codebase | high | Known Issues |
|
||||
| 3 | `.venv/.../agent_framework/_types.py:366,374` (`TextSpanRegion`, `Annotation` TypedDict) | codebase | high | Dim 3 |
|
||||
| 4 | `.venv/.../agent_framework/_sessions.py:351,370,391,253,266` (`ContextProvider`, `before_run`/`after_run`, `extend_instructions`) | codebase | high | Dim 4 |
|
||||
| 5 | <https://github.com/modelcontextprotocol/servers/blob/main/src/filesystem/README.md> (tool surface, no locators) | official | high | Dim 1 |
|
||||
| 6 | <https://www.npmjs.com/package/@modelcontextprotocol/server-filesystem> | official | high | Dim 1,5 |
|
||||
| 7 | <https://learn.microsoft.com/python/api/agent-framework-core/agent_framework.mcpstdiotool?view=agent-framework-python-latest> | official | high | Dim 5, Best Practice |
|
||||
| 8 | <https://learn.microsoft.com/agent-framework/agents/tools/local-mcp-tools> | official | high | Best Practice |
|
||||
| 9 | <https://learn.microsoft.com/agent-framework/agents/rag> (TextSearchProvider .NET-only; SK VectorStore path) | official | high | Dim 2, 6 |
|
||||
| 10 | <https://learn.microsoft.com/agent-framework/support/upgrade/python-2026-significant-changes> (Annotation TypedDict PR #3252) | official | high | Dim 3 |
|
||||
| 11 | <https://github.com/microsoft/agent-framework/issues/4316> (Python streaming citation drop, OPEN) | community | high | Dim 2, Synthesis |
|
||||
| 12 | <https://github.com/microsoft/agent-framework/issues/3313> (structuredContent dropped, fixed) | community | high | Dim 2, Known Issues |
|
||||
| 13 | <https://github.com/microsoft/agent-framework/issues/2884> (stdio session not invalidated, fixed #3154) | community | high | Known Issues |
|
||||
| 14 | <https://github.com/microsoft/agent-framework/issues/2284> (`_meta` discarded) | community | medium | Known Issues |
|
||||
| 15 | <https://github.com/modelcontextprotocol/servers/issues/64> (npx/nvm PATH handshake) | community | high | Dim 5, Known Issues |
|
||||
| 16 | <https://github.com/modelcontextprotocol/servers/issues/3232> (startup fails on unavailable dir) | community | medium | Known Issues |
|
||||
| 17 | <https://cymulate.com/blog/cve-2025-53109-53110-escaperoute-anthropic/> (EscapeRoute CVEs) | community | high | Dim 5, Security |
|
||||
| 18 | <https://github.com/advisories/GHSA-q66q-fx2p-7w4m> (CVE-2025-53109) | official | high | Security |
|
||||
| 19 | <https://github.com/advisories/GHSA-hc55-p739-j48w> (CVE-2025-53110) | official | high | Security |
|
||||
| 20 | <https://security.snyk.io/package/npm/@modelcontextprotocol%2Fserver-filesystem> (no advisories > 2025.7.1) | community | high | Security |
|
||||
| 21 | <https://stacklok.com/blog/examining-the-impact-of-npm-supply-chain-attacks-on-mcp> (chalk/debug, Shai-Hulud) | community | high | Dim 5, Security |
|
||||
| 22 | <https://github.com/shinpr/mcp-local-rag> (local-only citation-grade RAG reference) | community | medium | Dim 5 |
|
||||
| 23 | <https://docs.claude.com/en/api/agent-sdk/custom-tools> (Claude SDK in-process MCP idiom) | official | high | Dim 5, Synthesis |
|
||||
| 24 | <https://learn.microsoft.com/azure/foundry/agents/how-to/tools/file-search> (hosted FileSearch cites service-side) | official | medium | Dim 2 |
|
||||
| 25 | <https://www.anthropic.com/engineering/code-execution-with-mcp> (MCP token bloat) | community | medium | Known Issues |
|
||||
|
|
@ -0,0 +1,237 @@
|
|||
---
|
||||
type: trekresearch-brief
|
||||
created: 2026-06-24
|
||||
question: "How does agent-framework-core 1.9.0 run agents against a local model on the free/local profile (OpenAI-compatible endpoint, Ollama, or other local chat client), and does that path populate UsageDetails token counts or return None?"
|
||||
confidence: 0.80
|
||||
dimensions: 6
|
||||
mcp_servers_used: [microsoft-learn, tavily]
|
||||
local_agents_used: [installed-source-introspection (orchestrator, main context)]
|
||||
external_agents_used: [docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (unavailable)]
|
||||
topic: 3
|
||||
brief: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md
|
||||
---
|
||||
|
||||
# Real local-profile chat client for agent-framework 1.9.0 + UsageDetails
|
||||
|
||||
> Generated by trekresearch (Voyage 5.6.0) on 2026-06-24. Topic 3 of 3 for the
|
||||
> Fase 2 MVP-vertical-slice brief. Scope: external swarm + installed-source
|
||||
> introspection (installed API truth wins). Gemini unavailable (MCP SDK broke on
|
||||
> Google's May-2026 API change).
|
||||
|
||||
## Research Question
|
||||
|
||||
How does `agent-framework-core` 1.9.0 (+ `agent-framework-openai`) run agents
|
||||
against a LOCAL model on the free/local profile, and does that path populate
|
||||
`UsageDetails` token counts or return `None`? Decision it feeds: the backend-profile
|
||||
abstraction (local path), the budget middleware's `None`-handling, the end-to-end
|
||||
run, and cost-discipline verification.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Two local paths exist, and the brief's framing needs two corrections.** Path A —
|
||||
`OpenAIChatCompletionClient(base_url=<local /v1>)` — is **installed (1.8.2), needs
|
||||
zero new dependencies, and populates `UsageDetails` None-safely from the local
|
||||
server's `usage` object**. Path B — the native `OllamaChatClient` — is a **`--pre`
|
||||
beta package (`agent-framework-ollama 1.0.0b260521`) that is NOT installed** and
|
||||
adds the `ollama` client dependency. **Correction 1 (client):** use
|
||||
`OpenAIChatCompletionClient` (Chat Completions API — the surface Ollama/LM Studio
|
||||
`/v1` exposes), NOT `OpenAIChatClient` (Responses API). **Correction 2 (streaming):**
|
||||
run the debate/validator path **NON-STREAMING** — that simultaneously dodges the
|
||||
well-documented `/v1` streaming+tool-calling drop bugs AND the streaming
|
||||
`usage=None` problem (`UsageDetails` is reliably populated only non-streaming).
|
||||
**Token accounting works** (confirmed in installed source), so the "real token
|
||||
accounting" success criterion is satisfiable on the local profile. The real limits
|
||||
are model capability and Intel-Mac CPU speed: small CPU-runnable models are weak at
|
||||
tool-calling, and an Intel Mac (no Metal/MLX, ~4–6 tok/s on 7B) makes a multi-round
|
||||
debate take minutes — so local is for plumbing/smoke verification, with
|
||||
representative-quality runs pushed to Foundry. Confidence 0.80.
|
||||
|
||||
## Dimensions
|
||||
|
||||
### 1. Local chat-client paths — Confidence: high
|
||||
|
||||
**Installed-source findings (ground truth):**
|
||||
- `agent-framework-openai` 1.8.2 exports **both** `OpenAIChatClient` (Responses API) and `OpenAIChatCompletionClient` (Chat Completions API) — both import OK in the venv. `RawOpenAIChatClient`/`RawOpenAIChatCompletionClient` are the lower-level variants.
|
||||
- Both accept `base_url` (→ env `OPENAI_BASE_URL`) and `api_key` (required even for local — pass a dummy like `"ollama"`/`"not-needed"`). `model_id` selects the model.
|
||||
- The native Ollama path: `agent_framework/ollama/__init__.pyi` re-exports `OllamaChatClient`, `OllamaChatOptions`, `OllamaEmbeddingClient`, … from `agent_framework_ollama` — but **`agent_framework_ollama` is NOT installed** (`import agent_framework_ollama` → ModuleNotFoundError; only the lazy re-export stub exists). Installing it (`uv add agent-framework-ollama --prerelease=allow`) pulls `ollama >=0.5.3,<0.5.4`.
|
||||
|
||||
**External findings:**
|
||||
- **Correct client for local `/v1`:** Ollama/LM Studio/vLLM expose a Chat Completions surface (`/v1/chat/completions`), so `OpenAIChatCompletionClient` is the technically correct client; `OpenAIChatClient` targets the Responses API which local servers don't expose. Two official MS docs disagree on which to use for Ollama — **flagged conflict; the Chat-Completions client is correct for `/v1`.** <https://learn.microsoft.com/agent-framework/integrations/openai-endpoints>
|
||||
- Official local-endpoint table: Ollama `http://localhost:11434/v1/`, LM Studio `http://localhost:1234/v1/`, vLLM `http://localhost:8000/v1/`. The `/v1/` suffix matters; the native `OllamaChatClient` uses `:11434` WITHOUT `/v1`.
|
||||
- Native `OllamaChatClient` is **preview/beta** (`1.0.0b260521`, "4-Beta", `--pre`), wraps `ollama.AsyncClient` (native `/api/chat`), `OTEL_PROVIDER_NAME='ollama'`. MS docs describe it as having "full support for function tools and streaming." <https://learn.microsoft.com/agent-framework/agents/providers/ollama>
|
||||
|
||||
### 2. UsageDetails population — Confidence: high
|
||||
|
||||
**Installed-source findings (the load-bearing answer):**
|
||||
- `OpenAIChatCompletionClient`: `_chat_completion_client.py:705` — `usage_details=self._parse_usage_from_openai(response.usage) if response.usage else None`. **None-safe.** `_parse_usage_from_openai` (`:757-776`) maps `usage.prompt_tokens`→`input_token_count`, `usage.completion_tokens`→`output_token_count`, `usage.total_tokens`→`total_token_count` (+ reasoning/cached/audio details).
|
||||
- `UsageDetails` is the TypedDict in `agent_framework/_types.py` (`input/output/total_token_count: int | None`); `add_usage_details(u1, u2)` helper exists (`_types.py:417`) — use it to accumulate across calls in the shared budget meter. Lands on `response.usage_details` (`_types.py:547`); read `response.usage_details["total_token_count"]` (None-safe).
|
||||
- Streaming: `_chat_completion_client.py:724-726` — `if chunk.usage: Content.from_usage(...)` — usage only arrives if the provider emits a usage chunk.
|
||||
|
||||
**External findings:**
|
||||
- **Non-streaming:** Ollama/LM Studio/llama.cpp `/v1` all return a populated `usage` object → `UsageDetails` populated. Confirmed multi-source. <https://docs.ollama.com/api/openai-compatibility>
|
||||
- **Streaming:** OpenAI semantics require `stream_options={"include_usage": true}`; whether MAF sets it automatically is **unverified** — so streamed `UsageDetails` may be `None`. → run non-streaming for the accounting path.
|
||||
- **Native `OllamaChatClient`:** maps Ollama's `prompt_eval_count`→input, `eval_count`→output; **`total_token_count` is NOT set (derive it)**. Streaming updates carry no usage. <https://docs.ollama.com/api/usage>
|
||||
- MAF usage plumbing is young: "Fix OTel usage detail attributes" landed in python-1.8.0 (already in our 1.9.0).
|
||||
|
||||
### 3. Tool-calling / structured output on local models — Confidence: high (load-bearing risk)
|
||||
|
||||
**External findings (strong multi-source pattern):**
|
||||
- Small CPU-runnable models are markedly weak at tool-calling: 7–8B tool-selection F1 ≈ 0.48–0.57; failure modes = never calling the tool (answers from memory), emitting the tool call as **plain text/JSON** instead of a structured `tool_calls` object, and argument type drift (`"5"` vs `5`). <https://www.docker.com/blog/local-llm-tool-calling-a-practical-evaluation/>
|
||||
- Qwen3-class is the most reliable small family (F1 ≈ 0.93 in the same eval); MAF docs explicitly bless `qwen3:4b` / `llama3.2` for tools and warn "not all models support function calling." `num_ctx ≥ 32k` improves reliability.
|
||||
- Streaming + tools on `/v1` historically silently dropped/leaked tool calls (5+ repos; fixed 2025-05-28 but a 0.12.3 regression remains) — **another reason to run non-streaming.** <https://ollama.com/blog/streaming-tool>
|
||||
- Structured output: `response_format: type[BaseModel]` is available on both local-path clients, but reliability is model-dependent.
|
||||
- MAF bug #1772: `OpenAIChatClient` + Ollama + ChatContext middleware inserts the system prompt twice — verify against 1.9.0; the native client may avoid it. <https://github.com/microsoft/agent-framework/issues/1772>
|
||||
|
||||
**Design fit:** our **blocking Pydantic validator becomes the mitigation** — it must retry/repair on parse failure, turning model unreliability into a bounded retry loop rather than silent corruption.
|
||||
|
||||
### 4. Intel-Mac CPU performance reality — Confidence: medium-high
|
||||
|
||||
**External findings:**
|
||||
- Intel Macs run Ollama **CPU-only** (no Metal/MLX — Apple-Silicon-only). Reported ~4–6 tok/s on a 7B model (i9 2019); 3–8 tok/s CPU-only generally — "usable for testing, painful for actual work." <https://localaimaster.com/blog/ollama-system-requirements>
|
||||
- A maker-checker debate (N agents × M rounds × tool round-trips × validator retries) at single-digit tok/s runs into minutes per candidate.
|
||||
|
||||
**Conclusion:** Intel-Mac local is a **plumbing-verification + smoke-test** environment (does the slice wire up, do tool calls fire, does the validator parse, does usage come back?), not an iterative/batch run environment — exactly matching D6 ("develop on local, verify minimally on Foundry"). Keep local runs tiny: 1 maker + 1 checker, ≤3 rounds, tiny synthetic data, hard token caps, a 3–4B model.
|
||||
|
||||
### 5. Determinism boundary — Confidence: high
|
||||
|
||||
- Local LLM output is non-deterministic even at temperature 0 / fixed seed (documented edge cases). This does **not** conflict with the success criterion **iff** the determinism criterion is scoped to the **validator + Monte-Carlo** (fixed seed + fixed candidate input → identical accept/reject + identical P10/P50/P90), NOT to the LLM that produces the candidate.
|
||||
- **Trap to avoid:** never assert end-to-end determinism *through* the LLM. Golden tests pin the validator's *inputs*, not the LLM's *outputs*. Our architecture already separates "agents propose (non-deterministic) → validator decides (deterministic)", so the boundary is sound — make it explicit in the test design.
|
||||
|
||||
### 6. Security / no-egress posture — Confidence: high
|
||||
|
||||
**External findings:**
|
||||
- `OpenAIChatCompletionClient(base_url=localhost)` keeps inference on-box; the OpenAI SDK sends only to `base_url` and never contacts `api.openai.com`; a dummy `api_key` is safe. **No new dependency** (uses already-installed `agent-framework-openai`). This is the lower-supply-chain-cost path.
|
||||
- Native `agent-framework-ollama` is official MS but **beta** and adds the `ollama` client dep (pinned `<0.5.4`) — a supply-chain + GA-discipline cost.
|
||||
- Ollama daemon: bind **`127.0.0.1`** (never `0.0.0.0` — it has NO auth by design, CVE-2025-63389); pin Ollama **≥ 0.17.1** (clears CVE-2024-37032 "Probllama" RCE, the 39719-22 set, and CVE-2026-7482 "Bleeding Llama" GGUF heap-read); keep `OLLAMA_DEBUG` unset (else prompts are written to `server.log`).
|
||||
- **Model-pull is explicit egress** (downloads from a registry) — document it in provenance; not "silent" but real. Air-gap option: pre-stage model blobs offline. LM Studio persists chats/RAG locally by default (treat as data-at-rest).
|
||||
- No native telemetry/phone-home from MAF client (instrumentation off by default), Ollama (no inference telemetry), or LM Studio (off by default).
|
||||
|
||||
## External Knowledge
|
||||
|
||||
### Best Practice
|
||||
- Local wiring: `OpenAIChatCompletionClient(base_url="http://127.0.0.1:11434/v1/", api_key="ollama", model_id="qwen3:4b")` (non-streaming); or native `OllamaChatClient(host=..., model_id=...)` after `--pre` install. Cost: no native cost metric — derive `cost = tokens × per-model price` (local = 0); token usage on OTel `gen_ai.client.token.usage`.
|
||||
|
||||
### Known Issues
|
||||
- Streaming+tools `/v1` drop/leak (run non-streaming); `#1772` double-system-prompt with OpenAIChatClient+Ollama+middleware (verify vs 1.9.0); native client `total_token_count` not set (derive); `prompt_eval_count` historically flaky for cached/large prompts. **No community report exists of MAF *Python* against a *local* model on an *Intel Mac* with token usage — this scenario is under-documented; verify empirically.**
|
||||
|
||||
## Gemini Second Opinion
|
||||
|
||||
Unavailable (same MCP-SDK/Interactions-API failure as Topics 1–2). No independent
|
||||
Gemini triangulation; treat as absent, not negative.
|
||||
|
||||
## Synthesis
|
||||
|
||||
The brief's premise — "run locally via `OpenAIChatClient(base_url=...)`; `UsageDetails`
|
||||
works because the local server returns OpenAI-style `usage`" — is *directionally
|
||||
right but wrong in two specifics that would fail success criteria on the very
|
||||
profile we develop on*. First, the **Chat Completions** client
|
||||
(`OpenAIChatCompletionClient`), not the **Responses** client (`OpenAIChatClient`),
|
||||
is the one that talks to a local `/v1` endpoint and the one whose verified
|
||||
`_parse_usage_from_openai` populates `UsageDetails`. Second, **streaming is the
|
||||
trap**: it is exactly where local `/v1` silently drops/leaks tool calls AND where
|
||||
`usage` comes back `None` unless `include_usage` is set (uncertain in MAF). Running
|
||||
the debate/validator path **non-streaming** fixes both at once — and non-streaming
|
||||
is what we want anyway for deterministic accounting.
|
||||
|
||||
The contrarian's "use the native `OllamaChatClient` instead" is a real option (MS
|
||||
docs route you to it; it sidesteps `/v1` translation edges and gets
|
||||
`prompt_eval_count`/`eval_count` directly), but it conflicts with the project's
|
||||
**GA-pin dependency discipline** (it is a `--pre` beta + a new `ollama` dep). The
|
||||
resolution is a **spike-gate**: start with the zero-new-dep Chat-Completions
|
||||
`base_url` path non-streaming; if a quick spike shows tool-calls leaking as text, or
|
||||
`usage=None`, or the #1772 middleware bug biting, switch to the native client and
|
||||
accept the scoped `--pre` dependency. Either way the model-map → chat-client ctor
|
||||
abstraction (D2) hides the choice behind the profile.
|
||||
|
||||
The deeper truth the swarm surfaces: on Intel-Mac CPU, "local primary" means
|
||||
**local proves the slice RUNS** (plumbing, tool-calls fire, validator parses, usage
|
||||
populated) — it does **not** prove the agents produce *good* candidates. "Both
|
||||
profiles exercised" therefore means: local = it runs; one minimal Foundry pass
|
||||
(cheapest model, hard cap) = the candidates are substantively sound. And the model's
|
||||
tool-calling weakness is absorbed by our **blocking validator-as-retry** — the
|
||||
obligatory validator stops being just a gate and becomes the reliability mechanism
|
||||
that makes a weak local model usable.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **Concrete local model + endpoint** (brief `[OPEN]`, operator-supplied): default
|
||||
to `qwen3:4b` via Ollama on `127.0.0.1:11434` for dev/smoke; `qwen2.5:7b`/`qwen3:8b`
|
||||
for reliability checks. The whole free-local run hinges on a working local endpoint
|
||||
existing on the machine.
|
||||
- **Foundry deployment names** (brief `[OPEN]`, tenant-specific, operator-supplied) —
|
||||
back the role→deployment map for the minimal Azure/Foundry-profile check.
|
||||
- **base_url vs native client** — resolve via a one-call spike in /trekexecute:
|
||||
does `OpenAIChatCompletionClient(base_url)` non-streaming return populated
|
||||
`UsageDetails` and structured tool-calls on the chosen model? If yes, no new dep;
|
||||
if no, switch to native `OllamaChatClient`.
|
||||
- **Does MAF set `include_usage` on streaming?** Unverified — irrelevant if we run
|
||||
non-streaming, but confirm if streaming is ever needed for narration.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Run the local profile via `OpenAIChatCompletionClient(base_url=<local /v1>)`,
|
||||
non-streaming, behind the D2 model-map abstraction; keep the native `OllamaChatClient`
|
||||
as a spike-gated fallback.**
|
||||
|
||||
1. **Client:** `OpenAIChatCompletionClient(base_url="http://127.0.0.1:11434/v1/",
|
||||
api_key="ollama", model_id="qwen3:4b")` — installed, **zero new deps**, honours the
|
||||
GA-pin discipline. NOT `OpenAIChatClient` (Responses API).
|
||||
2. **Non-streaming for debate + validator** — dodges `/v1` streaming+tools drop AND
|
||||
streaming `usage=None` in one move. Streaming only for non-tool narration, if ever.
|
||||
3. **Budget middleware: treat `usage_details is None` as a HARD FAIL in dev**
|
||||
(assertion) so a usage-reporting regression can never silently disable the budget
|
||||
cap. Maps directly to the "real token accounting" success criterion (assert the
|
||||
meter is populated from `UsageDetails` on a real run). Accumulate via
|
||||
`add_usage_details(...)`. No `len(...split())` proxy in `src/`.
|
||||
4. **Validator-as-retry:** the blocking Pydantic validator must retry/repair on parse
|
||||
failure — small models emit unparseable/text-leaked tool calls; this turns the
|
||||
obligatory validator into the reliability mechanism.
|
||||
5. **Model:** `qwen3:4b` for dev/smoke (MAF-blessed, tool-capable, CPU-tolerable);
|
||||
`qwen2.5:7b`/`qwen3:8b` for reliability checks; `num_ctx=32k`.
|
||||
6. **Determinism golden tests:** pin validator + Monte-Carlo inputs (fixed seed →
|
||||
identical decision/percentiles); never assert determinism through the LLM.
|
||||
7. **"Both profiles exercised":** local (full run) proves the slice RUNS; one minimal
|
||||
Foundry pass (cheapest model, hard token cap, per D6) proves candidates are
|
||||
substantively sound. Keep local runs tiny (1 maker + 1 checker, ≤3 rounds).
|
||||
8. **Spike-gate the client choice** in /trekexecute: if `base_url` non-streaming shows
|
||||
tool-call text-leakage, `usage=None`, or #1772, switch to native `OllamaChatClient`
|
||||
(accept the `--pre` `agent-framework-ollama` dep + verify its `prompt_eval_count`/
|
||||
`eval_count` mapping; derive `total_token_count`).
|
||||
9. **Security hardening:** Ollama bind `127.0.0.1`, pin **≥ 0.17.1**, `OLLAMA_DEBUG`
|
||||
unset; document model-pull as explicit egress; dummy `api_key`.
|
||||
|
||||
Risks to carry into the plan: (a) Intel-CPU latency → local is plumbing/smoke, not
|
||||
fast iteration; (b) small-model tool-calling weakness → lean on validator-retry +
|
||||
Qwen3; (c) native-client fallback is `--pre` (GA-discipline tension) — adopt only if
|
||||
spike-gated.
|
||||
|
||||
## Sources
|
||||
|
||||
| # | Source | Type | Quality | Used in |
|
||||
|---|--------|------|---------|---------|
|
||||
| 1 | `.venv/.../agent_framework_openai/__init__.py:33-43` (`OpenAIChatClient`, `OpenAIChatCompletionClient`) | codebase | high | Dim 1 |
|
||||
| 2 | `.venv/.../agent_framework_openai/_chat_completion_client.py:705,724-726,757-776` (UsageDetails mapping, None-safe) | codebase | high | Dim 2 |
|
||||
| 3 | `.venv/.../agent_framework/_types.py:402-417,547` (`UsageDetails`, `add_usage_details`, `response.usage_details`) | codebase | high | Dim 2 |
|
||||
| 4 | `.venv/.../agent_framework_openai/_chat_client.py:376,382` (`base_url`, `api_key`) | codebase | high | Dim 1 |
|
||||
| 5 | `agent_framework/ollama/__init__.pyi` + `import agent_framework_ollama` fails (not installed) | codebase | high | Dim 1, 6 |
|
||||
| 6 | <https://learn.microsoft.com/agent-framework/integrations/openai-endpoints> (base_url table, ChatCompletion client) | official | high | Dim 1, 2 |
|
||||
| 7 | <https://learn.microsoft.com/agent-framework/agents/providers/ollama> (native client, tool caveat, qwen3:4b) | official | high | Dim 1,3 |
|
||||
| 8 | <https://learn.microsoft.com/python/api/agent-framework-core/agent_framework.usagedetails?view=agent-framework-python-latest> | official | high | Dim 2 |
|
||||
| 9 | <https://pypi.org/project/agent-framework-ollama/> (1.0.0b260521 beta, --pre) | official | high | Dim 1, 6 |
|
||||
| 10 | <https://github.com/microsoft/agent-framework/blob/main/python/packages/ollama/pyproject.toml> (deps: ollama <0.5.4) | official | high | Dim 1, 6 |
|
||||
| 11 | <https://docs.ollama.com/api/openai-compatibility> (usage; streaming needs include_usage) | official | high | Dim 2 |
|
||||
| 12 | <https://docs.ollama.com/api/usage> (prompt_eval_count/eval_count) | official | high | Dim 2 |
|
||||
| 13 | <https://github.com/microsoft/agent-framework/releases> (1.8.0 OTel usage fix; 1.9.0 ollama tools fix) | official | high | Dim 2 |
|
||||
| 14 | <https://github.com/microsoft/agent-framework/issues/1772> (double system prompt, Ollama+middleware) | community | medium | Dim 3 |
|
||||
| 15 | <https://ollama.com/blog/streaming-tool> (streaming+tools fixed 2025-05-28) | official | high | Dim 3 |
|
||||
| 16 | <https://github.com/ollama/ollama/issues/12557> (streaming tool-call regression 0.12.3) | community | medium | Dim 3 |
|
||||
| 17 | <https://www.docker.com/blog/local-llm-tool-calling-a-practical-evaluation/> (tool-call F1 by model) | community | high | Dim 3 |
|
||||
| 18 | <https://www.morphllm.com/best-ollama-models> (Qwen3 best tool-calling) | community | medium | Dim 3 |
|
||||
| 19 | <https://localaimaster.com/blog/ollama-system-requirements> (Intel i9 4-6 tok/s, no MLX) | community | medium | Dim 4 |
|
||||
| 20 | <https://github.com/ollama/ollama/issues/586> (seed+temp0 not deterministic) | community | medium | Dim 5 |
|
||||
| 21 | <https://www.wiz.io/blog/probllama-ollama-vulnerability-cve-2024-37032> (CVE-2024-37032) | community | high | Dim 6 |
|
||||
| 22 | <https://github.com/advisories/GHSA-f6mr-38g8-39rg> (CVE-2025-63389 no-auth by design) | official | high | Dim 6 |
|
||||
| 23 | <https://www.runzero.com/blog/ollama/> (CVE-2026-7482 Bleeding Llama, < 0.17.1) | community | high | Dim 6 |
|
||||
| 24 | <https://www.indusface.com/blog/exposed-ollama-servers-llm-security-risks/> (bind 127.0.0.1) | community | high | Dim 6 |
|
||||
| 25 | <https://learn.microsoft.com/agent-framework/agents/observability> (no native cost metric) | official | high | Best Practice |
|
||||
| 26 | <https://lmstudio.ai/app-privacy> (LM Studio local persistence, no telemetry default) | official | medium | Dim 6 |
|
||||
Loading…
Add table
Add a link
Reference in a new issue