portfolio-optimiser/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/research/03-local-profile-chat-client.md

type	created	question	confidence	dimensions	mcp_servers_used	local_agents_used	external_agents_used	topic	brief
trekresearch-brief	2026-06-24	How does agent-framework-core 1.9.0 run agents against a local model on the free/local profile (OpenAI-compatible endpoint, Ollama, or other local chat client), and does that path populate UsageDetails token counts or return None?	0.80	6	microsoft-learn tavily	installed-source-introspection (orchestrator main context)	docs-researcher community-researcher security-researcher contrarian-researcher gemini-bridge (unavailable)	3	.claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md

Real local-profile chat client for agent-framework 1.9.0 + UsageDetails

Generated by trekresearch (Voyage 5.6.0) on 2026-06-24. Topic 3 of 3 for the Fase 2 MVP-vertical-slice brief. Scope: external swarm + installed-source introspection (installed API truth wins). Gemini unavailable (MCP SDK broke on Google's May-2026 API change).

Research Question

How does agent-framework-core 1.9.0 (+ agent-framework-openai) run agents against a LOCAL model on the free/local profile, and does that path populate UsageDetails token counts or return None? Decision it feeds: the backend-profile abstraction (local path), the budget middleware's None-handling, the end-to-end run, and cost-discipline verification.

Executive Summary

Two local paths exist, and the brief's framing needs two corrections. Path A — OpenAIChatCompletionClient(base_url=<local /v1>) — is installed (1.8.2), needs zero new dependencies, and populates UsageDetails None-safely from the local server's usage object. Path B — the native OllamaChatClient — is a --pre beta package (agent-framework-ollama 1.0.0b260521) that is NOT installed and adds the ollama client dependency. Correction 1 (client): use OpenAIChatCompletionClient (Chat Completions API — the surface Ollama/LM Studio /v1 exposes), NOT OpenAIChatClient (Responses API). Correction 2 (streaming): run the debate/validator path NON-STREAMING — that simultaneously dodges the well-documented /v1 streaming+tool-calling drop bugs AND the streaming usage=None problem (UsageDetails is reliably populated only non-streaming). Token accounting works (confirmed in installed source), so the "real token accounting" success criterion is satisfiable on the local profile. The real limits are model capability and Intel-Mac CPU speed: small CPU-runnable models are weak at tool-calling, and an Intel Mac (no Metal/MLX, ~4–6 tok/s on 7B) makes a multi-round debate take minutes — so local is for plumbing/smoke verification, with representative-quality runs pushed to Foundry. Confidence 0.80.

Dimensions

1. Local chat-client paths — Confidence: high

Installed-source findings (ground truth):

• agent-framework-openai 1.8.2 exports both OpenAIChatClient (Responses API) and OpenAIChatCompletionClient (Chat Completions API) — both import OK in the venv. RawOpenAIChatClient/RawOpenAIChatCompletionClient are the lower-level variants.
• Both accept base_url (→ env OPENAI_BASE_URL) and api_key (required even for local — pass a dummy like "ollama"/"not-needed"). model_id selects the model.
• The native Ollama path: agent_framework/ollama/__init__.pyi re-exports OllamaChatClient, OllamaChatOptions, OllamaEmbeddingClient, … from agent_framework_ollama — but agent_framework_ollama is NOT installed (import agent_framework_ollama → ModuleNotFoundError; only the lazy re-export stub exists). Installing it (uv add agent-framework-ollama --prerelease=allow) pulls ollama >=0.5.3,<0.5.4.

External findings:

• Correct client for local /v1: Ollama/LM Studio/vLLM expose a Chat Completions surface (/v1/chat/completions), so OpenAIChatCompletionClient is the technically correct client; OpenAIChatClient targets the Responses API which local servers don't expose. Two official MS docs disagree on which to use for Ollama — flagged conflict; the Chat-Completions client is correct for /v1. https://learn.microsoft.com/agent-framework/integrations/openai-endpoints
• Official local-endpoint table: Ollama http://localhost:11434/v1/, LM Studio http://localhost:1234/v1/, vLLM http://localhost:8000/v1/. The /v1/ suffix matters; the native OllamaChatClient uses :11434 WITHOUT /v1.
• Native OllamaChatClient is preview/beta (1.0.0b260521, "4-Beta", --pre), wraps ollama.AsyncClient (native /api/chat), OTEL_PROVIDER_NAME='ollama'. MS docs describe it as having "full support for function tools and streaming." https://learn.microsoft.com/agent-framework/agents/providers/ollama

2. UsageDetails population — Confidence: high

Installed-source findings (the load-bearing answer):

• OpenAIChatCompletionClient: _chat_completion_client.py:705 — usage_details=self._parse_usage_from_openai(response.usage) if response.usage else None. None-safe. _parse_usage_from_openai (:757-776) maps usage.prompt_tokens→input_token_count, usage.completion_tokens→output_token_count, usage.total_tokens→total_token_count (+ reasoning/cached/audio details).
• UsageDetails is the TypedDict in agent_framework/_types.py (input/output/total_token_count: int | None); add_usage_details(u1, u2) helper exists (_types.py:417) — use it to accumulate across calls in the shared budget meter. Lands on response.usage_details (_types.py:547); read response.usage_details["total_token_count"] (None-safe).
• Streaming: _chat_completion_client.py:724-726 — if chunk.usage: Content.from_usage(...) — usage only arrives if the provider emits a usage chunk.

External findings:

• Non-streaming: Ollama/LM Studio/llama.cpp /v1 all return a populated usage object → UsageDetails populated. Confirmed multi-source. https://docs.ollama.com/api/openai-compatibility
• Streaming: OpenAI semantics require stream_options={"include_usage": true}; whether MAF sets it automatically is unverified — so streamed UsageDetails may be None. → run non-streaming for the accounting path.
• Native OllamaChatClient: maps Ollama's prompt_eval_count→input, eval_count→output; total_token_count is NOT set (derive it). Streaming updates carry no usage. https://docs.ollama.com/api/usage
• MAF usage plumbing is young: "Fix OTel usage detail attributes" landed in python-1.8.0 (already in our 1.9.0).

3. Tool-calling / structured output on local models — Confidence: high (load-bearing risk)

External findings (strong multi-source pattern):

• Small CPU-runnable models are markedly weak at tool-calling: 7–8B tool-selection F1 ≈ 0.48–0.57; failure modes = never calling the tool (answers from memory), emitting the tool call as plain text/JSON instead of a structured tool_calls object, and argument type drift ("5" vs 5). https://www.docker.com/blog/local-llm-tool-calling-a-practical-evaluation/
• Qwen3-class is the most reliable small family (F1 ≈ 0.93 in the same eval); MAF docs explicitly bless qwen3:4b / llama3.2 for tools and warn "not all models support function calling." num_ctx ≥ 32k improves reliability.
• Streaming + tools on /v1 historically silently dropped/leaked tool calls (5+ repos; fixed 2025-05-28 but a 0.12.3 regression remains) — another reason to run non-streaming. https://ollama.com/blog/streaming-tool
• Structured output: response_format: type[BaseModel] is available on both local-path clients, but reliability is model-dependent.
• MAF bug #1772: OpenAIChatClient + Ollama + ChatContext middleware inserts the system prompt twice — verify against 1.9.0; the native client may avoid it. https://github.com/microsoft/agent-framework/issues/1772

Design fit: our blocking Pydantic validator becomes the mitigation — it must retry/repair on parse failure, turning model unreliability into a bounded retry loop rather than silent corruption.

4. Intel-Mac CPU performance reality — Confidence: medium-high

External findings:

• Intel Macs run Ollama CPU-only (no Metal/MLX — Apple-Silicon-only). Reported ~4–6 tok/s on a 7B model (i9 2019); 3–8 tok/s CPU-only generally — "usable for testing, painful for actual work." https://localaimaster.com/blog/ollama-system-requirements
• A maker-checker debate (N agents × M rounds × tool round-trips × validator retries) at single-digit tok/s runs into minutes per candidate.

Conclusion: Intel-Mac local is a plumbing-verification + smoke-test environment (does the slice wire up, do tool calls fire, does the validator parse, does usage come back?), not an iterative/batch run environment — exactly matching D6 ("develop on local, verify minimally on Foundry"). Keep local runs tiny: 1 maker + 1 checker, ≤3 rounds, tiny synthetic data, hard token caps, a 3–4B model.

5. Determinism boundary — Confidence: high

• Local LLM output is non-deterministic even at temperature 0 / fixed seed (documented edge cases). This does not conflict with the success criterion iff the determinism criterion is scoped to the validator + Monte-Carlo (fixed seed + fixed candidate input → identical accept/reject + identical P10/P50/P90), NOT to the LLM that produces the candidate.
• Trap to avoid: never assert end-to-end determinism through the LLM. Golden tests pin the validator's inputs, not the LLM's outputs. Our architecture already separates "agents propose (non-deterministic) → validator decides (deterministic)", so the boundary is sound — make it explicit in the test design.

6. Security / no-egress posture — Confidence: high

External findings:

• OpenAIChatCompletionClient(base_url=localhost) keeps inference on-box; the OpenAI SDK sends only to base_url and never contacts api.openai.com; a dummy api_key is safe. No new dependency (uses already-installed agent-framework-openai). This is the lower-supply-chain-cost path.
• Native agent-framework-ollama is official MS but beta and adds the ollama client dep (pinned <0.5.4) — a supply-chain + GA-discipline cost.
• Ollama daemon: bind 127.0.0.1 (never 0.0.0.0 — it has NO auth by design, CVE-2025-63389); pin Ollama ≥ 0.17.1 (clears CVE-2024-37032 "Probllama" RCE, the 39719-22 set, and CVE-2026-7482 "Bleeding Llama" GGUF heap-read); keep OLLAMA_DEBUG unset (else prompts are written to server.log).
• Model-pull is explicit egress (downloads from a registry) — document it in provenance; not "silent" but real. Air-gap option: pre-stage model blobs offline. LM Studio persists chats/RAG locally by default (treat as data-at-rest).
• No native telemetry/phone-home from MAF client (instrumentation off by default), Ollama (no inference telemetry), or LM Studio (off by default).

External Knowledge

Best Practice

• Local wiring: OpenAIChatCompletionClient(base_url="http://127.0.0.1:11434/v1/", api_key="ollama", model_id="qwen3:4b") (non-streaming); or native OllamaChatClient(host=..., model_id=...) after --pre install. Cost: no native cost metric — derive cost = tokens × per-model price (local = 0); token usage on OTel gen_ai.client.token.usage.

Known Issues

• Streaming+tools /v1 drop/leak (run non-streaming); #1772 double-system-prompt with OpenAIChatClient+Ollama+middleware (verify vs 1.9.0); native client total_token_count not set (derive); prompt_eval_count historically flaky for cached/large prompts. No community report exists of MAF Python against a local model on an Intel Mac with token usage — this scenario is under-documented; verify empirically.

Gemini Second Opinion

Unavailable (same MCP-SDK/Interactions-API failure as Topics 1–2). No independent Gemini triangulation; treat as absent, not negative.

Synthesis

The brief's premise — "run locally via OpenAIChatClient(base_url=...); UsageDetails works because the local server returns OpenAI-style usage" — is directionally right but wrong in two specifics that would fail success criteria on the very profile we develop on. First, the Chat Completions client (OpenAIChatCompletionClient), not the Responses client (OpenAIChatClient), is the one that talks to a local /v1 endpoint and the one whose verified _parse_usage_from_openai populates UsageDetails. Second, streaming is the trap: it is exactly where local /v1 silently drops/leaks tool calls AND where usage comes back None unless include_usage is set (uncertain in MAF). Running the debate/validator path non-streaming fixes both at once — and non-streaming is what we want anyway for deterministic accounting.

The contrarian's "use the native OllamaChatClient instead" is a real option (MS docs route you to it; it sidesteps /v1 translation edges and gets prompt_eval_count/eval_count directly), but it conflicts with the project's GA-pin dependency discipline (it is a --pre beta + a new ollama dep). The resolution is a spike-gate: start with the zero-new-dep Chat-Completions base_url path non-streaming; if a quick spike shows tool-calls leaking as text, or usage=None, or the #1772 middleware bug biting, switch to the native client and accept the scoped --pre dependency. Either way the model-map → chat-client ctor abstraction (D2) hides the choice behind the profile.

The deeper truth the swarm surfaces: on Intel-Mac CPU, "local primary" means local proves the slice RUNS (plumbing, tool-calls fire, validator parses, usage populated) — it does not prove the agents produce good candidates. "Both profiles exercised" therefore means: local = it runs; one minimal Foundry pass (cheapest model, hard cap) = the candidates are substantively sound. And the model's tool-calling weakness is absorbed by our blocking validator-as-retry — the obligatory validator stops being just a gate and becomes the reliability mechanism that makes a weak local model usable.

Open Questions

• Concrete local model + endpoint (brief [OPEN], operator-supplied): default to qwen3:4b via Ollama on 127.0.0.1:11434 for dev/smoke; qwen2.5:7b/qwen3:8b for reliability checks. The whole free-local run hinges on a working local endpoint existing on the machine.
• Foundry deployment names (brief [OPEN], tenant-specific, operator-supplied) — back the role→deployment map for the minimal Azure/Foundry-profile check.
• base_url vs native client — resolve via a one-call spike in /trekexecute: does OpenAIChatCompletionClient(base_url) non-streaming return populated UsageDetails and structured tool-calls on the chosen model? If yes, no new dep; if no, switch to native OllamaChatClient.
• Does MAF set include_usage on streaming? Unverified — irrelevant if we run non-streaming, but confirm if streaming is ever needed for narration.

Recommendation

Run the local profile via OpenAIChatCompletionClient(base_url=<local /v1>), non-streaming, behind the D2 model-map abstraction; keep the native OllamaChatClient as a spike-gated fallback.

1. Client: OpenAIChatCompletionClient(base_url="http://127.0.0.1:11434/v1/", api_key="ollama", model_id="qwen3:4b") — installed, zero new deps, honours the GA-pin discipline. NOT OpenAIChatClient (Responses API).
2. Non-streaming for debate + validator — dodges /v1 streaming+tools drop AND streaming usage=None in one move. Streaming only for non-tool narration, if ever.
3. Budget middleware: treat usage_details is None as a HARD FAIL in dev (assertion) so a usage-reporting regression can never silently disable the budget cap. Maps directly to the "real token accounting" success criterion (assert the meter is populated from UsageDetails on a real run). Accumulate via add_usage_details(...). No len(...split()) proxy in src/.
4. Validator-as-retry: the blocking Pydantic validator must retry/repair on parse failure — small models emit unparseable/text-leaked tool calls; this turns the obligatory validator into the reliability mechanism.
5. Model: qwen3:4b for dev/smoke (MAF-blessed, tool-capable, CPU-tolerable); qwen2.5:7b/qwen3:8b for reliability checks; num_ctx=32k.
6. Determinism golden tests: pin validator + Monte-Carlo inputs (fixed seed → identical decision/percentiles); never assert determinism through the LLM.
7. "Both profiles exercised": local (full run) proves the slice RUNS; one minimal Foundry pass (cheapest model, hard token cap, per D6) proves candidates are substantively sound. Keep local runs tiny (1 maker + 1 checker, ≤3 rounds).
8. Spike-gate the client choice in /trekexecute: if base_url non-streaming shows tool-call text-leakage, usage=None, or #1772, switch to native OllamaChatClient (accept the --pre agent-framework-ollama dep + verify its prompt_eval_count/ eval_count mapping; derive total_token_count).
9. Security hardening: Ollama bind 127.0.0.1, pin ≥ 0.17.1, OLLAMA_DEBUG unset; document model-pull as explicit egress; dummy api_key.

Risks to carry into the plan: (a) Intel-CPU latency → local is plumbing/smoke, not fast iteration; (b) small-model tool-calling weakness → lean on validator-retry + Qwen3; (c) native-client fallback is --pre (GA-discipline tension) — adopt only if spike-gated.

Sources

#	Source	Type	Quality	Used in
1	.venv/.../agent_framework_openai/__init__.py:33-43 (OpenAIChatClient, OpenAIChatCompletionClient)	codebase	high	Dim 1
2	.venv/.../agent_framework_openai/_chat_completion_client.py:705,724-726,757-776 (UsageDetails mapping, None-safe)	codebase	high	Dim 2
3	.venv/.../agent_framework/_types.py:402-417,547 (UsageDetails, add_usage_details, response.usage_details)	codebase	high	Dim 2
4	.venv/.../agent_framework_openai/_chat_client.py:376,382 (base_url, api_key)	codebase	high	Dim 1
5	agent_framework/ollama/__init__.pyi + import agent_framework_ollama fails (not installed)	codebase	high	Dim 1, 6
6	https://learn.microsoft.com/agent-framework/integrations/openai-endpoints (base_url table, ChatCompletion client)	official	high	Dim 1, 2
7	https://learn.microsoft.com/agent-framework/agents/providers/ollama (native client, tool caveat, qwen3:4b)	official	high	Dim 1,3
8	https://learn.microsoft.com/python/api/agent-framework-core/agent_framework.usagedetails?view=agent-framework-python-latest	official	high	Dim 2
9	https://pypi.org/project/agent-framework-ollama/ (1.0.0b260521 beta, --pre)	official	high	Dim 1, 6
10	https://github.com/microsoft/agent-framework/blob/main/python/packages/ollama/pyproject.toml (deps: ollama <0.5.4)	official	high	Dim 1, 6
11	https://docs.ollama.com/api/openai-compatibility (usage; streaming needs include_usage)	official	high	Dim 2
12	https://docs.ollama.com/api/usage (prompt_eval_count/eval_count)	official	high	Dim 2
13	https://github.com/microsoft/agent-framework/releases (1.8.0 OTel usage fix; 1.9.0 ollama tools fix)	official	high	Dim 2
14	https://github.com/microsoft/agent-framework/issues/1772 (double system prompt, Ollama+middleware)	community	medium	Dim 3
15	https://ollama.com/blog/streaming-tool (streaming+tools fixed 2025-05-28)	official	high	Dim 3
16	https://github.com/ollama/ollama/issues/12557 (streaming tool-call regression 0.12.3)	community	medium	Dim 3
17	https://www.docker.com/blog/local-llm-tool-calling-a-practical-evaluation/ (tool-call F1 by model)	community	high	Dim 3
18	https://www.morphllm.com/best-ollama-models (Qwen3 best tool-calling)	community	medium	Dim 3
19	https://localaimaster.com/blog/ollama-system-requirements (Intel i9 4-6 tok/s, no MLX)	community	medium	Dim 4
20	https://github.com/ollama/ollama/issues/586 (seed+temp0 not deterministic)	community	medium	Dim 5
21	https://www.wiz.io/blog/probllama-ollama-vulnerability-cve-2024-37032 (CVE-2024-37032)	community	high	Dim 6
22	https://github.com/advisories/GHSA-f6mr-38g8-39rg (CVE-2025-63389 no-auth by design)	official	high	Dim 6
23	https://www.runzero.com/blog/ollama/ (CVE-2026-7482 Bleeding Llama, < 0.17.1)	community	high	Dim 6
24	https://www.indusface.com/blog/exposed-ollama-servers-llm-security-risks/ (bind 127.0.0.1)	community	high	Dim 6
25	https://learn.microsoft.com/agent-framework/agents/observability (no native cost metric)	official	high	Best Practice
26	https://lmstudio.ai/app-privacy (LM Studio local persistence, no telemetry default)	official	medium	Dim 6