portfolio-optimiser/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/research/03-local-profile-chat-client.md

---
type: trekresearch-brief
created: 2026-06-24
question: "How does agent-framework-core 1.9.0 run agents against a local model on the free/local profile (OpenAI-compatible endpoint, Ollama, or other local chat client), and does that path populate UsageDetails token counts or return None?"
confidence: 0.80
dimensions: 6
mcp_servers_used: [microsoft-learn, tavily]
local_agents_used: [installed-source-introspection (orchestrator, main context)]
external_agents_used: [docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (unavailable)]
topic: 3
brief: .claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md
---

# Real local-profile chat client for agent-framework 1.9.0 + UsageDetails

> Generated by trekresearch (Voyage 5.6.0) on 2026-06-24. Topic 3 of 3 for the
> Fase 2 MVP-vertical-slice brief. Scope: external swarm + installed-source
> introspection (installed API truth wins). Gemini unavailable (MCP SDK broke on
> Google's May-2026 API change).

## Research Question

How does `agent-framework-core` 1.9.0 (+ `agent-framework-openai`) run agents
against a LOCAL model on the free/local profile, and does that path populate
`UsageDetails` token counts or return `None`? Decision it feeds: the backend-profile
abstraction (local path), the budget middleware's `None`-handling, the end-to-end
run, and cost-discipline verification.

## Executive Summary

**Two local paths exist, and the brief's framing needs two corrections.** Path A —
`OpenAIChatCompletionClient(base_url=<local /v1>)` — is **installed (1.8.2), needs
zero new dependencies, and populates `UsageDetails` None-safely from the local
server's `usage` object**. Path B — the native `OllamaChatClient` — is a **`--pre`
beta package (`agent-framework-ollama 1.0.0b260521`) that is NOT installed** and
adds the `ollama` client dependency. **Correction 1 (client):** use
`OpenAIChatCompletionClient` (Chat Completions API — the surface Ollama/LM Studio
`/v1` exposes), NOT `OpenAIChatClient` (Responses API). **Correction 2 (streaming):**
run the debate/validator path **NON-STREAMING** — that simultaneously dodges the
well-documented `/v1` streaming+tool-calling drop bugs AND the streaming
`usage=None` problem (`UsageDetails` is reliably populated only non-streaming).
**Token accounting works** (confirmed in installed source), so the "real token
accounting" success criterion is satisfiable on the local profile. The real limits
are model capability and Intel-Mac CPU speed: small CPU-runnable models are weak at
tool-calling, and an Intel Mac (no Metal/MLX, ~4–6 tok/s on 7B) makes a multi-round
debate take minutes — so local is for plumbing/smoke verification, with
representative-quality runs pushed to Foundry. Confidence 0.80.

## Dimensions

### 1. Local chat-client paths — Confidence: high

**Installed-source findings (ground truth):**
- `agent-framework-openai` 1.8.2 exports **both** `OpenAIChatClient` (Responses API) and `OpenAIChatCompletionClient` (Chat Completions API) — both import OK in the venv. `RawOpenAIChatClient`/`RawOpenAIChatCompletionClient` are the lower-level variants.
- Both accept `base_url` (→ env `OPENAI_BASE_URL`) and `api_key` (required even for local — pass a dummy like `"ollama"`/`"not-needed"`). `model_id` selects the model.
- The native Ollama path: `agent_framework/ollama/__init__.pyi` re-exports `OllamaChatClient`, `OllamaChatOptions`, `OllamaEmbeddingClient`, … from `agent_framework_ollama` — but **`agent_framework_ollama` is NOT installed** (`import agent_framework_ollama` → ModuleNotFoundError; only the lazy re-export stub exists). Installing it (`uv add agent-framework-ollama --prerelease=allow`) pulls `ollama >=0.5.3,<0.5.4`.

**External findings:**
- **Correct client for local `/v1`:** Ollama/LM Studio/vLLM expose a Chat Completions surface (`/v1/chat/completions`), so `OpenAIChatCompletionClient` is the technically correct client; `OpenAIChatClient` targets the Responses API which local servers don't expose. Two official MS docs disagree on which to use for Ollama — **flagged conflict; the Chat-Completions client is correct for `/v1`.** <https://learn.microsoft.com/agent-framework/integrations/openai-endpoints>
- Official local-endpoint table: Ollama `http://localhost:11434/v1/`, LM Studio `http://localhost:1234/v1/`, vLLM `http://localhost:8000/v1/`. The `/v1/` suffix matters; the native `OllamaChatClient` uses `:11434` WITHOUT `/v1`.
- Native `OllamaChatClient` is **preview/beta** (`1.0.0b260521`, "4-Beta", `--pre`), wraps `ollama.AsyncClient` (native `/api/chat`), `OTEL_PROVIDER_NAME='ollama'`. MS docs describe it as having "full support for function tools and streaming." <https://learn.microsoft.com/agent-framework/agents/providers/ollama>

### 2. UsageDetails population — Confidence: high

**Installed-source findings (the load-bearing answer):**
- `OpenAIChatCompletionClient`: `_chat_completion_client.py:705` — `usage_details=self._parse_usage_from_openai(response.usage) if response.usage else None`. **None-safe.** `_parse_usage_from_openai` (`:757-776`) maps `usage.prompt_tokens`→`input_token_count`, `usage.completion_tokens`→`output_token_count`, `usage.total_tokens`→`total_token_count` (+ reasoning/cached/audio details).
- `UsageDetails` is the TypedDict in `agent_framework/_types.py` (`input/output/total_token_count: int | None`); `add_usage_details(u1, u2)` helper exists (`_types.py:417`) — use it to accumulate across calls in the shared budget meter. Lands on `response.usage_details` (`_types.py:547`); read `response.usage_details["total_token_count"]` (None-safe).
- Streaming: `_chat_completion_client.py:724-726` — `if chunk.usage: Content.from_usage(...)` — usage only arrives if the provider emits a usage chunk.

**External findings:**
- **Non-streaming:** Ollama/LM Studio/llama.cpp `/v1` all return a populated `usage` object → `UsageDetails` populated. Confirmed multi-source. <https://docs.ollama.com/api/openai-compatibility>
- **Streaming:** OpenAI semantics require `stream_options={"include_usage": true}`; whether MAF sets it automatically is **unverified** — so streamed `UsageDetails` may be `None`. → run non-streaming for the accounting path.
- **Native `OllamaChatClient`:** maps Ollama's `prompt_eval_count`→input, `eval_count`→output; **`total_token_count` is NOT set (derive it)**. Streaming updates carry no usage. <https://docs.ollama.com/api/usage>
- MAF usage plumbing is young: "Fix OTel usage detail attributes" landed in python-1.8.0 (already in our 1.9.0).

### 3. Tool-calling / structured output on local models — Confidence: high (load-bearing risk)

**External findings (strong multi-source pattern):**
- Small CPU-runnable models are markedly weak at tool-calling: 7–8B tool-selection F1 ≈ 0.48–0.57; failure modes = never calling the tool (answers from memory), emitting the tool call as **plain text/JSON** instead of a structured `tool_calls` object, and argument type drift (`"5"` vs `5`). <https://www.docker.com/blog/local-llm-tool-calling-a-practical-evaluation/>
- Qwen3-class is the most reliable small family (F1 ≈ 0.93 in the same eval); MAF docs explicitly bless `qwen3:4b` / `llama3.2` for tools and warn "not all models support function calling." `num_ctx ≥ 32k` improves reliability.
- Streaming + tools on `/v1` historically silently dropped/leaked tool calls (5+ repos; fixed 2025-05-28 but a 0.12.3 regression remains) — **another reason to run non-streaming.** <https://ollama.com/blog/streaming-tool>
- Structured output: `response_format: type[BaseModel]` is available on both local-path clients, but reliability is model-dependent.
- MAF bug #1772: `OpenAIChatClient` + Ollama + ChatContext middleware inserts the system prompt twice — verify against 1.9.0; the native client may avoid it. <https://github.com/microsoft/agent-framework/issues/1772>

**Design fit:** our **blocking Pydantic validator becomes the mitigation** — it must retry/repair on parse failure, turning model unreliability into a bounded retry loop rather than silent corruption.

### 4. Intel-Mac CPU performance reality — Confidence: medium-high

**External findings:**
- Intel Macs run Ollama **CPU-only** (no Metal/MLX — Apple-Silicon-only). Reported ~4–6 tok/s on a 7B model (i9 2019); 3–8 tok/s CPU-only generally — "usable for testing, painful for actual work." <https://localaimaster.com/blog/ollama-system-requirements>
- A maker-checker debate (N agents × M rounds × tool round-trips × validator retries) at single-digit tok/s runs into minutes per candidate.

**Conclusion:** Intel-Mac local is a **plumbing-verification + smoke-test** environment (does the slice wire up, do tool calls fire, does the validator parse, does usage come back?), not an iterative/batch run environment — exactly matching D6 ("develop on local, verify minimally on Foundry"). Keep local runs tiny: 1 maker + 1 checker, ≤3 rounds, tiny synthetic data, hard token caps, a 3–4B model.

### 5. Determinism boundary — Confidence: high

- Local LLM output is non-deterministic even at temperature 0 / fixed seed (documented edge cases). This does **not** conflict with the success criterion **iff** the determinism criterion is scoped to the **validator + Monte-Carlo** (fixed seed + fixed candidate input → identical accept/reject + identical P10/P50/P90), NOT to the LLM that produces the candidate.
- **Trap to avoid:** never assert end-to-end determinism *through* the LLM. Golden tests pin the validator's *inputs*, not the LLM's *outputs*. Our architecture already separates "agents propose (non-deterministic) → validator decides (deterministic)", so the boundary is sound — make it explicit in the test design.

### 6. Security / no-egress posture — Confidence: high

**External findings:**
- `OpenAIChatCompletionClient(base_url=localhost)` keeps inference on-box; the OpenAI SDK sends only to `base_url` and never contacts `api.openai.com`; a dummy `api_key` is safe. **No new dependency** (uses already-installed `agent-framework-openai`). This is the lower-supply-chain-cost path.
- Native `agent-framework-ollama` is official MS but **beta** and adds the `ollama` client dep (pinned `<0.5.4`) — a supply-chain + GA-discipline cost.
- Ollama daemon: bind **`127.0.0.1`** (never `0.0.0.0` — it has NO auth by design, CVE-2025-63389); pin Ollama **≥ 0.17.1** (clears CVE-2024-37032 "Probllama" RCE, the 39719-22 set, and CVE-2026-7482 "Bleeding Llama" GGUF heap-read); keep `OLLAMA_DEBUG` unset (else prompts are written to `server.log`).
- **Model-pull is explicit egress** (downloads from a registry) — document it in provenance; not "silent" but real. Air-gap option: pre-stage model blobs offline. LM Studio persists chats/RAG locally by default (treat as data-at-rest).
- No native telemetry/phone-home from MAF client (instrumentation off by default), Ollama (no inference telemetry), or LM Studio (off by default).

## External Knowledge

### Best Practice
- Local wiring: `OpenAIChatCompletionClient(base_url="http://127.0.0.1:11434/v1/", api_key="ollama", model_id="qwen3:4b")` (non-streaming); or native `OllamaChatClient(host=..., model_id=...)` after `--pre` install. Cost: no native cost metric — derive `cost = tokens × per-model price` (local = 0); token usage on OTel `gen_ai.client.token.usage`.

### Known Issues
- Streaming+tools `/v1` drop/leak (run non-streaming); `#1772` double-system-prompt with OpenAIChatClient+Ollama+middleware (verify vs 1.9.0); native client `total_token_count` not set (derive); `prompt_eval_count` historically flaky for cached/large prompts. **No community report exists of MAF *Python* against a *local* model on an *Intel Mac* with token usage — this scenario is under-documented; verify empirically.**

## Gemini Second Opinion

Unavailable (same MCP-SDK/Interactions-API failure as Topics 1–2). No independent
Gemini triangulation; treat as absent, not negative.

## Synthesis

The brief's premise — "run locally via `OpenAIChatClient(base_url=...)`; `UsageDetails`
works because the local server returns OpenAI-style `usage`" — is *directionally
right but wrong in two specifics that would fail success criteria on the very
profile we develop on*. First, the **Chat Completions** client
(`OpenAIChatCompletionClient`), not the **Responses** client (`OpenAIChatClient`),
is the one that talks to a local `/v1` endpoint and the one whose verified
`_parse_usage_from_openai` populates `UsageDetails`. Second, **streaming is the
trap**: it is exactly where local `/v1` silently drops/leaks tool calls AND where
`usage` comes back `None` unless `include_usage` is set (uncertain in MAF). Running
the debate/validator path **non-streaming** fixes both at once — and non-streaming
is what we want anyway for deterministic accounting.

The contrarian's "use the native `OllamaChatClient` instead" is a real option (MS
docs route you to it; it sidesteps `/v1` translation edges and gets
`prompt_eval_count`/`eval_count` directly), but it conflicts with the project's
**GA-pin dependency discipline** (it is a `--pre` beta + a new `ollama` dep). The
resolution is a **spike-gate**: start with the zero-new-dep Chat-Completions
`base_url` path non-streaming; if a quick spike shows tool-calls leaking as text, or
`usage=None`, or the #1772 middleware bug biting, switch to the native client and
accept the scoped `--pre` dependency. Either way the model-map → chat-client ctor
abstraction (D2) hides the choice behind the profile.

The deeper truth the swarm surfaces: on Intel-Mac CPU, "local primary" means
**local proves the slice RUNS** (plumbing, tool-calls fire, validator parses, usage
populated) — it does **not** prove the agents produce *good* candidates. "Both
profiles exercised" therefore means: local = it runs; one minimal Foundry pass
(cheapest model, hard cap) = the candidates are substantively sound. And the model's
tool-calling weakness is absorbed by our **blocking validator-as-retry** — the
obligatory validator stops being just a gate and becomes the reliability mechanism
that makes a weak local model usable.

## Open Questions

- **Concrete local model + endpoint** (brief `[OPEN]`, operator-supplied): default
  to `qwen3:4b` via Ollama on `127.0.0.1:11434` for dev/smoke; `qwen2.5:7b`/`qwen3:8b`
  for reliability checks. The whole free-local run hinges on a working local endpoint
  existing on the machine.
- **Foundry deployment names** (brief `[OPEN]`, tenant-specific, operator-supplied) —
  back the role→deployment map for the minimal Azure/Foundry-profile check.
- **base_url vs native client** — resolve via a one-call spike in /trekexecute:
  does `OpenAIChatCompletionClient(base_url)` non-streaming return populated
  `UsageDetails` and structured tool-calls on the chosen model? If yes, no new dep;
  if no, switch to native `OllamaChatClient`.
- **Does MAF set `include_usage` on streaming?** Unverified — irrelevant if we run
  non-streaming, but confirm if streaming is ever needed for narration.

## Recommendation

**Run the local profile via `OpenAIChatCompletionClient(base_url=<local /v1>)`,
non-streaming, behind the D2 model-map abstraction; keep the native `OllamaChatClient`
as a spike-gated fallback.**

1. **Client:** `OpenAIChatCompletionClient(base_url="http://127.0.0.1:11434/v1/",
   api_key="ollama", model_id="qwen3:4b")` — installed, **zero new deps**, honours the
   GA-pin discipline. NOT `OpenAIChatClient` (Responses API).
2. **Non-streaming for debate + validator** — dodges `/v1` streaming+tools drop AND
   streaming `usage=None` in one move. Streaming only for non-tool narration, if ever.
3. **Budget middleware: treat `usage_details is None` as a HARD FAIL in dev**
   (assertion) so a usage-reporting regression can never silently disable the budget
   cap. Maps directly to the "real token accounting" success criterion (assert the
   meter is populated from `UsageDetails` on a real run). Accumulate via
   `add_usage_details(...)`. No `len(...split())` proxy in `src/`.
4. **Validator-as-retry:** the blocking Pydantic validator must retry/repair on parse
   failure — small models emit unparseable/text-leaked tool calls; this turns the
   obligatory validator into the reliability mechanism.
5. **Model:** `qwen3:4b` for dev/smoke (MAF-blessed, tool-capable, CPU-tolerable);
   `qwen2.5:7b`/`qwen3:8b` for reliability checks; `num_ctx=32k`.
6. **Determinism golden tests:** pin validator + Monte-Carlo inputs (fixed seed →
   identical decision/percentiles); never assert determinism through the LLM.
7. **"Both profiles exercised":** local (full run) proves the slice RUNS; one minimal
   Foundry pass (cheapest model, hard token cap, per D6) proves candidates are
   substantively sound. Keep local runs tiny (1 maker + 1 checker, ≤3 rounds).
8. **Spike-gate the client choice** in /trekexecute: if `base_url` non-streaming shows
   tool-call text-leakage, `usage=None`, or #1772, switch to native `OllamaChatClient`
   (accept the `--pre` `agent-framework-ollama` dep + verify its `prompt_eval_count`/
   `eval_count` mapping; derive `total_token_count`).
9. **Security hardening:** Ollama bind `127.0.0.1`, pin **≥ 0.17.1**, `OLLAMA_DEBUG`
   unset; document model-pull as explicit egress; dummy `api_key`.

Risks to carry into the plan: (a) Intel-CPU latency → local is plumbing/smoke, not
fast iteration; (b) small-model tool-calling weakness → lean on validator-retry +
Qwen3; (c) native-client fallback is `--pre` (GA-discipline tension) — adopt only if
spike-gated.

## Sources

| # | Source | Type | Quality | Used in |
|---|--------|------|---------|---------|
| 1 | `.venv/.../agent_framework_openai/__init__.py:33-43` (`OpenAIChatClient`, `OpenAIChatCompletionClient`) | codebase | high | Dim 1 |
| 2 | `.venv/.../agent_framework_openai/_chat_completion_client.py:705,724-726,757-776` (UsageDetails mapping, None-safe) | codebase | high | Dim 2 |
| 3 | `.venv/.../agent_framework/_types.py:402-417,547` (`UsageDetails`, `add_usage_details`, `response.usage_details`) | codebase | high | Dim 2 |
| 4 | `.venv/.../agent_framework_openai/_chat_client.py:376,382` (`base_url`, `api_key`) | codebase | high | Dim 1 |
| 5 | `agent_framework/ollama/__init__.pyi` + `import agent_framework_ollama` fails (not installed) | codebase | high | Dim 1, 6 |
| 6 | <https://learn.microsoft.com/agent-framework/integrations/openai-endpoints> (base_url table, ChatCompletion client) | official | high | Dim 1, 2 |
| 7 | <https://learn.microsoft.com/agent-framework/agents/providers/ollama> (native client, tool caveat, qwen3:4b) | official | high | Dim 1,3 |
| 8 | <https://learn.microsoft.com/python/api/agent-framework-core/agent_framework.usagedetails?view=agent-framework-python-latest> | official | high | Dim 2 |
| 9 | <https://pypi.org/project/agent-framework-ollama/> (1.0.0b260521 beta, --pre) | official | high | Dim 1, 6 |
| 10 | <https://github.com/microsoft/agent-framework/blob/main/python/packages/ollama/pyproject.toml> (deps: ollama <0.5.4) | official | high | Dim 1, 6 |
| 11 | <https://docs.ollama.com/api/openai-compatibility> (usage; streaming needs include_usage) | official | high | Dim 2 |
| 12 | <https://docs.ollama.com/api/usage> (prompt_eval_count/eval_count) | official | high | Dim 2 |
| 13 | <https://github.com/microsoft/agent-framework/releases> (1.8.0 OTel usage fix; 1.9.0 ollama tools fix) | official | high | Dim 2 |
| 14 | <https://github.com/microsoft/agent-framework/issues/1772> (double system prompt, Ollama+middleware) | community | medium | Dim 3 |
| 15 | <https://ollama.com/blog/streaming-tool> (streaming+tools fixed 2025-05-28) | official | high | Dim 3 |
| 16 | <https://github.com/ollama/ollama/issues/12557> (streaming tool-call regression 0.12.3) | community | medium | Dim 3 |
| 17 | <https://www.docker.com/blog/local-llm-tool-calling-a-practical-evaluation/> (tool-call F1 by model) | community | high | Dim 3 |
| 18 | <https://www.morphllm.com/best-ollama-models> (Qwen3 best tool-calling) | community | medium | Dim 3 |
| 19 | <https://localaimaster.com/blog/ollama-system-requirements> (Intel i9 4-6 tok/s, no MLX) | community | medium | Dim 4 |
| 20 | <https://github.com/ollama/ollama/issues/586> (seed+temp0 not deterministic) | community | medium | Dim 5 |
| 21 | <https://www.wiz.io/blog/probllama-ollama-vulnerability-cve-2024-37032> (CVE-2024-37032) | community | high | Dim 6 |
| 22 | <https://github.com/advisories/GHSA-f6mr-38g8-39rg> (CVE-2025-63389 no-auth by design) | official | high | Dim 6 |
| 23 | <https://www.runzero.com/blog/ollama/> (CVE-2026-7482 Bleeding Llama, < 0.17.1) | community | high | Dim 6 |
| 24 | <https://www.indusface.com/blog/exposed-ollama-servers-llm-security-risks/> (bind 127.0.0.1) | community | high | Dim 6 |
| 25 | <https://learn.microsoft.com/agent-framework/agents/observability> (no native cost metric) | official | high | Best Practice |
| 26 | <https://lmstudio.ai/app-privacy> (LM Studio local persistence, no telemetry default) | official | medium | Dim 6 |