diff --git a/docs/research/2026-06-24-maf-capability-map.md b/docs/research/2026-06-24-maf-capability-map.md new file mode 100644 index 0000000..5fc76fc --- /dev/null +++ b/docs/research/2026-06-24-maf-capability-map.md @@ -0,0 +1,159 @@ +# MAF capability map — feature-utilization for portfolio-optimiser + +> **Purpose.** A living map of what Microsoft Agent Framework (MAF) `agent-framework-core` +> **1.9.0** actually offers, what this repo currently hand-rolls, and a per-need decision +> on whether to adopt the MAF feature or keep our own. It exists so Fase 2 wires MAF into +> core *using its features well* instead of reinventing them. Front-end input to the Fase 2 +> `/trekbrief`. +> +> **Provenance & trust.** Grounded by two independent passes on 2026-06-24: +> (1) source-introspection of the *installed* `agent_framework` 1.9.0 (+ `agent_framework_orchestrations` 1.0.0) +> with `file:line` citations, and (2) official Microsoft Learn docs via the `microsoft-learn` MCP. +> Each verdict is tagged **[verified-source]** (read in installed bytes), **[verified-docs]** +> (official Learn), or **[hypothesis]** (needs a confirming spike). Version caveat: Learn pages +> are not version-pinned (the .NET API ref shows `v1.0.0-rc2`); where source and docs disagree, +> **the installed 1.9.0 source wins** for our code. Triggered the search-first + verification +> duties: every symbol below was read or cited, not recalled. + +## 0. Headline: Microsoft documents the exact footgun Spike B(b) found + +Today's `/trekreview` BLOCKER was that Spike B(b) "confirmed" fan-out state isolation with a +tautological call-counter. The rebuilt experiment showed a reused `ConcurrentBuilder` `Workflow` +accumulates the conversation thread across `.run()` calls (project N contaminates N+1). **MAF's +own Workflows "State Isolation" page states this verbatim** [verified-docs]: + +> "Agent threads are persisted across workflow runs… content generated by the agent will be +> available in subsequent runs of the same workflow instance… it can also lead to unintended +> state sharing if the same workflow instance is reused for different tasks. To ensure each task +> has isolated agent state, wrap agent and workflow creation inside a helper method so each call +> produces new agent instances with their own threads. **It is not recommended to reuse a single +> workflow instance for multiple tasks or requests.**" +> — learn.microsoft.com/agent-framework/workflows/state + +So our `fresh_workflow()` factory **is the documented Microsoft pattern**, and the de-risk +assumption is confirmed both empirically (Spike B) and against official guidance. The lesson is +the thesis of this document: **knowing MAF's semantics is load-bearing — a missing piece of that +knowledge produced a false-confirm that a counter-test would have shipped.** + +Mechanism, traced in installed source [verified-source]: cross-run memory flows exclusively +through `AgentSession.state` (`_sessions.py:772`). With a session and no explicit context +provider, `Agent.run` auto-injects an `InMemoryHistoryProvider` (`_agents.py:1200-1209`); +`save_messages` appends into `state["messages"]` (`_sessions.py:889-890`); the next run reloads +the full history via `SessionContext.get_messages(include_input=True)` (`_sessions.py:315-348`). +A reused workflow threads one `AgentSession` into every run → accumulation. A fresh instance per +run gets a clean session. + +## 1. Feature-utilization decision table (the core deliverable) + +| Need | We hand-roll now | MAF 1.9.0 feature | Status | Verdict | Fase 2 action | +|---|---|---|---|---|---| +| **Token *measurement*** | `len(text.split())` proxy (`spikes/_harness.py`) | `UsageDetails` on `ChatResponse`/`AgentResponse` (`_types.py:393-414`), real provider counts | GA [verified-source] | **ADOPT** | Read `response.usage_details["total_token_count"]`; delete word-count. Handle `None` (not all providers report). | +| **Token *cap* / budget stop** | `TokenMeter.charge` raises `BudgetExceeded` | Stateful `ChatMiddleware` reading `context.result.usage_details`, short-circuits (`_middleware.py:592`); officially the "Guardrails & termination" use case | GA [verified-source+docs] | **ADOPT** | Implement budget as a `ChatMiddleware`; share one meter across agents. No native *orchestration*-level token cap exists. | +| **Round / iteration cap** | `Budget.max_rounds` + `tick_round()` | `GroupChatBuilder.with_max_rounds(n)` (`_group_chat.py:832`); Magentic `max_round_count` ctor kwarg | GA (orchestrations 1.0.0) [verified-source] | **ADOPT native; drop counter** | Use builder round caps for GroupChat/Magentic. Concurrent is single-shot (no rounds). Confirm "1 turn = 1 round" matches intent. | +| **Fan-out state isolation** | `fresh_workflow()` factory (Spike B) | Documented "factory / helper per run" pattern | core [verified-docs] | **KEEP — it is the official pattern** | Promote `fresh_workflow()` into a core fan-out factory; never reuse a built workflow across projects. | +| **Stop / resume around budget** | — | Workflow checkpointing: `WorkflowBuilder(checkpoint_storage=…)` + `run(checkpoint_id=…)` (`_workflow.py:707-731`) | GA [verified-source] | **ADOPT (later)** | Optional for resumable budget-stops; superstep-granular (not mid-LLM-call); resume needs identical graph. Defer past MVP unless needed. | +| **Cost control on long debates** | — | `_compaction.py` (`TokenBudgetComposedStrategy`, `ContextWindowCompactionStrategy`) | GA [verified-source] | **COMPLEMENT** | Reduces per-call tokens; does NOT cap spend or stop. Pair with the meter only if debates grow long (D5: not MVP). | +| **VerdictStore retrieval (ExpeL)** | Structural Jaccard over typed cost-codes + measure-type + magnitude bucket (`spikes/d_verdictstore.py`) | `MemoryStore`/`MemoryContextProvider` (keyword/substring); `mem0` (embeddings) | experimental / Preview, **wrong shape** [verified-source+docs] | **KEEP ours** | MAF retrieval is bag-of-words or embedding; cannot do structural-typed matching. Our store stays. | +| **ExpeL injection seam** | `ExpeLContextProvider(ContextProvider)` | `ContextProvider` base (`_sessions.py:351`) — `source_id` **required** | GA [verified-source] | **KEEP seam, FIX call** | Real contract. Correct the call: `extend_instructions(source_id, instructions)` — two args, `source_id` first (`_sessions.py:253-262`), not the single-list form in the spike. | +| **Deterministic blocking validator** | Hybrid IR+solver+MC returning `ValidatedProposal`\|`Rejection` (`spikes/c_validator.py`) | `_evaluation.py` (`Evaluator`/`LocalEvaluator`) | experimental, **wrong shape** [verified-source] | **KEEP ours** | MAF eval is offline batch returning a *quality score*; our validator is an inline gate returning a *domain object the system branches on*. Not interchangeable. | +| **Maker-checker regression scoring** | — | `LocalEvaluator` + `@evaluator` (`_evaluation.py`) | experimental [verified-source] | **OPTIONAL (CI only)** | "Did the checker catch the planted flaw?" fits an offline CI harness. Not for the live loop. Nice-to-have, not Fase 2 core. | +| **Model map (role→deployment)** | Planned config | `declarative/` (`AgentFactory.create_agent_from_yaml`) | Preview, **not installed** [verified-source+docs] | **ROLL OWN** | No first-class role→model registry in MAF; declarative is per-agent YAML + prerelease. Build a tiny role→deployment dict/YAML → chat-client ctor. | +| **Wrap validator / data fns as tools** | Planned | `@tool` / `FunctionTool` with `schema=` (`_tools.py`) | GA [verified-source] | **ADOPT** | GA tool surface with explicit JSON-Schema — fits "JSON-Schema-validated, fail-fast". | +| **Data access via MCP** | Planned | `MCPStdioTool` (local) / `MCPStreamableHTTPTool` (remote+SSE) / `MCPWebsocketTool` (`_mcp.py`) | GA core [verified-source+docs] | **ADOPT** | `MCPStdioTool` for local-profile data servers; `MCPStreamableHTTPTool` for remote. Pass creds per-run via header provider, never persisted. | +| **Method as Agent Skill** | Planned (CLAUDE.md) | `SkillsProvider.from_paths(...)` consumes standard `SKILL.md` folders (`_skills.py`) | experimental [verified-source] | **ADOPT (pin API)** | MAF natively consumes agentskills.io `SKILL.md` + `references/`+`scripts/`. Surface is experimental → pin version, watch for breaks. | + +## 2. Verified MAF semantics worth codifying (the digest) + +These are the behaviors whose absence caused today's BLOCKER — capture them so the next session +doesn't relearn them the hard way: + +- **Agents are stateless per run; state lives in the session.** `agent.run(...)` with no session + carries no history; pass `session=agent.create_session()` to accumulate. Sessions are bound to + the agent that created them (only same-provider agents can share). [verified-docs] +- **Reused workflow ⇒ shared thread ⇒ contamination.** Build fresh per task via a factory. The + *only* cross-run channel is `AgentSession.state`. [verified-source+docs] +- **`ContextProvider` requires `source_id`** (positional) for attribution; providers filter each + other's contributions by it (`_sessions.py:362-368`). A history provider instance is shared + across sessions — never store per-task state on the provider; store it in the session. + [verified-source+docs] +- **Orchestration round semantics:** GroupChat round = one orchestrator selection/dispatch cycle; + `with_max_rounds` enforced in `_handle_response` (not the initial handler). If you subclass + `BaseGroupChatOrchestrator`, the builder's `max_rounds`/`termination_condition` are **ignored** — + you must set them yourself. [verified-source+docs] +- **Magentic caps are constructor kwargs**, not `with_*` methods: `max_round_count`, + `max_stall_count` (default 3), `max_reset_count`. There is NO `with_max_round_count`. [verified-source] +- **Token usage is on every response** as `UsageDetails` and as the OTel metric + `gen_ai.client.token.usage`; **there is no cost/dollar metric** — derive cost = tokens × your + per-model pricing downstream. [verified-source+docs] +- **Middleware short-circuit = the official termination/guardrail mechanism** (return without + calling `next`); function-middleware blocking needs a `FunctionInvokingChatClient`. [verified-docs] + +## 3. Corrections to existing repo claims (verification duty) + +- **CLAUDE.md "Magentic is experimental (NOT the default)" — stands, with a nuance.** The installed + `agent_framework_orchestrations` 1.0.0 carries **no `@experimental` code gate** [verified-source], + but Microsoft's *docs* explicitly mark agent-orchestration features experimental, Magentic most so + ("untested outside the original Magentic-One design") [verified-docs]. So the invariant is correct + at the documentation/maturity level; it is just not enforced by a decorator. Keep the invariant; + understand it is a doc-level stance, not a code signal. +- **STATE.md "agent-framework-core 1.9.0" — confirmed** [verified-source] (`importlib.metadata`). + CLAUDE.md was corrected 1.8.0→1.9.0 in Fase 1. +- **Spike D `extend_instructions([...])` (single list) is the wrong signature.** Installed GA is + `extend_instructions(source_id, instructions)` — two args, `source_id` first (`_sessions.py:253-262`) + [verified-source]. Spike D's tests pass because they exercise `retrieve()` ranking, not the + injection path; fix the call when promoting the ExpeL seam to core in Fase 2. +- **API-naming to confirm in 1.9.0:** docs use `AgentSession`/`ContextProvider`; older docs say + `AgentThread` and note `BaseContextProvider`/`BaseHistoryProvider` as deprecated aliases. The + installed source uses `SessionContext`/`ContextProvider`/`AgentSession`/`InMemoryHistoryProvider` + [verified-source] — trust the source for our code. + +## 4. The Skills question (explicit answer to "should we skillify MAF knowledge?") + +**No to a MAF-documentation-mirror Skill; yes to two narrow, real uses.** + +- **Why not a docs-mirror suite:** MAF docs are live via the `microsoft-learn` MCP and the + installed source. Mirroring them into local Skills duplicates a moving target (1.9→1.10→…), + rots, and violates DRY + the verification duty (a stale local copy can drift from truth). It is + the docs-mirror anti-pattern for Skills and gold-plating against D5/D6. +- **Yes #1 — the *method* as a Skill (already in CLAUDE.md).** Our portfolio-optimization method + (debate → validate → learn) is correctly an Agent Skill, and MAF can **natively consume** a + standard `SKILL.md` folder via `SkillsProvider.from_paths(...)` [verified-source]. This is a + capability, not a reference dump. Caveat: the Skills surface is experimental — pin and watch. +- **Yes #2 — a *digest*, which is THIS document.** §2 (verified semantics) + §3 (corrections) are + exactly the curated "MAF patterns + footguns" an agent needs while authoring. Their durable home + is this living capability-map (and, when the method-Skill lands, its `references/`), **not** a + separate skill suite. A standalone "MAF-expert Skill" only pays off if many future sessions need + *invokable* MAF fluency on demand — defer that decision until Fase 2 shows the need; the raw + material (this doc) will already exist. + +**Net:** the capability-map is the no-regret first artifact; it feeds both the Fase 2 design and +any later skillification, and it answers "skillify?" with "the method yes, the reference no." + +## 5. Fase 2 implications (what changes in the brief) + +1. **Replace** hand-rolled token measurement with `UsageDetails`; **re-implement** the budget cap + as a shared `ChatMiddleware`; **use native** builder round caps. The hand-rolled `Budget`/ + `TokenMeter` shrink to a thin shared-meter object the middleware drives. +2. **Promote** `fresh_workflow()` to a core fan-out factory — it is the documented isolation pattern. +3. **Keep** the hand-rolled VerdictStore (structural retrieval) and validator (inline gate); MAF's + memory and eval are the wrong shape. Promote the `ExpeLContextProvider` seam with the two-arg fix. +4. **Adopt** GA `@tool` + `MCPStdioTool`/`MCPStreamableHTTPTool` for tools/data access. +5. **Roll** a tiny role→deployment config map (declarative is preview + not installed). +6. **Wire** observability (`gen_ai.client.token.usage`) for cost tracking; derive cost from pricing. +7. **Production-risk flags (D2/D6):** mem0/redis memory, Cosmos/Redis history, and declarative are + all **Preview** — do not put them on the MVP critical path; the GA core (sessions, middleware, + observability, tools, MCP, checkpointing) is enough for the vertical slice. + +## 6. D7 sibling note + +The Claude Agent SDK sibling (eget repo, deler kun spec+golden-suite) faces the same +feature-utilization discipline against a different substrate. This map's *structure* (need → +feature → adopt/keep verdict) is the reusable analog; the *answers* differ. The shared golden-suite +must not bake in MAF-specific assumptions (e.g., MAF's session-accumulation semantics). + +--- + +*Sources: installed `agent_framework` 1.9.0 source (file:line cited inline); Microsoft Learn — +workflows/state, orchestrations/{group-chat,magentic}, agents/{middleware,observability,evaluation, +conversations/storage}, integrations/chat-history-memory-provider, agents/declarative, +agents/tools/local-mcp-tools. Two independent research passes, 2026-06-24.*