open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 4 Activity
portfolio-optimiser/.claude/projects/2026-06-24-fase2-mvp-vertical-slice/research/01-native-hitl-maf-workflows.md
Kjell Tore Guttormsen c72730941e docs(fase2): 3 /trekresearch briefs — installed-source-verified [skip-docs] 
Topic 1 (HITL): native HITL is GA (ctx.request_info/@response_handler/
run(responses=); GroupChatBuilder.with_request_info → AgentApprovalExecutor),
but durable checkpoint-resume is fragile (open #5818/#6127/#6372 into 1.9.0)
→ capture verdict out-of-band in VerdictStore, defer checkpointing off MVP path.

Topic 2 (MCP citation): REVERSES brief lean — official server-filesystem cannot
cite (raw text + bare paths) → build thin custom local-folder MCP server
returning {file,locator,snippet,score} over a framework-agnostic in-process
retriever (D7 seam). Corrected docs error: ContextProvider(source_id) +
before_run/after_run + extend_instructions(source_id,...) DO exist in 1.9.0.

Topic 3 (local chat client): use OpenAIChatCompletionClient(base_url) NON-STREAMING
(not OpenAIChatClient/Responses) — installed, 0 new deps, UsageDetails None-safe
and populated non-streaming. Native OllamaChatClient is --pre fallback (spike-gated).
validator-as-retry mitigates weak small-model tool-calling; Intel-CPU = plumbing only.

All grounded in installed 1.9.0 source (source wins over Learn docs). Gemini
bridge unavailable (MCP SDK predates Google May-2026 API change).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fif1r1En5W542HbZV88yMH
2026-06-24 12:33:42 +02:00

25 KiB
Raw Blame History

type created question confidence dimensions mcp_servers_used local_agents_used external_agents_used topic brief
trekresearch-brief 2026-06-24 Does Microsoft Agent Framework 1.9.0 (core + orchestrations 1.0.0) provide a native human-in-the-loop primitive to pause a workflow for external input and resume it, and how does it interact with checkpointing and session state? 0.85 6
microsoft-learn
tavily
installed-source-introspection (orchestrator
main context)
docs-researcher
community-researcher
security-researcher
contrarian-researcher
gemini-bridge (unavailable)
1 .claude/projects/2026-06-24-fase2-mvp-vertical-slice/brief.md

Native human-in-the-loop in MAF 1.9.0 workflows

Generated by trekresearch (Voyage 5.6.0) on 2026-06-24. Topic 1 of 3 for the Fase 2 MVP-vertical-slice brief. Scope: external swarm + installed-source introspection (API truth from the installed 1.9.0 package wins over Learn docs, per CLAUDE.md). Gemini second opinion was unavailable (MCP SDK broke on Google's May-2026 Interactions API change) — triangulation rests on docs + community + contrarian + installed source.

Research Question

Does agent-framework 1.9.0 (agent-framework-core + agent-framework-orchestrations 1.0.0) provide a native human-in-the-loop (HITL) mechanism to pause a workflow for external/human input and resume it, and how does it interact with checkpointing and session/conversation state? Decision it feeds: the Fase 2 "expert verdict captured via HITL" success criterion — design the two-layer HITL + verdict-capture

  • feed-forward seam, and the per-project workflow graph.

Executive Summary

Yes — native HITL exists and is GA in 1.9.0, in three distinct surfaces: (1) a workflow-level request/response primitive (ctx.request_info() + @response_handler, resume via run(responses=...)), (2) durable checkpoint-and-resume that persists pending requests, and (3) an orchestration-level human-approval gate that — verified in installed source — works on our Group Chat maker-checker default via GroupChatBuilder.with_request_info(agents=[...]). Confidence on the mechanism is HIGH (installed-source-verified); confidence on durable checkpoint-resume RELIABILITY is LOW/contradictory — official docs promise lossless durable resume, but the bug tracker shows the pause half is solid while the persist-and-resume half has multiple open/silent-failure bugs surviving into the 1.9.0 line (#5818, #6127, #6372, #5621) plus a pickle type-allowlist that rejects custom Pydantic types on restore (#5810). Key caveat: our learned-verdict path is an indefinite, out-of-band, never-resume-the- original-run decision whose durable artifact is the hand-rolled VerdictStore — so the recommendation is to use the GA in-run approval gate for synchronous Layer-1 review, keep the durable learned verdict OUT-OF-BAND in the VerdictStore, and defer checkpointing off the MVP critical path.

Dimensions

1. Native HITL primitive — Confidence: high

Installed-source findings (ground truth, 1.9.0):

  • WorkflowContext.request_info(request_data: object, response_type: type, *, request_id: str | None = None) — any executor calls this to suspend the workflow and request external input (agent_framework/_workflows/_workflow_context.py:393).
  • Every Executor inherits RequestInfoMixin (_workflows/_executor.py:30); a method decorated @response_handler receives the typed response on resume; handlers are matched by the request/response type annotations (_workflows/_request_info_mixin.py).
  • RequestInfoExecutor is NOT present in installed 1.9.0 (only RequestInfoMixin + response_handler + ctx.request_info). send_responses/send_responses_streaming are NOT public on Workflow — internal _send_responses_internal only.

External findings:

Contradictions:

  • The auto-generated API-reference page (agent-framework-python-latest) still lists send_responses* and prose-mentions RequestInfoExecutor. This conflicts with both the dated changes-guide and the installed source. Resolution: installed source wins (CLAUDE.md invariant) — those names are doc-lag / the C# (RequestPort) idiom. Do not write them into Python 1.9.0 code.

2. Pause/resume mechanics — Confidence: high

Installed-source findings:

  • Workflow.run(message=None, *, responses=None, checkpoint_id=None, checkpoint_storage=None) — three intents, exactly one of message / responses / checkpoint_id per call (_workflows/_workflow.py:675+). responses is Mapping[str, Any] keyed by request_id; mutually exclusive with message; can be combined with checkpoint_id ("restore then send responses in a single call" per the docstring).
  • Run-state enum (_workflows/_events.py:58-65): IDLE, IDLE_WITH_PENDING_REQUESTS (paused awaiting input — non-terminal), IN_PROGRESS_PENDING_REQUESTS. result.get_final_state() returns it.
  • RunnerContext.send_request_info_response(request_id, response) validates the response type against the original request and raises ValueError on unknown request_id or type mismatch (_workflows/_runner_context.py:457-471).
  • Events collapsed to a generic WorkflowEvent[DataT] carrying .request_id / .data / .type (no RequestInfoEvent subclass).

External findings:

  • Canonical loop (official sample guessing_game_with_human_input.py): run(..., stream=True) → collect request_info events → run(stream=True, responses=pending) → repeat until no pending requests. In-process resume needs NO checkpoint — "state is preserved across multiple calls to run." https://learn.microsoft.com/agent-framework/workflows/human-in-the-loop
  • Request IDs are caller-supplied or auto-UUID; responses strongly typed via response_type.

3. Checkpointing interaction — Confidence: high (mechanism) / contradictory (reliability)

Installed-source + docs findings (mechanism — high):

  • WorkflowCheckpoint.pending_request_info_events: dict[str, ...] — pending HITL requests ARE serialized into the checkpoint (_workflows/_checkpoint.py:81). On restore the runner rehydrates them (_runner_context.py:424-425) and re-emits them as request_info events; you then answer with a separate run(responses=...) call (you cannot inject responses during the restore call itself).
  • Storage backends (all implement the CheckpointStorage protocol, swap without code change): InMemoryCheckpointStorage (ephemeral), FileCheckpointStorage (local disk, explicit storage_path, pickle + restricted unpickler), CosmosCheckpointStorage (Azure, preview — egress). Checkpoints fire at superstep boundaries, so a HITL pause lands on one cleanly. https://learn.microsoft.com/agent-framework/workflows/checkpoints

Contradictions (reliability — LOW):

4. Session / conversation-state interaction — Confidence: high

External findings:

  • The built-in AgentExecutor (wraps an agent inside a workflow) serializes on checkpoint: internal message cache, full conversation history, agent session state, and pending requests/responses — and restores them. So a HITL-paused agent-bearing workflow does NOT lose chat history. https://learn.microsoft.com/agent-framework/workflows/advanced/agent-executor
  • Material gap for the Azure/Foundry profile: "Checkpointing with agents that use server-side sessions (e.g. FoundryAgent) has limitations. Server-side session state is not captured in checkpoints." A durable HITL pause with a Foundry-hosted agent will not have its conversation reliably restored from the checkpoint alone. (Same doc.)
  • For pure custom-executor workflows (no AgentExecutor): only shared state, in-transit messages, and pending requests are captured; executor-local fields persist only if you override on_checkpoint_save() / on_checkpoint_restore().
  • Pre-1.9.0 rename relevant to code around a pause: SharedState→State, ctx.shared_state→ctx.state, state getters/setters now synchronous (PR #3667).

Cross-link: this is the Fase 1 B7 bleed vector — cross-run conversation state lives in AgentSession.state + InMemoryHistoryProvider. The HITL/checkpoint state is workflow-level and distinct, but fresh_workflow() isolation still governs whether a restored conversation contaminates the next project run. (See docs/research/2026-06-24-maf-capability-map.md, Fase 1 B7.)

5. Group-chat / orchestration-level approval gate — Confidence: high

Installed-source findings (decisive for our debate default):

  • GroupChatBuilder.with_request_info(*, agents: Sequence[str | SupportsAgentRun] | None = None) EXISTS (agent_framework_orchestrations/_group_chat.py:882). It pauses after the named agent(s) respond and emits a request_info event (type='request_info') "that allows the caller to review the conversation and optionally [approve/edit] … the standard response_handler/request_info pattern." Same method exists on SequentialBuilder (_sequential.py:154).
  • Participants matching the filter are wrapped as AgentApprovalExecutor(WorkflowExecutor) (_orchestration_request_info.py:168), constructed allow_direct_output=True so the user-approved final response surfaces as workflow output.
  • The human reply object is AgentRequestInfoResponse (public export) with .approve() (accept as-is), .from_strings([text]), .from_messages([...]) (_orchestration_request_info.py:44-85). Supplied via run(responses={request_id: AgentRequestInfoResponse...}).
  • resolve_request_info_filter(agents) selects which agents pause for approval — e.g. pause only before the checker in maker-checker.
  • A separate, lighter gate also exists: tool-approval via @tool(approval_mode="always_require") → function_approval_request content → approve/deny (in-run, no graph).

External findings:

  • Docs/maintainer confirm there is no single universal "approve/edit/reject" object; the documented routes are (a) with_request_info + AgentRequestInfoResponse, (b) tool-approval, (c) Magentic plan-review (enable_plan_review) — Magentic is experimental and OFF our path. For arbitrary maker-checker over agent output, the intended route is with_request_info (now installed-source-confirmed for GroupChat) or a custom executor calling ctx.request_info(). https://github.com/microsoft/agent-framework/discussions/1287
  • Caveat: community shows the orchestration approval resume path is where bugs cluster (#5818, #6127, #6006) — the gate fires reliably; persisting/resuming the approval across serialization is fragile.

6. Design implication for our two-layer HITL + verdict capture — Confidence: high (recommendation)

This is decision-relevant — see Recommendation below. Short form: the brief's fallback assumption ("if MAF lacks native HITL, capture out-of-band") is partly inverted: MAF has native HITL, and the right split is to use the GA in-run pieces for synchronous review but keep the durable learned verdict out-of-band.

External Knowledge

Best Practice

  • Canonical 1.9.0 HITL = ctx.request_info() + @response_handler + run(responses=); detect pause via IDLE_WITH_PENDING_REQUESTS / get_request_info_events(). Official samples: guessing_game_with_human_input.py, sequential_request_info.py, checkpoint/checkpoint_with_human_in_the_loop.py, magentic_human_plan_review.py (all under microsoft/agent-framework python/samples/03-workflows/).

Security (relevant to D3 no-silent-egress + local-only)

  • FileCheckpointStorage writes unencrypted pickle blobs containing conversation history + pending HITL payloads + shared state. Encryption-at-rest / permissions / ACLs are the developer's responsibility (no built-in encryption). Lock the storage_path down (dedicated dir, 0700/0600, encrypted volume; macOS FileVault helps).
  • Restore is hardened: restricted unpickler ON by default since 1.0.1 (we're on 1.9.0); non-safe types throw WorkflowCheckpointException unless registered in allowed_checkpoint_types. Treat it as a safety net, not the control — "never load checkpoints from untrusted sources."
  • No CVEs against agent-framework* (GitHub Security Advisories empty; OSV empty). Two transitive Starlette advisories on deps.dev — confirm via uv run pip-audit. Semantic Kernel CVEs (CVE-2026-26030/-25592) are a different package and not in the checkpoint path; only relevant if semantic-kernel is pulled in (it is not, per our pinned tree).
  • Telemetry OFF by default — no exporter ships; ENABLE_INSTRUMENTATION/ENABLE_SENSITIVE_DATA/ENABLE_CONSOLE_EXPORTERS all default false. No-silent-egress holds out of the box if we (a) don't set those, (b) set no OTEL_EXPORTER_OTLP_*, (c) use File/InMemory (never Cosmos). One subtlety: MAF auto-propagates OTel trace context into MCP tools/call _meta when a span is active — inert with instrumentation off; keep MCP servers local (stdio) regardless.

Known Issues

  • See Dimension 3 contradictions. Plus: with_request_info() naming is opaque and there is no event-type filter for which events trigger a HITL pause (#3534, open). Local-model + HITL is unverified by anyone — zero community signal; several worst resume bugs are Azure-server-side-persistence-specific and may simply not apply locally, but then we own conversation-history persistence ourselves. Spike locally; do not rely on precedent.

Gemini Second Opinion

Unavailable. The gemini-mcp server's client SDK predates Google's May-2026 Interactions API breaking change and returned 400 BadRequestError before any research ran. No independent Gemini triangulation was obtained for this topic; treat the second opinion as absent (not negative). To restore: upgrade the gemini-mcp server's client SDK to ≥ 2.0.0.

Synthesis

The triangulation surfaces an insight no single source states: MAF 1.9.0 has three different HITL surfaces, and the strongest one for our need is the lightest one — while the heaviest one (durable checkpoint-resume) is both the shakiest in practice and a poor fit for the problem.

  1. In-run synchronous review (GroupChatBuilder.with_request_info → AgentApprovalExecutor → run(responses={id: AgentRequestInfoResponse...})) is GA, installed-source-confirmed for our Group Chat maker-checker default, and needs no checkpointing (state persists across run() calls in-process). This is the solid, happy-path piece.

  2. Durable cross-process pause (checkpoint + restore + responses) is where the docs promise and the bug tracker diverge hardest: open/silent-failure resume bugs into the 1.9.0 line, a pickle type-allowlist that fights our Pydantic IR, and "checkpoints cannot be resumed between versions." The contrarian pass is right that coupling our highest-value data path (the verdict the system learns from) to MAF's most-churned, least-durable, Azure-favoring surface is a self-inflicted risk — and a lock-in against the D7 Claude-SDK sibling, which has no executor/checkpoint model and can only share a framework-agnostic verdict seam.

  3. The actual shape of "fagekspert enters a verdict the next run learns from" is an indefinite, out-of-band, never-resume-the-original-run decision. Its durable artifact is the hand-rolled VerdictStore, not an in-flight workflow checkpoint. A workflow checkpoint is engineered for "pause seconds-to-minutes, resume the same process" — the wrong tool for "pause indefinitely, decide elsewhere."

So the brief's binary ("native HITL → use it; else out-of-band") resolves to a split: adopt the GA in-run approval gate for the synchronous Layer-1 review where it fits; keep the durable, learning-loop verdict out-of-band in the VerdictStore; and defer checkpointing off the MVP critical path (matches the brief's [OPEN] default — and Topic 1 confirms the default rather than overturning it, because durable checkpoint-resume is the fragile part).

Open Questions

  • Layer-1 review: native gate vs. simplest possible? with_request_info is GA and fits, but a custom executor calling ctx.request_info() gives full control over the request payload (the ValidatedProposal + provenance). Decide in /trekplan: native with_request_info(agents=[checker]) vs custom request-info executor. Either way: in-process, no checkpoint.
  • Is Layer-1 even in the MVP, or is it async-only? The brief marks two-layer HITL semantics (sync review vs async+notification stub) as [OPEN]. If Fase 2 ships only the async/out-of-band verdict + notification stub (B11), the native in-run gate may be deferred too — fewer moving parts. Resolve in /trekplan.
  • Local-profile HITL behaviour — unverified by anyone. Needs a self-spike: run the approval gate / request_info loop against the local OpenAI-compatible endpoint (Topic 3) and confirm it fires and resumes. Tie to Topic 3's outcome.
  • If durable pause is ever needed: the acceptance gate must be a resume-integrity test (pause → checkpoint → restore → assert pending requests match + conversation intact), given #6372/#5621-class silent failures. Out of MVP scope but record the condition.

Recommendation

For the Fase 2 MVP, capture the expert verdict OUT-OF-BAND and keep the durable artifact in the hand-rolled VerdictStore; do NOT couple the learned-verdict path to native checkpoint-resume. Concretely:

  1. Layer 2 (durable, learning loop) = out-of-band VerdictStore. The emitted ValidatedProposal + provenance is presented to the fagekspert out-of-band; the verdict is written to the VerdictStore; the next run retrieves it via the ExpeL ContextProvider seam (extend_instructions(source_id, instructions)). This is framework-agnostic, D7-portable, and the success-criterion ("second run retrieves the prior verdict") is satisfied without any MAF checkpoint.
  2. Layer 1 (optional in-run synchronous review) = GA native gate IF included. If Fase 2 ships a synchronous review, use GroupChatBuilder.with_request_info( agents=[checker]) (or a custom ctx.request_info() executor for a richer typed payload) — in-process, resume via run(responses=...), no checkpointing. Register any custom Pydantic response type's expectations now so it survives if checkpointing is ever added.
  3. Defer checkpointing off the MVP critical path (brief [OPEN] default upheld). InMemoryCheckpointStorage is fine for tests; do not put FileCheckpointStorage durable HITL on the critical path. If added later: File/InMemory only (never Cosmos — egress), locked-down storage_path, allowed_checkpoint_types for our IR, a resume-integrity acceptance test, and awareness of cross-version checkpoint invalidation.
  4. Provenance, not checkpoint, carries the audit trail — the single emitted proposal's provenance stamp (citations + model/role + validator decision + token usage) is the durable record, consistent with the no-silent-egress + provenance NFRs.

Risks to carry into the plan: (a) MAF HITL/checkpoint API churn → any MAF-native HITL code carries upgrade cost; pin the surface and watch the changes guide. (b) Local-profile HITL is unverified → spike with Topic 3's local client. (c) If a synchronous in-run gate is used, the orchestration resume path is where community bugs cluster — keep it in-process (no serialization boundary) to dodge that whole class.

Sources

# Source Type Quality Used in
1 .venv/.../agent_framework/_workflows/_workflow_context.py:393 (request_info) codebase high Dim 1, 2
2 .venv/.../agent_framework/_workflows/_request_info_mixin.py (response_handler, RequestInfoMixin) codebase high Dim 1
3 .venv/.../agent_framework/_workflows/_workflow.py:675+ (run(responses=, checkpoint_id=)) codebase high Dim 2, 3
4 .venv/.../agent_framework/_workflows/_events.py:58-65 (WorkflowRunState) codebase high Dim 2
5 .venv/.../agent_framework/_workflows/_runner_context.py:424,457-471 (rehydrate + send_request_info_response) codebase high Dim 2, 3
6 .venv/.../agent_framework/_workflows/_checkpoint.py:81 (pending_request_info_events) codebase high Dim 3
7 .venv/.../agent_framework_orchestrations/_group_chat.py:882 (GroupChatBuilder.with_request_info) codebase high Dim 5
8 .venv/.../agent_framework_orchestrations/_orchestration_request_info.py:44-85,168 (AgentRequestInfoResponse, AgentApprovalExecutor) codebase high Dim 5
9 https://learn.microsoft.com/agent-framework/workflows/human-in-the-loop official high Dim 1,2,3,5
10 https://learn.microsoft.com/agent-framework/workflows/checkpoints official high Dim 3, Security
11 https://learn.microsoft.com/agent-framework/workflows/advanced/agent-executor official high Dim 4
12 https://learn.microsoft.com/agent-framework/support/upgrade/python-2026-significant-changes official high Dim 1,2,3
13 https://learn.microsoft.com/agent-framework/support/upgrade/requests-and-responses-upgrade-guide-python official high Dim 1
14 https://learn.microsoft.com/agent-framework/migration-guide/from-autogen/ official high Dim 1
15 https://github.com/microsoft/agent-framework/issues/5818 (Magentic AgentSession resume, OPEN) community high Dim 3,5
16 https://github.com/microsoft/agent-framework/discussions/6127 (Sequential re-prompt+re-exec) community medium Dim 3,5
17 https://github.com/microsoft/agent-framework/issues/6372 (fan-in barrier silent loss, 1.1.0–1.9.0) community high Dim 3
18 https://github.com/microsoft/agent-framework/issues/5621 (Handoff restore fails, OPEN) community high Dim 3
19 https://github.com/microsoft/agent-framework/issues/5810 (checkpoint type-allowlist) community high Dim 3, Security
20 https://github.com/microsoft/agent-framework/issues/3255 (sub-workflow dup request, fixed) community high Dim 3
21 https://github.com/microsoft/agent-framework/issues/3534 (with_request_info naming/filter, OPEN) community medium Known Issues
22 https://github.com/microsoft/agent-framework/discussions/1287 (no universal approval object; custom executor) community high Dim 5
23 https://www.diagrid.io/blog/still-not-durable-how-microsoft-agent-framework-and-strands-agents-repeat-the-same-mistake community medium Synthesis
24 https://github.com/microsoft/agent-framework/discussions/2305 (checkpoint limitations) community medium Synthesis
25 https://github.com/microsoft/agent-framework/issues/4078 (stability/production timeline, unanswered) community medium Synthesis
26 https://github.com/microsoft/agent-framework/security/advisories (no advisories) official high Security
27 https://osv.dev/list?ecosystem=PyPI&q=agent-framework (no results) official high Security
28 https://learn.microsoft.com/agent-framework/agents/observability (telemetry off by default) official high Security
29 https://learn.microsoft.com/agent-framework/agents/providers/ollama (local provider exists) official medium Known Issues
30 https://pypi.org/project/agent-framework-core/ (1.9.0 = 2026-06-18) official high Exec summary