From 7a42701f2c9ad178e31e0da1af8ff320d8036f36 Mon Sep 17 00:00:00 2001 From: Kjell Tore Guttormsen Date: Wed, 24 Jun 2026 11:38:25 +0200 Subject: [PATCH] =?UTF-8?q?docs(ms-ai-architect):=20KB-refresh=20MEDIUM=20?= =?UTF-8?q?28/33=20=E2=80=94=20agent-evaluation:=20nye=20agent-evaluatorer?= =?UTF-8?q?=20(Task=20Completion/Tool=20Selection=20m.fl.)=20+=20FoundryEv?= =?UTF-8?q?als=20+=204.1-mini=20judge?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../agent-evaluation-testing-frameworks.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/skills/ms-ai-engineering/references/agent-orchestration/agent-evaluation-testing-frameworks.md b/skills/ms-ai-engineering/references/agent-orchestration/agent-evaluation-testing-frameworks.md index 091b65a..86a491c 100644 --- a/skills/ms-ai-engineering/references/agent-orchestration/agent-evaluation-testing-frameworks.md +++ b/skills/ms-ai-engineering/references/agent-orchestration/agent-evaluation-testing-frameworks.md @@ -1,6 +1,6 @@ # Agent Evaluation and Testing Frameworks -**Last updated:** 2026-04 | Verified: MCP 2026-04 +**Last updated:** 2026-06-24 | Verified: MCP 2026-06 **Status:** GA (Azure AI Evaluation SDK), Preview (Agent-specific evaluators) **Category:** Agent Orchestration & Automation @@ -31,6 +31,15 @@ Microsoft Foundry støtter både Foundry Agent Service (built-in agents), Semant | **ContentSafetyEvaluator** | Detekterer harmful content (violence, hate, sexual, self-harm) | query, response | 0-7 severity | Ja (Azure AI Content Safety) | | **IndirectAttackEvaluator** | Sjekker jailbreak attempts via indirect injection | query, response | Pass/Fail | Ja | | **CodeVulnerabilityEvaluator** | Identifiserer usikker kode i agentsvar | response | Pass/Fail | Ja | +| **TaskCompletionEvaluator** (preview) | Måler om agenten fullførte oppgaven end-to-end med brukbar leveranse | query, response, (tool_definitions) | Pass/Fail | Ja | +| **CustomerSatisfactionEvaluator** (preview) | Predikerer brukertilfredshet over en samtale (6 dimensjoner) | messages | 1-5 Likert | Ja | +| **ToolSelectionEvaluator** | Måler om agenten valgte riktige (og kun nødvendige) tools | query, response, tool_definitions | Pass/Fail | Ja | +| **ToolInputAccuracyEvaluator** | Validerer at alle tool-parametere er korrekte (6 strenge kriterier) | query, response, tool_definitions | Pass/Fail | Ja | +| **ToolOutputUtilizationEvaluator** | Måler om agenten brukte tool-outputs riktig i svaret | query, response, tool_definitions | Pass/Fail | Ja | +| **ToolCallSuccessEvaluator** | Sjekker om tool-kall ble utført uten tekniske feil | response | Pass/Fail | Ja | +| **TaskNavigationEfficiencyEvaluator** | Sammenligner agentens steg mot optimal/forventet sti (krever ground truth) | actions, expected_actions | Pass/Fail | Nei | + +> **Microsoft Agent Framework — `FoundryEvals`:** I tillegg til `AIAgentConverter` kan en `AIAgent` kobles direkte til Foundrys evalueringstjeneste via `FoundryEvals` (kjører som standard relevance/coherence/task_adherence og legger automatisk til tool_call_accuracy når tool-definisjoner finnes; resultater vises i Foundry-portalen). For kompleks evaluering anbefaler Microsoft nå en sterk reasoning-modell som `gpt-4.1-mini` som judge for balanse mellom ytelse og kostnad. ### Evaluator output format