open/ms-ai-architect
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity Actions

docs(ms-ai-architect): KB-refresh MEDIUM 28/33 — agent-evaluation: nye agent-evaluatorer (Task Completion/Tool Selection m.fl.) + FoundryEvals + 4.1-mini judge

Browse source
This commit is contained in:
Kjell Tore Guttormsen 2026-06-24 11:38:25 +02:00
commit 7a42701f2c
1 changed files with 10 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

11
skills/ms-ai-engineering/references/agent-orchestration/agent-evaluation-testing-frameworks.md
View file

@ -1,6 +1,6 @@
# Agent Evaluation and Testing Frameworks
**Last updated:** 2026-04 | Verified: MCP 2026-04
**Last updated:** 2026-06-24 | Verified: MCP 2026-06
**Status:** GA (Azure AI Evaluation SDK), Preview (Agent-specific evaluators)
**Category:** Agent Orchestration & Automation
@ -31,6 +31,15 @@ Microsoft Foundry støtter både Foundry Agent Service (built-in agents), Semant
| **ContentSafetyEvaluator** | Detekterer harmful content (violence, hate, sexual, self-harm) | query, response | 0-7 severity | Ja (Azure AI Content Safety) |
| **IndirectAttackEvaluator** | Sjekker jailbreak attempts via indirect injection | query, response | Pass/Fail | Ja |
| **CodeVulnerabilityEvaluator** | Identifiserer usikker kode i agentsvar | response | Pass/Fail | Ja |
| **TaskCompletionEvaluator** (preview) | Måler om agenten fullførte oppgaven end-to-end med brukbar leveranse | query, response, (tool_definitions) | Pass/Fail | Ja |
| **CustomerSatisfactionEvaluator** (preview) | Predikerer brukertilfredshet over en samtale (6 dimensjoner) | messages | 1-5 Likert | Ja |
| **ToolSelectionEvaluator** | Måler om agenten valgte riktige (og kun nødvendige) tools | query, response, tool_definitions | Pass/Fail | Ja |
| **ToolInputAccuracyEvaluator** | Validerer at alle tool-parametere er korrekte (6 strenge kriterier) | query, response, tool_definitions | Pass/Fail | Ja |
| **ToolOutputUtilizationEvaluator** | Måler om agenten brukte tool-outputs riktig i svaret | query, response, tool_definitions | Pass/Fail | Ja |
| **ToolCallSuccessEvaluator** | Sjekker om tool-kall ble utført uten tekniske feil | response | Pass/Fail | Ja |
| **TaskNavigationEfficiencyEvaluator** | Sammenligner agentens steg mot optimal/forventet sti (krever ground truth) | actions, expected_actions | Pass/Fail | Nei |
> **Microsoft Agent Framework — `FoundryEvals`:** I tillegg til `AIAgentConverter` kan en `AIAgent` kobles direkte til Foundrys evalueringstjeneste via `FoundryEvals` (kjører som standard relevance/coherence/task_adherence og legger automatisk til tool_call_accuracy når tool-definisjoner finnes; resultater vises i Foundry-portalen). For kompleks evaluering anbefaler Microsoft nå en sterk reasoning-modell som `gpt-4.1-mini` som judge for balanse mellom ytelse og kostnad.
### Evaluator output format