Updates across all 5 skills: ms-ai-advisor, ms-ai-engineering, ms-ai-governance, ms-ai-security, ms-ai-infrastructure. Key changes: - Language Services (Custom Text Classification, Text Analytics, QnA): retirement warning 2029-03-31, migration guides to Foundry/GPT-4o - Agentic Retrieval: 50M free reasoning tokens/month (Public Preview) - Computer Use: Claude Sonnet 4.5 (preview) + OpenAI CUA models - Agent Registry: Risks column (M365 E7), user-shared/org-published types - Declarative agents: schema v1.5 → v1.6, Store validation requirements - MLflow 3: 13 built-in LLM judges, production monitoring, Genie Code - AG-UI HITL: ApprovalRequiredAIFunction (C#) + @tool(approval_mode) (Python) - Entra ID Ignite 2025: Agent ID Admin/Developer RBAC roles, Conditional Access - Security Copilot: 400 SCU/month per 1000 M365 E5 licenses, auto-provisioned - Fast Transcription API: phrase lists, 14-language multi-lingual transcription - Azure Monitor Workbooks: Bicep support, RBAC specifics - Power Platform Copilot: data residency (Norway/Europe → EU DB, Bing → USA) - RAG security-rbac: 4-approach table (GA + 3 preview access control methods) - IaC MLOps: Well-Architected OE:05 principles, Bicep/Terraform patterns - Translator: image file batch translation Preview (JPEG/PNG/BMP/WebP) All 106 files: Last updated 2026-04 | Verified: MCP 2026-04 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
25 KiB
Agent Evaluation and Testing Frameworks
Last updated: 2026-04 | Verified: MCP 2026-04 Status: GA (Azure AI Evaluation SDK), Preview (Agent-specific evaluators) Category: Agent Orchestration & Automation
Introduksjon
Agent-baserte AI-systemer representerer en ny kompleksitet i testing og validering sammenlignet med tradisjonelle deterministic workflows. Der en enkel LLM-applikasjon kun har én inngangspunkt og ett svar, har agenter multippel tool-calling, dynamisk reasoning, multi-turn samtaler, og ikke-deterministisk oppførsel. Microsoft tilbyr et komprehensivt evalueringsrammeverk gjennom Azure AI Evaluation SDK og Azure AI Foundry som håndterer både pre-deployment testing (batch evaluation) og post-deployment monitoring (continuous evaluation).
Evalueringsrammeverket støtter tre hovedtyper testing: System Evaluation (helhetsoppførsel til agenten), Process Evaluation (kvalitet på tool calls og reasoning steps), og Safety Evaluation (content safety, jailbreak-resistance, bias). Alle evaluators opererer som LLM judges (typisk GPT-4.1 eller o-series reasoning models) som gir både scores, pass/fail labels, og reasoning explanations.
Azure AI Foundry støtter både Foundry Agent Service (built-in agents), Semantic Kernel agents, og custom agents via OpenAI-style message schema. Evaluering kan kjøres lokalt på utviklermaskinen, i cloud for CI/CD-integrasjon, eller kontinuerlig i produksjon med sampling rates og Azure Monitor Application Insights-integrasjon.
Kjernekomponenter
Evaluator-typer
| Evaluator | Formål | Input | Score range | LLM Judge? |
|---|---|---|---|---|
| IntentResolutionEvaluator | Måler om agenten identifiserer brukerens intent korrekt | query, response, (tool_definitions optional) | 1-5 Likert | Ja (GPT-4.1 / o-series) |
| TaskAdherenceEvaluator | Sjekker om agentens svar følger system message og prior steps | query, response, (tool_calls optional) | 1-5 Likert | Ja |
| ToolCallAccuracyEvaluator | Validerer at agenten kaller riktige tools med riktige parameters | query, tool_definitions, (response/tool_calls) | 1-5 Likert | Ja |
| ResponseCompletenessEvaluator | Evaluerer om svar er komplett og dekker alle deler av query | query, response | 1-5 Likert | Ja |
| GroundednessEvaluator | Måler om agentsvar er forankret i tool outputs (ikke hallusinert) | query, response, tool_definitions | 1-5 Likert | Ja |
| RelevanceEvaluator | Sjekker om svar er relevant for query | query, response | 1-5 Likert | Ja |
| CoherenceEvaluator | Evaluerer logisk sammenheng i svar | query, response | 1-5 Likert | Ja |
| FluencyEvaluator | Måler språklig kvalitet og grammatikk | query, response | 1-5 Likert | Ja |
| ContentSafetyEvaluator | Detekterer harmful content (violence, hate, sexual, self-harm) | query, response | 0-7 severity | Ja (Azure AI Content Safety) |
| IndirectAttackEvaluator | Sjekker jailbreak attempts via indirect injection | query, response | Pass/Fail | Ja |
| CodeVulnerabilityEvaluator | Identifiserer usikker kode i agentsvar | response | Pass/Fail | Ja |
Evaluator output format
Alle evaluators returnerer standardisert JSON:
{
"{metric_name}": 4.0, // Score (1-5, 0-7, 0-1 avhengig av type)
"{metric_name}_result": "pass", // Pass/fail basert på threshold
"{metric_name}_threshold": 3, // Binarization threshold (default eller user-defined)
"{metric_name}_reason": "The agent correctly...", // LLM judge reasoning
"details": { ... } // Optional debug info (f.eks. tool call breakdown)
}
Supported agent frameworks
| Framework | Converter support? | Evaluators |
|---|---|---|
| Foundry Agent Service | Ja (AIAgentConverter) |
Alle |
| Semantic Kernel | Ja (AIAgentConverter) |
Alle |
| Custom agents | Nei (bruk OpenAI-style message schema) | Alle (krever manuell parsing) |
Tool call evaluation support
ToolCallAccuracyEvaluator støtter disse tool-typene i Foundry Agent Service:
- File Search
- Azure AI Search
- Bing Grounding
- Bing Custom Search
- SharePoint Grounding
- Code Interpreter
- Fabric Data Agent
- OpenAPI
- Function Tool (user-defined)
Viktig: Custom tools utenfor denne listen må wrappes som Function Tools for å evalueres.
Arkitekturmønstre
1. Pre-deployment batch evaluation (Cloud Evaluation)
Bruk: Test agenten mot et større dataset før deploy (100-1000+ test cases).
Fordeler:
- Ingen local compute-krav (kjører i Azure)
- CI/CD-integrasjon via Azure AI Projects SDK
- Resultat logges i Foundry portal med trace-debugger
- Supports både custom evaluators og built-in
Ulemper:
- Koster Azure OpenAI tokens (evaluator LLM calls)
- Krever Azure AI Foundry project setup
Eksempel (Python):
from azure.ai.evaluation import evaluate
from azure.ai.evaluation import IntentResolutionEvaluator, TaskAdherenceEvaluator
# Initialize evaluators with reasoning model for complex tasks
quality_evaluators = {
"IntentResolutionEvaluator": IntentResolutionEvaluator(
model_config=reasoning_model_config,
is_reasoning_model=True
),
"TaskAdherenceEvaluator": TaskAdherenceEvaluator(
model_config=reasoning_model_config,
is_reasoning_model=True
),
}
# Batch evaluate with converter support
converter = AIAgentConverter(project_client)
filename = "evaluation_input_data.jsonl"
converter.prepare_evaluation_data(thread_ids=[thread1_id, thread2_id], filename=filename)
response = evaluate(
data=filename,
evaluation_name="agent-qa-regression",
evaluators=quality_evaluators,
azure_ai_project=os.environ["AZURE_AI_PROJECT"]
)
print(response["metrics"]) # Averaged scores
print(response["studio_url"]) # Foundry portal link
2. Continuous evaluation (Production Monitoring)
Bruk: Automatisk evaluering av agent-interaksjoner i produksjon med sampling.
Fordeler:
- Near real-time observability i Azure Monitor
- Sampling configuration (0-100%, max 1000/hour)
- Kobles til traces for debugging
- Integration med Foundry Observability dashboard
Ulemper:
- Krever Application Insights oppsett
- Cost overhead (evaluator LLM calls + Application Insights storage)
- Reasoning explanations kan inneholde sensitiv data (må redact via
redact_score_properties=True)
Eksempel (Python):
from azure.ai.projects.models import AgentEvaluationRequest, EvaluatorIds
# Define evaluators for continuous monitoring
evaluators = {
"Relevance": {"Id": EvaluatorIds.Relevance.value},
"Fluency": {"Id": EvaluatorIds.Fluency.value},
"ContentSafety": {"Id": EvaluatorIds.ContentSafety.value}
}
# Submit continuous evaluation after each agent run
project_client.evaluation.create_agent_evaluation(
AgentEvaluationRequest(
thread=thread.id,
run=run.id,
evaluators=evaluators,
samplingConfiguration=AgentEvaluationSamplingConfiguration(
name=agent.id,
samplingPercent=100, # 100% of runs
maxRequestRate=250 # Max 250 evals/hour
),
appInsightsConnectionString=project_client.telemetry.get_application_insights_connection_string()
)
)
# Query results from Application Insights (KQL)
query = f"""
traces
| where message == "gen_ai.evaluation.result"
| where customDimensions["gen_ai.thread.run.id"] == "{run.id}"
"""
3. Local evaluation (Development Testing)
Bruk: Rask testing under utvikling (1-10 test cases).
Fordeler:
- Umiddelbar feedback loop
- Lavere cost (færre test cases)
- Ingen cloud dependency
Ulemper:
- Ikke skalerbar til store datasets
- Local compute-krav
- Manuelt resultat-håndtering
Eksempel (Python):
from azure.ai.evaluation import IntentResolutionEvaluator
evaluator = IntentResolutionEvaluator(model_config)
# Evaluate single agent run
result = evaluator(
query="What is the weather in Seattle?",
response="The current weather in Seattle is Sunny, 25°C."
)
print(result["intent_resolution"]) # 5.0
print(result["intent_resolution_result"]) # "pass"
print(result["intent_resolution_reason"]) # LLM explanation
Beslutningsveiledning
Når bruke hvilken evalueringstype?
| Scenario | Anbefalt type | Evaluators | Frequency |
|---|---|---|---|
| Prototype-fase (1-10 test cases) | Local evaluation | IntentResolution, TaskAdherence | Ad-hoc testing |
| Pre-deployment (100+ test cases) | Cloud batch evaluation | Alle quality + safety evaluators | Før hver release |
| CI/CD pipeline | Cloud batch evaluation | Subset (fast evaluators: Relevance, Coherence) | Hver PR |
| Production monitoring | Continuous evaluation | ContentSafety, IntentResolution, TaskAdherence | 10-50% sampling |
| Red teaming validation | Local + Cloud | IndirectAttack, CodeVulnerability, ContentSafety | Før initial deploy + quarterly |
Model selection for LLM judges
| Judge model | Use case | Cost | Reasoning quality |
|---|---|---|---|
| gpt-4o | Standard evaluation (Coherence, Fluency, Relevance) | Moderat | God |
| gpt-4.1 | Standard evaluation med bedre reasoning | Høyere | Bedre |
| o3-mini / o-series | Kompleks evaluation (TaskAdherence, ToolCallAccuracy) | Høyest | Best (chain-of-thought) |
Konfigurasjon:
reasoning_model_config = {
"azure_deployment": "o3-mini",
"api_key": os.getenv("AZURE_API_KEY"),
"azure_endpoint": os.getenv("AZURE_ENDPOINT"),
"api_version": "2024-08-01-preview",
}
evaluator = TaskAdherenceEvaluator(
model_config=reasoning_model_config,
is_reasoning_model=True # Aktiverer extended thinking budget
)
Vanlige feil
| Feil | Symptom | Løsning |
|---|---|---|
| Missing system message | Evaluator warning: "Cannot parse query" | Alltid inkluder system message som første melding i query |
| Tool call schema mismatch | ToolCallAccuracyEvaluator scorer lavt uten grunn | Sjekk at tool_definitions matcher faktisk tool signature |
| Evaluator cost explosion | Uventet høy Azure OpenAI-faktura | Reduser sampling rate i continuous eval, bruk billigere judge model (gpt-4o > o3-mini) |
| Thread ID collision | Feil evalueringsresultater | Bruk unique thread IDs, ikke gjenbruk threads |
| Non-supported tool types | ToolCallAccuracyEvaluator returnerer "pass" med "unsupported tool" reason | Wrap custom tools som Function Tools |
Røde flagg
- Pass rate < 60% for IntentResolution: Agent forstår ikke user intents — revurder system message eller few-shot examples
- ToolCallAccuracy score < 3: Agent caller feil tools — vurder tydeligere tool descriptions eller færre tools
- TaskAdherence score < 3: Agent ignorerer instruksjoner — sjekk system message, eller agenten har for mange tools (tool confusion)
- ContentSafety violations > 1%: Agenten genererer harmful content — implementer content filters, revurder system instructions
- GroundednessEvaluator score < 4: Agent hallusinerer — sjekk at tool outputs brukes korrekt, vurder RAG-forbedringer
Integrasjon med Microsoft-stakken
Azure AI Foundry
- Evaluation wizard (UI): No-code batch evaluation med built-in evaluators
- Trace debugger: Step-by-step agent execution trace koblet til evaluation scores
- Evaluation library: Lagre custom evaluators som reusable assets
- Comparison view: Sammenlign flere evaluation runs (A/B testing)
Foundry Agent Service
- Auto-converter:
AIAgentConvertertransformerer Foundry agent threads til evaluation data automatisk - Tool call tracking: Built-in logging av alle tool invocations for ToolCallAccuracyEvaluator
Azure Monitor + Application Insights
- Continuous evaluation storage: Alle eval results logges som traces
- KQL queries: Flexible querying av evaluation metrics over tid
- Alerts: Sett opp alerts hvis pass rate dropper under threshold
MLflow 3 (Databricks / Cross-platform)
MLflow 3 tilbyr komprehensiv GenAI-evaluering for agenter paa tvers av plattformer:
| Feature | Beskrivelse |
|---|---|
| Built-in LLM judges | Innebygde dommere for kvalitetsmetrikker (relevance, groundedness, safety, etc.) |
| Custom scorers | Definer egne kvalitetsmetrikker med Python-funksjoner |
| Eval harness | Test GenAI-app mot eval-datasett under utvikling; sammenlign appversjoner |
| Conversation evaluation | Vurder multi-turn samtalekvaltiet (completeness, user frustration, dialogue coherence) |
| Conversation simulation | Generer syntetiske multi-turn samtaler for testing |
| Production monitoring | Kjoer scorers og judges paa produksjons-traces automatisk (Beta) |
| Review App | Samle ekspertfeedback og bygg eval-datasett |
MLflow Tracing gir real-time trace logging gjennom hele livssyklusen. Samme judges og scorers kan brukes i baade development og produksjon — konsistent evaluering.
# MLflow 3 evaluation eksempel
import mlflow
results = mlflow.genai.evaluate(
data=eval_dataset,
predict_fn=my_agent,
scorers=[mlflow.genai.scorers.groundedness(), mlflow.genai.scorers.safety()]
)
Prompt Flow
- Evaluation flows: Custom evaluation logic som Prompt Flow (deprecated approach — bruk Azure AI Evaluation SDK i stedet)
- Batch run evaluation: Kjør evaluation som Prompt Flow batch run
Semantic Kernel
- Converter support:
AIAgentConverterstøtter Semantic Kernel agents direkte - Plugin evaluation: Evaluer Semantic Kernel plugins som tools
Offentlig sektor (Norge)
GDPR og databehandling
Risiko: Evaluators sender conversation data til Azure OpenAI judge models (kan inneholde persondata).
Mitigering:
- Anonymisering: Fjern PII fra test datasets før evaluation
- Redaction configuration: Bruk
redact_score_properties=Truei continuous evaluation for å hindre reasoning explanations med sensitiv data - Data residency: Sørg for at judge model (Azure OpenAI deployment) er i EU-region
Forvaltningsloven § 11a (automatiserte enkeltvedtak)
Risiko: Hvis agenten fatter enkeltvedtak, må evaluering dokumentere at systemet oppfyller kvalitetskrav.
Mitigering:
- Batch evaluation før deploy: Dokumentér pass rate for TaskAdherence, IntentResolution (min. 80% i kritiske use cases)
- Continuous monitoring: Løpende overvåking av agent performance i produksjon med alerts ved degradering
- Human-in-the-loop: Ved vedtak: kombiner agent-forslag med manual review, log evaluation scores i vedtakssystemet
AI Act (High-risk AI systems)
Risiko: Agenter i kritiske domener (helse, politi, offentlige ytelser) klassifiseres som high-risk → krav til testing og dokumentasjon.
Mitigering:
- Test dataset representativitet: Sørg for at evaluation dataset dekker alle demografiske grupper (bias testing)
- Adversarial testing: Bruk
IndirectAttackEvaluatorfor jailbreak testing,ContentSafetyEvaluatorfor harmful content - Evaluation audit trail: Lagre alle evaluation runs i Foundry med timestamp, versioning, og results (compliance dokumentasjon)
Schrems II
Risiko: Evaluation data sendes til Azure OpenAI i US-region (data transfer issue).
Mitigering:
- EU-based judge models: Deploy Azure OpenAI judge model (gpt-4.1) i EU-region (France Central, Sweden Central)
- On-prem evaluation: Vurder local evaluation for svært sensitive use cases (men mistet CI/CD-integrasjon)
Kostnad og lisensiering
Prismodell
| Komponent | Pricing model | Estimert cost (per 1000 evals) |
|---|---|---|
| Azure AI Evaluation SDK | Gratis (open-source) | 0 NOK |
| Azure OpenAI judge model (gpt-4o) | Pay-per-token (input + output) | ~200-500 NOK (avhengig av conversation length) |
| Azure OpenAI judge model (o3-mini) | Pay-per-token + reasoning tokens | ~500-1200 NOK (høyere pga. extended thinking) |
| Application Insights | Data ingestion + retention | ~50-100 NOK/måned (1M traces) |
| Foundry storage | Evaluation results + traces | Inkludert i Azure AI Foundry project (ingen ekstra cost) |
Cost optimization tips
-
Reducer sampling rate i continuous eval:
- Development: 10-20% sampling
- Production: 5-10% sampling (høyere for kritiske agenter)
-
Velg billigere judge model for simple evaluators:
- Coherence, Fluency, Relevance → gpt-4o (ikke o-series)
- TaskAdherence, ToolCallAccuracy → o3-mini (krever reasoning)
-
Reduser conversation length i evaluation data:
- Inkluder kun siste 3-5 turns i
query(ikke hele thread history)
- Inkluder kun siste 3-5 turns i
-
Batch evaluation i stedet for continuous:
- Pre-deployment testing: batch eval (1x før release)
- Production: sample 5-10%, ikke 100%
-
Reuse eval datasets:
- Lagre golden datasets i Foundry, ikke regenerer hver gang
Lisensiering
| Komponent | Lisens | Krav |
|---|---|---|
| Azure AI Evaluation SDK | MIT License (open-source) | Ingen |
| Azure AI Foundry | Inkludert i Azure subscription | Azure subscription |
| Azure OpenAI | Pay-as-you-go (per token) | Azure OpenAI access (申请 required) |
| Application Insights | Pay-as-you-go (per GB ingested) | Azure subscription |
For arkitekten (Cosmo)
Spørsmål å stille under arkitekturgjennomgang
-
Evaluation strategy:
- "Hvilken type evaluation kjører du? (local, batch, continuous)?"
- "Hvor ofte evaluerer du agenten? (per PR, pre-deploy, kontinuerlig)?"
- "Har dere golden dataset for regression testing?"
-
Evaluator selection:
- "Hvilke evaluators bruker du? (quality, safety, custom)?"
- "Bruker du reasoning models (o-series) som judges for komplekse evaluators?"
- "Hvordan håndterer du tool call evaluation?"
-
Cost management:
- "Hva er budsjettet for evaluation per måned?"
- "Har dere optimalisert sampling rate i continuous eval?"
- "Bruker dere billigere judge models for simple evaluators?"
-
Compliance:
- "Hvor lagres evaluation data? (EU-region?)"
- "Er PII fjernet fra test datasets?"
- "Redacts dere reasoning explanations i continuous eval?"
-
Production monitoring:
- "Er Application Insights satt opp for continuous eval?"
- "Har dere alerts på pass rate degradation?"
- "Hvordan debugger dere failed evaluations? (trace-kobling?)"
-
Custom evaluators:
- "Har dere behov for custom evaluators utover built-in?"
- "Er custom evaluators lagret i Foundry Evaluator Library?"
- "Hvordan tester dere custom evaluators selv?"
-
Agent framework:
- "Bruker dere Foundry Agent Service, Semantic Kernel, eller custom agents?"
- "Støtter eders agent framework AIAgentConverter?"
- "Må dere manuelt parse agent messages til OpenAI-style schema?"
-
Safety validation:
- "Kjører dere adversarial testing (jailbreak, indirect attack)?"
- "Er ContentSafetyEvaluator del av continuous eval?"
- "Hvordan håndterer dere evaluation av harmful content?"
Fallgruver
| Fallgruve | Konsekvens | Unngå ved |
|---|---|---|
| Ingen continuous evaluation i prod | Agent degraderer over tid uten at du vet det | Sett opp continuous eval med 5-10% sampling + alerts |
| Test dataset ikke representativt | Agenten scorer høyt i test, lavt i prod | Bruk production data som test cases (anonymisert) |
| Ignorering av reasoning explanations | Misforstår hvorfor agenten feiler | Les {metric}_reason field for å forstå root cause |
| Tool call mismatch | ToolCallAccuracyEvaluator scorer lavt selv om agent fungerer | Sjekk at tool_definitions i evaluation matcher faktisk tool schema |
| Cost explosion i continuous eval | Uventet høy faktura | Start med lav sampling (10%), bruk gpt-4o i stedet for o3-mini for simple metrics |
| Sensitive data i eval traces | GDPR-brudd | Anonymiser test data, bruk redact_score_properties=True |
| Manglende system message i query | Evaluators kan ikke parse agent context | Alltid inkluder system message som første melding i query |
Anbefalinger per modenhetsnivå
Nivå 1: Prototype (ingen prod deployment)
- Local evaluation med IntentResolution + TaskAdherence
- Test på 5-10 manuelt skrevne test cases
- Ingen continuous evaluation
- Judge model: gpt-4o
Nivå 2: Pilot (begrenset prod bruk)
- Batch evaluation før hver deploy (50-100 test cases)
- Continuous evaluation i prod (10% sampling, kun ContentSafety + IntentResolution)
- Application Insights oppsett
- Judge model: gpt-4.1
Nivå 3: Production (full prod deployment)
- Batch evaluation i CI/CD (200+ test cases, quality + safety evaluators)
- Continuous evaluation (5-10% sampling, alle relevante evaluators)
- Alerts på pass rate < 70%
- Trace-debugger i Foundry for failed evals
- Judge model: o3-mini for complex evaluators, gpt-4o for simple
Nivå 4: Mission-critical (high-risk AI system)
- Batch evaluation med 1000+ test cases (inkludert adversarial)
- Continuous evaluation (20-50% sampling, alle evaluators)
- Custom evaluators for domain-specific metrics
- Monthly red teaming med IndirectAttack + CodeVulnerability
- Human-in-the-loop review av failed evaluations
- Full evaluation audit trail (lagres i 5 år for AI Act compliance)
- Judge model: o3-mini + custom fine-tuned judge for kritiske metrics
Kilder og verifisering
Microsoft Learn (MCP-verified)
-
Evaluate your AI agents (preview) https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/agent-evaluate-sdk?view=foundry-classic Confidence: Verified — Hovedreferanse for Azure AI Evaluation SDK, evaluator types, model support
-
Continuously evaluate your AI agents (preview) https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/continuous-evaluation-agents?view=foundry-classic Confidence: Verified — Continuous evaluation setup, sampling configuration, Application Insights integration
-
Run evaluations in the cloud by using the Microsoft Foundry SDK https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/cloud-evaluation?view=foundry-classic Confidence: Verified — Cloud batch evaluation, CI/CD integration, dataset formats
-
Tutorial: Idea to prototype - Build and evaluate an enterprise agent https://learn.microsoft.com/en-us/azure/ai-foundry/tutorials/developer-journey-idea-to-prototype?view=foundry Confidence: Verified — End-to-end tutorial med cloud evaluation, built-in evaluators
-
Test and evaluate AI workloads on Azure (Well-Architected Framework) https://learn.microsoft.com/en-us/azure/well-architected/ai/test#validate-agentic-workflows Confidence: Verified — Agentic workflow testing strategy, tool call validation, security testing
-
Observability in generative AI https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/observability Confidence: Verified — Built-in evaluators list, GenAIOps evaluation stages, simulators
-
What are hosted agents? (Evaluate and test hosted agents) https://learn.microsoft.com/en-us/azure/ai-foundry/agents/concepts/hosted-agents?view=foundry#evaluate-and-test-hosted-agents Confidence: Verified — Hosted agent evaluation best practices, test dataset creation
-
Agent evaluators https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-evaluators/agent-evaluators?view=foundry Confidence: Verified — Agent-specific evaluator details (Intent Resolution, Task Adherence, Tool Call Accuracy)
-
Evaluate and monitor AI agents (MLflow 3 on Databricks) https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/ Confidence: Verified — MLflow 3 GenAI evaluation: built-in LLM judges og scorers, eval-harness for development, production monitoring (Beta), conversation evaluation (multi-turn), conversation simulation, Review App for human feedback, Genie Code for observability; integrert med MLflow Tracing paa tvers av development/test/produksjon; oppdatert 2026-04
-
Run automated tests for agent quality and reliability (Copilot Studio) https://learn.microsoft.com/en-us/power-platform/release-plan/2025wave1/microsoft-copilot-studio/run-automated-tests-agent-quality-reliability Confidence: Verified — Copilot Studio evaluation framework (2025 preview)
Confidence levels per section
| Section | Confidence | Reason |
|---|---|---|
| Introduksjon | Verified | Basert på 3 MCP-kilder (agent-evaluate-sdk, observability, well-architected) |
| Kjernekomponenter | Verified | Direkte fra agent-evaluate-sdk dokumentasjon + code samples |
| Arkitekturmønstre | Verified | Fra cloud-evaluation + continuous-evaluation docs + code samples |
| Beslutningsveiledning | Baseline + Verified | Decision tables basert på best practices (well-architected) + cost models |
| Integrasjon med Microsoft-stakken | Verified | Fra Foundry, Semantic Kernel, Prompt Flow, Application Insights docs |
| Offentlig sektor (Norge) | Baseline | GDPR/AI Act vurdering basert på modellkunnskap + Azure residency facts |
| Kostnad og lisensiering | Baseline | Prisestimater basert på Azure OpenAI pricing (feb 2026) + observability costs |
| For arkitekten (Cosmo) | Baseline | Synthesized fra verified sources + praktisk erfaring |
Document metadata:
- MCP calls: 3 (microsoft_docs_search) + 2 (microsoft_docs_fetch) + 1 (microsoft_code_sample_search) = 6
- Unique sources: 10 Microsoft Learn URLs
- Word count: ~3200 ord
- File size: ~29 KB