ms-ai-architect/skills/ms-ai-engineering/references/agent-orchestration/agent-evaluation-testing-frameworks.md

Agent Evaluation and Testing Frameworks

Last updated: 2026-06-24 | Verified: MCP 2026-06 Status: GA (Azure AI Evaluation SDK), Preview (Agent-specific evaluators) Category: Agent Orchestration & Automation Type: reference Source: https://learn.microsoft.com/azure/foundry/concepts/observability

---

Innhold

• Introduksjon
• Kjernekomponenter
• Arkitekturmønstre
• Beslutningsveiledning
• Integrasjon med Microsoft-stakken
• Offentlig sektor (Norge)
• Kostnad og lisensiering
• For arkitekten (Cosmo)
• Kilder og verifisering

Introduksjon

Agent-baserte AI-systemer representerer en ny kompleksitet i testing og validering sammenlignet med tradisjonelle deterministic workflows. Der en enkel LLM-applikasjon kun har én inngangspunkt og ett svar, har agenter multippel tool-calling, dynamisk reasoning, multi-turn samtaler, og ikke-deterministisk oppførsel. Microsoft tilbyr et komprehensivt evalueringsrammeverk gjennom Azure AI Evaluation SDK og Microsoft Foundry som håndterer både pre-deployment testing (batch evaluation) og post-deployment monitoring (continuous evaluation).

Evalueringsrammeverket støtter tre hovedtyper testing: System Evaluation (helhetsoppførsel til agenten), Process Evaluation (kvalitet på tool calls og reasoning steps), og Safety Evaluation (content safety, jailbreak-resistance, bias). Alle evaluators opererer som LLM judges (typisk GPT-4.1 eller o-series reasoning models) som gir både scores, pass/fail labels, og reasoning explanations.

Microsoft Foundry støtter både Foundry Agent Service (built-in agents), Semantic Kernel agents, og custom agents via OpenAI-style message schema. Evaluering kan kjøres lokalt på utviklermaskinen, i cloud for CI/CD-integrasjon, eller kontinuerlig i produksjon med sampling rates og Azure Monitor Application Insights-integrasjon.

Kjernekomponenter

Evaluator-typer

Evaluator	Formål	Input	Score range	LLM Judge?
IntentResolutionEvaluator	Måler om agenten identifiserer brukerens intent korrekt	query, response, (tool_definitions optional)	1-5 Likert	Ja (GPT-4.1 / o-series)
TaskAdherenceEvaluator	Sjekker om agentens svar følger system message og prior steps	query, response, (tool_calls optional)	1-5 Likert	Ja
ToolCallAccuracyEvaluator	Validerer at agenten kaller riktige tools med riktige parameters	query, tool_definitions, (response/tool_calls)	1-5 Likert	Ja
ResponseCompletenessEvaluator	Evaluerer om svar er komplett og dekker alle deler av query	query, response	1-5 Likert	Ja
GroundednessEvaluator	Måler om agentsvar er forankret i tool outputs (ikke hallusinert)	query, response, tool_definitions	1-5 Likert	Ja
RelevanceEvaluator	Sjekker om svar er relevant for query	query, response	1-5 Likert	Ja
CoherenceEvaluator	Evaluerer logisk sammenheng i svar	query, response	1-5 Likert	Ja
FluencyEvaluator	Måler språklig kvalitet og grammatikk	query, response	1-5 Likert	Ja
ContentSafetyEvaluator	Detekterer harmful content (violence, hate, sexual, self-harm)	query, response	0-7 severity	Ja (Azure AI Content Safety)
IndirectAttackEvaluator	Sjekker jailbreak attempts via indirect injection	query, response	Pass/Fail	Ja
CodeVulnerabilityEvaluator	Identifiserer usikker kode i agentsvar	response	Pass/Fail	Ja
TaskCompletionEvaluator (preview)	Måler om agenten fullførte oppgaven end-to-end med brukbar leveranse	query, response, (tool_definitions)	Pass/Fail	Ja
CustomerSatisfactionEvaluator (preview)	Predikerer brukertilfredshet over en samtale (6 dimensjoner)	messages	1-5 Likert	Ja
ToolSelectionEvaluator	Måler om agenten valgte riktige (og kun nødvendige) tools	query, response, tool_definitions	Pass/Fail	Ja
ToolInputAccuracyEvaluator	Validerer at alle tool-parametere er korrekte (6 strenge kriterier)	query, response, tool_definitions	Pass/Fail	Ja
ToolOutputUtilizationEvaluator	Måler om agenten brukte tool-outputs riktig i svaret	query, response, tool_definitions	Pass/Fail	Ja
ToolCallSuccessEvaluator	Sjekker om tool-kall ble utført uten tekniske feil	response	Pass/Fail	Ja
TaskNavigationEfficiencyEvaluator	Sammenligner agentens steg mot optimal/forventet sti (krever ground truth)	actions, expected_actions	Pass/Fail	Nei

Microsoft Agent Framework — FoundryEvals: I tillegg til AIAgentConverter kan en AIAgent kobles direkte til Foundrys evalueringstjeneste via FoundryEvals (kjører som standard relevance/coherence/task_adherence og legger automatisk til tool_call_accuracy når tool-definisjoner finnes; resultater vises i Foundry-portalen). For kompleks evaluering anbefaler Microsoft nå en sterk reasoning-modell som gpt-4.1-mini som judge for balanse mellom ytelse og kostnad.

Evaluator output format

Alle evaluators returnerer standardisert JSON:

{
  "{metric_name}": 4.0,                      // Score (1-5, 0-7, 0-1 avhengig av type)
  "{metric_name}_result": "pass",            // Pass/fail basert på threshold
  "{metric_name}_threshold": 3,              // Binarization threshold (default eller user-defined)
  "{metric_name}_reason": "The agent correctly...", // LLM judge reasoning
  "details": { ... }                         // Optional debug info (f.eks. tool call breakdown)
}

Supported agent frameworks

Framework	Converter support?	Evaluators
Foundry Agent Service	Ja (AIAgentConverter)	Alle
Semantic Kernel	Ja (AIAgentConverter)	Alle
Custom agents	Nei (bruk OpenAI-style message schema)	Alle (krever manuell parsing)

Tool call evaluation support

ToolCallAccuracyEvaluator støtter disse tool-typene i Foundry Agent Service:

1. File Search
2. Azure AI Search
3. Bing Grounding
4. Bing Custom Search
5. SharePoint Grounding
6. Code Interpreter
7. Fabric Data Agent
8. OpenAPI
9. Function Tool (user-defined)

Viktig: Custom tools utenfor denne listen må wrappes som Function Tools for å evalueres.

Arkitekturmønstre

1. Pre-deployment batch evaluation (Cloud Evaluation)

Bruk: Test agenten mot et større dataset før deploy (100-1000+ test cases).

Fordeler:

• Ingen local compute-krav (kjører i Azure)
• CI/CD-integrasjon via Azure AI Projects SDK
• Resultat logges i Foundry portal med trace-debugger
• Supports både custom evaluators og built-in

Ulemper:

• Koster Azure OpenAI tokens (evaluator LLM calls)
• Krever Microsoft Foundry project setup

Eksempel (Python):

from azure.ai.evaluation import evaluate
from azure.ai.evaluation import IntentResolutionEvaluator, TaskAdherenceEvaluator

# Initialize evaluators with reasoning model for complex tasks
quality_evaluators = {
    "IntentResolutionEvaluator": IntentResolutionEvaluator(
        model_config=reasoning_model_config,
        is_reasoning_model=True
    ),
    "TaskAdherenceEvaluator": TaskAdherenceEvaluator(
        model_config=reasoning_model_config,
        is_reasoning_model=True
    ),
}

# Batch evaluate with converter support
converter = AIAgentConverter(project_client)
filename = "evaluation_input_data.jsonl"
converter.prepare_evaluation_data(thread_ids=[thread1_id, thread2_id], filename=filename)

response = evaluate(
    data=filename,
    evaluation_name="agent-qa-regression",
    evaluators=quality_evaluators,
    azure_ai_project=os.environ["AZURE_AI_PROJECT"]
)

print(response["metrics"])  # Averaged scores
print(response["studio_url"])  # Foundry portal link

2. Continuous evaluation (Production Monitoring)

Bruk: Automatisk evaluering av agent-interaksjoner i produksjon med sampling.

Fordeler:

• Near real-time observability i Azure Monitor
• Sampling configuration (0-100%, max 1000/hour)
• Kobles til traces for debugging
• Integration med Foundry Observability dashboard

Ulemper:

• Krever Application Insights oppsett
• Cost overhead (evaluator LLM calls + Application Insights storage)
• Reasoning explanations kan inneholde sensitiv data (må redact via redact_score_properties=True)

Eksempel (Python):

from azure.ai.projects.models import AgentEvaluationRequest, EvaluatorIds

# Define evaluators for continuous monitoring
evaluators = {
    "Relevance": {"Id": EvaluatorIds.Relevance.value},
    "Fluency": {"Id": EvaluatorIds.Fluency.value},
    "ContentSafety": {"Id": EvaluatorIds.ContentSafety.value}
}

# Submit continuous evaluation after each agent run
project_client.evaluation.create_agent_evaluation(
    AgentEvaluationRequest(
        thread=thread.id,
        run=run.id,
        evaluators=evaluators,
        samplingConfiguration=AgentEvaluationSamplingConfiguration(
            name=agent.id,
            samplingPercent=100,  # 100% of runs
            maxRequestRate=250    # Max 250 evals/hour
        ),
        appInsightsConnectionString=project_client.telemetry.get_application_insights_connection_string()
    )
)

# Query results from Application Insights (KQL)
query = f"""
traces
| where message == "gen_ai.evaluation.result"
| where customDimensions["gen_ai.thread.run.id"] == "{run.id}"
"""

3. Local evaluation (Development Testing)

Bruk: Rask testing under utvikling (1-10 test cases).

Fordeler:

• Umiddelbar feedback loop
• Lavere cost (færre test cases)
• Ingen cloud dependency

Ulemper:

• Ikke skalerbar til store datasets
• Local compute-krav
• Manuelt resultat-håndtering

Eksempel (Python):

from azure.ai.evaluation import IntentResolutionEvaluator

evaluator = IntentResolutionEvaluator(model_config)

# Evaluate single agent run
result = evaluator(
    query="What is the weather in Seattle?",
    response="The current weather in Seattle is Sunny, 25°C."
)

print(result["intent_resolution"])  # 5.0
print(result["intent_resolution_result"])  # "pass"
print(result["intent_resolution_reason"])  # LLM explanation

Beslutningsveiledning

Når bruke hvilken evalueringstype?

Scenario	Anbefalt type	Evaluators	Frequency
Prototype-fase (1-10 test cases)	Local evaluation	IntentResolution, TaskAdherence	Ad-hoc testing
Pre-deployment (100+ test cases)	Cloud batch evaluation	Alle quality + safety evaluators	Før hver release
CI/CD pipeline	Cloud batch evaluation	Subset (fast evaluators: Relevance, Coherence)	Hver PR
Production monitoring	Continuous evaluation	ContentSafety, IntentResolution, TaskAdherence	10-50% sampling
Red teaming validation	Local + Cloud	IndirectAttack, CodeVulnerability, ContentSafety	Før initial deploy + quarterly

Model selection for LLM judges

Judge model	Use case	Cost	Reasoning quality
gpt-4o	Standard evaluation (Coherence, Fluency, Relevance)	Moderat	God
gpt-4.1	Standard evaluation med bedre reasoning	Høyere	Bedre
o3-mini / o-series	Kompleks evaluation (TaskAdherence, ToolCallAccuracy)	Høyest	Best (chain-of-thought)

Konfigurasjon:

reasoning_model_config = {
    "azure_deployment": "o3-mini",
    "api_key": os.getenv("AZURE_API_KEY"),
    "azure_endpoint": os.getenv("AZURE_ENDPOINT"),
    "api_version": "2024-08-01-preview",
}

evaluator = TaskAdherenceEvaluator(
    model_config=reasoning_model_config,
    is_reasoning_model=True  # Aktiverer extended thinking budget
)

Vanlige feil

Feil	Symptom	Løsning
Missing system message	Evaluator warning: "Cannot parse query"	Alltid inkluder system message som første melding i query
Tool call schema mismatch	ToolCallAccuracyEvaluator scorer lavt uten grunn	Sjekk at tool_definitions matcher faktisk tool signature
Evaluator cost explosion	Uventet høy Azure OpenAI-faktura	Reduser sampling rate i continuous eval, bruk billigere judge model (gpt-4o > o3-mini)
Thread ID collision	Feil evalueringsresultater	Bruk unique thread IDs, ikke gjenbruk threads
Non-supported tool types	ToolCallAccuracyEvaluator returnerer "pass" med "unsupported tool" reason	Wrap custom tools som Function Tools

Røde flagg

• Pass rate < 60% for IntentResolution: Agent forstår ikke user intents — revurder system message eller few-shot examples
• ToolCallAccuracy score < 3: Agent caller feil tools — vurder tydeligere tool descriptions eller færre tools
• TaskAdherence score < 3: Agent ignorerer instruksjoner — sjekk system message, eller agenten har for mange tools (tool confusion)
• ContentSafety violations > 1%: Agenten genererer harmful content — implementer content filters, revurder system instructions
• GroundednessEvaluator score < 4: Agent hallusinerer — sjekk at tool outputs brukes korrekt, vurder RAG-forbedringer

Integrasjon med Microsoft-stakken

Microsoft Foundry

• Evaluation wizard (UI): No-code batch evaluation med built-in evaluators
• Trace debugger: Step-by-step agent execution trace koblet til evaluation scores
• Evaluation library: Lagre custom evaluators som reusable assets
• Comparison view: Sammenlign flere evaluation runs (A/B testing)

Foundry Agent Service

• Auto-converter: AIAgentConverter transformerer Foundry agent threads til evaluation data automatisk
• Tool call tracking: Built-in logging av alle tool invocations for ToolCallAccuracyEvaluator

Azure Monitor + Application Insights

• Continuous evaluation storage: Alle eval results logges som traces
• KQL queries: Flexible querying av evaluation metrics over tid
• Alerts: Sett opp alerts hvis pass rate dropper under threshold

MLflow 3 (Databricks / Cross-platform)

MLflow 3 tilbyr komprehensiv GenAI-evaluering for agenter paa tvers av plattformer:

Feature	Beskrivelse
Built-in LLM judges	Innebygde dommere for kvalitetsmetrikker (relevance, groundedness, safety, etc.)
Custom scorers	Definer egne kvalitetsmetrikker med Python-funksjoner
Eval harness	Test GenAI-app mot eval-datasett under utvikling; sammenlign appversjoner
Conversation evaluation	Vurder multi-turn samtalekvaltiet (completeness, user frustration, dialogue coherence)
Conversation simulation	Generer syntetiske multi-turn samtaler for testing
Production monitoring	Kjoer scorers og judges paa produksjons-traces automatisk (Beta)
Review App	Samle ekspertfeedback og bygg eval-datasett

MLflow Tracing gir real-time trace logging gjennom hele livssyklusen. Samme judges og scorers kan brukes i baade development og produksjon — konsistent evaluering.

# MLflow 3 evaluation eksempel
import mlflow

results = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=my_agent,
    scorers=[mlflow.genai.scorers.groundedness(), mlflow.genai.scorers.safety()]
)

Prompt Flow

• Evaluation flows: Custom evaluation logic som Prompt Flow (deprecated approach — bruk Azure AI Evaluation SDK i stedet)
• Batch run evaluation: Kjør evaluation som Prompt Flow batch run

Semantic Kernel

• Converter support: AIAgentConverter støtter Semantic Kernel agents direkte
• Plugin evaluation: Evaluer Semantic Kernel plugins som tools

Offentlig sektor (Norge)

GDPR og databehandling

Risiko: Evaluators sender conversation data til Azure OpenAI judge models (kan inneholde persondata).

Mitigering:

• Anonymisering: Fjern PII fra test datasets før evaluation
• Redaction configuration: Bruk redact_score_properties=True i continuous evaluation for å hindre reasoning explanations med sensitiv data
• Data residency: Sørg for at judge model (Azure OpenAI deployment) er i EU-region

Forvaltningsloven § 11a (automatiserte enkeltvedtak)

Risiko: Hvis agenten fatter enkeltvedtak, må evaluering dokumentere at systemet oppfyller kvalitetskrav.

Mitigering:

• Batch evaluation før deploy: Dokumentér pass rate for TaskAdherence, IntentResolution (min. 80% i kritiske use cases)
• Continuous monitoring: Løpende overvåking av agent performance i produksjon med alerts ved degradering
• Human-in-the-loop: Ved vedtak: kombiner agent-forslag med manual review, log evaluation scores i vedtakssystemet

AI Act (High-risk AI systems)

Risiko: Agenter i kritiske domener (helse, politi, offentlige ytelser) klassifiseres som high-risk → krav til testing og dokumentasjon.

Mitigering:

• Test dataset representativitet: Sørg for at evaluation dataset dekker alle demografiske grupper (bias testing)
• Adversarial testing: Bruk IndirectAttackEvaluator for jailbreak testing, ContentSafetyEvaluator for harmful content
• Evaluation audit trail: Lagre alle evaluation runs i Foundry med timestamp, versioning, og results (compliance dokumentasjon)

Schrems II

Risiko: Evaluation data sendes til Azure OpenAI i US-region (data transfer issue).

Mitigering:

• EU-based judge models: Deploy Azure OpenAI judge model (gpt-4.1) i EU-region (France Central, Sweden Central)
• On-prem evaluation: Vurder local evaluation for svært sensitive use cases (men mistet CI/CD-integrasjon)

Kostnad og lisensiering

Prismodell

Komponent	Pricing model	Estimert cost (per 1000 evals)
Azure AI Evaluation SDK	Gratis (open-source)	0 NOK
Azure OpenAI judge model (gpt-4o)	Pay-per-token (input + output)	~200-500 NOK (avhengig av conversation length)
Azure OpenAI judge model (o3-mini)	Pay-per-token + reasoning tokens	~500-1200 NOK (høyere pga. extended thinking)
Application Insights	Data ingestion + retention	~50-100 NOK/måned (1M traces)
Foundry storage	Evaluation results + traces	Inkludert i Microsoft Foundry project (ingen ekstra cost)

Cost optimization tips

1. Reducer sampling rate i continuous eval:

   • Development: 10-20% sampling
   • Production: 5-10% sampling (høyere for kritiske agenter)

2. Velg billigere judge model for simple evaluators:

   • Coherence, Fluency, Relevance → gpt-4o (ikke o-series)
   • TaskAdherence, ToolCallAccuracy → o3-mini (krever reasoning)

3. Reduser conversation length i evaluation data:

   • Inkluder kun siste 3-5 turns i query (ikke hele thread history)

4. Batch evaluation i stedet for continuous:

   • Pre-deployment testing: batch eval (1x før release)
   • Production: sample 5-10%, ikke 100%

5. Reuse eval datasets:

   • Lagre golden datasets i Foundry, ikke regenerer hver gang

Lisensiering

Komponent	Lisens	Krav
Azure AI Evaluation SDK	MIT License (open-source)	Ingen
Microsoft Foundry	Inkludert i Azure subscription	Azure subscription
Azure OpenAI	Pay-as-you-go (per token)	Azure OpenAI access (申请 required)
Application Insights	Pay-as-you-go (per GB ingested)	Azure subscription

For arkitekten (Cosmo)

Spørsmål å stille under arkitekturgjennomgang

1. Evaluation strategy:

   • "Hvilken type evaluation kjører du? (local, batch, continuous)?"
   • "Hvor ofte evaluerer du agenten? (per PR, pre-deploy, kontinuerlig)?"
   • "Har dere golden dataset for regression testing?"

2. Evaluator selection:

   • "Hvilke evaluators bruker du? (quality, safety, custom)?"
   • "Bruker du reasoning models (o-series) som judges for komplekse evaluators?"
   • "Hvordan håndterer du tool call evaluation?"

3. Cost management:

   • "Hva er budsjettet for evaluation per måned?"
   • "Har dere optimalisert sampling rate i continuous eval?"
   • "Bruker dere billigere judge models for simple evaluators?"

4. Compliance:

   • "Hvor lagres evaluation data? (EU-region?)"
   • "Er PII fjernet fra test datasets?"
   • "Redacts dere reasoning explanations i continuous eval?"

5. Production monitoring:

   • "Er Application Insights satt opp for continuous eval?"
   • "Har dere alerts på pass rate degradation?"
   • "Hvordan debugger dere failed evaluations? (trace-kobling?)"

6. Custom evaluators:

   • "Har dere behov for custom evaluators utover built-in?"
   • "Er custom evaluators lagret i Foundry Evaluator Library?"
   • "Hvordan tester dere custom evaluators selv?"

7. Agent framework:

   • "Bruker dere Foundry Agent Service, Semantic Kernel, eller custom agents?"
   • "Støtter eders agent framework AIAgentConverter?"
   • "Må dere manuelt parse agent messages til OpenAI-style schema?"

8. Safety validation:

   • "Kjører dere adversarial testing (jailbreak, indirect attack)?"
   • "Er ContentSafetyEvaluator del av continuous eval?"
   • "Hvordan håndterer dere evaluation av harmful content?"

Fallgruver

Fallgruve	Konsekvens	Unngå ved
Ingen continuous evaluation i prod	Agent degraderer over tid uten at du vet det	Sett opp continuous eval med 5-10% sampling + alerts
Test dataset ikke representativt	Agenten scorer høyt i test, lavt i prod	Bruk production data som test cases (anonymisert)
Ignorering av reasoning explanations	Misforstår hvorfor agenten feiler	Les {metric}_reason field for å forstå root cause
Tool call mismatch	ToolCallAccuracyEvaluator scorer lavt selv om agent fungerer	Sjekk at tool_definitions i evaluation matcher faktisk tool schema
Cost explosion i continuous eval	Uventet høy faktura	Start med lav sampling (10%), bruk gpt-4o i stedet for o3-mini for simple metrics
Sensitive data i eval traces	GDPR-brudd	Anonymiser test data, bruk redact_score_properties=True
Manglende system message i query	Evaluators kan ikke parse agent context	Alltid inkluder system message som første melding i query

Anbefalinger per modenhetsnivå

Nivå 1: Prototype (ingen prod deployment)

• Local evaluation med IntentResolution + TaskAdherence
• Test på 5-10 manuelt skrevne test cases
• Ingen continuous evaluation
• Judge model: gpt-4o

Nivå 2: Pilot (begrenset prod bruk)

• Batch evaluation før hver deploy (50-100 test cases)
• Continuous evaluation i prod (10% sampling, kun ContentSafety + IntentResolution)
• Application Insights oppsett
• Judge model: gpt-4.1

Nivå 3: Production (full prod deployment)

• Batch evaluation i CI/CD (200+ test cases, quality + safety evaluators)
• Continuous evaluation (5-10% sampling, alle relevante evaluators)
• Alerts på pass rate < 70%
• Trace-debugger i Foundry for failed evals
• Judge model: o3-mini for complex evaluators, gpt-4o for simple

Nivå 4: Mission-critical (high-risk AI system)

• Batch evaluation med 1000+ test cases (inkludert adversarial)
• Continuous evaluation (20-50% sampling, alle evaluators)
• Custom evaluators for domain-specific metrics
• Monthly red teaming med IndirectAttack + CodeVulnerability
• Human-in-the-loop review av failed evaluations
• Full evaluation audit trail (lagres i 5 år for AI Act compliance)
• Judge model: o3-mini + custom fine-tuned judge for kritiske metrics

Kilder og verifisering

Microsoft Learn (MCP-verified)

1. Evaluate your AI agents (preview) https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/agent-evaluate-sdk?view=foundry-classic Confidence: Verified — Hovedreferanse for Azure AI Evaluation SDK, evaluator types, model support

2. Continuously evaluate your AI agents (preview) https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/continuous-evaluation-agents?view=foundry-classic Confidence: Verified — Continuous evaluation setup, sampling configuration, Application Insights integration

3. Run evaluations in the cloud by using the Microsoft Foundry SDK https://learn.microsoft.com/en-us/azure/foundry/how-to/develop/cloud-evaluation?view=foundry-classic Confidence: Verified — Cloud batch evaluation, CI/CD integration, dataset formats

4. Tutorial: Idea to prototype - Build and evaluate an enterprise agent https://learn.microsoft.com/en-us/azure/foundry/tutorials/developer-journey-idea-to-prototype?view=foundry Confidence: Verified — End-to-end tutorial med cloud evaluation, built-in evaluators

5. Test and evaluate AI workloads on Azure (Well-Architected Framework) https://learn.microsoft.com/en-us/azure/well-architected/ai/test#validate-agentic-workflows Confidence: Verified — Agentic workflow testing strategy, tool call validation, security testing

6. Observability in generative AI https://learn.microsoft.com/en-us/azure/foundry/concepts/observability Confidence: Verified — Built-in evaluators list, GenAIOps evaluation stages, simulators

7. What are hosted agents? (Evaluate and test hosted agents) https://learn.microsoft.com/en-us/azure/foundry/agents/concepts/hosted-agents?view=foundry#evaluate-and-test-hosted-agents Confidence: Verified — Hosted agent evaluation best practices, test dataset creation

8. Agent evaluators https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/agent-evaluators?view=foundry Confidence: Verified — Agent-specific evaluator details (Intent Resolution, Task Adherence, Tool Call Accuracy)

9. Evaluate and monitor AI agents (MLflow 3 on Databricks) https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/ Confidence: Verified — MLflow 3 GenAI evaluation: built-in LLM judges og scorers, eval-harness for development, production monitoring (Beta), conversation evaluation (multi-turn), conversation simulation, Review App for human feedback, Genie Code for observability; integrert med MLflow Tracing paa tvers av development/test/produksjon; oppdatert 2026-04

10. Run automated tests for agent quality and reliability (Copilot Studio) https://learn.microsoft.com/en-us/power-platform/release-plan/2025wave1/microsoft-copilot-studio/run-automated-tests-agent-quality-reliability Confidence: Verified — Copilot Studio evaluation framework (2025 preview)

Confidence levels per section

Section	Confidence	Reason
Introduksjon	Verified	Basert på 3 MCP-kilder (agent-evaluate-sdk, observability, well-architected)
Kjernekomponenter	Verified	Direkte fra agent-evaluate-sdk dokumentasjon + code samples
Arkitekturmønstre	Verified	Fra cloud-evaluation + continuous-evaluation docs + code samples
Beslutningsveiledning	Baseline + Verified	Decision tables basert på best practices (well-architected) + cost models
Integrasjon med Microsoft-stakken	Verified	Fra Foundry, Semantic Kernel, Prompt Flow, Application Insights docs
Offentlig sektor (Norge)	Baseline	GDPR/AI Act vurdering basert på modellkunnskap + Azure residency facts
Kostnad og lisensiering	Baseline	Prisestimater basert på Azure OpenAI pricing (feb 2026) + observability costs
For arkitekten (Cosmo)	Baseline	Synthesized fra verified sources + praktisk erfaring

---

Document metadata:

• MCP calls: 3 (microsoft_docs_search) + 2 (microsoft_docs_fetch) + 1 (microsoft_code_sample_search) = 6
• Unique sources: 10 Microsoft Learn URLs
• Word count: ~3200 ord
• File size: ~29 KB