ms-ai-architect/skills/ms-ai-engineering/references/agent-orchestration/agent-evaluation-testing-frameworks.md

# Agent Evaluation and Testing Frameworks

**Last updated:** 2026-06-24 | Verified: MCP 2026-06
**Status:** GA (Azure AI Evaluation SDK), Preview (Agent-specific evaluators)
**Category:** Agent Orchestration & Automation
**Type:** reference
**Source:** https://learn.microsoft.com/azure/foundry/concepts/observability

---

## Innhold

- [Introduksjon](#introduksjon)
- [Kjernekomponenter](#kjernekomponenter)
- [Arkitekturmønstre](#arkitekturmønstre)
- [Beslutningsveiledning](#beslutningsveiledning)
- [Integrasjon med Microsoft-stakken](#integrasjon-med-microsoft-stakken)
- [Offentlig sektor (Norge)](#offentlig-sektor-norge)
- [Kostnad og lisensiering](#kostnad-og-lisensiering)
- [For arkitekten (Cosmo)](#for-arkitekten-cosmo)
- [Kilder og verifisering](#kilder-og-verifisering)

## Introduksjon

Agent-baserte AI-systemer representerer en ny kompleksitet i testing og validering sammenlignet med tradisjonelle deterministic workflows. Der en enkel LLM-applikasjon kun har én inngangspunkt og ett svar, har agenter multippel tool-calling, dynamisk reasoning, multi-turn samtaler, og ikke-deterministisk oppførsel. Microsoft tilbyr et komprehensivt evalueringsrammeverk gjennom Azure AI Evaluation SDK og Microsoft Foundry som håndterer både pre-deployment testing (batch evaluation) og post-deployment monitoring (continuous evaluation).

Evalueringsrammeverket støtter tre hovedtyper testing: **System Evaluation** (helhetsoppførsel til agenten), **Process Evaluation** (kvalitet på tool calls og reasoning steps), og **Safety Evaluation** (content safety, jailbreak-resistance, bias). Alle evaluators opererer som LLM judges (typisk GPT-4.1 eller o-series reasoning models) som gir både scores, pass/fail labels, og reasoning explanations.

Microsoft Foundry støtter både Foundry Agent Service (built-in agents), Semantic Kernel agents, og custom agents via OpenAI-style message schema. Evaluering kan kjøres lokalt på utviklermaskinen, i cloud for CI/CD-integrasjon, eller kontinuerlig i produksjon med sampling rates og Azure Monitor Application Insights-integrasjon.

## Kjernekomponenter

### Evaluator-typer

| Evaluator | Formål | Input | Score range | LLM Judge? |
|-----------|--------|-------|-------------|-----------|
| **IntentResolutionEvaluator** | Måler om agenten identifiserer brukerens intent korrekt | query, response, (tool_definitions optional) | 1-5 Likert | Ja (GPT-4.1 / o-series) |
| **TaskAdherenceEvaluator** | Sjekker om agentens svar følger system message og prior steps | query, response, (tool_calls optional) | 1-5 Likert | Ja |
| **ToolCallAccuracyEvaluator** | Validerer at agenten kaller riktige tools med riktige parameters | query, tool_definitions, (response/tool_calls) | 1-5 Likert | Ja |
| **ResponseCompletenessEvaluator** | Evaluerer om svar er komplett og dekker alle deler av query | query, response | 1-5 Likert | Ja |
| **GroundednessEvaluator** | Måler om agentsvar er forankret i tool outputs (ikke hallusinert) | query, response, tool_definitions | 1-5 Likert | Ja |
| **RelevanceEvaluator** | Sjekker om svar er relevant for query | query, response | 1-5 Likert | Ja |
| **CoherenceEvaluator** | Evaluerer logisk sammenheng i svar | query, response | 1-5 Likert | Ja |
| **FluencyEvaluator** | Måler språklig kvalitet og grammatikk | query, response | 1-5 Likert | Ja |
| **ContentSafetyEvaluator** | Detekterer harmful content (violence, hate, sexual, self-harm) | query, response | 0-7 severity | Ja (Azure AI Content Safety) |
| **IndirectAttackEvaluator** | Sjekker jailbreak attempts via indirect injection | query, response | Pass/Fail | Ja |
| **CodeVulnerabilityEvaluator** | Identifiserer usikker kode i agentsvar | response | Pass/Fail | Ja |
| **TaskCompletionEvaluator** (preview) | Måler om agenten fullførte oppgaven end-to-end med brukbar leveranse | query, response, (tool_definitions) | Pass/Fail | Ja |
| **CustomerSatisfactionEvaluator** (preview) | Predikerer brukertilfredshet over en samtale (6 dimensjoner) | messages | 1-5 Likert | Ja |
| **ToolSelectionEvaluator** | Måler om agenten valgte riktige (og kun nødvendige) tools | query, response, tool_definitions | Pass/Fail | Ja |
| **ToolInputAccuracyEvaluator** | Validerer at alle tool-parametere er korrekte (6 strenge kriterier) | query, response, tool_definitions | Pass/Fail | Ja |
| **ToolOutputUtilizationEvaluator** | Måler om agenten brukte tool-outputs riktig i svaret | query, response, tool_definitions | Pass/Fail | Ja |
| **ToolCallSuccessEvaluator** | Sjekker om tool-kall ble utført uten tekniske feil | response | Pass/Fail | Ja |
| **TaskNavigationEfficiencyEvaluator** | Sammenligner agentens steg mot optimal/forventet sti (krever ground truth) | actions, expected_actions | Pass/Fail | Nei |

> **Microsoft Agent Framework — `FoundryEvals`:** I tillegg til `AIAgentConverter` kan en `AIAgent` kobles direkte til Foundrys evalueringstjeneste via `FoundryEvals` (kjører som standard relevance/coherence/task_adherence og legger automatisk til tool_call_accuracy når tool-definisjoner finnes; resultater vises i Foundry-portalen). For kompleks evaluering anbefaler Microsoft nå en sterk reasoning-modell som `gpt-4.1-mini` som judge for balanse mellom ytelse og kostnad.

### Evaluator output format

Alle evaluators returnerer standardisert JSON:

```json
{
  "{metric_name}": 4.0,                      // Score (1-5, 0-7, 0-1 avhengig av type)
  "{metric_name}_result": "pass",            // Pass/fail basert på threshold
  "{metric_name}_threshold": 3,              // Binarization threshold (default eller user-defined)
  "{metric_name}_reason": "The agent correctly...", // LLM judge reasoning
  "details": { ... }                         // Optional debug info (f.eks. tool call breakdown)
}
```

### Supported agent frameworks

| Framework | Converter support? | Evaluators |
|-----------|-------------------|-----------|
| **Foundry Agent Service** | Ja (`AIAgentConverter`) | Alle |
| **Semantic Kernel** | Ja (`AIAgentConverter`) | Alle |
| **Custom agents** | Nei (bruk OpenAI-style message schema) | Alle (krever manuell parsing) |

### Tool call evaluation support

`ToolCallAccuracyEvaluator` støtter disse tool-typene i Foundry Agent Service:

1. File Search
2. Azure AI Search
3. Bing Grounding
4. Bing Custom Search
5. SharePoint Grounding
6. Code Interpreter
7. Fabric Data Agent
8. OpenAPI
9. Function Tool (user-defined)

**Viktig:** Custom tools utenfor denne listen må wrappes som Function Tools for å evalueres.

## Arkitekturmønstre

### 1. Pre-deployment batch evaluation (Cloud Evaluation)

**Bruk:** Test agenten mot et større dataset før deploy (100-1000+ test cases).

**Fordeler:**
- Ingen local compute-krav (kjører i Azure)
- CI/CD-integrasjon via Azure AI Projects SDK
- Resultat logges i Foundry portal med trace-debugger
- Supports både custom evaluators og built-in

**Ulemper:**
- Koster Azure OpenAI tokens (evaluator LLM calls)
- Krever Microsoft Foundry project setup

**Eksempel (Python):**

```python
from azure.ai.evaluation import evaluate
from azure.ai.evaluation import IntentResolutionEvaluator, TaskAdherenceEvaluator

# Initialize evaluators with reasoning model for complex tasks
quality_evaluators = {
    "IntentResolutionEvaluator": IntentResolutionEvaluator(
        model_config=reasoning_model_config,
        is_reasoning_model=True
    ),
    "TaskAdherenceEvaluator": TaskAdherenceEvaluator(
        model_config=reasoning_model_config,
        is_reasoning_model=True
    ),
}

# Batch evaluate with converter support
converter = AIAgentConverter(project_client)
filename = "evaluation_input_data.jsonl"
converter.prepare_evaluation_data(thread_ids=[thread1_id, thread2_id], filename=filename)

response = evaluate(
    data=filename,
    evaluation_name="agent-qa-regression",
    evaluators=quality_evaluators,
    azure_ai_project=os.environ["AZURE_AI_PROJECT"]
)

print(response["metrics"])  # Averaged scores
print(response["studio_url"])  # Foundry portal link
```

### 2. Continuous evaluation (Production Monitoring)

**Bruk:** Automatisk evaluering av agent-interaksjoner i produksjon med sampling.

**Fordeler:**
- Near real-time observability i Azure Monitor
- Sampling configuration (0-100%, max 1000/hour)
- Kobles til traces for debugging
- Integration med Foundry Observability dashboard

**Ulemper:**
- Krever Application Insights oppsett
- Cost overhead (evaluator LLM calls + Application Insights storage)
- Reasoning explanations kan inneholde sensitiv data (må redact via `redact_score_properties=True`)

**Eksempel (Python):**

```python
from azure.ai.projects.models import AgentEvaluationRequest, EvaluatorIds

# Define evaluators for continuous monitoring
evaluators = {
    "Relevance": {"Id": EvaluatorIds.Relevance.value},
    "Fluency": {"Id": EvaluatorIds.Fluency.value},
    "ContentSafety": {"Id": EvaluatorIds.ContentSafety.value}
}

# Submit continuous evaluation after each agent run
project_client.evaluation.create_agent_evaluation(
    AgentEvaluationRequest(
        thread=thread.id,
        run=run.id,
        evaluators=evaluators,
        samplingConfiguration=AgentEvaluationSamplingConfiguration(
            name=agent.id,
            samplingPercent=100,  # 100% of runs
            maxRequestRate=250    # Max 250 evals/hour
        ),
        appInsightsConnectionString=project_client.telemetry.get_application_insights_connection_string()
    )
)

# Query results from Application Insights (KQL)
query = f"""
traces
| where message == "gen_ai.evaluation.result"
| where customDimensions["gen_ai.thread.run.id"] == "{run.id}"
"""
```

### 3. Local evaluation (Development Testing)

**Bruk:** Rask testing under utvikling (1-10 test cases).

**Fordeler:**
- Umiddelbar feedback loop
- Lavere cost (færre test cases)
- Ingen cloud dependency

**Ulemper:**
- Ikke skalerbar til store datasets
- Local compute-krav
- Manuelt resultat-håndtering

**Eksempel (Python):**

```python
from azure.ai.evaluation import IntentResolutionEvaluator

evaluator = IntentResolutionEvaluator(model_config)

# Evaluate single agent run
result = evaluator(
    query="What is the weather in Seattle?",
    response="The current weather in Seattle is Sunny, 25°C."
)

print(result["intent_resolution"])  # 5.0
print(result["intent_resolution_result"])  # "pass"
print(result["intent_resolution_reason"])  # LLM explanation
```

## Beslutningsveiledning

### Når bruke hvilken evalueringstype?

| Scenario | Anbefalt type | Evaluators | Frequency |
|----------|---------------|-----------|-----------|
| **Prototype-fase (1-10 test cases)** | Local evaluation | IntentResolution, TaskAdherence | Ad-hoc testing |
| **Pre-deployment (100+ test cases)** | Cloud batch evaluation | Alle quality + safety evaluators | Før hver release |
| **CI/CD pipeline** | Cloud batch evaluation | Subset (fast evaluators: Relevance, Coherence) | Hver PR |
| **Production monitoring** | Continuous evaluation | ContentSafety, IntentResolution, TaskAdherence | 10-50% sampling |
| **Red teaming validation** | Local + Cloud | IndirectAttack, CodeVulnerability, ContentSafety | Før initial deploy + quarterly |

### Model selection for LLM judges

| Judge model | Use case | Cost | Reasoning quality |
|-------------|----------|------|-------------------|
| **gpt-4o** | Standard evaluation (Coherence, Fluency, Relevance) | Moderat | God |
| **gpt-4.1** | Standard evaluation med bedre reasoning | Høyere | Bedre |
| **o3-mini / o-series** | Kompleks evaluation (TaskAdherence, ToolCallAccuracy) | Høyest | Best (chain-of-thought) |

**Konfigurasjon:**

```python
reasoning_model_config = {
    "azure_deployment": "o3-mini",
    "api_key": os.getenv("AZURE_API_KEY"),
    "azure_endpoint": os.getenv("AZURE_ENDPOINT"),
    "api_version": "2024-08-01-preview",
}

evaluator = TaskAdherenceEvaluator(
    model_config=reasoning_model_config,
    is_reasoning_model=True  # Aktiverer extended thinking budget
)
```

### Vanlige feil

| Feil | Symptom | Løsning |
|------|---------|---------|
| **Missing system message** | Evaluator warning: "Cannot parse query" | Alltid inkluder system message som første melding i `query` |
| **Tool call schema mismatch** | ToolCallAccuracyEvaluator scorer lavt uten grunn | Sjekk at tool_definitions matcher faktisk tool signature |
| **Evaluator cost explosion** | Uventet høy Azure OpenAI-faktura | Reduser sampling rate i continuous eval, bruk billigere judge model (gpt-4o > o3-mini) |
| **Thread ID collision** | Feil evalueringsresultater | Bruk unique thread IDs, ikke gjenbruk threads |
| **Non-supported tool types** | ToolCallAccuracyEvaluator returnerer "pass" med "unsupported tool" reason | Wrap custom tools som Function Tools |

### Røde flagg

- **Pass rate < 60% for IntentResolution:** Agent forstår ikke user intents — revurder system message eller few-shot examples
- **ToolCallAccuracy score < 3:** Agent caller feil tools — vurder tydeligere tool descriptions eller færre tools
- **TaskAdherence score < 3:** Agent ignorerer instruksjoner — sjekk system message, eller agenten har for mange tools (tool confusion)
- **ContentSafety violations > 1%:** Agenten genererer harmful content — implementer content filters, revurder system instructions
- **GroundednessEvaluator score < 4:** Agent hallusinerer — sjekk at tool outputs brukes korrekt, vurder RAG-forbedringer

## Integrasjon med Microsoft-stakken

### Microsoft Foundry

- **Evaluation wizard (UI):** No-code batch evaluation med built-in evaluators
- **Trace debugger:** Step-by-step agent execution trace koblet til evaluation scores
- **Evaluation library:** Lagre custom evaluators som reusable assets
- **Comparison view:** Sammenlign flere evaluation runs (A/B testing)

### Foundry Agent Service

- **Auto-converter:** `AIAgentConverter` transformerer Foundry agent threads til evaluation data automatisk
- **Tool call tracking:** Built-in logging av alle tool invocations for ToolCallAccuracyEvaluator

### Azure Monitor + Application Insights

- **Continuous evaluation storage:** Alle eval results logges som traces
- **KQL queries:** Flexible querying av evaluation metrics over tid
- **Alerts:** Sett opp alerts hvis pass rate dropper under threshold

### MLflow 3 (Databricks / Cross-platform)

MLflow 3 tilbyr komprehensiv GenAI-evaluering for agenter paa tvers av plattformer:

| Feature | Beskrivelse |
|---------|-------------|
| **Built-in LLM judges** | Innebygde dommere for kvalitetsmetrikker (relevance, groundedness, safety, etc.) |
| **Custom scorers** | Definer egne kvalitetsmetrikker med Python-funksjoner |
| **Eval harness** | Test GenAI-app mot eval-datasett under utvikling; sammenlign appversjoner |
| **Conversation evaluation** | Vurder multi-turn samtalekvaltiet (completeness, user frustration, dialogue coherence) |
| **Conversation simulation** | Generer syntetiske multi-turn samtaler for testing |
| **Production monitoring** | Kjoer scorers og judges paa produksjons-traces automatisk (Beta) |
| **Review App** | Samle ekspertfeedback og bygg eval-datasett |

MLflow Tracing gir real-time trace logging gjennom hele livssyklusen. Samme judges og scorers kan brukes i baade development og produksjon — konsistent evaluering.

```python
# MLflow 3 evaluation eksempel
import mlflow

results = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=my_agent,
    scorers=[mlflow.genai.scorers.groundedness(), mlflow.genai.scorers.safety()]
)
```

### Prompt Flow

- **Evaluation flows:** Custom evaluation logic som Prompt Flow (deprecated approach — bruk Azure AI Evaluation SDK i stedet)
- **Batch run evaluation:** Kjør evaluation som Prompt Flow batch run

### Semantic Kernel

- **Converter support:** `AIAgentConverter` støtter Semantic Kernel agents direkte
- **Plugin evaluation:** Evaluer Semantic Kernel plugins som tools

## Offentlig sektor (Norge)

### GDPR og databehandling

**Risiko:** Evaluators sender conversation data til Azure OpenAI judge models (kan inneholde persondata).

**Mitigering:**
- **Anonymisering:** Fjern PII fra test datasets før evaluation
- **Redaction configuration:** Bruk `redact_score_properties=True` i continuous evaluation for å hindre reasoning explanations med sensitiv data
- **Data residency:** Sørg for at judge model (Azure OpenAI deployment) er i EU-region

### Forvaltningsloven § 11a (automatiserte enkeltvedtak)

**Risiko:** Hvis agenten fatter enkeltvedtak, må evaluering dokumentere at systemet oppfyller kvalitetskrav.

**Mitigering:**
- **Batch evaluation før deploy:** Dokumentér pass rate for TaskAdherence, IntentResolution (min. 80% i kritiske use cases)
- **Continuous monitoring:** Løpende overvåking av agent performance i produksjon med alerts ved degradering
- **Human-in-the-loop:** Ved vedtak: kombiner agent-forslag med manual review, log evaluation scores i vedtakssystemet

### AI Act (High-risk AI systems)

**Risiko:** Agenter i kritiske domener (helse, politi, offentlige ytelser) klassifiseres som high-risk → krav til testing og dokumentasjon.

**Mitigering:**
- **Test dataset representativitet:** Sørg for at evaluation dataset dekker alle demografiske grupper (bias testing)
- **Adversarial testing:** Bruk `IndirectAttackEvaluator` for jailbreak testing, `ContentSafetyEvaluator` for harmful content
- **Evaluation audit trail:** Lagre alle evaluation runs i Foundry med timestamp, versioning, og results (compliance dokumentasjon)

### Schrems II

**Risiko:** Evaluation data sendes til Azure OpenAI i US-region (data transfer issue).

**Mitigering:**
- **EU-based judge models:** Deploy Azure OpenAI judge model (gpt-4.1) i EU-region (France Central, Sweden Central)
- **On-prem evaluation:** Vurder local evaluation for svært sensitive use cases (men mistet CI/CD-integrasjon)

## Kostnad og lisensiering

### Prismodell

| Komponent | Pricing model | Estimert cost (per 1000 evals) |
|-----------|---------------|--------------------------------|
| **Azure AI Evaluation SDK** | Gratis (open-source) | 0 NOK |
| **Azure OpenAI judge model (gpt-4o)** | Pay-per-token (input + output) | ~200-500 NOK (avhengig av conversation length) |
| **Azure OpenAI judge model (o3-mini)** | Pay-per-token + reasoning tokens | ~500-1200 NOK (høyere pga. extended thinking) |
| **Application Insights** | Data ingestion + retention | ~50-100 NOK/måned (1M traces) |
| **Foundry storage** | Evaluation results + traces | Inkludert i Microsoft Foundry project (ingen ekstra cost) |

### Cost optimization tips

1. **Reducer sampling rate i continuous eval:**
   - Development: 10-20% sampling
   - Production: 5-10% sampling (høyere for kritiske agenter)

2. **Velg billigere judge model for simple evaluators:**
   - Coherence, Fluency, Relevance → gpt-4o (ikke o-series)
   - TaskAdherence, ToolCallAccuracy → o3-mini (krever reasoning)

3. **Reduser conversation length i evaluation data:**
   - Inkluder kun siste 3-5 turns i `query` (ikke hele thread history)

4. **Batch evaluation i stedet for continuous:**
   - Pre-deployment testing: batch eval (1x før release)
   - Production: sample 5-10%, ikke 100%

5. **Reuse eval datasets:**
   - Lagre golden datasets i Foundry, ikke regenerer hver gang

### Lisensiering

| Komponent | Lisens | Krav |
|-----------|--------|------|
| **Azure AI Evaluation SDK** | MIT License (open-source) | Ingen |
| **Microsoft Foundry** | Inkludert i Azure subscription | Azure subscription |
| **Azure OpenAI** | Pay-as-you-go (per token) | Azure OpenAI access (申请 required) |
| **Application Insights** | Pay-as-you-go (per GB ingested) | Azure subscription |

## For arkitekten (Cosmo)

### Spørsmål å stille under arkitekturgjennomgang

1. **Evaluation strategy:**
   - "Hvilken type evaluation kjører du? (local, batch, continuous)?"
   - "Hvor ofte evaluerer du agenten? (per PR, pre-deploy, kontinuerlig)?"
   - "Har dere golden dataset for regression testing?"

2. **Evaluator selection:**
   - "Hvilke evaluators bruker du? (quality, safety, custom)?"
   - "Bruker du reasoning models (o-series) som judges for komplekse evaluators?"
   - "Hvordan håndterer du tool call evaluation?"

3. **Cost management:**
   - "Hva er budsjettet for evaluation per måned?"
   - "Har dere optimalisert sampling rate i continuous eval?"
   - "Bruker dere billigere judge models for simple evaluators?"

4. **Compliance:**
   - "Hvor lagres evaluation data? (EU-region?)"
   - "Er PII fjernet fra test datasets?"
   - "Redacts dere reasoning explanations i continuous eval?"

5. **Production monitoring:**
   - "Er Application Insights satt opp for continuous eval?"
   - "Har dere alerts på pass rate degradation?"
   - "Hvordan debugger dere failed evaluations? (trace-kobling?)"

6. **Custom evaluators:**
   - "Har dere behov for custom evaluators utover built-in?"
   - "Er custom evaluators lagret i Foundry Evaluator Library?"
   - "Hvordan tester dere custom evaluators selv?"

7. **Agent framework:**
   - "Bruker dere Foundry Agent Service, Semantic Kernel, eller custom agents?"
   - "Støtter eders agent framework AIAgentConverter?"
   - "Må dere manuelt parse agent messages til OpenAI-style schema?"

8. **Safety validation:**
   - "Kjører dere adversarial testing (jailbreak, indirect attack)?"
   - "Er ContentSafetyEvaluator del av continuous eval?"
   - "Hvordan håndterer dere evaluation av harmful content?"

### Fallgruver

| Fallgruve | Konsekvens | Unngå ved |
|-----------|-----------|-----------|
| **Ingen continuous evaluation i prod** | Agent degraderer over tid uten at du vet det | Sett opp continuous eval med 5-10% sampling + alerts |
| **Test dataset ikke representativt** | Agenten scorer høyt i test, lavt i prod | Bruk production data som test cases (anonymisert) |
| **Ignorering av reasoning explanations** | Misforstår hvorfor agenten feiler | Les `{metric}_reason` field for å forstå root cause |
| **Tool call mismatch** | ToolCallAccuracyEvaluator scorer lavt selv om agent fungerer | Sjekk at tool_definitions i evaluation matcher faktisk tool schema |
| **Cost explosion i continuous eval** | Uventet høy faktura | Start med lav sampling (10%), bruk gpt-4o i stedet for o3-mini for simple metrics |
| **Sensitive data i eval traces** | GDPR-brudd | Anonymiser test data, bruk `redact_score_properties=True` |
| **Manglende system message i query** | Evaluators kan ikke parse agent context | Alltid inkluder system message som første melding i query |

### Anbefalinger per modenhetsnivå

#### Nivå 1: Prototype (ingen prod deployment)
- **Local evaluation** med IntentResolution + TaskAdherence
- Test på 5-10 manuelt skrevne test cases
- Ingen continuous evaluation
- Judge model: gpt-4o

#### Nivå 2: Pilot (begrenset prod bruk)
- **Batch evaluation** før hver deploy (50-100 test cases)
- Continuous evaluation i prod (10% sampling, kun ContentSafety + IntentResolution)
- Application Insights oppsett
- Judge model: gpt-4.1

#### Nivå 3: Production (full prod deployment)
- **Batch evaluation** i CI/CD (200+ test cases, quality + safety evaluators)
- Continuous evaluation (5-10% sampling, alle relevante evaluators)
- Alerts på pass rate < 70%
- Trace-debugger i Foundry for failed evals
- Judge model: o3-mini for complex evaluators, gpt-4o for simple

#### Nivå 4: Mission-critical (high-risk AI system)
- **Batch evaluation** med 1000+ test cases (inkludert adversarial)
- Continuous evaluation (20-50% sampling, alle evaluators)
- Custom evaluators for domain-specific metrics
- Monthly red teaming med IndirectAttack + CodeVulnerability
- Human-in-the-loop review av failed evaluations
- Full evaluation audit trail (lagres i 5 år for AI Act compliance)
- Judge model: o3-mini + custom fine-tuned judge for kritiske metrics

## Kilder og verifisering

### Microsoft Learn (MCP-verified)

1. **Evaluate your AI agents (preview)**
   https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/agent-evaluate-sdk?view=foundry-classic
   *Confidence: Verified* — Hovedreferanse for Azure AI Evaluation SDK, evaluator types, model support

2. **Continuously evaluate your AI agents (preview)**
   https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/continuous-evaluation-agents?view=foundry-classic
   *Confidence: Verified* — Continuous evaluation setup, sampling configuration, Application Insights integration

3. **Run evaluations in the cloud by using the Microsoft Foundry SDK**
   https://learn.microsoft.com/en-us/azure/foundry/how-to/develop/cloud-evaluation?view=foundry-classic
   *Confidence: Verified* — Cloud batch evaluation, CI/CD integration, dataset formats

4. **Tutorial: Idea to prototype - Build and evaluate an enterprise agent**
   https://learn.microsoft.com/en-us/azure/foundry/tutorials/developer-journey-idea-to-prototype?view=foundry
   *Confidence: Verified* — End-to-end tutorial med cloud evaluation, built-in evaluators

5. **Test and evaluate AI workloads on Azure (Well-Architected Framework)**
   https://learn.microsoft.com/en-us/azure/well-architected/ai/test#validate-agentic-workflows
   *Confidence: Verified* — Agentic workflow testing strategy, tool call validation, security testing

6. **Observability in generative AI**
   https://learn.microsoft.com/en-us/azure/foundry/concepts/observability
   *Confidence: Verified* — Built-in evaluators list, GenAIOps evaluation stages, simulators

7. **What are hosted agents? (Evaluate and test hosted agents)**
   https://learn.microsoft.com/en-us/azure/foundry/agents/concepts/hosted-agents?view=foundry#evaluate-and-test-hosted-agents
   *Confidence: Verified* — Hosted agent evaluation best practices, test dataset creation

8. **Agent evaluators**
   https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/agent-evaluators?view=foundry
   *Confidence: Verified* — Agent-specific evaluator details (Intent Resolution, Task Adherence, Tool Call Accuracy)

9. **Evaluate and monitor AI agents (MLflow 3 on Databricks)**
   https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/
   *Confidence: Verified* — MLflow 3 GenAI evaluation: built-in LLM judges og scorers, eval-harness for development, production monitoring (Beta), conversation evaluation (multi-turn), conversation simulation, Review App for human feedback, Genie Code for observability; integrert med MLflow Tracing paa tvers av development/test/produksjon; oppdatert 2026-04

10. **Run automated tests for agent quality and reliability (Copilot Studio)**
    https://learn.microsoft.com/en-us/power-platform/release-plan/2025wave1/microsoft-copilot-studio/run-automated-tests-agent-quality-reliability
    *Confidence: Verified* — Copilot Studio evaluation framework (2025 preview)

### Confidence levels per section

| Section | Confidence | Reason |
|---------|-----------|--------|
| Introduksjon | Verified | Basert på 3 MCP-kilder (agent-evaluate-sdk, observability, well-architected) |
| Kjernekomponenter | Verified | Direkte fra agent-evaluate-sdk dokumentasjon + code samples |
| Arkitekturmønstre | Verified | Fra cloud-evaluation + continuous-evaluation docs + code samples |
| Beslutningsveiledning | Baseline + Verified | Decision tables basert på best practices (well-architected) + cost models |
| Integrasjon med Microsoft-stakken | Verified | Fra Foundry, Semantic Kernel, Prompt Flow, Application Insights docs |
| Offentlig sektor (Norge) | Baseline | GDPR/AI Act vurdering basert på modellkunnskap + Azure residency facts |
| Kostnad og lisensiering | Baseline | Prisestimater basert på Azure OpenAI pricing (feb 2026) + observability costs |
| For arkitekten (Cosmo) | Baseline | Synthesized fra verified sources + praktisk erfaring |

---

**Document metadata:**
- **MCP calls:** 3 (microsoft_docs_search) + 2 (microsoft_docs_fetch) + 1 (microsoft_code_sample_search) = 6
- **Unique sources:** 10 Microsoft Learn URLs
- **Word count:** ~3200 ord
- **File size:** ~29 KB