ms-ai-architect/skills/ms-ai-advisor/references/prompt-engineering/prompt-testing-and-evaluation.md

# Prompt Testing, Evaluation and Iteration

**Last updated:** 2026-06-19
**Status:** GA
**Category:** Prompt Engineering & LLM Optimization

---

## Introduksjon

Evaluering av prompt-baserte LLM-løsninger er kritisk for å måle ytelse, kvalitet og sikkerhet i generative AI-applikasjoner. Microsoft tilbyr en omfattende evalueringsplattform gjennom Microsoft Foundry og Prompt Flow som støtter både automatisert testing, AI-assistert evaluering og kontinuerlig overvåking.

Denne referansen dekker evalueringsrammeverket for prompt testing, iterasjon og optimalisering på tvers av Microsoft AI-stakken — fra utviklingsfasen (prototyping), via eksperimentering (evaluation flows), til produksjon (continuous evaluation).

**Hovedkomponenter:**
- **Microsoft Foundry Evaluation**: UI-basert evalueringsportal med innebygde metrics
- **Prompt Flow Evaluation**: SDK-basert rammeverk for programmatisk evaluering
- **Azure AI Evaluation SDK**: Python SDK for custom evaluators og batch-evaluering
- **Continuous Evaluation**: Automatisk evaluering av agent-responser i produksjon
- **Evaluation Metrics**: AI-assisterte, NLP-baserte og safety-fokuserte metrics

**Evalueringstyper:**
- **Model evaluation**: Evaluerer output fra en modell mot et datasett
- **Agent evaluation**: Evaluerer agent-responser (inkl. tool calls og reasoning)
- **Dataset evaluation**: Evaluerer pre-genererte outputs i et datasett
- **Synthetic evaluation**: Evaluerer modell mot syntetisk genererte testdata

---

## Kjernekomponenter

### 1. Microsoft Foundry Evaluation Portal

**Beskrivelse:** UI-basert evalueringsverktøy i Microsoft Foundry portalen som lar deg opprette evaluation runs med innebygde metrics, visualisere resultater og sammenligne evalueringer.

**Kapabiliteter:**
- Wizard-basert opprettelse av evaluation runs (Evaluation → Create)
- Test mot model deployments, agents eller forhåndsgenererte datasets
- Støtte for CSV/JSONL datasets
- Automatisk field mapping mellom dataset og evaluators
- Synthetic dataset generation (GPT-genererte spørsmål basert på topic)
- Evaluator library for versjonering og gjenbruk av evaluators

**Built-in Evaluation Metrics (3 kategorier):**

| Kategori | Metrics | Krever | Beskrivelse |
|----------|---------|--------|-------------|
| **AI Quality (AI-assisted)** | Groundedness, Relevance, Coherence, Fluency, GPT Similarity | GPT-4/GPT-3.5 deployment | AI-vurdert kvalitet med Likert-skala (1-5) |
| **AI Quality (NLP)** | F1 Score, ROUGE, BLEU, GLEU, METEOR | Ground truth data | Matematiske metrics for tekstlikhet |
| **Risk & Safety** | Violence, Hate/Unfairness, Self-Harm, Sexual Content, Protected Material, Indirect Attack | Ingen (Foundry provisjonerer GPT-4) | Content safety scoring (0-7 severity) |

**Data Mapping Requirements:**

| Metric | Query | Response | Context | Ground Truth |
|--------|-------|----------|---------|--------------|
| Groundedness | ✅ | ✅ | ✅ | — |
| Relevance | ✅ | ✅ | ✅ | — |
| Coherence | ✅ | ✅ | — | — |
| Fluency | ✅ | ✅ | — | — |
| GPT Similarity | ✅ | ✅ | — | ✅ |
| F1/BLEU/ROUGE | — | ✅ | — | ✅ |
| Safety metrics | ✅ | ✅ | — | — |

**Regiontilgjengelighet (Safety Metrics):**
AI-assisted risk and safety metrics er hostet av Foundry safety evaluations og tilgjengelig i: **East US 2, France Central, UK South, Sweden Central**.

**Synthetic Data Generation (Preview):**
Tilgjengelig i regioner som støtter Response API. Genererer testdata basert på en prompt + optional file upload for kontekst.

---

### 2. Prompt Flow Evaluation Framework

> **Utfasing (verifisert MCP 2026-06-19):** Prompt Flow i Microsoft Foundry og Azure Machine Learning utfases **20. april 2027** og anbefales ikke lenger for ny utvikling. Migrer eksisterende Prompt Flow-applikasjoner og -deployments til **Microsoft Agent Framework (MAF)** før denne datoen. Prompt Flow-runtime-images (`promptflow-runtime`, `promptflow-runtime-stable`, `promptflow-python`) får ikke lenger oppdateringer, inkludert sikkerhetsoppdateringer. Evaluation-mønstrene under er fortsatt gyldige for eksisterende løsninger, men velg MAF / Azure AI Evaluation SDK for nye prosjekter.

**Beskrivelse:** SDK-basert evalueringsrammeverk som lar deg bygge custom evaluation flows som Python-kode eller Prompty-filer, kjøre batch evaluations og logge metrics programmatisk.

**Evaluation Flow Lifecycle:**

```
1. Input Definition     → Definer inputs (query, response, context, ground_truth)
2. Line Processing      → Kalkuler score per data row (Python/LLM node)
3. Output Specification → Spesifiser outputs (scores, reasoning)
4. Aggregation          → Kalkuler overall metrics (mean, median, pass rate)
5. Metric Logging       → Log metrics med `log_metric()` funksjon
```

**Evaluation Flow Structure:**

| Node Type | Formål | Input | Output |
|-----------|--------|-------|--------|
| **Line Process** | Kalkuler score per rad | Single row data | Score (float/int), reasoning (str) |
| **Aggregation** | Kalkuler overall metrics | List of scores | Aggregated metric (float/int) |

**Kodeeksempel: Custom Evaluator (Python node):**

```python
from typing import List
from promptflow import tool, log_metric

@tool
def calculate_accuracy(grades: List[str]):
    """
    Aggregation node som kalkulerer overall accuracy.
    """
    accuracy = round((grades.count("Correct") / len(grades)), 2)
    log_metric("accuracy", accuracy)
    return accuracy
```

**Built-in Evaluators (Prompt Flow SDK):**

```python
from azure.ai.evaluation import (
    RelevanceEvaluator,
    CoherenceEvaluator,
    GroundednessProEvaluator,
    ViolenceEvaluator,
    BleuScoreEvaluator
)

# AI-assisted evaluator
model_config = {
    "azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
    "api_key": os.environ["AZURE_OPENAI_API_KEY"],
    "azure_deployment": "gpt-4o"
}

relevance_eval = RelevanceEvaluator(model_config)
result = relevance_eval(
    query="What is the capital of Japan?",
    response="The capital of Japan is Tokyo.",
    context="Japan is a country in East Asia."
)

# NLP evaluator (no model required)
bleu_eval = BleuScoreEvaluator()
result = bleu_eval(
    response="Tokyo is the capital of Japan.",
    ground_truth="The capital of Japan is Tokyo."
)
```

**Prompt Flow CLI for Batch Evaluation:**

```bash
# Kjør evaluation flow mot et batch run
pfazure run create --file run_evaluation.yml

# Vis evaluation metrics
pfazure run show-metrics --name <evaluation-run-name>

# Stream evaluation logs
pfazure run stream --name <evaluation-run-name>
```

---

### 3. Azure AI Evaluation SDK

**Beskrivelse:** Python SDK (`azure-ai-evaluation`) for programmatisk evaluering av LLM-applikasjoner, med støtte for custom evaluators, batch evaluation og integration med Microsoft Foundry.

**Installasjon:**

```bash
pip install azure-ai-evaluation
pip install "azure-ai-evaluation[remote]"  # For remote evaluation
pip install "azure-ai-evaluation[redteam]"  # Inkluderer PyRIT for red teaming
```

**Evaluate Function (Core API):**

```python
from azure.ai.evaluation import evaluate, RelevanceEvaluator, CoherenceEvaluator

model_config = {
    "azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
    "api_key": os.environ["AZURE_OPENAI_KEY"],
    "azure_deployment": "gpt-4o"
}

# Batch evaluation mot JSONL dataset
result = evaluate(
    data="evaluation_data.jsonl",  # CSV eller JSONL
    evaluators={
        "coherence": CoherenceEvaluator(model_config=model_config),
        "relevance": RelevanceEvaluator(model_config=model_config)
    },
    evaluator_config={
        "coherence": {
            "column_mapping": {
                "response": "${data.response}",
                "query": "${data.query}"
            }
        },
        "relevance": {
            "column_mapping": {
                "response": "${data.response}",
                "context": "${data.context}",
                "query": "${data.query}"
            }
        }
    },
    tags={"environment": "production", "version": "v1.2"}
)

# Access results
print(f"Coherence score: {result['metrics']['coherence']}")
print(f"Relevance score: {result['metrics']['relevance']}")
```

**Custom Evaluators (AzureOpenAIPythonGrader):**

```python
from azure.ai.evaluation import AzureOpenAIPythonGrader

# Custom evaluator med Python-basert grading logic
custom_grader = AzureOpenAIPythonGrader(
    model_config=model_config,
    name="custom_accuracy",
    pass_threshold=0.8,
    source="""
def grade(sample: dict, item: dict) -> float:
    output = item.get("response", "").lower()
    label = item.get("ground_truth", "").lower()

    if output == label:
        return 1.0
    elif output in label or label in output:
        return 0.5
    return 0.0
"""
)

# Kjør evaluation
result = evaluate(
    data="test_data.jsonl",
    evaluators={"custom_accuracy": custom_grader}
)

print(f"Pass rate: {result['metrics']['custom_accuracy.pass_rate']}")
```

**Agent-Specific Evaluators:**

```python
from azure.ai.evaluation import (
    IntentResolutionEvaluator,
    ResponseCompletenessEvaluator
)

intent_eval = IntentResolutionEvaluator(model_config)
result = intent_eval(
    query="What are the opening hours of the Eiffel Tower?",
    response="Opening hours of the Eiffel Tower are 9:00 AM to 11:00 PM."
)
print(result["score"])  # 1-5 skala
```

---

### 4. Continuous Evaluation (Production Monitoring)

**Beskrivelse:** Automatisk evaluering av agent-responser i produksjon ved hjelp av Evaluation Rules som trigger på agent events (f.eks. `RESPONSE_COMPLETED`).

**Setup via Azure AI Projects SDK:**

```python
from azure.ai.projects.models import (
    EvaluationRule,
    ContinuousEvaluationRuleAction,
    EvaluationRuleFilter,
    EvaluationRuleEventType
)

# Opprett evaluation object (som i batch evaluation)
data_source_config = {"type": "azure_ai_source", "scenario": "responses"}
testing_criteria = [
    {
        "type": "azure_ai_evaluator",
        "name": "violence_detection",
        "evaluator_name": "builtin.violence"
    }
]

eval_object = openai_client.evals.create(
    name="Continuous Evaluation",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria
)

# Opprett continuous evaluation rule
continuous_eval_rule = project_client.evaluation_rules.create_or_update(
    id="my-continuous-eval-rule",
    evaluation_rule=EvaluationRule(
        display_name="Production Agent Safety Monitor",
        description="Evaluerer alle agent-responser for violence content",
        action=ContinuousEvaluationRuleAction(
            eval_id=eval_object.id,
            max_hourly_runs=100  # Rate limiting
        ),
        event_type=EvaluationRuleEventType.RESPONSE_COMPLETED,
        filter=EvaluationRuleFilter(agent_name="MyProductionAgent"),
        enabled=True
    )
)
```

**Event Types:**
- `RESPONSE_COMPLETED`: Trigger når agent ferdigstiller en respons
- `RESPONSE_FAILED`: Trigger ved agent errors

**Use Cases:**
- Real-time safety monitoring (violence, hate speech)
- Quality drift detection (relevance, coherence)
- Compliance logging (protected material, GDPR)

---

### 5. Evaluator Library & Version Management

**Beskrivelse:** Sentralisert bibliotek i Microsoft Foundry for lagring, versjonering og deling av custom evaluators.

**Registrere Custom Evaluator:**

```python
from azure.ai.ml import MLClient
from azure.ai.ml.entities import Model
from promptflow.client import PFClient

# Opprett MLClient for Azure AI Project
ml_client = MLClient(
    subscription_id=os.environ["AZURE_SUBSCRIPTION_ID"],
    resource_group_name=os.environ["AZURE_RESOURCE_GROUP"],
    workspace_name=os.environ["AZURE_PROJECT_NAME"],
    credential=DefaultAzureCredential()
)

# Konverter evaluator til Prompt Flow format
pf_client = PFClient()
pf_client.flows.save(entry=MyCustomEvaluator, path="custom_eval_local")

# Registrer evaluator i Evaluator Library
custom_evaluator = Model(
    path="custom_eval_local",
    name="MyCustomEvaluator",
    description="Evaluator som måler svar-lengde og relevans."
)

registered_evaluator = ml_client.evaluators.create_or_update(custom_evaluator)
print(f"Registered evaluator: {registered_evaluator.id}")

# Hent spesifikk versjon
versioned_evaluator = ml_client.evaluators.get("MyCustomEvaluator", version=1)
```

**Fordeler:**
- **Versjonering**: Spor endringer i evaluators over tid
- **Gjenbruk**: Del evaluators på tvers av team og prosjekter
- **Governance**: Sentralisert kontroll over evaluation logic

---

## Arkitekturmønstre

### Mønster 1: Iterativ Prompt Development Workflow

**Bruksområde:** Utvikle og tune prompts gjennom systematisk evaluering og iterasjon.

**Prosess:**

```
1. Initialization
   └─ Definer business use case
   └─ Samle sample data (50-100 eksempler)
   └─ Utvikle baseline prompt

2. Experimentation (Inner Loop)
   └─ Test prompt i Playground/SDK
   └─ Kjør batch evaluation (5-10 samples)
   └─ Analyser failure cases
   └─ Iterer prompt (instruksjoner, few-shot examples)
   └─ Repeat til tilfredsstillende results

3. Evaluation & Refinement (Outer Loop)
   └─ Kjør batch evaluation (100-500 samples)
   └─ Mål metrics: quality (coherence, relevance), safety (violence, hate)
   └─ Sammenlign prompt variants (A/B testing)
   └─ Analyser edge cases og failure modes
   └─ Refiner prompt basert på metrics

4. Production
   └─ Deploy prompt til production
   └─ Aktiver continuous evaluation
   └─ Monitor metrics over time (drift detection)
   └─ Feedback loop til steg 1 for continuous improvement
```

**Best Practices:**
- **Start smått**: 5-10 samples i inner loop, 100-500 i outer loop
- **Diverse metrics**: Kombiner AI-assisted (coherence, relevance) + safety (violence, hate)
- **Ground truth data**: Kuratér høy-kvalitet ground truth for NLP metrics
- **Human-in-the-loop**: Kombiner automated evaluation med human feedback
- **Versjonering**: Bruk Evaluator Library for å tracke prompt changes

---

### Mønster 2: Multi-Evaluator Testing Strategy

**Bruksområde:** Evaluere prompts på tvers av flere dimensjoner (quality, safety, task-specific metrics) for helhetlig vurdering.

**Evaluator Stack:**

| Layer | Evaluator Type | Metrics | Threshold |
|-------|----------------|---------|-----------|
| **Layer 1: Safety** | Risk & Safety Evaluators | Violence, Hate, Self-Harm, Sexual | 100% pass rate (severity < 2) |
| **Layer 2: Quality** | AI-Assisted Quality | Groundedness, Relevance, Coherence | Avg score ≥ 4/5 |
| **Layer 3: Task Performance** | NLP/Custom Evaluators | F1 Score, ROUGE, Custom Logic | F1 ≥ 0.8 |
| **Layer 4: User Experience** | Human Feedback | Thumbs up/down, CSAT | ≥ 80% positive |

**Implementasjon:**

```python
# Layer 1: Safety evaluators (blokkerende)
safety_evaluators = {
    "violence": ViolenceEvaluator(azure_ai_project),
    "hate": HateUnfairnessEvaluator(azure_ai_project),
    "self_harm": SelfHarmEvaluator(azure_ai_project)
}

# Layer 2: Quality evaluators (krav: avg ≥ 4/5)
quality_evaluators = {
    "groundedness": GroundednessProEvaluator(azure_ai_project, threshold=4),
    "relevance": RelevanceEvaluator(model_config),
    "coherence": CoherenceEvaluator(model_config)
}

# Layer 3: Task performance
task_evaluators = {
    "f1_score": F1ScoreEvaluator(),
    "custom_accuracy": AzureOpenAIPythonGrader(...)
}

# Kjør evaluation i sekvens
safety_result = evaluate(data=data, evaluators=safety_evaluators)
if safety_result["metrics"]["violence.defect_rate"] == 0:
    quality_result = evaluate(data=data, evaluators=quality_evaluators)
    if quality_result["metrics"]["relevance"] >= 4:
        task_result = evaluate(data=data, evaluators=task_evaluators)
```

**Når bruke:**
- **RAG-applikasjoner**: Safety → Groundedness → Relevance → F1 Score
- **Conversational agents**: Safety → Coherence → IntentResolution → CSAT
- **Classification tasks**: Safety → Custom Logic → F1/Accuracy

---

### Mønster 3: Dataset-Driven Evaluation (Golden Dataset Strategy)

**Bruksområde:** Opprette et kuratert "golden dataset" for konsistent evaluering av prompt changes over tid.

**Dataset Structure (JSONL format):**

```json
{"query": "What is the capital of France?", "context": "France is a country in Europe.", "ground_truth": "Paris", "category": "geography"}
{"query": "Explain photosynthesis", "context": "Photosynthesis is a process...", "ground_truth": "Photosynthesis converts light to energy...", "category": "science"}
```

**Golden Dataset Characteristics:**
- **Size**: 300-1000 samples (representative of production distribution)
- **Diversity**: Dekker edge cases, common queries, failure modes
- **Quality**: Manuelt validert ground truth av domain experts
- **Version Control**: Lagret i Git, oppdatert ved nye use cases
- **Stratification**: Balansert på tvers av kategorier (f.eks. 30% geography, 30% science, 40% history)

**Evaluation Workflow:**

```python
# Last inn golden dataset
golden_dataset = "golden_dataset_v3.jsonl"

# Evaluer prompt variant
result = evaluate(
    data=golden_dataset,
    evaluators={
        "relevance": RelevanceEvaluator(model_config),
        "f1_score": F1ScoreEvaluator()
    },
    tags={"prompt_version": "v2.1", "dataset_version": "v3"}
)

# Sammenlign med baseline
baseline_metrics = load_baseline_metrics("v1.0")
improvement = result["metrics"]["f1_score"] - baseline_metrics["f1_score"]
print(f"F1 Score improvement: {improvement:.2%}")
```

**Best Practices:**
- **Versjonering**: Tag både dataset version og prompt version i evaluation runs
- **Regression Testing**: Kjør golden dataset evaluation ved hver prompt change
- **Continuous Update**: Legg til nye failure cases fra production til golden dataset
- **Stratified Sampling**: Sikre balansert distribusjon av query types

---

### Mønster 4: Continuous Evaluation + Human-in-the-Loop (Production)

**Bruksområde:** Kombinere automated continuous evaluation med human feedback i produksjon for å fange kvalitetsproblemer og safety issues i real-time.

**Arkitektur:**

```
Production Agent
    ↓ (response_completed event)
Continuous Evaluation Rule
    ↓ (automated metrics)
Evaluation Dashboard
    ↓ (flagged samples)
Human Review Queue
    ↓ (feedback)
Feedback Loop → Retraining/Prompt Tuning
```

**Implementasjon:**

```python
# Setup continuous evaluation
continuous_eval_rule = project_client.evaluation_rules.create_or_update(
    id="production-safety-monitor",
    evaluation_rule=EvaluationRule(
        action=ContinuousEvaluationRuleAction(eval_id=eval_object.id, max_hourly_runs=100),
        event_type=EvaluationRuleEventType.RESPONSE_COMPLETED,
        filter=EvaluationRuleFilter(agent_name="CustomerSupportAgent"),
        enabled=True
    )
)

# Query flagged samples for human review
flagged_samples = project_client.evaluations.query_samples(
    filter="violence_score > 2 OR groundedness_score < 3"
)

# Human reviewer workflow
for sample in flagged_samples:
    print(f"Query: {sample['query']}")
    print(f"Response: {sample['response']}")
    print(f"Flags: Violence={sample['violence_score']}, Groundedness={sample['groundedness_score']}")
    feedback = input("Approve (y/n)? ")

    if feedback == "n":
        # Log to feedback dataset for retraining
        feedback_dataset.append({
            "query": sample["query"],
            "response": sample["response"],
            "feedback": "rejected",
            "reason": "low_groundedness"
        })
```

**Alerting Strategy:**

| Metric | Threshold | Alert Level | Action |
|--------|-----------|-------------|--------|
| Violence Score > 4 | Immediate | Critical | Block response, manual review |
| Groundedness < 3 | > 5% of responses | Warning | Review prompt, update context |
| Relevance < 3 | > 10% of responses | Warning | Retrain/tune prompt |
| Response Time > 10s | > 20% of responses | Info | Optimize inference |

---

### Mønster 5: A/B Testing for Prompt Optimization

**Bruksområde:** Teste flere prompt variants i produksjon for å identifisere beste prompt basert på real-world metrics.

**Workflow:**

```python
# Definer prompt variants
prompt_a = "You are a helpful assistant. Answer briefly."
prompt_b = "You are an expert assistant. Provide detailed answers with examples."

# Deploy variants med traffic split
traffic_split = {"prompt_a": 0.5, "prompt_b": 0.5}

# Continuous evaluation per variant
for variant in ["prompt_a", "prompt_b"]:
    continuous_eval_rule = project_client.evaluation_rules.create_or_update(
        id=f"ab-test-{variant}",
        evaluation_rule=EvaluationRule(
            action=ContinuousEvaluationRuleAction(eval_id=eval_object.id),
            filter=EvaluationRuleFilter(agent_name=f"Agent-{variant}"),
            enabled=True
        )
    )

# Analyser results etter 1 uke
results_a = query_evaluation_metrics(agent="Agent-prompt_a", time_range="7d")
results_b = query_evaluation_metrics(agent="Agent-prompt_b", time_range="7d")

# Statistical significance test (t-test)
from scipy.stats import ttest_ind
t_stat, p_value = ttest_ind(results_a["relevance_scores"], results_b["relevance_scores"])

if p_value < 0.05:
    winner = "prompt_a" if results_a["avg_relevance"] > results_b["avg_relevance"] else "prompt_b"
    print(f"Winner: {winner} (p={p_value:.4f})")
```

**Evaluering Metrics (A/B Test):**
- **Primary Metrics**: Relevance, Coherence, Task Completion Rate
- **Secondary Metrics**: Response Time, User Satisfaction (CSAT)
- **Guardrail Metrics**: Safety (violence, hate), Groundedness

---

## Beslutningsveiledning

### Spørsmål 1: Hvilken evalueringsmetode passer for mitt use case?

| Use Case | Evalueringsmetode | Begrunnelse |
|----------|-------------------|-------------|
| **Prototyping (5-50 samples)** | Playground + Manual Review | Rask iterasjon, minimal overhead |
| **Development (100-500 samples)** | Prompt Flow Batch Evaluation | Strukturert testing, metrics logging |
| **Pre-Production (1000+ samples)** | Microsoft Foundry Evaluation (UI/SDK) | Golden dataset testing, A/B comparison |
| **Production Monitoring** | Continuous Evaluation + HITL | Real-time safety, drift detection |

---

### Spørsmål 2: Hvilke metrics skal jeg bruke?

| Scenario | Primary Metrics | Secondary Metrics | Rationale |
|----------|----------------|-------------------|-----------|
| **RAG (Q&A)** | Groundedness, Relevance | F1 Score, ROUGE | Sikre factuelt korrekte svar basert på context |
| **Conversational Agent** | Coherence, Fluency, IntentResolution | CSAT, Response Time | Sikre naturlig dialog og user intent-oppfyllelse |
| **Classification** | F1 Score, Accuracy | Precision, Recall | Måle task performance matematisk |
| **Content Generation** | Coherence, Fluency, GPT Similarity | BLEU, Human Feedback | Kvalitet og likhet til menneskeskrevne tekster |
| **Public Sector (Norge)** | **Safety metrics** (Violence, Hate), Groundedness | Relevance, Coherence | Compliance med AI-loven, GDPR, etiske retningslinjer |

**Best Practice:** Kombiner alltid AI-assisted quality metrics (relevance, coherence) med safety metrics (violence, hate) for helhetlig evaluering.

---

### Spørsmål 3: Hvor mange samples trenger jeg i evalueringen?

| Fase | Sample Count | Begrunnelse |
|------|--------------|-------------|
| **Inner Loop (Rapid Iteration)** | 5-20 | Rask feedback på prompt changes |
| **Outer Loop (Validation)** | 100-500 | Statistisk signifikante resultater |
| **Pre-Production (Golden Dataset)** | 500-1000 | Representativ for production distribution |
| **Continuous Evaluation (Production)** | Alle responses (sampled) | Kontinuerlig overvåking av quality drift |

**Rule of Thumb:** Minimum 100 samples for pålitelig metric calculation (confidence interval < 5%).

---

### Spørsmål 4: Når skal jeg bruke custom evaluators vs. built-in evaluators?

| Situasjon | Anbefaling | Eksempel |
|-----------|------------|----------|
| Standardiserte use cases (RAG, classification) | **Built-in evaluators** | Groundedness, Relevance, F1 Score |
| Domain-spesifikk logikk | **Custom evaluators** | Medical terminology accuracy, Legal citation format |
| Business-spesifikke KPIs | **Custom evaluators** | Customer satisfaction scoring, Brand compliance |
| Regulatory compliance (Norge) | **Custom evaluators** | GDPR-compliance check, Norwegian language quality |
| Cost optimization | **Built-in evaluators** | Raskere utvikling, ingen custom logic vedlikehold |

**Best Practice:** Start med built-in evaluators, utvikle custom evaluators kun når nødvendig for spesifikke krav.

---

## Integrasjon med Microsoft-stakken

### Microsoft Foundry

**Evaluation Workflow:**

```
1. Develop Prompt (Playground)
   └─ Test interaktivt med sample queries

2. Batch Test (Evaluation Portal)
   └─ Upload dataset (CSV/JSONL)
   └─ Select evaluators (Groundedness, Relevance, Safety)
   └─ Map fields (query, response, context, ground_truth)
   └─ Submit evaluation run

3. View Results (Evaluation Portal)
   └─ Metrics dashboard (avg scores, pass rate)
   └─ Per-sample analysis (drill-down)
   └─ Comparison view (A vs B)

4. Iterate Prompt
   └─ Refiner prompt basert på failure cases
   └─ Re-run evaluation → Compare metrics
```

**Integration Points:**
- **Model Catalog**: Evaluer modeller i katalogen med egne data
- **Playground**: Test prompts interaktivt før batch evaluation
- **Deployments**: Evaluer deployed models og agenter
- **Evaluator Library**: Lagre og versjonere custom evaluators

---

### Prompt Flow

**SDK-Based Evaluation Workflow:**

```python
from promptflow import PFClient
from azure.ai.evaluation import evaluate

# Step 1: Opprett PFClient
pf_client = PFClient()

# Step 2: Kjør batch run
batch_run = pf_client.run(
    flow="./my_flow",
    data="./test_data.jsonl",
    column_mapping={"query": "${data.query}"}
)

# Step 3: Kjør evaluation
eval_result = evaluate(
    data="./test_data.jsonl",
    evaluators={
        "relevance": RelevanceEvaluator(model_config),
        "coherence": CoherenceEvaluator(model_config)
    },
    evaluator_config={
        "relevance": {
            "column_mapping": {
                "query": "${data.query}",
                "response": "${run.outputs.response}",
                "context": "${data.context}"
            }
        }
    }
)

# Step 4: Analyser metrics
print(f"Relevance: {eval_result['metrics']['relevance']}")
```

**DevOps Integration:**

```yaml
# Azure Pipelines YAML
trigger:
  branches:
    include:
      - main

steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.11'

  - script: |
      pip install promptflow azure-ai-evaluation
      pfazure run create --file run.yml
      pfazure run create --file run_evaluation.yml
    displayName: 'Run Prompt Flow Evaluation'

  - script: |
      python validate_metrics.py  # Fail pipeline hvis metrics under threshold
    displayName: 'Validate Metrics'
```

---

### Copilot Studio

**Limitation:** Copilot Studio har begrenset native evaluation support (ingen built-in evaluation framework).

**Workaround:**
1. **Eksporter conversation logs** fra Copilot Studio til Dataverse
2. **Sync til Microsoft Foundry** via API
3. **Kjør evaluation** i Microsoft Foundry mot eksporterte logs

**Alternativ:** Bruk **Power Automate** flow for å samle conversation logs og kalle Azure AI Evaluation API.

---

### Power Platform AI Builder

**Limitation:** AI Builder har ikke native evaluation support for prompt-baserte modeller.

**Workaround:**
1. Test prompts i **Microsoft Foundry Playground**
2. Evaluer via **Microsoft Foundry Evaluation Portal**
3. Deploy finalized prompt til AI Builder (via custom connector til Azure OpenAI)

---

### Microsoft 365 Copilot

**Limitation:** M365 Copilot er closed-source, ingen direkte evaluation access.

**Enterprise-Level Monitoring:**
- **Microsoft Purview**: Compliance monitoring (DLP, sensitivity labels)
- **Microsoft Viva Insights**: User adoption metrics (ikke quality metrics)
- **Azure Monitor**: Latency, error rates (ikke semantic quality)

**Recommendation:** For custom Copilot Extensions (via Copilot Studio), bruk Copilot Studio evaluation workflow ovenfor.

---

## Offentlig sektor (Norge)

### Compliance-Krav

| Regulering | Krav | Evaluation Metrics |
|------------|------|-------------------|
| **EU AI Act (Article 52)** | Transparency om AI-generert innhold | Groundedness, Source Attribution (custom evaluator) |
| **GDPR (Article 22)** | No automated decision-making uten human review | Human-in-the-Loop metrics (% human-reviewed) |
| **Diskrimineringsloven** | No bias mot beskyttede grupper | Fairness metrics (custom evaluator for Norwegian context) |
| **Språkkrav (Norsk offentlig sektor)** | Norwegian language quality | Language Quality Evaluator (custom, trained on Norwegian corpus) |

---

### Anbefalt Evaluation Stack for Norske Myndigheter

| Layer | Evaluator | Threshold | Begrunnelse |
|-------|-----------|-----------|-------------|
| **Safety (Obligatorisk)** | Violence, Hate, Self-Harm | 100% pass rate (severity < 2) | AI-loven krav til innholdssikkerhet |
| **Factuality (Obligatorisk)** | Groundedness | 100% pass rate (score ≥ 4/5) | Forhindre feilinformasjon i offentlig sektor |
| **Language Quality** | Norwegian Language Evaluator (custom) | 95% pass rate | Sikre korrekt norsk grammatikk og terminologi |
| **Transparency** | Source Attribution Evaluator (custom) | 100% (alle claims må ha kilde) | AI-loven transparency requirement |
| **Quality** | Relevance, Coherence | Avg ≥ 4/5 | Brukerkvalitet |

---

### Custom Evaluator: Norwegian Language Quality

**Bruksområde:** Sjekke at AI-generert tekst følger norsk grammatikk, terminologi og bokmål/nynorsk-standarder.

**Implementasjon:**

```python
from azure.ai.evaluation import AzureOpenAIPythonGrader

norwegian_language_evaluator = AzureOpenAIPythonGrader(
    model_config=model_config,
    name="norwegian_language_quality",
    pass_threshold=0.9,
    source="""
def grade(sample: dict, item: dict) -> float:
    response = item.get("response", "")

    # Sjekk 1: Ingen engelske ord (unntatt tekniske termer)
    english_words = ["the", "and", "is", "are", "to", "for"]
    has_english = any(word in response.lower() for word in english_words)

    # Sjekk 2: Korrekt bokmål/nynorsk (basert på terminologi)
    # Implementer custom logic basert på LanguageTool API eller spaCy Norwegian model

    # Sjekk 3: Formell tone (offentlig sektor krav)
    informal_words = ["hei", "sånn", "skjønner"]
    has_informal = any(word in response.lower() for word in informal_words)

    if has_english or has_informal:
        return 0.6
    return 1.0
"""
)
```

**Best Practice:** Integrer LanguageTool API eller GPT-4o med Norwegian system prompt for mer avansert grammatikksjekk.

---

### Custom Evaluator: Source Attribution (GDPR Transparency)

**Bruksområde:** Sikre at alle factual claims i AI-generert tekst har en identifiserbar kilde (GDPR Article 22, AI Act Article 52).

**Implementasjon:**

```python
source_attribution_evaluator = AzureOpenAIPythonGrader(
    model_config=model_config,
    name="source_attribution",
    pass_threshold=1.0,  # Alle claims må ha kilde
    source="""
def grade(sample: dict, item: dict) -> float:
    response = item.get("response", "")
    context = item.get("context", "")

    # Prompt GPT-4o til å identifisere claims
    claims = extract_claims(response)  # Custom function via LLM

    # Sjekk at hver claim kan traces til context
    attributed_claims = 0
    for claim in claims:
        if is_claim_in_context(claim, context):  # Custom function via LLM
            attributed_claims += 1

    attribution_rate = attributed_claims / len(claims) if claims else 1.0
    return attribution_rate
"""
)
```

---

## Kostnad og lisensiering

### Microsoft Foundry Evaluation Costs

| Komponent | Kostnadsmodell | Estimat (NOK/måned) |
|-----------|----------------|---------------------|
| **AI-Assisted Evaluators** | Charged per GPT-4 token consumption | NOK 500-2000 (avhenger av dataset size) |
| **Safety Evaluators** | **Gratis** (Foundry-provisjonert GPT-4) | NOK 0 |
| **NLP Evaluators** | **Gratis** (matematisk beregning) | NOK 0 |
| **Synthetic Data Generation** | Charged per GPT-4 token consumption | NOK 100-500 per 1000 samples |
| **Continuous Evaluation** | Charged per GPT-4 token consumption | NOK 2000-10 000 (avhenger av traffic volume) |

**Optimalisering:**
- Bruk **NLP evaluators** (F1, ROUGE) for bulk testing (gratis)
- Bruk **AI-assisted evaluators** kun for final validation (mindre dataset)
- Limit **max_hourly_runs** i continuous evaluation for cost control

---

### Lisensiering

| Komponent | Lisenskrav | Inkludert i |
|-----------|------------|-------------|
| **Microsoft Foundry Evaluation Portal** | Azure-subscription | Microsoft Foundry Hub |
| **Prompt Flow SDK** | Ingen lisens (open-source) | Gratis (pip install) |
| **Azure AI Evaluation SDK** | Ingen lisens (open-source) | Gratis (pip install) |
| **Azure OpenAI (for GPT-4 judges)** | Azure-subscription + model deployment | Pay-as-you-go pricing |
| **Foundry Safety Evaluators** | Inkludert i Foundry-subscription | Gratis (limited regions) |

**Note:** Foundry Safety Evaluators er kun tilgjengelig i **East US 2, France Central, UK South, Sweden Central**.

---

### Kostnadsestimat: Typisk Evaluation Workflow

| Fase | Dataset Size | Evaluators | GPT-4 Token Consumption | Kostnad (NOK) |
|------|--------------|------------|-------------------------|---------------|
| **Inner Loop (Development)** | 10 samples | Relevance, Coherence | ~10K tokens | NOK 10 |
| **Outer Loop (Validation)** | 500 samples | Groundedness, Relevance, Safety | ~500K tokens | NOK 500 |
| **Golden Dataset (Pre-Prod)** | 1000 samples | Full stack (6 evaluators) | ~2M tokens | NOK 2000 |
| **Continuous Eval (Production)** | 10K responses/month | Safety only (gratis) | 0 tokens | NOK 0 |

**Total estimat (per måned i produksjon):** **NOK 2000-5000** (avhenger av evaluation frequency og dataset size).

---

## For arkitekten (Cosmo)

### Når foreslå Microsoft Foundry Evaluation?

✅ **JA, når:**
- Kunden jobber med RAG, conversational agents eller content generation
- Kunden trenger **systematisk prompt testing** for å sikre kvalitet før produksjon
- Kunden er underlagt **compliance-krav** (AI Act, GDPR, norsk offentlig sektor)
- Kunden har **eksisterende Microsoft Foundry infrastructure**
- Kunden trenger **continuous evaluation** for production monitoring

❌ **NEI, når:**
- Kunden har simple keyword-based eller rule-based logic (ikke LLM-based)
- Kunden mangler resurser til å kuratere golden dataset (100+ samples)
- Kunden har **svært lave budsjetter** (< NOK 5000/måned) og høy traffic volume
- Kunden har **ingen Azure-subscription** og vil unngå cloud lock-in

---

### Diskusjonsspørsmål til kunden

1. **"Har dere en testdatasett med 100-500 eksempler som representerer typiske bruksscenarioer?"**
   - Hvis NEI → Foreslå synthetic data generation (kostnad: ~NOK 500)

2. **"Hvilke kvalitetsdimensjoner er viktigst for dere: faktakorrekthet (groundedness), relevans, eller sikkerhet (safety)?"**
   - Tailor evaluator stack basert på svar

3. **"Trenger dere compliance-dokumentasjon for AI-loven eller GDPR?"**
   - Hvis JA → Inkluder source attribution evaluator + human-in-the-loop

4. **"Hvor ofte planlegger dere å oppdatere prompts i produksjon?"**
   - Hvis ofte (ukentlig) → Foreslå golden dataset + regression testing
   - Hvis sjelden (kvartalsvis) → Enklere ad-hoc evaluation

5. **"Har dere kapasitet til manuell review av 5-10% av AI-genererte responses?"**
   - Hvis JA → Foreslå continuous evaluation + human-in-the-loop
   - Hvis NEI → Fokuser på automated safety evaluators

---

### Red Flags (Advarselssignaler)

⚠️ **Ingen testdata tilgjengelig**
→ *Løsning:* Start med synthetic data generation (50-100 samples), deretter kuratér golden dataset over tid.

⚠️ **Kunden forventer 100% accuracy fra AI**
→ *Løsning:* Eduker om LLM-limitasjoner, foreslå human-in-the-loop for kritiske use cases.

⚠️ **Kunden vil hoppe rett til produksjon uten evaluering**
→ *Løsning:* Påpek risiko for reputational damage, compliance issues. Minimum krav: Safety evaluators (gratis).

⚠️ **Ingen budsjettkontroll for GPT-4 evaluation costs**
→ *Løsning:* Kombiner NLP evaluators (gratis) med AI-assisted (payg). Sett `max_hourly_runs` limit.

---

### Trinnvis Anbefalingsstrategi

#### Steg 1: Minimal Viable Evaluation (MVE)
**Kostnad:** NOK 0-500/måned
**Komponenter:**
- Safety evaluators (gratis) for violence, hate, self-harm
- NLP evaluators (F1 Score, ROUGE) for task performance
- Manual testing i Playground (5-10 samples)

**Når bruke:** Early-stage prototyping, tight budget.

---

#### Steg 2: Standard Evaluation Stack
**Kostnad:** NOK 2000-5000/måned
**Komponenter:**
- Safety evaluators (gratis)
- AI-assisted quality evaluators (Groundedness, Relevance, Coherence)
- Golden dataset (500-1000 samples)
- Batch evaluation via Prompt Flow SDK

**Når bruke:** Pre-production, medium-sized deployments (< 10K responses/month).

---

#### Steg 3: Enterprise Evaluation (Production-Grade)
**Kostnad:** NOK 10 000-50 000/måned
**Komponenter:**
- Full evaluator stack (safety + quality + custom)
- Continuous evaluation + human-in-the-loop
- A/B testing framework
- Custom evaluators for compliance (Norwegian language, source attribution)
- Dedicated evaluation team (manual review 5-10% of responses)

**Når bruke:** Large-scale production (> 50K responses/month), public sector, regulated industries.

---

### Confidence Markers

**High Confidence (>95%):**
- Built-in evaluators (Groundedness, Relevance, Safety) er production-ready og widely used
- Prompt Flow SDK evaluation workflow er stable (GA siden 2023), men utfases 20. april 2027 — migrer til Microsoft Agent Framework / Azure AI Evaluation SDK for ny utvikling
- Microsoft Foundry Evaluation Portal er GA (as of 2024)

**Medium Confidence (70-95%):**
- Synthetic data generation quality (Preview-feature, limited regions)
- Custom evaluator performance (avhenger av prompt engineering quality)
- Continuous evaluation pricing (can vary significantly based on traffic patterns)

**Low Confidence (<70%):**
- Copilot Studio native evaluation support (mangler offisiell løsning per Feb 2026)
- M365 Copilot evaluation (closed-source, ingen official API)
- Cross-region safety evaluator availability (kun 4 regioner støttet)

---

## Kilder og verifisering

**Primary Sources (Microsoft Learn):**
1. [Evaluate generative AI models and applications - Microsoft Foundry](https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app?view=foundry-classic) — GA
2. [Evaluation flows and metrics - Azure Machine Learning Prompt Flow](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-develop-an-evaluation-flow?view=azureml-api-2) — GA. Re-verifisert MCP 2026-06-19: Prompt Flow utfases 20. april 2027 → migrer til Microsoft Agent Framework. `log_metric()`/aggregation-mønster uendret.
3. [Azure AI Evaluation SDK - Python API](https://learn.microsoft.com/en-us/python/api/overview/azure/ai-evaluation-readme?view=azure-python) — GA
4. [Agent evaluation with Azure AI Evaluation SDK](https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/agent-evaluate-sdk?view=foundry-classic) — GA

**Code Samples (Microsoft Learn):**
1. [Cloud evaluation with Azure AI Projects SDK](https://learn.microsoft.com/en-us/azure/foundry/how-to/develop/cloud-evaluation?view=foundry-classic)
2. [Continuous evaluation setup](https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/how-to-monitor-agents-dashboard?view=foundry)
3. [Custom evaluator registration](https://learn.microsoft.com/en-us/azure/foundry/how-to/develop/cloud-evaluation?view=foundry-classic#specify-custom-evaluators)

**Last Verified:** 2026-06-19
**Version:** Microsoft Foundry v2 (2024-2026), Prompt Flow v1.13+ (2024-2026; utfases 2027-04-20 → Microsoft Agent Framework)