ms-ai-architect/skills/ms-ai-advisor/references/prompt-engineering/prompt-testing-and-evaluation.md

Prompt Testing, Evaluation and Iteration

Last updated: 2026-06-19 Status: GA Category: Prompt Engineering & LLM Optimization

---

Introduksjon

Evaluering av prompt-baserte LLM-løsninger er kritisk for å måle ytelse, kvalitet og sikkerhet i generative AI-applikasjoner. Microsoft tilbyr en omfattende evalueringsplattform gjennom Azure AI Foundry og Prompt Flow som støtter både automatisert testing, AI-assistert evaluering og kontinuerlig overvåking.

Denne referansen dekker evalueringsrammeverket for prompt testing, iterasjon og optimalisering på tvers av Microsoft AI-stakken — fra utviklingsfasen (prototyping), via eksperimentering (evaluation flows), til produksjon (continuous evaluation).

Hovedkomponenter:

• Azure AI Foundry Evaluation: UI-basert evalueringsportal med innebygde metrics
• Prompt Flow Evaluation: SDK-basert rammeverk for programmatisk evaluering
• Azure AI Evaluation SDK: Python SDK for custom evaluators og batch-evaluering
• Continuous Evaluation: Automatisk evaluering av agent-responser i produksjon
• Evaluation Metrics: AI-assisterte, NLP-baserte og safety-fokuserte metrics

Evalueringstyper:

• Model evaluation: Evaluerer output fra en modell mot et datasett
• Agent evaluation: Evaluerer agent-responser (inkl. tool calls og reasoning)
• Dataset evaluation: Evaluerer pre-genererte outputs i et datasett
• Synthetic evaluation: Evaluerer modell mot syntetisk genererte testdata

---

Kjernekomponenter

1. Azure AI Foundry Evaluation Portal

Beskrivelse: UI-basert evalueringsverktøy i Azure AI Foundry portalen som lar deg opprette evaluation runs med innebygde metrics, visualisere resultater og sammenligne evalueringer.

Kapabiliteter:

• Wizard-basert opprettelse av evaluation runs (Evaluation → Create)
• Test mot model deployments, agents eller forhåndsgenererte datasets
• Støtte for CSV/JSONL datasets
• Automatisk field mapping mellom dataset og evaluators
• Synthetic dataset generation (GPT-genererte spørsmål basert på topic)
• Evaluator library for versjonering og gjenbruk av evaluators

Built-in Evaluation Metrics (3 kategorier):

Kategori	Metrics	Krever	Beskrivelse
AI Quality (AI-assisted)	Groundedness, Relevance, Coherence, Fluency, GPT Similarity	GPT-4/GPT-3.5 deployment	AI-vurdert kvalitet med Likert-skala (1-5)
AI Quality (NLP)	F1 Score, ROUGE, BLEU, GLEU, METEOR	Ground truth data	Matematiske metrics for tekstlikhet
Risk & Safety	Violence, Hate/Unfairness, Self-Harm, Sexual Content, Protected Material, Indirect Attack	Ingen (Foundry provisjonerer GPT-4)	Content safety scoring (0-7 severity)

Data Mapping Requirements:

Metric	Query	Response	Context	Ground Truth
Groundedness	✅	✅	✅	—
Relevance	✅	✅	✅	—
Coherence	✅	✅	—	—
Fluency	✅	✅	—	—
GPT Similarity	✅	✅	—	✅
F1/BLEU/ROUGE	—	✅	—	✅
Safety metrics	✅	✅	—	—

Regiontilgjengelighet (Safety Metrics): AI-assisted risk and safety metrics er hostet av Foundry safety evaluations og tilgjengelig i: East US 2, France Central, UK South, Sweden Central.

Synthetic Data Generation (Preview): Tilgjengelig i regioner som støtter Response API. Genererer testdata basert på en prompt + optional file upload for kontekst.

---

2. Prompt Flow Evaluation Framework

Utfasing (verifisert MCP 2026-06-19): Prompt Flow i Microsoft Foundry og Azure Machine Learning utfases 20. april 2027 og anbefales ikke lenger for ny utvikling. Migrer eksisterende Prompt Flow-applikasjoner og -deployments til Microsoft Agent Framework (MAF) før denne datoen. Prompt Flow-runtime-images (promptflow-runtime, promptflow-runtime-stable, promptflow-python) får ikke lenger oppdateringer, inkludert sikkerhetsoppdateringer. Evaluation-mønstrene under er fortsatt gyldige for eksisterende løsninger, men velg MAF / Azure AI Evaluation SDK for nye prosjekter.

Beskrivelse: SDK-basert evalueringsrammeverk som lar deg bygge custom evaluation flows som Python-kode eller Prompty-filer, kjøre batch evaluations og logge metrics programmatisk.

Evaluation Flow Lifecycle:

1. Input Definition     → Definer inputs (query, response, context, ground_truth)
2. Line Processing      → Kalkuler score per data row (Python/LLM node)
3. Output Specification → Spesifiser outputs (scores, reasoning)
4. Aggregation          → Kalkuler overall metrics (mean, median, pass rate)
5. Metric Logging       → Log metrics med `log_metric()` funksjon

Evaluation Flow Structure:

Node Type	Formål	Input	Output
Line Process	Kalkuler score per rad	Single row data	Score (float/int), reasoning (str)
Aggregation	Kalkuler overall metrics	List of scores	Aggregated metric (float/int)

Kodeeksempel: Custom Evaluator (Python node):

from typing import List
from promptflow import tool, log_metric

@tool
def calculate_accuracy(grades: List[str]):
    """
    Aggregation node som kalkulerer overall accuracy.
    """
    accuracy = round((grades.count("Correct") / len(grades)), 2)
    log_metric("accuracy", accuracy)
    return accuracy

Built-in Evaluators (Prompt Flow SDK):

from azure.ai.evaluation import (
    RelevanceEvaluator,
    CoherenceEvaluator,
    GroundednessProEvaluator,
    ViolenceEvaluator,
    BleuScoreEvaluator
)

# AI-assisted evaluator
model_config = {
    "azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
    "api_key": os.environ["AZURE_OPENAI_API_KEY"],
    "azure_deployment": "gpt-4o"
}

relevance_eval = RelevanceEvaluator(model_config)
result = relevance_eval(
    query="What is the capital of Japan?",
    response="The capital of Japan is Tokyo.",
    context="Japan is a country in East Asia."
)

# NLP evaluator (no model required)
bleu_eval = BleuScoreEvaluator()
result = bleu_eval(
    response="Tokyo is the capital of Japan.",
    ground_truth="The capital of Japan is Tokyo."
)

Prompt Flow CLI for Batch Evaluation:

# Kjør evaluation flow mot et batch run
pfazure run create --file run_evaluation.yml

# Vis evaluation metrics
pfazure run show-metrics --name <evaluation-run-name>

# Stream evaluation logs
pfazure run stream --name <evaluation-run-name>

---

3. Azure AI Evaluation SDK

Beskrivelse: Python SDK (azure-ai-evaluation) for programmatisk evaluering av LLM-applikasjoner, med støtte for custom evaluators, batch evaluation og integration med Azure AI Foundry.

Installasjon:

pip install azure-ai-evaluation
pip install "azure-ai-evaluation[remote]"  # For remote evaluation
pip install "azure-ai-evaluation[redteam]"  # Inkluderer PyRIT for red teaming

Evaluate Function (Core API):

from azure.ai.evaluation import evaluate, RelevanceEvaluator, CoherenceEvaluator

model_config = {
    "azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
    "api_key": os.environ["AZURE_OPENAI_KEY"],
    "azure_deployment": "gpt-4o"
}

# Batch evaluation mot JSONL dataset
result = evaluate(
    data="evaluation_data.jsonl",  # CSV eller JSONL
    evaluators={
        "coherence": CoherenceEvaluator(model_config=model_config),
        "relevance": RelevanceEvaluator(model_config=model_config)
    },
    evaluator_config={
        "coherence": {
            "column_mapping": {
                "response": "${data.response}",
                "query": "${data.query}"
            }
        },
        "relevance": {
            "column_mapping": {
                "response": "${data.response}",
                "context": "${data.context}",
                "query": "${data.query}"
            }
        }
    },
    tags={"environment": "production", "version": "v1.2"}
)

# Access results
print(f"Coherence score: {result['metrics']['coherence']}")
print(f"Relevance score: {result['metrics']['relevance']}")

Custom Evaluators (AzureOpenAIPythonGrader):

from azure.ai.evaluation import AzureOpenAIPythonGrader

# Custom evaluator med Python-basert grading logic
custom_grader = AzureOpenAIPythonGrader(
    model_config=model_config,
    name="custom_accuracy",
    pass_threshold=0.8,
    source="""
def grade(sample: dict, item: dict) -> float:
    output = item.get("response", "").lower()
    label = item.get("ground_truth", "").lower()

    if output == label:
        return 1.0
    elif output in label or label in output:
        return 0.5
    return 0.0
"""
)

# Kjør evaluation
result = evaluate(
    data="test_data.jsonl",
    evaluators={"custom_accuracy": custom_grader}
)

print(f"Pass rate: {result['metrics']['custom_accuracy.pass_rate']}")

Agent-Specific Evaluators:

from azure.ai.evaluation import (
    IntentResolutionEvaluator,
    ResponseCompletenessEvaluator
)

intent_eval = IntentResolutionEvaluator(model_config)
result = intent_eval(
    query="What are the opening hours of the Eiffel Tower?",
    response="Opening hours of the Eiffel Tower are 9:00 AM to 11:00 PM."
)
print(result["score"])  # 1-5 skala

---

4. Continuous Evaluation (Production Monitoring)

Beskrivelse: Automatisk evaluering av agent-responser i produksjon ved hjelp av Evaluation Rules som trigger på agent events (f.eks. RESPONSE_COMPLETED).

Setup via Azure AI Projects SDK:

from azure.ai.projects.models import (
    EvaluationRule,
    ContinuousEvaluationRuleAction,
    EvaluationRuleFilter,
    EvaluationRuleEventType
)

# Opprett evaluation object (som i batch evaluation)
data_source_config = {"type": "azure_ai_source", "scenario": "responses"}
testing_criteria = [
    {
        "type": "azure_ai_evaluator",
        "name": "violence_detection",
        "evaluator_name": "builtin.violence"
    }
]

eval_object = openai_client.evals.create(
    name="Continuous Evaluation",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria
)

# Opprett continuous evaluation rule
continuous_eval_rule = project_client.evaluation_rules.create_or_update(
    id="my-continuous-eval-rule",
    evaluation_rule=EvaluationRule(
        display_name="Production Agent Safety Monitor",
        description="Evaluerer alle agent-responser for violence content",
        action=ContinuousEvaluationRuleAction(
            eval_id=eval_object.id,
            max_hourly_runs=100  # Rate limiting
        ),
        event_type=EvaluationRuleEventType.RESPONSE_COMPLETED,
        filter=EvaluationRuleFilter(agent_name="MyProductionAgent"),
        enabled=True
    )
)

Event Types:

• RESPONSE_COMPLETED: Trigger når agent ferdigstiller en respons
• RESPONSE_FAILED: Trigger ved agent errors

Use Cases:

• Real-time safety monitoring (violence, hate speech)
• Quality drift detection (relevance, coherence)
• Compliance logging (protected material, GDPR)

---

5. Evaluator Library & Version Management

Beskrivelse: Sentralisert bibliotek i Azure AI Foundry for lagring, versjonering og deling av custom evaluators.

Registrere Custom Evaluator:

from azure.ai.ml import MLClient
from azure.ai.ml.entities import Model
from promptflow.client import PFClient

# Opprett MLClient for Azure AI Project
ml_client = MLClient(
    subscription_id=os.environ["AZURE_SUBSCRIPTION_ID"],
    resource_group_name=os.environ["AZURE_RESOURCE_GROUP"],
    workspace_name=os.environ["AZURE_PROJECT_NAME"],
    credential=DefaultAzureCredential()
)

# Konverter evaluator til Prompt Flow format
pf_client = PFClient()
pf_client.flows.save(entry=MyCustomEvaluator, path="custom_eval_local")

# Registrer evaluator i Evaluator Library
custom_evaluator = Model(
    path="custom_eval_local",
    name="MyCustomEvaluator",
    description="Evaluator som måler svar-lengde og relevans."
)

registered_evaluator = ml_client.evaluators.create_or_update(custom_evaluator)
print(f"Registered evaluator: {registered_evaluator.id}")

# Hent spesifikk versjon
versioned_evaluator = ml_client.evaluators.get("MyCustomEvaluator", version=1)

Fordeler:

• Versjonering: Spor endringer i evaluators over tid
• Gjenbruk: Del evaluators på tvers av team og prosjekter
• Governance: Sentralisert kontroll over evaluation logic

---

Arkitekturmønstre

Mønster 1: Iterativ Prompt Development Workflow

Bruksområde: Utvikle og tune prompts gjennom systematisk evaluering og iterasjon.

Prosess:

1. Initialization
   └─ Definer business use case
   └─ Samle sample data (50-100 eksempler)
   └─ Utvikle baseline prompt

2. Experimentation (Inner Loop)
   └─ Test prompt i Playground/SDK
   └─ Kjør batch evaluation (5-10 samples)
   └─ Analyser failure cases
   └─ Iterer prompt (instruksjoner, few-shot examples)
   └─ Repeat til tilfredsstillende results

3. Evaluation & Refinement (Outer Loop)
   └─ Kjør batch evaluation (100-500 samples)
   └─ Mål metrics: quality (coherence, relevance), safety (violence, hate)
   └─ Sammenlign prompt variants (A/B testing)
   └─ Analyser edge cases og failure modes
   └─ Refiner prompt basert på metrics

4. Production
   └─ Deploy prompt til production
   └─ Aktiver continuous evaluation
   └─ Monitor metrics over time (drift detection)
   └─ Feedback loop til steg 1 for continuous improvement

Best Practices:

• Start smått: 5-10 samples i inner loop, 100-500 i outer loop
• Diverse metrics: Kombiner AI-assisted (coherence, relevance) + safety (violence, hate)
• Ground truth data: Kuratér høy-kvalitet ground truth for NLP metrics
• Human-in-the-loop: Kombiner automated evaluation med human feedback
• Versjonering: Bruk Evaluator Library for å tracke prompt changes

---

Mønster 2: Multi-Evaluator Testing Strategy

Bruksområde: Evaluere prompts på tvers av flere dimensjoner (quality, safety, task-specific metrics) for helhetlig vurdering.

Evaluator Stack:

Layer	Evaluator Type	Metrics	Threshold
Layer 1: Safety	Risk & Safety Evaluators	Violence, Hate, Self-Harm, Sexual	100% pass rate (severity < 2)
Layer 2: Quality	AI-Assisted Quality	Groundedness, Relevance, Coherence	Avg score ≥ 4/5
Layer 3: Task Performance	NLP/Custom Evaluators	F1 Score, ROUGE, Custom Logic	F1 ≥ 0.8
Layer 4: User Experience	Human Feedback	Thumbs up/down, CSAT	≥ 80% positive

Implementasjon:

# Layer 1: Safety evaluators (blokkerende)
safety_evaluators = {
    "violence": ViolenceEvaluator(azure_ai_project),
    "hate": HateUnfairnessEvaluator(azure_ai_project),
    "self_harm": SelfHarmEvaluator(azure_ai_project)
}

# Layer 2: Quality evaluators (krav: avg ≥ 4/5)
quality_evaluators = {
    "groundedness": GroundednessProEvaluator(azure_ai_project, threshold=4),
    "relevance": RelevanceEvaluator(model_config),
    "coherence": CoherenceEvaluator(model_config)
}

# Layer 3: Task performance
task_evaluators = {
    "f1_score": F1ScoreEvaluator(),
    "custom_accuracy": AzureOpenAIPythonGrader(...)
}

# Kjør evaluation i sekvens
safety_result = evaluate(data=data, evaluators=safety_evaluators)
if safety_result["metrics"]["violence.defect_rate"] == 0:
    quality_result = evaluate(data=data, evaluators=quality_evaluators)
    if quality_result["metrics"]["relevance"] >= 4:
        task_result = evaluate(data=data, evaluators=task_evaluators)

Når bruke:

• RAG-applikasjoner: Safety → Groundedness → Relevance → F1 Score
• Conversational agents: Safety → Coherence → IntentResolution → CSAT
• Classification tasks: Safety → Custom Logic → F1/Accuracy

---

Mønster 3: Dataset-Driven Evaluation (Golden Dataset Strategy)

Bruksområde: Opprette et kuratert "golden dataset" for konsistent evaluering av prompt changes over tid.

Dataset Structure (JSONL format):

{"query": "What is the capital of France?", "context": "France is a country in Europe.", "ground_truth": "Paris", "category": "geography"}
{"query": "Explain photosynthesis", "context": "Photosynthesis is a process...", "ground_truth": "Photosynthesis converts light to energy...", "category": "science"}

Golden Dataset Characteristics:

• Size: 300-1000 samples (representative of production distribution)
• Diversity: Dekker edge cases, common queries, failure modes
• Quality: Manuelt validert ground truth av domain experts
• Version Control: Lagret i Git, oppdatert ved nye use cases
• Stratification: Balansert på tvers av kategorier (f.eks. 30% geography, 30% science, 40% history)

Evaluation Workflow:

# Last inn golden dataset
golden_dataset = "golden_dataset_v3.jsonl"

# Evaluer prompt variant
result = evaluate(
    data=golden_dataset,
    evaluators={
        "relevance": RelevanceEvaluator(model_config),
        "f1_score": F1ScoreEvaluator()
    },
    tags={"prompt_version": "v2.1", "dataset_version": "v3"}
)

# Sammenlign med baseline
baseline_metrics = load_baseline_metrics("v1.0")
improvement = result["metrics"]["f1_score"] - baseline_metrics["f1_score"]
print(f"F1 Score improvement: {improvement:.2%}")

Best Practices:

• Versjonering: Tag både dataset version og prompt version i evaluation runs
• Regression Testing: Kjør golden dataset evaluation ved hver prompt change
• Continuous Update: Legg til nye failure cases fra production til golden dataset
• Stratified Sampling: Sikre balansert distribusjon av query types

---

Mønster 4: Continuous Evaluation + Human-in-the-Loop (Production)

Bruksområde: Kombinere automated continuous evaluation med human feedback i produksjon for å fange kvalitetsproblemer og safety issues i real-time.

Arkitektur:

Production Agent
    ↓ (response_completed event)
Continuous Evaluation Rule
    ↓ (automated metrics)
Evaluation Dashboard
    ↓ (flagged samples)
Human Review Queue
    ↓ (feedback)
Feedback Loop → Retraining/Prompt Tuning

Implementasjon:

# Setup continuous evaluation
continuous_eval_rule = project_client.evaluation_rules.create_or_update(
    id="production-safety-monitor",
    evaluation_rule=EvaluationRule(
        action=ContinuousEvaluationRuleAction(eval_id=eval_object.id, max_hourly_runs=100),
        event_type=EvaluationRuleEventType.RESPONSE_COMPLETED,
        filter=EvaluationRuleFilter(agent_name="CustomerSupportAgent"),
        enabled=True
    )
)

# Query flagged samples for human review
flagged_samples = project_client.evaluations.query_samples(
    filter="violence_score > 2 OR groundedness_score < 3"
)

# Human reviewer workflow
for sample in flagged_samples:
    print(f"Query: {sample['query']}")
    print(f"Response: {sample['response']}")
    print(f"Flags: Violence={sample['violence_score']}, Groundedness={sample['groundedness_score']}")
    feedback = input("Approve (y/n)? ")

    if feedback == "n":
        # Log to feedback dataset for retraining
        feedback_dataset.append({
            "query": sample["query"],
            "response": sample["response"],
            "feedback": "rejected",
            "reason": "low_groundedness"
        })

Alerting Strategy:

Metric	Threshold	Alert Level	Action
Violence Score > 4	Immediate	Critical	Block response, manual review
Groundedness < 3	> 5% of responses	Warning	Review prompt, update context
Relevance < 3	> 10% of responses	Warning	Retrain/tune prompt
Response Time > 10s	> 20% of responses	Info	Optimize inference

---

Mønster 5: A/B Testing for Prompt Optimization

Bruksområde: Teste flere prompt variants i produksjon for å identifisere beste prompt basert på real-world metrics.

Workflow:

# Definer prompt variants
prompt_a = "You are a helpful assistant. Answer briefly."
prompt_b = "You are an expert assistant. Provide detailed answers with examples."

# Deploy variants med traffic split
traffic_split = {"prompt_a": 0.5, "prompt_b": 0.5}

# Continuous evaluation per variant
for variant in ["prompt_a", "prompt_b"]:
    continuous_eval_rule = project_client.evaluation_rules.create_or_update(
        id=f"ab-test-{variant}",
        evaluation_rule=EvaluationRule(
            action=ContinuousEvaluationRuleAction(eval_id=eval_object.id),
            filter=EvaluationRuleFilter(agent_name=f"Agent-{variant}"),
            enabled=True
        )
    )

# Analyser results etter 1 uke
results_a = query_evaluation_metrics(agent="Agent-prompt_a", time_range="7d")
results_b = query_evaluation_metrics(agent="Agent-prompt_b", time_range="7d")

# Statistical significance test (t-test)
from scipy.stats import ttest_ind
t_stat, p_value = ttest_ind(results_a["relevance_scores"], results_b["relevance_scores"])

if p_value < 0.05:
    winner = "prompt_a" if results_a["avg_relevance"] > results_b["avg_relevance"] else "prompt_b"
    print(f"Winner: {winner} (p={p_value:.4f})")

Evaluering Metrics (A/B Test):

• Primary Metrics: Relevance, Coherence, Task Completion Rate
• Secondary Metrics: Response Time, User Satisfaction (CSAT)
• Guardrail Metrics: Safety (violence, hate), Groundedness

---

Beslutningsveiledning

Spørsmål 1: Hvilken evalueringsmetode passer for mitt use case?

Use Case	Evalueringsmetode	Begrunnelse
Prototyping (5-50 samples)	Playground + Manual Review	Rask iterasjon, minimal overhead
Development (100-500 samples)	Prompt Flow Batch Evaluation	Strukturert testing, metrics logging
Pre-Production (1000+ samples)	Azure AI Foundry Evaluation (UI/SDK)	Golden dataset testing, A/B comparison
Production Monitoring	Continuous Evaluation + HITL	Real-time safety, drift detection

---

Spørsmål 2: Hvilke metrics skal jeg bruke?

Scenario	Primary Metrics	Secondary Metrics	Rationale
RAG (Q&A)	Groundedness, Relevance	F1 Score, ROUGE	Sikre factuelt korrekte svar basert på context
Conversational Agent	Coherence, Fluency, IntentResolution	CSAT, Response Time	Sikre naturlig dialog og user intent-oppfyllelse
Classification	F1 Score, Accuracy	Precision, Recall	Måle task performance matematisk
Content Generation	Coherence, Fluency, GPT Similarity	BLEU, Human Feedback	Kvalitet og likhet til menneskeskrevne tekster
Public Sector (Norge)	Safety metrics (Violence, Hate), Groundedness	Relevance, Coherence	Compliance med AI-loven, GDPR, etiske retningslinjer

Best Practice: Kombiner alltid AI-assisted quality metrics (relevance, coherence) med safety metrics (violence, hate) for helhetlig evaluering.

---

Spørsmål 3: Hvor mange samples trenger jeg i evalueringen?

Fase	Sample Count	Begrunnelse
Inner Loop (Rapid Iteration)	5-20	Rask feedback på prompt changes
Outer Loop (Validation)	100-500	Statistisk signifikante resultater
Pre-Production (Golden Dataset)	500-1000	Representativ for production distribution
Continuous Evaluation (Production)	Alle responses (sampled)	Kontinuerlig overvåking av quality drift

Rule of Thumb: Minimum 100 samples for pålitelig metric calculation (confidence interval < 5%).

---

Spørsmål 4: Når skal jeg bruke custom evaluators vs. built-in evaluators?

Situasjon	Anbefaling	Eksempel
Standardiserte use cases (RAG, classification)	Built-in evaluators	Groundedness, Relevance, F1 Score
Domain-spesifikk logikk	Custom evaluators	Medical terminology accuracy, Legal citation format
Business-spesifikke KPIs	Custom evaluators	Customer satisfaction scoring, Brand compliance
Regulatory compliance (Norge)	Custom evaluators	GDPR-compliance check, Norwegian language quality
Cost optimization	Built-in evaluators	Raskere utvikling, ingen custom logic vedlikehold

Best Practice: Start med built-in evaluators, utvikle custom evaluators kun når nødvendig for spesifikke krav.

---

Integrasjon med Microsoft-stakken

Azure AI Foundry

Evaluation Workflow:

1. Develop Prompt (Playground)
   └─ Test interaktivt med sample queries

2. Batch Test (Evaluation Portal)
   └─ Upload dataset (CSV/JSONL)
   └─ Select evaluators (Groundedness, Relevance, Safety)
   └─ Map fields (query, response, context, ground_truth)
   └─ Submit evaluation run

3. View Results (Evaluation Portal)
   └─ Metrics dashboard (avg scores, pass rate)
   └─ Per-sample analysis (drill-down)
   └─ Comparison view (A vs B)

4. Iterate Prompt
   └─ Refiner prompt basert på failure cases
   └─ Re-run evaluation → Compare metrics

Integration Points:

• Model Catalog: Evaluer modeller i katalogen med egne data
• Playground: Test prompts interaktivt før batch evaluation
• Deployments: Evaluer deployed models og agenter
• Evaluator Library: Lagre og versjonere custom evaluators

---

Prompt Flow

SDK-Based Evaluation Workflow:

from promptflow import PFClient
from azure.ai.evaluation import evaluate

# Step 1: Opprett PFClient
pf_client = PFClient()

# Step 2: Kjør batch run
batch_run = pf_client.run(
    flow="./my_flow",
    data="./test_data.jsonl",
    column_mapping={"query": "${data.query}"}
)

# Step 3: Kjør evaluation
eval_result = evaluate(
    data="./test_data.jsonl",
    evaluators={
        "relevance": RelevanceEvaluator(model_config),
        "coherence": CoherenceEvaluator(model_config)
    },
    evaluator_config={
        "relevance": {
            "column_mapping": {
                "query": "${data.query}",
                "response": "${run.outputs.response}",
                "context": "${data.context}"
            }
        }
    }
)

# Step 4: Analyser metrics
print(f"Relevance: {eval_result['metrics']['relevance']}")

DevOps Integration:

# Azure Pipelines YAML
trigger:
  branches:
    include:
      - main

steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.11'

  - script: |
      pip install promptflow azure-ai-evaluation
      pfazure run create --file run.yml
      pfazure run create --file run_evaluation.yml
    displayName: 'Run Prompt Flow Evaluation'

  - script: |
      python validate_metrics.py  # Fail pipeline hvis metrics under threshold
    displayName: 'Validate Metrics'

---

Copilot Studio

Limitation: Copilot Studio har begrenset native evaluation support (ingen built-in evaluation framework).

Workaround:

1. Eksporter conversation logs fra Copilot Studio til Dataverse
2. Sync til Azure AI Foundry via API
3. Kjør evaluation i Azure AI Foundry mot eksporterte logs

Alternativ: Bruk Power Automate flow for å samle conversation logs og kalle Azure AI Evaluation API.

---

Power Platform AI Builder

Limitation: AI Builder har ikke native evaluation support for prompt-baserte modeller.

Workaround:

1. Test prompts i Azure AI Foundry Playground
2. Evaluer via Azure AI Foundry Evaluation Portal
3. Deploy finalized prompt til AI Builder (via custom connector til Azure OpenAI)

---

Microsoft 365 Copilot

Limitation: M365 Copilot er closed-source, ingen direkte evaluation access.

Enterprise-Level Monitoring:

• Microsoft Purview: Compliance monitoring (DLP, sensitivity labels)
• Microsoft Viva Insights: User adoption metrics (ikke quality metrics)
• Azure Monitor: Latency, error rates (ikke semantic quality)

Recommendation: For custom Copilot Extensions (via Copilot Studio), bruk Copilot Studio evaluation workflow ovenfor.

---

Offentlig sektor (Norge)

Compliance-Krav

Regulering	Krav	Evaluation Metrics
EU AI Act (Article 52)	Transparency om AI-generert innhold	Groundedness, Source Attribution (custom evaluator)
GDPR (Article 22)	No automated decision-making uten human review	Human-in-the-Loop metrics (% human-reviewed)
Diskrimineringsloven	No bias mot beskyttede grupper	Fairness metrics (custom evaluator for Norwegian context)
Språkkrav (Norsk offentlig sektor)	Norwegian language quality	Language Quality Evaluator (custom, trained on Norwegian corpus)

---

Anbefalt Evaluation Stack for Norske Myndigheter

Layer	Evaluator	Threshold	Begrunnelse
Safety (Obligatorisk)	Violence, Hate, Self-Harm	100% pass rate (severity < 2)	AI-loven krav til innholdssikkerhet
Factuality (Obligatorisk)	Groundedness	100% pass rate (score ≥ 4/5)	Forhindre feilinformasjon i offentlig sektor
Language Quality	Norwegian Language Evaluator (custom)	95% pass rate	Sikre korrekt norsk grammatikk og terminologi
Transparency	Source Attribution Evaluator (custom)	100% (alle claims må ha kilde)	AI-loven transparency requirement
Quality	Relevance, Coherence	Avg ≥ 4/5	Brukerkvalitet

---

Custom Evaluator: Norwegian Language Quality

Bruksområde: Sjekke at AI-generert tekst følger norsk grammatikk, terminologi og bokmål/nynorsk-standarder.

Implementasjon:

from azure.ai.evaluation import AzureOpenAIPythonGrader

norwegian_language_evaluator = AzureOpenAIPythonGrader(
    model_config=model_config,
    name="norwegian_language_quality",
    pass_threshold=0.9,
    source="""
def grade(sample: dict, item: dict) -> float:
    response = item.get("response", "")

    # Sjekk 1: Ingen engelske ord (unntatt tekniske termer)
    english_words = ["the", "and", "is", "are", "to", "for"]
    has_english = any(word in response.lower() for word in english_words)

    # Sjekk 2: Korrekt bokmål/nynorsk (basert på terminologi)
    # Implementer custom logic basert på LanguageTool API eller spaCy Norwegian model

    # Sjekk 3: Formell tone (offentlig sektor krav)
    informal_words = ["hei", "sånn", "skjønner"]
    has_informal = any(word in response.lower() for word in informal_words)

    if has_english or has_informal:
        return 0.6
    return 1.0
"""
)

Best Practice: Integrer LanguageTool API eller GPT-4o med Norwegian system prompt for mer avansert grammatikksjekk.

---

Custom Evaluator: Source Attribution (GDPR Transparency)

Bruksområde: Sikre at alle factual claims i AI-generert tekst har en identifiserbar kilde (GDPR Article 22, AI Act Article 52).

Implementasjon:

source_attribution_evaluator = AzureOpenAIPythonGrader(
    model_config=model_config,
    name="source_attribution",
    pass_threshold=1.0,  # Alle claims må ha kilde
    source="""
def grade(sample: dict, item: dict) -> float:
    response = item.get("response", "")
    context = item.get("context", "")

    # Prompt GPT-4o til å identifisere claims
    claims = extract_claims(response)  # Custom function via LLM

    # Sjekk at hver claim kan traces til context
    attributed_claims = 0
    for claim in claims:
        if is_claim_in_context(claim, context):  # Custom function via LLM
            attributed_claims += 1

    attribution_rate = attributed_claims / len(claims) if claims else 1.0
    return attribution_rate
"""
)

---

Kostnad og lisensiering

Azure AI Foundry Evaluation Costs

Komponent	Kostnadsmodell	Estimat (NOK/måned)
AI-Assisted Evaluators	Charged per GPT-4 token consumption	NOK 500-2000 (avhenger av dataset size)
Safety Evaluators	Gratis (Foundry-provisjonert GPT-4)	NOK 0
NLP Evaluators	Gratis (matematisk beregning)	NOK 0
Synthetic Data Generation	Charged per GPT-4 token consumption	NOK 100-500 per 1000 samples
Continuous Evaluation	Charged per GPT-4 token consumption	NOK 2000-10 000 (avhenger av traffic volume)

Optimalisering:

• Bruk NLP evaluators (F1, ROUGE) for bulk testing (gratis)
• Bruk AI-assisted evaluators kun for final validation (mindre dataset)
• Limit max_hourly_runs i continuous evaluation for cost control

---

Lisensiering

Komponent	Lisenskrav	Inkludert i
Azure AI Foundry Evaluation Portal	Azure-subscription	Azure AI Foundry Hub
Prompt Flow SDK	Ingen lisens (open-source)	Gratis (pip install)
Azure AI Evaluation SDK	Ingen lisens (open-source)	Gratis (pip install)
Azure OpenAI (for GPT-4 judges)	Azure-subscription + model deployment	Pay-as-you-go pricing
Foundry Safety Evaluators	Inkludert i Foundry-subscription	Gratis (limited regions)

Note: Foundry Safety Evaluators er kun tilgjengelig i East US 2, France Central, UK South, Sweden Central.

---

Kostnadsestimat: Typisk Evaluation Workflow

Fase	Dataset Size	Evaluators	GPT-4 Token Consumption	Kostnad (NOK)
Inner Loop (Development)	10 samples	Relevance, Coherence	~10K tokens	NOK 10
Outer Loop (Validation)	500 samples	Groundedness, Relevance, Safety	~500K tokens	NOK 500
Golden Dataset (Pre-Prod)	1000 samples	Full stack (6 evaluators)	~2M tokens	NOK 2000
Continuous Eval (Production)	10K responses/month	Safety only (gratis)	0 tokens	NOK 0

Total estimat (per måned i produksjon): NOK 2000-5000 (avhenger av evaluation frequency og dataset size).

---

For arkitekten (Cosmo)

Når foreslå Azure AI Foundry Evaluation?

✅ JA, når:

• Kunden jobber med RAG, conversational agents eller content generation
• Kunden trenger systematisk prompt testing for å sikre kvalitet før produksjon
• Kunden er underlagt compliance-krav (AI Act, GDPR, norsk offentlig sektor)
• Kunden har eksisterende Azure AI Foundry infrastructure
• Kunden trenger continuous evaluation for production monitoring

❌ NEI, når:

• Kunden har simple keyword-based eller rule-based logic (ikke LLM-based)
• Kunden mangler resurser til å kuratere golden dataset (100+ samples)
• Kunden har svært lave budsjetter (< NOK 5000/måned) og høy traffic volume
• Kunden har ingen Azure-subscription og vil unngå cloud lock-in

---

Diskusjonsspørsmål til kunden

1. "Har dere en testdatasett med 100-500 eksempler som representerer typiske bruksscenarioer?"

   • Hvis NEI → Foreslå synthetic data generation (kostnad: ~NOK 500)

2. "Hvilke kvalitetsdimensjoner er viktigst for dere: faktakorrekthet (groundedness), relevans, eller sikkerhet (safety)?"

   • Tailor evaluator stack basert på svar

3. "Trenger dere compliance-dokumentasjon for AI-loven eller GDPR?"

   • Hvis JA → Inkluder source attribution evaluator + human-in-the-loop

4. "Hvor ofte planlegger dere å oppdatere prompts i produksjon?"

   • Hvis ofte (ukentlig) → Foreslå golden dataset + regression testing
   • Hvis sjelden (kvartalsvis) → Enklere ad-hoc evaluation

5. "Har dere kapasitet til manuell review av 5-10% av AI-genererte responses?"

   • Hvis JA → Foreslå continuous evaluation + human-in-the-loop
   • Hvis NEI → Fokuser på automated safety evaluators

---

Red Flags (Advarselssignaler)

⚠️ Ingen testdata tilgjengelig → Løsning: Start med synthetic data generation (50-100 samples), deretter kuratér golden dataset over tid.

⚠️ Kunden forventer 100% accuracy fra AI → Løsning: Eduker om LLM-limitasjoner, foreslå human-in-the-loop for kritiske use cases.

⚠️ Kunden vil hoppe rett til produksjon uten evaluering → Løsning: Påpek risiko for reputational damage, compliance issues. Minimum krav: Safety evaluators (gratis).

⚠️ Ingen budsjettkontroll for GPT-4 evaluation costs → Løsning: Kombiner NLP evaluators (gratis) med AI-assisted (payg). Sett max_hourly_runs limit.

---

Trinnvis Anbefalingsstrategi

Steg 1: Minimal Viable Evaluation (MVE)

Kostnad: NOK 0-500/måned Komponenter:

• Safety evaluators (gratis) for violence, hate, self-harm
• NLP evaluators (F1 Score, ROUGE) for task performance
• Manual testing i Playground (5-10 samples)

Når bruke: Early-stage prototyping, tight budget.

---

Steg 2: Standard Evaluation Stack

Kostnad: NOK 2000-5000/måned Komponenter:

• Safety evaluators (gratis)
• AI-assisted quality evaluators (Groundedness, Relevance, Coherence)
• Golden dataset (500-1000 samples)
• Batch evaluation via Prompt Flow SDK

Når bruke: Pre-production, medium-sized deployments (< 10K responses/month).

---

Steg 3: Enterprise Evaluation (Production-Grade)

Kostnad: NOK 10 000-50 000/måned Komponenter:

• Full evaluator stack (safety + quality + custom)
• Continuous evaluation + human-in-the-loop
• A/B testing framework
• Custom evaluators for compliance (Norwegian language, source attribution)
• Dedicated evaluation team (manual review 5-10% of responses)

Når bruke: Large-scale production (> 50K responses/month), public sector, regulated industries.

---

Confidence Markers

High Confidence (>95%):

• Built-in evaluators (Groundedness, Relevance, Safety) er production-ready og widely used
• Prompt Flow SDK evaluation workflow er stable (GA siden 2023), men utfases 20. april 2027 — migrer til Microsoft Agent Framework / Azure AI Evaluation SDK for ny utvikling
• Azure AI Foundry Evaluation Portal er GA (as of 2024)

Medium Confidence (70-95%):

• Synthetic data generation quality (Preview-feature, limited regions)
• Custom evaluator performance (avhenger av prompt engineering quality)
• Continuous evaluation pricing (can vary significantly based on traffic patterns)

Low Confidence (<70%):

• Copilot Studio native evaluation support (mangler offisiell løsning per Feb 2026)
• M365 Copilot evaluation (closed-source, ingen official API)
• Cross-region safety evaluator availability (kun 4 regioner støttet)

---

Kilder og verifisering

Primary Sources (Microsoft Learn):

1. Evaluate generative AI models and applications - Azure AI Foundry — GA
2. Evaluation flows and metrics - Azure Machine Learning Prompt Flow — GA. Re-verifisert MCP 2026-06-19: Prompt Flow utfases 20. april 2027 → migrer til Microsoft Agent Framework. log_metric()/aggregation-mønster uendret.
3. Azure AI Evaluation SDK - Python API — GA
4. Agent evaluation with Azure AI Evaluation SDK — GA

Code Samples (Microsoft Learn):

1. Cloud evaluation with Azure AI Projects SDK
2. Continuous evaluation setup
3. Custom evaluator registration

Last Verified: 2026-06-19 Version: Azure AI Foundry v2 (2024-2026), Prompt Flow v1.13+ (2024-2026; utfases 2027-04-20 → Microsoft Agent Framework) MCP Calls: 3 (microsoft_docs_search × 2, microsoft_docs_fetch × 2, microsoft_code_sample_search × 1)